What I see here is an ecosystem. There are narrow real-time feeds such as expertly curated Twitter accounts, and big general reference works like Wikipedia. There are armies of reporters working in their niches, but also colonies of computer scientists. There are curators both human and algorithmic. And I have no problem imagining that this ecosystem includes certain kinds of artists and artworks. Let’s say it includes all public acts and systems which come down to one person trying to tell another, “I didn’t just make this up. There’s something here of the world we share.”

I asked people what to call it. Some said “media.” That captures a lot of it, but I’m not really talking about the art or entertainment aspects of media. Also I wanted to include something of where ideas come from, something about discussions, collaborative investigation, and the generation of new knowledge. Other people said “information” but there is much more here than being informed. Information alone doesn’t make us care or act. It is part of, but only part of, what it means to connect to another human being at a distance.  Someone else said “the fourth estate” and this is much closer, because it pulls in all the ideas around civic participation and public discourse and speaking truth to power, loads of stuff we generally file under “democracy.” But the fourth estate today means “the press” and what I want to talk about is broader than journalism.

I’m just going to call this the “digital public sphere”, building on Jürgen Habermas’ idea of a place for the discussion of shared concerns, public yet apart from the state. Maybe that’s not a great name — it’s a bit dry for my taste — but perhaps it’s the best that can be done in three words, and it’s already in use as a phrase to refer to many of the sorts of things I want to talk about. “Public sphere” captures something important, something about the societal goals of the system, and “digital” is a modifier that means we have to account for interactivity, networks, and computation. Taking inspiration from Michael Schudson’s essay ”Six or seven things that news can do for democracy,” I want to ask what the digital public sphere can do for us. I think I see three broad categories, which are also three goals to keep in mind as we build our institutions and systems.

1. Information. It should be possible for people to find things out, whatever they want to know. Our institutions should help people organize to produce valuable new knowledge. And important information should automatically reach each person at just the right moment.

2. Empathy. The vast majority of people in the world, we will only know through media. We must strive to represent the “other” to each-other with compassion and reality. We can’t forget that there are people on the other end of the wire.

3. Collective action. What good is public deliberation if we can’t eventually come to a decision and act? But truly enabling the formation of broad agreement also requires that our information systems support conflict resolution. In this age of complex overlapping communities, this role spans everything from the local to the global.

Each of these is its own rich area, and each of these roles already cuts across many different forms and institutions of media.

Information
I’d like to live in a world where it’s cheap and easy for anyone to satisfy the following desires:

1. “I want to learn about X.”
2. “How do we know that about X?”
3. “What are the most interesting things we don’t know about X?”
4. “Please keep me informed about X.”
5. “I think we should know more about X.”
6. “I know something about X and want to tell others.”

These desires span everything from mundane queries (“what time does the store close?”) to complex questions of fact (“what will be the effects of global climate change?”) And they apply at all scales; I might have a burning desire to know how the city government is going to deal with bike lanes, or I might be curious about the sum total of humanity’s knowledge of breast cancer — everything we know today, plus all the good questions we can’t yet answer. Different institutions exist to address each of these needs in various ways. Libraries have historically served the need to answer specific questions, desires number #1 and #2, but search engines also do this. Journalism strives to keep people abreast of current events, the essence of #4. Academia has focused on how we know and what we don’t yet know, which is #2 and #3.

This list includes two functions related to the production of new knowledge, because it seems to me that the public information ecosystem should support people working together to become collectively smarter. That’s why I’ve included #5, which is something like casting a vote for an unanswered question, and #6, the peer-to-peer ability to provide an answer. These seem like key elements in the democratic production of knowledge, because the resources which can be devoted to investigating answers are limited. There will always be a finite number of people well placed to answer any particular question, whether those people are researchers, reporters, subject matter experts, or simply well-informed. I like to imagine that their collective output is dwarfed by human curiosity. So efficiency matters, and we need to find ways to aggregate the questions of a community, and route each question to the person or people best positioned to find out the answer.

In the context of professional journalism, this amounts to asking what unanswered questions are most pressing to the community served by a newsroom. One could devise systems of asking the audience (like Quora and StackExchange) or analyze search logs (ala Demand Media.) That newsrooms don’t frequently do these things is, I think, an artifact of industrial history — and an unfilled niche in the current ecosystem. Search engines know where the gaps between supply and demand lie, but they’re not in the business of researching new answers. Newsrooms can produce the supply, but they don’t have an understanding of the demand. Today, these two sides of the industry do not work together to close this loop. Some symbiotic hybrid of Google and The Associated Press might be an uncannily good system for answering civic questions.

When new information does become available, there’s the issue of timing and routing. This is #4 again, “please keep me informed.” Traditionally, journalism has answered the question “who should know when?” with “everyone everything as fast as possible” but this is ridiculous today. I really don’t want my phone to vibrate for every news article ever written, which is why only “important” stories generate alerts. But taste and specialization dictate different definitions of “important” for each person, and old answers delivered when I need them might be just as valuable as new information delivered hot and fresh. Google is far down this track with its thinking on knowing what I want before I search for it.

Empathy 
There is no better way to show one person to another, across a distance, than the human story. These stories about other people may be informative, sure, but maybe their real purpose is to help us feel what it is like to be someone else. This is an old art; one journalist friend credits Homer with the last major innovation in the form.

But we also have to show whole groups to each other, a very “mass media” goal. If I’ve never met a Cambodian or hung out with a union organizer, I only know what I see in the media. How can and should entire communities, groups, cultures, races, interests or nations be represented?

A good journalist, anthropologist, or writer can live with a community for a while, observing and learning, then articulate generalizations. This is important and useful. It’s also wildly subjective. But then, so is empathy. Curation and amplification can also be empathetic processes: someone can direct attention to the genuine voices of a community. This “don’t speak, point” role has been articulated by Ethan Zuckerman and practiced by Andy Carvin.

But these are still at the level of individual stories. Who is representative? If I can only talk to five people, which five people should I know? Maybe a human story, no matter how effective, is just a single sample in the sense of a tiny part standing for the whole. Turning this notion around, making it personal, I come to an ideal: If I am to be seen as part of some group, then I want representations of that group to include me in some way. This is an argument that mass media coverage of a community should try to account for every person in that community. This is absurd in practical terms, but it can serve as a signpost, a core idea, something to aim for.

Fortunately, more inclusive representations are getting easier. Most profoundly, the widespread availability of peer-to-peer communication networks makes it easier than ever for a single member of a community to speak and be heard widely.

We also have data. We can compile the demographics of social movements, or conduct polls to find “public opinion.” We can learn a lot from the numbers that describe a particular population, which is why surveys and censuses persist. But data are terrible at producing the emotional response at the core of empathy. For most people, learning that 23% of the children in some state live in poverty lacks the gut-punch of a story about a child who goes hungry at the end of every month. In fact there is evidence that making someone think analytically about an issue actually makes them less compassionate.

The best reporting might combine human stories with broader data. I am impressed by CNN’s interactive exploration of American casualties in Iraq, which links mass visualization with photographs and stories about each individual. But that piece covers a comparatively small population, only a few thousand people. There are emerging techniques to understand much larger groups, such as by visualizing the data trails of online life, all of the personal information that we leave behind. We can visualize communities, using aggregate information to see the patterns of human association at all scales. I suspect that mass data visualization represents a fundamentally new way of understanding large groups, a way that is perhaps more inclusive than anecdotes yet richer than demographics. Also, visualization forces us into conversations about who exactly is a member of the community in question, because each person is either included in a particular visualization or not. Drawing such a hard boundary is often difficult, but it’s good to talk about the meanings of our labels.

And yet, for all this new technology, empathy remains a deeply human pursuit. Do we really want statistically unbiased samples of a community? My friend Quinn Norton says that journalism should “strive to show us our better selves.” Sometimes, what we need is brutal honesty. At other times, what we need is kindness and inspiration.

Collective action

What a difficult challenge advances in communication have become in recent decades. On the one hand they are definitely bringing us closer to each other, but are they really bringing us together?

- Ryszard Kapuściński, The Other

I am sensitive to the idea of filter bubbles and concerns about the fragmentation of media, the worry that the personalization of information will create a series of insular and homogenous communities, but I cannot abide the implied nostalgia for the broadcast era. I do not see how one-size-fits-all media can ever serve a diverse and specialized society, and so: let a million micro-cultures bloom! But I do see a need for powerful unifying forces within the public sphere, because everything from keeping a park clean to tackling global climate change requires the agreement and cooperation of a community.

We have long had decision making systems at all scales — from the neighborhood to the United Nations — and these mechanisms span a range from very lightweight and informal to global and ritualized. In many cases decision-making is built upon voting, with some majority required to pass, such as 51% or 66%. But is a vicious, hard-fought 51% in a polarized society really the best we can do? And what about all the issues that we will not be voting on — that is to say, most of them?

Unfortunately, getting agreement among even very moderate numbers of people seems phenomenally difficult. People disagree about methods, but in a pluralistic society they often disagree even more strongly about goals. Sometimes presenting all sides with credible information is enough, but strongly held disagreements usually cannot be resolved by shared facts; experimental work shows that, in many circumstances, polarization deepens with more information. This is the painful truth that blows a hole in ideas like “informed public” and “deliberative democracy.”

Something else is needed here. I want to bring the field of conflict resolution into the digital public sphere. As a named pursuit with its own literature and community, this is a young subject, really only begun after World War II. I love the field, but it’s in its infancy; I think it’s safe to say that we really don’t know very much about how to help groups with incompatible values find acceptable common solutions. We know even less about how to do this in an online setting.

But we can say for sure that “moderator” is an important role in the digital public sphere. This is old-school internet culture, dating back to the pre-web Usenet days, and we have evolved very many tools for keeping online discussions well-ordered, from classic comment moderation to collaborative filtering, reputation systems, online polls, and various other tricks. At the edges, moderation turns into conflict resolution, and there are tools for this too. I’m particularly intrigued by visualizations that show where a community agrees or disagrees along multiple axes, because the conceptually similar process of “peace polls” has had some success in real-world conflict situations such as Northern Ireland. I bet we could also learn from the arduously evolved dispute resolution processes of Wikipedia.

It seems to me that the ideal of legitimate community decision making is consensus, 100% agreement. This is very difficult, another unreachable goal, but we could define a scale from 51% agreement to 100%, and say that the goal is  ”as consensus as possible” decision making, which would also be “as legitimate as possible.” With this sort of metric — and always remembering that the goal is to reach a decision on a collective action, not to make people agree for the sake of it — we could undertake a systematic study of online consensus formation. For any given community, for any given issue, how fragmented is the discourse? Do people with different opinions hang out in different places online? Can we document examples of successful and unsuccessful online consensus formation, as has been done in the offline case? What role do human moderators play, and how can well-designed social software contribute? How do the processes of online agreement and disagreement play out at different scales and under different circumstances? How we do know when the process has converged to a “good” answer, and when it has degraded into hegemony or groupthink? These are mostly unexplored questions. Fortunately, there’s a huge amount of related work to draw on: voting systems and public choice theory, social network analysis, cognitive psychology, information flow and media ecosystems, social software design, issues of identity and culture, language and semiotics, epistemology…

I would like conflict resolution to be an explicit goal of our media platforms and processes, because we cannot afford to be polarized and grid-locked while there are important collective problems to be solved. We may have lost the unifying narrative of the front page, but that narrative was neither comprehensive nor inclusive: it didn’t always address the problems of concern to me, nor did it ask me what I thought. Effective collective action, at all relevant scales, seems a better and more concrete goal than “shared narrative.” It is also an exceptionally hard problem — in some ways it is the problem of democracy itself — but there’s lots to try, and our public sphere must be designed to support this.

Why now?
I began writing this essay because I wanted to say something very simple: all of these things — journalism, search engines, Wikipedia, social media and the lot — have to work together to common ends. There is today no one profession which encompasses the entirety of the public sphere. Journalism used to be the primary bearer of these responsibilities — or perhaps that was a well-meaning illusion sprung from near monopolies on mass information distribution channels. Either way, that era is now approaching two decades gone. Now what we have is an ecosystem, and in true networked fashion there may not ever again be a central authority. From algorithm designers to dedicated curators to, yes, traditional on-the-scene pro journalists, a great many people in different fields now have a part in shaping the digital public sphere. I wanted try to understand what all of us are working toward. I hope that I have at least articulated goals that we can agree are important.

My friend Maha Atal – whose profile of incoming IMF chief Christine Lagarde recently graced the cover of Forbes – tells me there are two histories of financial journalism, two types that have been practiced since the dawn of newsprint. One tradition began with lists of market prices and insurance rates and evolved into the financial data services and newswires we have today, a journalism of utility for people who want to make money. The other tradition she called “muckraking,” a journalism which interests itself in shady deals, insider trading, and undue influence. It looks for hypocrisy and offenses against the interests of the broader public.

Service to the status quo, and zealous suspicion of power. Are these really the only two stands that a journalist can take? When I imagine the global financial system improving, actually improving in the sense of changing in a way that makes the lives of very many people better — say, becoming less prone to the sort of systemic collapse that puts tens of millions out of work — I don’t see it much assisted by either of these approaches to reporting, necessary though they might be.

The financial system is just that: a system, sprawling, messy, very complex, built of people and laws and machines. It serves a great many ends, both humane and monstrously avaricious. It won’t be much improved by forcing a few traders to resign in disgrace, or focusing public fury on the bonuses of bank executives (which, obscene though they may be, remain just a drop in the bucket.) It seems rather that improvement will require international agreement on arcane solutions both political and technical, things like risk models, capital reserve requirements, and trading platforms. This is regulation both in the sense of law and in the sense of code-as-law, because software is a deep part of the infrastructure of modern finance. Markets don’t just happen; they are human desires channeled by what we have agreed to allow, and by what our technology has been built to support. Markets are designed things.

So maybe what we need are designers. Geeks who like to understand very complex systems, and tinker with them. I want to borrow from the culture of “makers,” because maker culture plants a flag on this idea. It draws on the hacker tradition of technical mastery, the DIY aesthetic perfected by the punks, and the best disruptive tendencies of global counter-culture. It lives in online forums and nerdy meetups and on the dingy couches of hack spaces. This is the chaotic ecosystem that powers Silicon Valley, and I bet it’s the secret ingredient that government planners miss when they build huge technology parks that end up empty.

But most of all, makers are deeply participatory. Where the political activist sees persuasion as the ultimate goal, the maker wants to personally rewire the system. This requires a deep love of the inner workings of things, the finicky, empirical details of how the real world is put together. A maker combines the democratic instinct with the technologist’s hands-on ability. And increasingly, makers are directing their attention to social problems. Efforts such as crisis mapping and Code For America and the whole information and communication technologies for development (ICT4D) movement are evidence of this. Maker language has recently been spotted at the White House and the United Nations.

The global financial system is just the sort of complex, intricate, part technical and part social system that makers would love, if only they could open it up and look inside. There are textbooks, but you can’t learn how the world actually works from textbooks. What would it take to open the global financial system to independent minds? Because it will be these independent minds — smart, deeply informed, creative — who will pore over the arcania of today in order to conceive of the better worlds to come.

Consider the latest draft of the Basel III standards for international banking. Who reads such dense and technical stuff? The professional regulator is obliged to sit at their desk and study this document. The financier wants only to understand how these rules will make or cost them money. The muckraker might ask who is making the rules and why. Another journalist will look for headlines of broad interest, but almost certainly won’t have the technical background to trace the subtle implications. But a maker would read these standards because they are changes in the operating system of global finance. And of these, it might be the maker, the specialized outsider, who is most qualified to understand the detailed, systemic effects on everyone else. The systems that underlie finance have become so fast and so complex that we don’t really understand the interactions. The people who know it best are mostly too busy making money to explain it to the rest of us. The public interest is in dire need of geeks who are not on the payroll.

There is a journalism to be done here, but it’s not the journalism of making people money, penning morality tales, or interesting articles in the Sunday paper. It’s a techno-social investigative journalism for those who have chosen to use their specialized knowledge in the interests of the rest of us. It’s a journalism that generalist reporters may be ill equipped to do.

We already have models for this. Dowser.org practices “solutions journalism,” writing about how best to solve societal problems. I appreciate that, but I don’t think they’ve conceived of their audience as the policy and technology geeks who will one day flesh out and implement those solutions. The contemporary science journalism ecosystem might be a better example. There are science reporters at news organizations, but the best science reporting now tends to come from elsewhere. Science, like finance, is absurdly specialized, and so its chronicling has been taken over by networks of specialists — very often scientists themselves, the ones who have learned to write. Science blogging is thriving. Its audience is the general public, yes, but also other scientists, because it’s the real thing. Even better, science writing exists in a web of knowledge: you can follow the links and go arbitrarily deep into the original research papers. And if you still have questions, the experts are already active online. Compare this to the experience of reading an economics article in the paper.

We don’t have much truly excellent journalism on deep, intricate topics, issues with enormous technical and institutional complexity. There’s some, but it’s mostly in trade publications with little sense of the social good, or tucked away in expensive journals which speak to us in grown-up tones and don’t know how to listen for the questions of the uninitiated. And yet our world abounds in complex problems! Sustainability, climate change, and energy production. Security, justice, and the delicate tradeoffs of civic freedoms. Health care for the individual, and for entire countries. The policies of government from the international to the municipal. And governments themselves, in all their gruesome operational detail. These things are not toys. But when journalists write about such issues, they satisfy themselves with discovering some flavor of corruption, or they end up removing so much of the substance that readers cannot hope to make a meaningful contribution. Perhaps this is because it has always been assumed that there is no audience for wonkish depth. And perhaps that’s true. Perhaps there won’t ever be a “mainstream” audience for this type of reporting, because the journalism of makers is directed to those who have some strange, burning desire to know the gory details, and are willing to invest years of their life acquiring background knowledge and building relationships. Can we not help these people? Could we encourage more of them to exist, if we served them better?

This is a departure from the broadcast-era idea of “the public.” It gives up on the romantic notion of great common narratives and tries instead to serve particular pieces of the vast mosaic of communities that comprise a society. But we are learning that when done well, this kind of deep, specialist journalism can strike surprising chords in a global population that is more educated than it has ever been. And the internet is very, very good at routing niche information to communities of interest. We have the data to show this. As Atlantic editor Alexis Madrigal put it, “I love analytics because I owe them my ability to write weird stories on the Internet.”

Where is the journalism for the idealist doer with a burning curiosity? I don’t think we have much right now, but we can imagine what it could be. The journalism of makers aligns itself with the tiny hotbeds of knowledge and practice where great things emerge, the nascent communities of change. Its aim is a deep understanding of the complex systems of the real world, so that plans for a better world may constructed one piece at a time by people who really know what they’re talking about. It never takes itself too seriously, because it knows that play is necessary for exploration and that a better understanding will come along tomorrow. It serves the talent pools that give rise to the people who are going to do the work of bringing us a potentially better world — regardless of where in society these people may be found, and whether or not they are already within existing systems of power. This is a theory of civic participation based on empowering the people who like to get their hands dirty tinkering with the future. Maybe that’s every bit as important as informing voters or getting politicians fired.

Different medium, different story form. It’s clear that each new technology — photography, radio, television — has brought with it different ways of constructing a narrative, and different ways those narratives fit into the audience’s lives. Online media are no different, and the step between analog and digital is in many ways much larger than any that has come before, because the internet connects the audience to each other as well as to the newsroom.

Here’s my attempt at a little reading list of recent work on the structure of stories. Pointers to additional material are welcome!

The Big Picture
What’s wrong with the article, anyway? Jeff Jarvis explores this question in “The article as luxury or by-product.” This essay provoked lots of interesting reaction, such as from Mathew Ingram.

So how do we understand ways to fix this? Vadim Lavrusik takes a shot at this question and comes up with the building blocks of context, social, personalization, mobile, participation. It’s a good taxonomy so I’m going to partially steal it for this post.

At the  NYC Hacks Hackers meetup last week, Trei Brundrett took us through SB Nation’s “story stream” product, and Gideon Lichfield of The Economist gave a really nice run through of the “news thing” concept that was fleshed-out collaboratively last month at Spark Camp by Gideon, Matt Thompson, and a room full of others. Very meaty, detailed, up-to-the-minute discussions, for serious news nerds. Video here.

Context
You just can’t do better than Matt Thompson’s “An antidote for web overload.” I also recommend Matt’s wonderful “The three key parts of news stories that are usually missing.” Another good primer is Jay Rosen’s “Future of Context” talk at SXSW.

See also my “Short doesn’t mean shallow,” about hyperlinks as a contextual storytelling form.

For an example of these ideas in action, consider Mother Jone’s Egypt Explainer page — which Gideon Lichfield critiques in the video linked above.

Social
What does it mean for news to be social anyway? Henry Jenkins argues for the power of “spreadable media” as a new distribution model.

In “What’s the point of social news?” I discuss two areas where social media have a huge impact on news: the use of social networks as a personalized filter, and distributed sourcing of tips and material.

Personalization
News is now personalized by a variety of filters, both social and algorithmic. Eli Pariser argues this puts us in a “filter bubble.” He may be right, but research by Pew and others [1,2] consistently shows that when users are allowed to recommend any URL to one other, the “news agenda” that the audience constructs has only 5%-30% of stories in common with mainstream media.

A comparison of questions asked of the White House by a Twitter audience vs. by journalists shows a remarkable difference in focus.  All of this this suggests to me that whatever else is happening, personalization meets an audience need that traditional broadcast journalism does not.

Besides, maybe not every person needs to see every story, if we view the goal of journalism as empowerment.

Participation
What do we know and what don’t we know about public participation in the journalism project, and what has worked or failed so far? Jay Rosen has an invaluable summary.

I also recommend the work of Amanda Michel as someone who does crowd-based reporting every day, and my own speculations on distributed investigative reporting.

Structured information
Is the product of journalism narratives or (potentially machine-readable) facts? Adrian Holovaty seems to be the first to have explored this in his 2006 essay “A fundamental way newspaper websites need to change.” This mantle has been more recently taken up by Stijn Debrouwere in his “Information Architecture for News Websites” series, and in Reg Chua’s “structured journalism,” and in a wide-ranging series at Xark.

There are close connections here to semantic web efforts, and occasional overlap between the semweb and journalism communities.

Mobile
I haven’t seen any truly good roundup posts on what mobile will mean for news story form, but there are some bits and pieces. Mobile is by-definition location aware, and Mathew Ingram examines how location is well used by Google News (and not by newsrooms.)

Meanwhile, Zach Seward of the Wall Street Journal has done some interesting news-related things with Foursquare.

Real time
Emily Bell, formerly of the Guardian and now at Columbia, explains why every news organization needs to be real-time.

For a granular look at how informations spreads in real time, consider Mathew Ingram on “Osama bin Laden and the new ecosystem of news.” For a case study of real-time mobile reporting, we have Brian Stelter’s “What I learned in Joplin.”

Huzzah!

Now I am looking for a pair of professional developers to make it a reality. It won’t be hard for the calibre of person I’m trying to find to get some job, but I’m going to try to convince you that this is the best job.

The project is called Overview. You can read about it at overview.ap.org. It’s going to be a system for the exploration of large to very large collections of unstructured text documents. We’re building it in New York in the main newsroom of The Associated Press, the original all-formats global news network. The AP has to deal with document dumps constantly. We download them from government sites. We file over 1000 freedom of information requests each year. We look at every single leak from Wikileaks, Anonymous, Lulzsec. We’re drowning in this stuff. We need better tools. So does everyone else.

So we’re going make the killer app for document set analysis. Overview will start with a visual programming language for computational linguistics algorithms. Like Max/MSP for text. The output of that will be connected to some large-scale visualization. All of this will be backed by a distributed file store and computed through map-reduce. Our target document set size is 10 million. The goal is to design a sort of visualization sketching system for large unstructured text document sets. Kinda like Processing, maybe, but data-flow instead of procedural.

We’ve already got a prototype working, which we pointed at the Wikileaks Iraq and Afghanistan data sets and learned some interesting things. Now we have to engineer an industrial-strength open-source product. It’s a challenging project, because it requires production implementation of state-of-the-art, research-level algorithms for distributed computing, statistical natural language processing, and high-throughput visualization. And, oh yeah, a web interface. So people can use it anywhere, to understand their world.

Because that’s what this is about: a step in the direction of applied transparency. Journalists badly need this tool. But everyone else needs it too. Transparency is not an end in itself — it’s what you can do with the data that counts. And right now, we suck at making sense of piles of documents. Have you ever looked at what comes back from a FOIA request? It’s not pretty. Governments have to give you the documents, but they don’t have to organize them. What you typically get is a 10,000 page PDF. Emails mixed in with meeting minutes and financial statements and god-knows what else. It’s like being let into a decrepit warehouse with paper stacked floor to ceiling. No boxes. No files. Good luck, kiddo.

Intelligence agencies have the necessary technology, but you can’t have it. The legal profession has some pretty good “e-discovery” software, but it’s wildly expensive. Law enforcement won’t share either. There are a few cheapish commercial products but they all choke above 10,000 documents because they’re not written with scalable, distributed algorithms. (Ask me how I know.) There simply isn’t an open, extensible tool for making sense of huge quantities of unstructured text. Not searching it, but finding the patterns you didn’t know you were looking for. The big picture. The Overview.

So we’re making one. Here are the buzzwords we are looking for in potential hires:

• We’re writing this in Java or maybe Scala. Plus JavaScript/WebGL on the client side.
• Be a genuine computer scientist, or at least be able to act like one. Know the technologies above, and know your math.
• But it’s not just research. We have to ship production software. So be someone who has done that, on a big project.
• This stuff is complicated! The UX has to make it simple for the user. Design, design, design!
• We’re open-source. I know you’re cool with that, but are you good at leading a distributed development community?

And that’s pretty much it. We’re hiring immediately. We need two. It’s a two-year contract to start. We’ve got a pair of desks in the newsroom in New York, with really nice views of the Hudson river. Yeah, you could write high-frequency trading software for a hedge fund. Or you could spend your time analyzing consumer data and trying to get people to click on ads. You could code any of a thousand other sophisticated projects. But I bet you’d rather work on Overview, because what we’re making has never been done before. And it will make the world a better place.

For more information, see :

• Writeups in Nieman Journalism Lab, O’Reilly Radar, Journalism.co.uk
• Video of a talk and live demo of the prototype.
• The official job posting.

Thanks for your time. Please contact jstray@ap.org if you’d like to work on this.

There have been many collaborative journalism projects, from the Huffington Post’s landmark “Off The Bus” election campaign coverage to the BBC’s sophisticated “user-generated content hub” to CNN’s iReport. One lesson in all of this is that form matters. Take the lowly comment section. News site owners have long complained, often with good reason, that comments are a mess of trolls and flame wars. But the prompt is supremely important in asking for online collaboration. Do journalists really want “comments”? Or do they want error corrections, smart additions, leads, and evidence that furthers the story?

Which leads me to investigative reporting. It’s considered a specialty within professional journalism, dedicated to getting answers to difficult questions — often answers that are embarrassing to those in power. I don’t claim to be very good at journalistic investigations, but I’ve done enough reporting to understand the basics. Investigative reporting is as much about convincing a source to talk as it is about filing a FOIA request, or running a statistical analysis on a government data feed. At heart, it seems to be a process of assembling widely dispersed pieces of information — connecting the distributed dots. Sounds like a perfect opportunity for collaborative work. How could we support that?

A system for tracking what’s already known
Reporters keep notes. They have files. They write down what was said in conversations, or make recordings. They collect documents. All of this material is typically somewhere on or around a reporter’s desk or sitting on their computer. That means it’s not online, which means no one else can build on it. Even within the same newsroom, notes and source materials are seldom shared. We have long had customer relationship management systems that track every contact with a customer. Why not a “source relationship management” system that tracks every contact with every source by every reporter in the newsroom? Ideally, such a system would be integrated into the reporter’s communications tools: when I make a phone call and hit record (after getting the source’s permission of course) that recording could be automatically entered into system’s files, stamped by time, date, and source, then transcribed by machine to make it searchable. Primary documents would be also be filed in the system, along with notes and links and comments from everyone working on the story. The entire story of the story could be in one place.

There have been experiments in collaborative journalistic files, such as OpenFile.ca or even good local wikis. But I don’t believe there has yet been a major professional newsroom which operated with open files. For that matter, I am not aware of this type of information filing system in existence anywhere in journalism, though I suspect it’s what intelligence services do.

Public verification processes
Journalism aims to be “true,” a goal which requires elaborate verification processes. But in every newsroom I’ve worked with, essential parts of the verification standards are not codified. “At least two sources” is a common maxim, but are there any situations where one is enough? For that matter, who counts as a definitive source? When is a conflict of interest serious enough to disqualify what someone is telling you? The answers to these questions and many more are a matter of professional practice and culture. This is confusing enough for a new reporter joining staff, let alone outsiders who might want to help.

Verification is necessarily contextual. Both the costs of verification and the consequences of being in error vary widely with circumstance, so journalists must make situational choices. How sure do we have to be before we say something is true, how do we measure that certainty, and what would it take to be more sure? Until this sort of nuanced guidance is made public, and the public is provided with experienced support to encourage good calls in complex or borderline cases, it won’t be possible to bring enthusiastic outsiders fully into the reporting process. They simply won’t know what’s expected of them, to be able to participate in the the production of a product to certain standards. Those standards depend on what accuracy/cost/speed tradeoffs best serve the communities that a newsroom writes for, which means that there is audience input here too.

What is secret, or, who gets to participate?
Traditionally, a big investigative story is kept completely secret until it’s published. This is shifting, as some journalists begin to view investigation as more of a process than a product. However, you may not want the subject of an investigation to know what you already know. It might, for example, make your interview with a bank CEO tricky if they know you’ve already got the goods on them from a former employee. There are also off-the-record interviews, embargoed material, documents which cannot legally be published, and a multitude of concerns around the privacy rights of individuals. I agree with Jay Rosen when he says that “everything a journalist learns that he cannot tell the public alienates him from the public,” but that doesn’t mean that complete openness is the solution in all cases. There are complex tradeoffs here.

So access to at least some files must be controlled, for at least some period of time. Ok then — who gets to see what, when? Is there a private section that only staff can see and a public section for everyone else? Or, what about opening some files up to trusted outsiders? That might be a powerful way to extend investigations outside the boundaries of the newsroom, but it brings in all the classic problems of distributed trust, and more generally, all the issues of “membership” in online communities. I can’t say I know any good answers. But because the open flow of information can be so dramatically productive, I’d prefer to start open and close down only where needed. In other words, probably the fastest way to learn what truly needs to be secret is to blow a few investigations when someone says something they shouldn’t have, then design processes and policies to minimize those failure modes.

There is also a professional cultural shift required here, towards open collaboration. Newsrooms don’t like to get scooped. Fair enough, but my answer to this is to ask what’s more important: being first, or collectively getting as much journalism done as possible?

Safe places for dangerous hypotheses
Investigative journalism requires speculation. “What if?” the reporter must say, then go looking for evidence. (And equally, “what if not?” so as not to fall prey to confirmation bias.) Unfortunately, “what if the district attorney is a child molester?” is not a question that most news organizations can tolerate on their web site. In the worst case, the news organization could be sued for libel. How can we make a safe and civil space — both legally and culturally — for following speculative trains of thought about the wrongdoings of the powerful? One idea, which is probably a good idea for many reasons, is to have very explicit marking of what material is considered “confirmed,” “vetted,” “verified,” etc. and what material is not. For example, iReport has such an endorsement system. A report marked “verified” would of course have been vetted according to the public verification process. In the US, that marking plus CDA section 230 might solve the legal issues.

A proposed design goal: maximum amplification of staff effort
There are very many possible stories, and very few paid journalists. The massive amplification of staff effort that community involvement can provide may be our only hope for getting the quantity and quality of journalism that we want. Consider, for example, Wikipedia. With a paid staff of about 35 they produce millions of near-real time topic pages in dozens of languages.

But this is also about the usability of the social software designed to facilitate collaborative investigations. We’ll know we have the design right when lots of people want to use it. Also: just how much and what types of journalism could volunteers produce collaboratively? To find out, we could try to get the audience to scale faster than newsroom staff size. To make that happen, communities of all descriptions would need to find the newsroom’s public interface a useful tool for uncovering new information about themselves even when very little staff time is available to help them. Perhaps the best way to design a platform for collaborative investigation would be to imagine it as encouraging and coordinating as many people as possible in the production of journalism in the broader society, with as few full time staff as possible. These staff would be experts in community management and information curation. I don’t believe that all types of journalism can be produced this way or that anything like a majority of people will contribute to the process of journalism. Likely, only a few percent will. But helping the audience to inform itself on the topics of its choice on a mass scale sounds like civic empowerment to me, which I believe to be a fundamental goal of journalism.

Accuracy is a hard thing to measure because it’s a hard thing to define. There are subjective and objective errors, and no standard way of determining whether a reported fact is true or false. But a small group of academics has been grappling with these questions since the early 20th century, and undertaking periodic news accuracy surveys. The results aren’t encouraging. The last big study of mainstream reporting accuracy found errors (defined below) in 59% of 4,800 stories across 14 metro newspapers. This level of inaccuracy — where about one in every two articles contains an error — has persisted for as long as news accuracy has been studied, over seven decades now.

With the explosion of available information, more than ever it’s time to get serious about accuracy, about knowing which sources can be trusted. Fortunately, there are emerging techniques that might help us to measure media accuracy cheaply, and then increase it. We could continuously sample a news source’s output to produce ongoing accuracy estimates, and build social software to help the audience report and filter errors. Meticulously applied, this approach would give a measure of the accuracy of each information source, and a measure of the efficiency of their corrections process (currently only about 3% of all errors are corrected.) The goal of any newsroom is to get the first number down and the second number up. I am tired of editorials proclaiming that a news organization is dedicated to the truth. That’s so easy to say that it’s meaningless. I want an accuracy process that gives us something more than a rosy feeling.

This is a long post, but there are lots of pretty pictures. Let’s begin with what we know about the problem.

An error in every other story
To measure news accuracy, you need a process for counting errors in published stories. This process has to be independent of the original reporting, otherwise you can’t learn anything about the story that the reporter didn’t already know. Real world reporting isn’t always clearly “right” or “wrong,” so it will often be hard to decide whether something is an error or not. But we’re not going for ultimate Truth here,  just a general way of measuring some important aspect of the idea we call “accuracy.” In practice it’s important that the error counting method is simple, clear and repeatable, so that you can compare error rates of different times and sources.

The first systematic measurements of media accuracy approached these problems by asking the story’s sources to find errors in finished articles. In 1936 Mitchell V. Charnley of the University of Minnesota ”mailed 1,000 news items clipped from three Minneapolis dailies to persons named in the stories, asking for their perceptions of inaccuracies,” according to a description in a later similar study (Charnley’s original paper isn’t online, boo.) This method of asking the sources whether the story is correct has some obvious shortcomings, and sources have their own agendas. However, it’s good at detecting basic errors of fact, such as incorrect names, dates, places, figures, occupations, etc. And because (roughly) this same methodology has been used in media accuracy studies ever since then, it’s possible to produce (rough) comparisons of accuracy over time.

This table is from Scott R. Maier’s 2005 paper “Accuracy Matters: A Cross-Market Assessment of Newspaper Error and Credibility,” which is both the largest and most recent news accuracy survey, and absolutely required reading for anyone with an interest in the subject. Maier checked 4,800 consecutive articles across 14 American newspapers, and,

This study’s central finding is sobering: More than 60% of local news and news feature stories in a cross-section of American daily newspapers were found in error by news sources, an inaccuracy rate among the highest reported in nearly seventy years of research, and empirical evidence corroborating the public’s impression that mistakes pervade the press. In about every other article, sources identified “hard” objective errors.

Maier and others counted both simple “objective” errors of fact and “subjective” errors such as over-emphasis or omission of an aspect of the story. Subjective errors are much fuzzier category and newsmakers are not necessarily neutral. But, says Maier,

Subjective errors, though by definition involving judgment, should not be dismissed as merely differences in opinion. Sources found such errors to be about as common as factual errors and often more egregious [as rated by the sources.]

But subjective errors are a very complex category, so for today let’s not count them at all. Purely “objective,” straightforward errors of fact were found in 48% of the stories Maier checked, whereas 61% of stories had errors of either type. There seems to be no escaping the conclusion that, according to the newsmakers, about half of all American newspaper stories contained a simple factual error in 2005. And this rate has held about steady since we started measuring it seven decades ago.

Maier’s work is amazing, and you should go read it. He also investigates which types of errors are most common (top three: misquotation, inaccurate headline, numbers wrong) and how error rate affects perceived story and newspaper credibility, and includes a fantastic bibliography of previous accuracy work. But all of this is still a very limited glimpse. The figures in the table above cover only newspapers, not online news sources, or television and radio. And it looks only at “old media” news organizations, not digital-only newsrooms, social media, and blogs. In the end, as extensive as academic accuracy research is, we only have accuracy measurements for a handful sources at a few points in time. This type of work is not useful for ongoing evaluation of accuracy strategies within a newsroom, and it doesn’t provide consumers with broad enough information to make good choices about where they get their news.

Continuous error sampling
One of the major problems with previous news accuracy metrics is the effort and time required to produce them. In short, existing accuracy measurement methods are expensive and slow. I’ve been wondering if we can do better, and a simple idea comes to mind: sampling.

The core idea is this: news sources could take an ongoing random sample of their output and check it for accuracy — a fact check spot check. Stories could be checked for errors by asking sources in the traditional manner, or through independent verification by a reporter who did not originally work on the story. I’m imagining that one person could check a couple stories per day in this fashion. Although this isn’t much, over time these samples will add up to an ongoing estimate of overall newsroom accuracy.

Standard statistical theory tells us what the error on that estimate will be for any given number of samples (If I’ve got this right, the relevant formula is standard error of a population proportion estimate without replacement.) At a sample rate of a few stories per day, daily estimates of error rate won’t be worth much. But weekly and monthly aggregates will start to produce useful accuracy estimates. For example, if a newsroom produces 1,000 stories per month and checks 60 of those — two per day — the error on the monthly estimate will be about ±10% (95% CI.) Or you could average three months of data and get an accuracy estimate to within ±6%. The result would be a graph like this:

This graph shows that NewsCo’s ongoing accuracy efforts are effective: the estimated error rate has decreased from about 50% to 35% over the course of a year. Given the margins of error on the estimates, this is a statistically significant difference indicating a real improvement, not just sampling noise. I would really like to see a news source that displayed an accuracy graph like this on their site. Otherwise, how can anyone really claim that they’re getting the facts right? Or even claim that they’re more accurate than a random blogger?

Fixing as many errors as possible
Meanwhile, the online corrections process is evolving. I now know of four different news outlets that have a “report an error” link or form on every story: The Washington Post, the Huffington Post, the Register Citizen of Lichfield County, CT, and the Daily Local of Chester County, PA. The MediaBugs project can also be used to report errors on any other news site.

Asking your users to report inaccuracies strikes me as a fabulous idea, and likely very productive (see: “someone is wrong on the internet!“) I have no knowledge of the quantity of errors submitted using these forms, or how the corrections process works. My suspicion is that each submitted correction sends an email to some hapless Engagement Editor who than has to cull the reports and route each plausible error to the story’s original reporter, if that reporter can be bothered to deal with audience feedback. (Excuse my snark, but more than one seasoned hack has told me how much they hate the idea that responding to users might be part of their job. Thankfully this attitude does seem to be on the way out.)

This very manual error correction system won’t scale. We can do better by asking the users to help filter error reports. It would be straightforward to implement a user-viewable queue of submitted errors for each story. Then the user who spots an error could a) first check to see if it’s already been reported, which would cut down duplicate reports, and b) vote on the severity of existing error reports. The idea is to do collaborative filtering on error reports, so that the most serious and plausible come to the attention of the corrections editor first. Users could be encouraged to submit supporting evidence of the error, in the form of URLs to primary sources, by automatically giving precedence to items which include links.

I would really like to see the day when every news story on every device includes a “submit addition or correction” button. And once the corrections process exists, we can start looking at the data it generates. One goal would be to increase the efficiency of the corrections process in catching errors. That will drive the number of detected errors up, which might be very scary for newsrooms which are in the habit of pretending that every story is perfect. No newsroom wants to be the first to let the audience see that half of their stories contain a factual error, even if most of those errors are going to be minor. And yet, if decades of news accuracy research are to be believed, this is inevitable, because those errors are already there across the industry, silent. This makes me suspect that good corrections processes — real, web-native, efficient crowd-sourced corrections — will not be quickly adopted.

But even if you were excited to get real corrections going, how would you know if your process was really catching all of the errors?

Combining accuracy and corrections measures
Suppose a newsroom was doing random samples of accuracy, and monitoring the number of errors corrected through the user-submission process and all other correction routes. Then we could plot errors corrected versus (estimated) errors found, like this:

A newsroom monitoring its accuracy and corrections processes in this way would have two goals. First, drive errors down, perhaps by instituting an accuracy checklist, like this one from Steve Buttry. Second, drive corrections up by making it easy and rewarding for users to submit corrections, building better error queuing and filtering systems, assigning more staff resources to corrections, etc. The goal is to have the blue line and the red line meet, meaning that all errors are rapidly corrected.

Used internally, this sort of chart could give a quantitative understanding of how well a newsroom is doing in terms of accuracy, or at least those aspects of the concept of “accuracy” that are well-captured by the metrics. One could even break it out by desk to see if certain parts of the newsroom are over- or under-performing — or perhaps we’ll learn that certain types of reporting are just harder to get right. Proudly posted externally, this sort of chart would demonstrate a tangible commitment to accuracy, far better than any values statement. And if the metric could be standardized across the industry — in much the same way that, e.g. accountants have standardized their reporting — then we would finally be able to compare the ongoing accuracy of two news sources in an empirical way. I think there might be some real hope for standardized, fair accuracy metrics, because news organizations are obliged to fix the errors they find on an audit, and this will leave a trail in the revision history that serves as a public record of what a newsroom considers an “error” to be.

Accuracy is not what it used to be
The explosion of information sources combined with the decades long decline in media trust makes this a great time to dream up ways to increase accuracy in journalism. One approach goes like this: the first step would be admitting how inaccurate journalism has historically been. Then we have to come up with standardized accuracy evaluation procedures, in pursuit of metrics that capture enough of what we mean by “true” to be worth optimizing. Meanwhile, we can ramp up the efficiency of our online corrections processes until we find as many useful, legitimate errors as possible with as little staff time as possible. It might also be possible do data mining on types of errors and types of stories to figure out if there are patterns in how an organization fails to get facts right.

There are some serious practical problems and gnarly missing details here. But think of the prize! I’d love to live in a world where I could compare the accuracy of information sources, where errors got found and fixed with crowd-sourced ease, and where news organizations weren’t shy about telling me what they did and did not know. Basic factual accuracy is far from the only measure of good journalism, but perhaps it’s an improvement over the current sad state of affairs: “aside from prizes, there really aren’t any other metrics for journalism quality.”

I do not think that giving equal time or credence to “left” and “right” points of view, or “pro” and “con”, or any other such pair, has anything to do with “balance.” First of all, there may be more enlightening ways to view the issue — not every story is best understood as a conflict. But more fundamentally, I think this word implies that journalists should strive to be equally acceptable (or equally unacceptable) to all sides. The often-used phrase “straight down the middle” has become a synonym for “fair,” but this metaphor gives me hives. What does being exactly between two poles have to do with truth? A judge doesn’t define fairness in this way, and neither should journalists. Rather, I believe that our job is to represent reality as best as we can discern it, with humility, intelligence, sensitivity, and transparency.

A better definition of balance might mean: the story leaves the viewer with an impression that matches what is actually “out there” in the world, plus a detailed critique (links are really helpful for that part.) There is room in this for pluralistic truths, because we live in a pluralistic world, but that doesn’t mean all truths are equal. It is surely important to acknowledge the existence of and fully understand the points of view of the various “sides” in a dispute, but I feel we would be failing significantly if we did not also convey:

• whether each viewpoint is in the majority or the minority, and by whom it is held.
• an evaluation of the evidence supporting each position, if the issue involves testable claims of fact.
• a thorough sense of who is affected by the events in the story, even if those people are unpopular or invisible.
• what is still uncertain, either because the question is inherently difficult to answer or because we have not yet completed deeper reporting.

In other words, I don’t think it’s enough to say “this person disagrees” without some comment on whether or not their account holds up to the truth, as best we can determine it. “He said, she said” is right out. In fact, forget “balance.” It’s just not the right word at all, because it implies an equivalence that may not exist. Yes, the journalist needs to be dispassionate with the facts, but the facts might not favor all parties equally. I’d much rather think about reporting in terms of completeness, representativeness, accuracy, context, accountability, and transparency. That’s what fairness is to me.

There is something extraordinarily rich in the intersection of computer science and journalism. It feels like there’s a nascent field in the making, tied to the rise of the internet. The last few years have seen calls for a new class of  “programmer journalist” and the birth of a community of hacks and hackers. Meanwhile, several schools are now offering joint degrees. But we’ll need more than competent programmers in newsrooms. What are the key problems of computational journalism? What other fields can we draw upon for ideas and theory? For that matter, what is it?

I’d like to propose a working definition of computational journalism as the application of computer science to the problems of public information, knowledge, and belief, by practitioners who see their mission as outside of both commerce and government. This includes the journalistic mainstay of “reporting” — because information not published is information not known — but my definition is intentionally much broader than that. To succeed, this young discipline will need to draw heavily from social science, computer science, public communications, cognitive psychology and other fields, as well as the traditional values and practices of the journalism profession.

“Computational journalism” has no textbooks yet. In fact the term barely is barely recognized. The phrase seems to have emerged at Georgia Tech in 2006 or 2007. Nonetheless I feel like there are already important topics and key references.

Data journalism
Data journalism is obtaining, reporting on, curating and publishing data in the public interest. The practice is often more about spreadsheets than algorithms, so I’ll suggest that not all data journalism is “computational,” in the same way that a novel written on a word processor isn’t “computational.” But data journalism is interesting and important and dovetails with computational journalism in many ways.

• The Nieman Journalism Lab’s interview with Guardian Data Blog editor Simon Rogers remains a solid introduction to (one kind of) contemporary practice.
• The best practical guides I know are Rogers’ “How to: get to grips with data journalism” and Dan Nguyen’s series of data-scraping tutorials at ProPublica.
• Stanford’s Journalism in the Age of Data is an hour-long documentary on data journalism and visualization.
• The web is a linked system of human-readable documents. Now Tim Berners-Lee wants to create a web of machine-readable linked data. The full potential is unclear, but it’s a big idea that may come to be the backbone of semantic web visions. The New York Times, The Guardian, and others are experimenting with open data APIs.
• Everyblock creator Adrian Holovaty seems to have been the first to suggest that reporters file structured data in his 2006 “A Fundamental Way Newspaper Websites Need to Change.” This idea is beautifully expanded in Stijn Debrouwere’s “Information Architecture for News Websites” series.

Visualization
Big data requires powerful exploration and storytelling tools, and increasingly that means visualization. But there’s good visualization and bad visualization, and the field has advanced tremendously since Tufte wrote The Visual Display of Quantitative Information. There is lots of good science that is too little known, and many open problems here.

• Tamara Munzner’s chapter on visualization is the essential primer. She puts visualization on rigorous perceptual footing, and discusses all the major categories of practice. Absolutely required reading for anyone who works with pictures of data.
• Ben Fry invented the Processing language and wrote his PhD thesis on “computational information design,” which is his powerful conception of the iterative, interactive practice of designing useful visualizations.
• How do we make visualization statistically rigorous? How do we know we’re not just fooling ourselves when we see patterns in the pixels? This amazing paper by Wickham et. al. has some answers.
• Is a visualization a story? Segal and Heer explore this question in “Narrative Visualization: Telling Stories with Data.”

Computational linguistics
Data is more than numbers. Given that the web is designed to be read by humans, it makes heavy use of human language. And then there are all the world’s books, and the archival recordings of millions of speeches and interviews. Computers are slowly getting better at dealing with language.

• Word frequency techniques like tf-idf and the vector space document model are very simple and very useful. See also stemming. Lots more in the wonderful (and free!) Introduction to Information Retrieval. This book explains how search engines are built, and  discusses tf-idf etc. in great technical detail.
• Statistical language models are increasingly important for all kinds of applications. Michael Nielsen has a great introduction to statistical machine translation. Google’s Peter Norvig discusses how he implemented statistical spelling correction on his laptop during a long plane flight. For the full deal, see the book Foundations of Statistical Natural Language Processing.
• On a related note, Google N-gram viewer lets you look at the frequency of short phrases within 4% of all books published, ever. The excellent paper gives examples of how to use this for cultural research. Dan Cohen has important criticisms.
• Speech-to-text algorithms enable automated transcription, and Matt Thompson explores the huge implications for journalism.
• Reuters maintains the OpenCalais entity extraction service, which parses text to contextually determine who and what is referenced.
• IBM’s Watson project built a question-answering system that reads reference books and wins at Jeopardy. Imagine how useful to journalists and curious readers this could be! This paper on the DeepQA system describes how they did it.

Communications technology and free speech
Code is law. Because our communications systems use software, the underlying mathematics of communication lead to staggering political consequences — including whether or not it is possible for governments to verify online identity or remove things from the internet. The key topics here are networks, cryptography, and information theory.

• The Handbook of Applied Cryptography is a classic, and free online. But despite the title it doesn’t really explain how crypto is used in the real world, like Wikipedia does.
• It’s important to know how the internet routes information, using TCP/IP and BGP, or at a somewhat higher level, things like the BitTorrent protocol. The technical details determine how hard it is to do things like block websites, suppress the dissemination of a file, or remove entire countries from the internet.
• Anonymity is deeply important to online free speech, and very hard. The Tor project is the outstanding leader in anonymity-related research.
• Information theory is stunningly useful across almost every technical discipline. Pierce’s short textbook is the classic introduction, while Tom Schneider’s Information Theory Primer seems to be the best free online reference.

Tracking the spread of information (and misinformation)
What do we know about how information spreads through society? Very little. But one nice side effect of our increasingly digital public sphere is the ability to track such things, at least in principle.

• Memetracker was (AFAIK) the first credible demonstration of whole-web information tracking, following quoted soundbites through blogs and mainstream news sites and everything in between. Zach Seward has cogent reflections on their findings.
• The Truthy Project aims for automated detection of astro-turfing on Twitter. They specialize in covert political messaging, or as I like to call it, computational propaganda.
• We badly need tools to help us determine the source of any given online “fact.” There are many existing techniques that could be applied to the problem, as I discussed in a previous post.
• If we had information provenance tools that worked across a spectrum of media outlets and feed types (web, social media, etc.) it would be much cheaper to do the sort of information ecosystem studies that Pew and others occasionally undertake. This would lead to a much better understanding of who does original reporting.

Filtering and recommendation
With vastly more information than ever before available to us, attention becomes the scarcest resource. Algorithms are an essential tool in filtering the flood of information that reaches each person. (Social media networks also act as filters.)

• The paper on preference networks by Turyen et. al. is probably as good an introduction as anything to the state of the art in recommendation engines, those algorithms that tell you what articles you might like to read or what movies you might like to watch.
• Before Google News there was Columbia News Blaster, which incorporated a number of interesting algorithms such as multi-lingual article clustering, automatic summarization, and more as described in this paper by McKeown et. al.
• Anyone playing with clustering algorithms needs to have a deep appreciation of the ugly duckling theorem, which says that there is no categorization without preconceptions. King and Grimmer explore this with their technique for visualizing the space of clusterings.
• Any digital journalism product which involves the audience to any degree — that should be all digital journalism products — is a piece of social software, well defined by Clay Shirky in his classic essay, “A Group Is Its Own Worst Enemy.” It’s also a “collective knowledge system” as articulated by Chris Dixon.

Measuring public knowledge
If journalism is about “informing the public” then we must consider what happens to stories after publication — this is the “last mile” problem in journalism. There is almost none of this happening in professional journalism today, aside from basic traffic analytics. The key question here is, how does journalism change ideas and action? Can we apply computers to help answer this question empirically?

• World Public Opinion’s recent survey of misinformation among American voters solves this problem in the classic way, by doing a randomly sampled opinion poll. I discuss their bleak results here.
• Blogosphere maps and other kinds of visualizations can help us understand the public information ecosystem, such as this interactive visualization of Iranian blogs. I have previously suggested using such maps as a navigation tool that might broaden our information horizons.
• UN Global Pulse is a serious attempt to create a real-time global monitoring system to detect humanitarian threats in crisis situations. They plan to do this by mining the “data exhaust” of entire societies — social media postings, online records, news reports, and whatever else they can get their hands on. Sounds like key technology for journalism.
• Vox Civitas is an ambitious social media mining tool designed for journalists. Computational linguistics, visualization, and more.

Research agenda
I know of only one work which proposes a research agenda for computational journalism.

• “Computational Journalism: A Call to Arms for Database Researchers” by Sarah Cohen et. al. raises the very intriguing possibility of building systems that automatically or semi-automatically scan databases for stories, document the rationale for believing certain facts, etc.

This paper presents a broad vision and is really a must-read. However, it deals almost exclusively with reporting, that is, finding new knowledge and making it public. I’d like to suggest that the following unsolved problems are also important:

• Tracing the source of any particular “fact” found online, and generally tracking the spread and mutation of information.
• Cheap metrics for the state of the public information ecosystem. How accurate is the web? How accurate is a particular source?
• Techniques for mapping public knowledge. What is it that people actually know and believe? How polarized is a population? What is under-reported? What is well reported but poorly appreciated?
• Information routing and timing: how can we route each story to the set of people who might be most concerned about it, or best in a position to act, at the moment when it will be most relevant to them?

This sort of attention to the health of the public information ecosystem as a whole, beyond just the traditional surfacing of new stories, seems essential to the project of making journalism work.

Live coverage

The Huffington Post, the New York Times, the Wall Street Journal, ABC, CNN, Mashable, and many others, including even Mother Jones had live web video. But you can get live video on television, so perhaps the digitally native form of the live blog is more interesting. This can include commentary from multiple reporters, reactions from social media, link round-ups, etc. The New York Times, the Boston Globe, The Wall Street Journal, CNN, MSNBC, and many others had a live blog. The Huffington Post’s effort was particularly comprehensive, continuing well into Wednesday afternoon.

Multi-format, socially-aware live coverage is now standard, and by my reckoning makes television look meagre. But the experience is not really available on tablet and mobile yet. For example, almost all of the live video feeds were in Flash and therefore unavailable on Apple devices, as CNET reports.

As far as tools, there was some use of Coveritlive, but most live blogs seemed to be using nondescript custom software.

Visualizations

Lots of visualization love this year. But visualizations take time to create, so most of them were rooted in previously available SOTU information. The Wall Street Journal did an interactive topic and keyword breakdown of Obama’s addresses to congress since 2009, which moved about an hour after Tuesday’s speech concluded.

The New York Times had a snazzy graphic comparing the topics of 75 years of SOTU addresses,  by looking at the rates of certain carefully chosen words. Rollovers for individual counts, but mostly a flat thing.

The Guardian Data Blog took a similar historical approach, with Wordles for SOTU speeches from Obama and seven other presidents back to Washington. Being the Data Blog, they also put the word frequencies for these speeches into a downloadable spreadsheet. It’s a huge image, definitely intended for big print pages.

A shout-out to my AP colleagues for all their hard work on our SOTU interactive, which included the video, a fact-checked transcript, and an animated visualization of Twitter responses before, during, and after the State of the Union.

But it’s not clear what, if anything, we can actually learn from such visualizations. In terms of solid journalism content, possibly the best visualization came not from a news organization but from Nick Diakopoulos and co. at Rutgers University. Their Vox Civitas tool does filtering, search, and visualization of over  100,000 tweets captured during the address.

I find this interface a little too complex for general audience consumption — definitely a power user’s tool. But the algorithms are second to none. For example, Vox Civitas compares tweets to the text of the speech within the previous two minutes to detect “relevance,” and the automated keyword extraction — you can see the keywords at the bottom of the interface above — is based on tf-idf and seems to choose really interesting and relevant words. The interactive graph of keyword frequency over time clearly shows the sort of information that I had hoped to reveal with the AP’s visualization.

Fact Checking

A number of organizations did real-time or near real-time fact checking, as Yahoo reports. The Sunlight Foundation used itsSunlight Live system fo real-time fact checks and commentary. This platform, incorporating live video, social media monitoring, and other components is expected to be available as an open-source web app, for the use of other news organizations, by mid-2011.

The Associated Press published a long fact check piece (also integrated into the AP interactive), ABC had their own story, and CNN took a stab at it.

But the heaviest hitter was Politifact, who had a number of fact check rulings within hours and several more by Wednesday evening. These are together in a nice summary article, but as is their custom the individual fact checks are extensively documented and linked to primary sources.

Audience engagement

Pretty much every news organization had some SUTO action on social media, though with varying degrees of aggressiveness and creativity. Some of the more interesting efforts involved solicitation of audience responses of a specific kind. NPR asked people to describe their reaction to the state of the union in three words. This was promoted aggressively on Twitter and Facebook. They also asked for political affiliation, and split out the 4000 responses into Democratic and Republican word clouds:

Apparently, Obama’s salmon joke went down well. The Wall Street Journal went live Tuesday morning with “The State of the Union is…” asking viewers to leave a one word answer. This was also promoted on Twitter. Their results were presented in the same interactive, as a popularity-sorted list.

Aside from this type of interactive, we saw lots of agressive social media engagement in general. The more social-media savvy organizations were all over this, promoting their upcoming coverage and responding to their audiences. As usual, the Huffington Post was pretty seriously tweeting the event, posting about updates to their live blog, etc. and going well into Wednesday morning. Perhaps inspired by NPR, they encouraged people to tweet their #3wordreaction to the speech. They also collected and highlighted reaction from teachers, Sarah Palin, etc.

But as an AP colleague of mine asked, engagement to what end? Getting people’s attention is great, but then how do we, as journalists, focus that attention in a way that makes people think or act?

The White House

No online media roundup of the SOTU would be complete without a discussion of the White House’s own efforts, including web and mobile app presences. Fortunately, Nieman Journalism Lab has done this for us. Here I’ll just add that the White House livestreamed a Q&A session in front of  an audience immediately after the speech, in which White House Office of Public Engagement’s Kal Penn (aka Kumar) read questions from social media. Then Obama himself did an intervew Thursday afternoon in which he answered questions submitted as videos on YouTube.

Abstract
There are now several websites which allow users to assemble news content from around the internet by means of voting systems. The result is a new kind of front page that directly reflects what the audience believes to be salient, as opposed to what the editorial staff of a newsroom believes the audience should know. Content analyses of such sites show that they have little overlap with mainstream media agendas (5% in a previous study). In fact, many of the items selected by users would not traditionally be considered “news” at all. This paper examines the shift from editor to audience agendas in the context of previous theories of news production, discusses existing content analysis work on the subject, and reports on a new 30 day study of Digg.com versus NYTimes.com.

Introduction
No news organization can cover everything. Traditionally, it is ultimately the editor of a news publication who decides what is newsworthy: what stories reporters will follow, and what stories will be published. It has been considered part of the value of a news organization to determine what its audiences need to know about.

It’s never been entirely clear how professional journalists decide which events are worth reporting, out of all the events taking place in the world. Neither has it been obvious how editorial choices relate to the audience’s personal judgments about what is important, but  such questions were largely theoretical before the advent of the web. “I own a newspaper, you do not” was always the implicit end to discussions about who got to decide what was news.

Today, publishing is near-free and the news package has been disaggregated. An online audience member can select single stories that interest them, without reading or even really being aware of the traditional news package. Alongside this disaggregation we find a new class of online applications that re-aggregate content from multiple sources. Readers vote on pages from across the web, and the top-rated items are displayed on the aggregator’s home page.

News consumers are literally tearing the world’s newspapers apart and re-assembling them to fit their own agendas, including lots of content not traditionally considered news at all.

This paper examines what we can learn about the online audience’s judgment not only of what is important but what is news at all, and how it differs from that of traditional newsrooms. I review previous work on “news values”  and “news agenda” in professional journalism, look at measurements of what audiences view online, and report on my own 30 day quantitative study of Digg as compared to the New York Times.

Features of the audience-generated agenda
Online audiences seem to be selecting for themselves a radically different set of stories and topics than that assembled for them by the mainstream media. The most relevant work on this topic is 2007 study by the Pew Research Center’s Project for Excellence in Journalism [1]. The PEJ investigation followed Digg, Del.icio.us  and Reddit for one week, as well as the more conventionally edited Yahoo News, collecting a total of 644 stories. It compared these to 1,395 stories from the same period published in the PEJ’s News Coverage Index, a collection of mainstream news sources in print, online, network TV, cable, and radio. The report’s “key findings” are worth excerpting:

• The news agenda of the three user-sites that week was markedly different from that of the mainstream press. Many of the stories users selected did not appear anywhere among the top stories in the mainstream media coverage studied.

…

• The sources user news sites draw on are strikingly different from the mainstream media. Seven in ten stories on the user sites come either from blogs or Web sites such as YouTube and WebMd that do not focus mostly on news.

…

• Despite claims that the Web would internationalize consumers’ news diets, coverage across the three user-news sites focused more on domestic events and less on news from abroad than the mainstream media that week.

These points suggest the major themes that will recur in this paper. User news judgment is vastly different from editor news judgment.  Users do not appear to care whether or not stories sources are produced in traditional journalistic fashion. And “serious” journalism (on, e.g. international topics) is unpopular.

Theories of news agenda
How do professional journalists decide what stories to follow and publish?

Schudson [2] examines research into this question beginning with the “gatekeeper” model. In this framework, those who are in a position to decide what is published control what information may pass from the world into the news.  The notion is that the gatekeepers will inject their particular perspectives and biases into the news. And yet, early studies revealed that there is little variation in the wire stories chosen for publication by different local editors. Moreover, the “gatekeeper” approach doesn’t answer the question of what comes to the gate and how.

The term “gatekeeper” is still in use and provides a handy, if not altogether appropriate, metaphor for the relation of news organizations to news products. A problem with the metaphor is that it leaves “information” sociologically untouched, a pristine material that comes to the gate already prepared; the journalist as “gatekeeper” simply decides which pieces of prefabricated news will be allowed through the gate.

In pursuit of a more sophisticated model of how the news content is decided, Schudson identifies three broad schools of theory.

In political economy theories of news production, structural constraints determine what it is possible to publish, regardless of the intentions of individual journalists. Chomsky’s “propaganda model” is the archetypal example. According to Chomsky the news is strictly limited to and complicit in reporting only what is favorable to maintaining the (unjust) status quo. Issues of political interest, commercial pressures and the capitalist structures of society are cogently discussed in this branch of theory.

“Organizational” theories are also structural, but don’t necessarily see collusion between newsrooms and elites. Instead, the focus is on how the individual journalist within these structures ends up having little choice in how they operate. Noting that the great majority of news stories are the result of official reports from government agencies, Schudson says that the reporter sees the world as “bureaucratically organized”:

One study after another comes up with essentially the same observation, and it matters not whether the study is at the national, state, or local level — the story of journalism, on a day-to-day basis, is the story of the interaction of reporters and officials. Some claim officials generally have the upper hand. Some media critics, including many government officials, say reporters do. But there is little doubt that the center of news generation is the link between reporter and official, the interaction of the representatives of the news bureaucracies and the government bureaucracies. This is clear especially when one examines the actual daily practices of journalists.

Constructivist theories that investigate how meaning is produced for journalists and audiences, taking into account the cultures in which they both live. It differs from the organizational perspective in that it examines the symbols and ideas available to journalists. This body of theory is the one best suited to deal with framing choices, and questions of what is and isn’t surprising (and therefore newsworthy) within a particular culture.

Each of these theories provides interesting analytical tools, yet there is no overall theory that reliably answers our basic question: what makes the news? Lacking a clear explanation of this point, it’s difficult to know how well the public is being served by traditional news sources.

What is newsworthy?
If we ask journalists how they decide what beats to follow, what leads to investigate, and what stories to produce, we typically get answers involving the “newsworthiness” of various events. Yet journalists are at a loss to explain what this actually means. One veteran editor described news judgment to me as “tribal,” i.e. publication dependent and essentially arbitrary — which is of course at odds with theories of “objective” reporting. Hall writes about the difficulty of defining “newsworthy” in a 1973 essay on photojournalism [3]:

“News values” are one of the most opaque structures of meaning in modern society. All “true journalists” are supposed to possess it: few can or are willing to identify and define it. Journalists speak of “the news” as if events select themselves. Further, they speak as if which is the “most significant”  news story, and which “news angles” are the most salient are divinely inspired. Yet of the millions of events which occur every day in the world, only a tiny portion ever become visible as “potential news stories”; and of this proportion, only a small fraction are actually produced as the day’s news in the news media. We appear to be dealing, then, with a “deep structure” whose function as a selective device is un-transparent event to those who professionally most know how to operate it.

Perhaps our most comprehensive understanding of what “newsworthiness” actually means comes from Shoemaker [4] and colleagues. They performed very diverse studies of what people consider to newsworthy and found something surprising: the news agenda doesn’t reflect anyone’s personal judgment!

In our study of news in ten countries, Akiba Cohen and I (2006) discovered a disconnect between what people think is newsworthy and how prominently newspapers display the stories. People in four types of focus groups — journalists, public relations practitioners, low socio-economic audience, and high socio-economic audience — were asked to rank ten headlines according to their newsworthiness, each set being taken from their local newspapers several months earlier. The stories ranged (in percentiles) from the most prominent as displayed in the newspaper to the least prominent.

As expected, people within each focus group ranked the stories in much the same way, but we also found that journalists agreed with public relations practitioners, high SES audience members agreed with low SES audience members, journalists with audience members, and so on — no matter what their station in life, people agreed on how newsworthy the events were. This was true in each of the ten countries we studied.

But when we compared the peoples’ newsworthiness rankings to how prominently their local newspapers had displayed the stories, agreement was much lower. In some countries there was actually a negative relationship between how newsworthy people thought an event was and how prominently it was covered by the newspaper. In most countries, the relationship was positive, but much weaker than the relationships between the various groups of people.

The newsworthiness of an event is only one of many factors that determines how prominently the story will be covered. We cannot assume that the most prominently covered stories are the ones that people (whether editors, reporters, PR practitioners, doctors, mechanics, or teachers) think are most newsworthy, and we cannot reasonably expect people’s mental judgments about what is newsworthy to correlate highly with what actually becomes the social artifact news.

In other words, it does not appear that mainstream news agenda is representative of what even journalists think is newsworthy, let alone the audience.

This finding demands an explanation. Does the news as produced not reflect individual judgment due to structural issues such as, say, political pressures? Might this be an example of groupthink, where each person produces what they imagine everyone else in their culture wants? It is also possible that the people in Shoemaker’s focus groups were mis-reporting their judgments in some way, whether through selective perception and cognitive bias, Hawthorne and other experimental effects, or social pressures.

But if we assume for the moment that Shoemaker’s finding is believable, we do not yet have any good explanations of why it could be that the “social artifact” of news represents the newsworthiness judgment of no one in particular. But we do get a prediction: given a free choice, audiences would construct a news agenda that is dramatically different from the mainstream status quo.

What are Audiences Actually Reading?
Consumption is one form of audience judgment. Newspaper publishers have long known or assumed that their readers don’t have the same priorities as the newsroom. Stereotypically, it is sports and celebrity gossip that draw the most readers. This wasn’t necessarily an economic concern in the newsprint era given that the customer could not buy less than a whole paper. In any case, reader story choice was not immediately measureable. In some studies readers have been asked what they read, but this technique is subject to deep problems related to memory and perception. In another design, readers have been directly observed as they read the paper, but this is an artificial situation and subject to well-known distortions such the Hawthorne effect (where the subject tries to please the researcher).

In contrast, online measurements can be completely unobtrusive. In fact the raw data is already routinely recorded in web server logs, and collected by companies such as Nielsen Ratings. Tewksbury [5] analyzed visits to 13 pre-selected news sites by 9,209 randomly-selected Americans in the months of March and May 2000.  From [5] table 2:

True to stereotype, sports lead with 26% of views, followed by business, entertainment, features, and national news. The political and international news considered so important by professional journalists together comprise just 11.5% of total page views.

Academic work on news agenda has taken a similarly narrow focus, in that it has not really come to grips with the implications of a readership who doesn’t much care for the news that journalists think is important. Media effects researchers have for decades used studies where subjects are asked “what is the most important problem facing the nation?” [6] Within this research paradigm, by definition only “problems facing the nation” can generate news. Where does sports and lifestyle reporting fit into this? Even before the era of user news sites, it seems that journalists and scholars alike had a skewed conception of how their audiences interacted with the news media.

The Audience Agenda-Generation Process
Audience-driven content aggregation sites such as Digg, Reddit, etc. all work on similar principles. Users can submit arbitrary URLs into the system, either directly on the aggregation site or via submission buttons that content publishers make available to readers on their site in the hope that their content will be promoted, e.g. “Digg this” buttons. Similar voting buttons are provided on the aggregation site for each item displayed. Votes collected from all locations are tallied for each item, and the ranked results constitute the user-generated “front page.”

The resulting rankings are not equivalent to a poll of readers. To begin with, votes must be counted in a time-limited way, or such sites would rank the most popular content of all time, as opposed to popular recent content.  The number of votes for an item also depends greatly on the number of people exposed to that item, and this has much to do with factors that influence the extent of “viral” transmission of an item through social media, including emotional response (see e.g. [7]) and the social network topology around the people who have an interest in the topic. Further, readers voting on the site are far more likely to vote for items that are more prominently displayed — and thus already more popular.

Nonetheless, this voting process produces some sort of snapshot of aggregate audience interest. It’s a relatively opaque sort of snapshot, and doesn’t clearly represent anything in particular. It will favor already popular items and items with emotional content — but so does pop culture. It’s not obvious that this user-generated news agenda is any “better” or “worse” than the agenda of a newsroom.

What we can say is that type of audience-generated agenda clearly draws on a wider array of sources than a traditional news publication. Any web page can be voted upon, not just content from “news” sites. Videos are relevant, as are blog postings on arbitrary topics. Crucially, the audience agenda generation process seems to involve only very weak preconceptions about what is potentially newsworthy — “anything on the web” — as compared to journalists and academics. Because users vote individually, mostly in private, not for pay, and effectively anonymously, we might also expect user agendas to be free from the structural and sociological constraints acting on a newsroom. Is it possible that user-generated agenda are simply a more honest reflection of what all of us actually consider newsworthy?

Audience-Generated Agendas in Detail
The PEJ study [1] examined user-edited sites in several ways. One of the most revealing is the (lack of) overlap between these sites and the stories in the Pew’s news coverage index of mainstream media outlets.

In the user-generated sites, [mainstream media] stories were barely visible. Overall, just 5% of the stories captured on these three sites overlapped with the ten most widely-covered stories in the Index (13% for Reddit, 4% for Digg, and 0% for Del.icio.us).

Again we see that there is very little overlap between what the mainstream media considers “news”, and the stories that users choose for themselves. Even accounting for the demographic skew of these sites — which are arguably still “early adopter” and over-represent tech geeks — the lack of agreement is astounding.

They further examine the content in terms of the top five subject categories for each site.

News Index (Mainstream Media)

Digg

Reddit

Del.icio.us

The difference of focus between editor- and audience-generated agenda stands out here. The audience sites had heavy coverage of technology and science issues, which again suggests demographic differences. Lifestyle was also much more popular. As in the online readership survey, what a traditional journalist would call “hard news” barely registers.

My own study of user vs. mainstream media agenda was a 30 day comparison of Digg and the New York Times, from 4 October 2009 to 4 November 2009. Each evening shortly after midnight, I took a screen capture of the front pages of both sites. With the browser window set to a full screen height of 1024 pixels, and ignoring the smallest sized links on the NYT page, both screen shots averaged about nine stories per day. I categorized the stories on each site and also tracked, for each day, how many stories appeared on both sites. In total, I collected 238 stories from Digg and 227 stories from the New York Times this way.

Rather than the default Digg page as used in the PEJ study, I used the “24 hour news” view in order to capture a list that was a little more comparable in time scale to a daily newspaper, and contained the “news” label which may tell content voters that “newsworthiness” is being asked for. Results by top 5 story categories:

Story Category — Digg

Story Category — NYT

In these tables, audience and editor agendas don’t seem that different. But consider this:

Although the selection of topics was similar, the actual stories almost never overlapped between Digg and the NYT. Only a few very prominent stories were covered in both agendas, such as Obama winning the Nobel Peace Prize. In total, stories in the Digg “news” category came from mainstream media reports only 29% of the time. This is high compared to the PEJ study which averaged 5%. The difference may be attributable to my choice of the  “news” category on Digg. The fact that this category exists, and the why and how of different user behavior when using this category, is an area ripe for future research.

Conclusion
Lacking a clear understanding of how a traditional newsroom selects its stories, we can say very little about how audiences might theoretically choose differently. Even journalists cannot explain how story selection really works. However, both online news readership surveys and user-aggregated news sites show very different agendas than the mainstream media. This is most visible when one looks at the overlap between audience and editor-generated agendas — 5% in the PEJ study and 29% for the Digg “news” category.

Is this because the media never really represented public tastes for information, due to structural constraints or value differences between journalists and their audiences? Structural and organizational reasons seem a more promising explanation then personal judgments, given that Shoemaker’s surveys suggest that not even journalists agree with the agendas of their publications.

But we must also contend with the deeper problem of “what is news?” What can thinkably be on a news agenda? In traditional media effects research, only “problems facing the nation” can be on the agenda. On the web, the answer is most naturally “any web page.” Most of the web is nothing like news from a traditional journalism perspective, but this doesn’t stop audiences from voting it on to the agenda.

And this paper has only scratched the surface. We’ve discussed only the web, not real-time messaging services such as Twitter and Facebook, and this leads us to a key underlying assumption of all of the work discussed in this paper: the “news” is the same for every member of the audience. This is a very “broadcast” mentality, and the web is not a broadcast medium. In the future, news will be personalized and it will also be personal: the machinations of my social network may not be newsworthy to a journalist, but it’s certainly newsworthy to me. The entire concept of “news” is undergoing a transformation.

So what is news in the age of the audience-as-editor? In 2007 entrepreneur Adrian Holovaty founded a site called EveryBlock.com that aggregates local police reports, blog posts, and other web content with determinable location. Users from all over the US can see what is happening literally in their neighborhood. It is a valuable source of news, yet EveryBlock employs no “journalists”. The success of this product sparked a lively debate around the question “is data journalism?” Holovaty’s answer was [8],

I no longer see the point in debating the definition of journalism. I’m interested in building products that improve people’s lives via information. Whether somebody calls that “journalism” is utterly uninteresting.

See also

• Rating items by number of votes: ur doing it rong. User-voting on items tends to favor the already popular. A statistical commentary on the problem, and a suggested solution.
• The divergent online news preferences of journalists and readers. A similar quantitative content study, comparing headlines with user-generated “most popular” story lists.
• The Latest News Headlines — Your Vote Counts. The classic PEJ study I discussed in this paper.

References

1. “The Latest News Headlines — Your Vote Counts.” Project for Excellence in Journalism, September 12, 2007. http://www.journalism.org/node/7493

2. Michael Schudson, “The Sociology of News Production”, Media Culture Society, 1989; 11; 263.

3. Stuart Hall, “The Determination of News Photographs”, pp 176-190 in S. Cohen and J. Young (eds), The Manufacture of News: A Reader, Beverly Hills: Sage, 1973

4. Pamela Shoemaker, “News and Newsworthiness: A Commentary.” Communications, 31(1):105-111, 2006

5. David Tewksbury,  “What do Americans Really Want to Know?” Journal of Communication Research, December 2003

6. Maxwell McCombs, “Building Consensus: The News Media’s Agenda Setting Roles.” Political Communication, 14:433-443, 1997.

7. Jonah Berger and Katy Milkman, “Social Transmission and Viral Culture.” Wharton School, University of Pennsylvania, 2009. http://marketing.wharton.upenn.edu/documents/research/Virality.pdf

8. http://www.holovaty.com/writing/data-is-journalism/