I made this image a couple months ago to introduce the question at a conference. A reporter researches and writes a story. The first arrow represents the process that gets that story published. We understand that process quite well, and the internet makes publishing really cheap and easy. Then there’s a process that takes published, accurate information and turns it into truth and justice for all. That’s the part that’s fuzzy. In fact I don’t think we understand it at all. I call this “the last mile problem” in journalism — how does journalism actually reach people?

Journalists occasionally claim a scalp, such as by embarrassing a politician enough to force them to resign, or focussing attention on some issue long enough to get legislation passed. Journalism also theoretically informs citizens so they can vote responsibly, in the elections which happen every few years. As I’ve argued before, these are weak levers by which to shift society. I’m less interested in what journalism does in extraordinary times, and more interested in how the journalist’s work improves the day-to-day operation of a society, and the experiences of the people living in it.

It’s possible that much of the journalism we have is effective. Maybe the mere existence of consistent reporting on the machinations of the powerful keeps them in line, and we’ll only know what journalism really gave us when it disappears and civilization collapses into a mire of secrecy and corruption. Or maybe that’s already happened. How would we know? How can we tell whether journalism, as a local or a global endeavor, is doing better this year than last?

Other fields have goals
I like to hang around the international development community, and those people have real problems. People working in public health are charged with improving access to clean water or preventing the spread of HIV. Others try to get more girls into school, or to raise entire communities out of poverty.

There are lots of ways to attack such complex social problems. An NGO or a foundation or a UN organ could lobby local politicians, produce research reports, provide services directly to affected populations, or launch a public awareness campaign. The way in which an organization proposes to have an effect is called their “theory of change.” This is a term I hear frequently at gatherings of development workers, and from the staff of NGOs and international organizations. Such organizations must continually develop and articulate their theory of change in order to secure philanthropic funding.

Journalism has no theory of change — at least not at the level of practice.

I’ve taken to asking editors, “what do you want your work to change in society?” The answer is generally along the lines of, “we aren’t here to change things. We are only here to publish information.” I don’t think that’s an acceptable answer. Journalism without effect does not deserve the special place in democracy that it tries to claim.

The question of “what change should journalism produce” is hard because it is unavoidably a normative question, a question about how journalists envision a “better” world. At the moment, the field of professional journalism is mired in intense confusion about its role and the meaning of classic standards such as “objectivity.” This has obscured discussion of the field’s goals at a moment of great transition brought on by new communications technology, precisely the time when clarity is most needed.

It’s telling that discussions of journalism’s fundamentals frequently harken back to the great debate of Lippman vs. Dewey. That happened in the 1920s. This was not only before live television and before the internet, it was before bastions of modern reasoning such as statistical inference, the study of cognitive biases, and the social construction of knowledge were fully developed. Other fields have done much better in adapting to the philosophical and technological revolutions of the last century.

Medicine in general and public health in particular have become relentlessly evidence-based. It’s no longer enough to run anti-smoking ads; we now require those responsible for public health to show that their preferred method of behavior modification actually reduces disease. Meanwhile, marketers have rallied around the idea that purpose of their work is to get targeted individuals to do something, whether that’s purchasing a product or voting for a particular candidate. That may not be an appropriate goal for non-advocacy journalism, but marketing and public relations researchers have made very careful studies of communication, recall, and belief.

Similar concerns over how messages are received arise in many fields, from crisis communications to public diplomacy. But not in journalism. If journalism does not change action it must change minds, but the tools and language of belief change seem to be entirely missing from the profession.

Journalism as surveillance of ignorance
It used to be the job of an editor to decide what to publish. Maybe it is now the job of an editor to decide what needs to be known. These are not at all the same thing. They used to be, when nothing could be done with a story after the ink hit paper. The internet allows so much more — promotion within specific communities, feedback on readership and reception, conversation as opposed to oratory. And potentially, cheap techniques to determine what people already believe.

We should expect that users will largely be choosing for themselves what to read and view. That’s reality, and that’s fine, and systems that make it easy to satisfy curiosity are systems that will make us smarter (even though we’ll mostly use them for entertainment.) But I believe there will still be an identifiable set of common content, the few things that the public — or some targeted fraction of it — absolutely has to know to participate meaningfully in the civic issues of the day. This is more or less what editors put on the front page today. But rather than the headlines reflecting the most important events, perhaps they should reflect the most pernicious misconceptions. Good journalists already have some sense of this, and every so often we learn of an alarming gap in public knowledge. A majority of Americans believed for years that Saddam Hussein was linked to 9/11, for example. Today, most Americans don’t know what’s actually in Obama’s new health care laws. (I apologize again to my international readers for the US-centric examples; I’d love to hear of similarly woeful tales from other countries.)

Combatting ignorance is harder than publishing. It’s my best guess for the second, mysterious arrow in the diagram above. Fortunately we also have new tools. We have reams and reams of data that people voluntarily put online, the “data exhaust” of entire societies. We also have old-fashioned public opinion polls, and their lightweight cousin online polls (though self-selection bias may render online surveys useless for all but the most casual work.) Somewhere in all this data and all this communication, it must be possible to figure out what it is that people actually believe — and where those beliefs are factually wrong in an uncomplicated way, precisely the way that an editor would say “that’s not true, we can’t print it.”

There are many possibilities for understanding the beliefs of an audience. I am particularly intrigued by opinion mapping, deliberative polling, and the attempts of UN Global Pulse to create data-driven societal monitoring systems. It may actually be possible to cheaply measure the state of public knowledge, which would also give us concrete metrics for improvement. We need new ways of thinking about the surveillance of ignorance, and we need software to implement them. But more than anything else, we need journalists attuned to what it is that people don’t know. Good journalists already are; they can see what is missing from discussion — whether that’s a question that no one has answered or a challenge to a prevalent belief — and do the hard work of adding it.

This effort applies at all scales. Each journalist has an audience or audiences, their communities of concern. Each could track what their audience already knows and believes. The job of the journalist, so conceived, is not merely to report the happenings, but to ensure that the audience is aware of and understands the most crucial of them. That won’t be easy. Aside from the challenges of determining what an audience already knows, people don’t like to be told they’re uninformed or wrong. This is why I believe a journalist needs to learn everything there is know about public communication, borrowing and adapting from marketing experts and public health planners. Genuine honesty and humility seems to me the ethical core, and newsroom transparency is a critical check on this power.

Of course, decisions would have to be made about what are misconceptions and which of them are important enough to combat. Decisions have to be made already about what to cover and promote with limited resources, and these hard choices are the iceberg that sinks any hope of a truly “impartial” journalism. It’s a reality that the profession has to deal with every day, and I wish we would get on with the work of crafting and communicating our normative stance, rather than insisting that “objectivity” means we don’t have one. (Even Wikipedia explains its norms in great detail.) I’d like to start with a list of things that journalists wish were better known. Be honest. I know you’ve already thought about this.

But if we can get over that hurdle — if we can admit that journalism needs concrete goals — then we stand a chance of doing better journalism, and knowing when we’re doing it. For me, the insane possibility of new communications technology carries with it the obligation to do better than we ever have before.

UPDATE: As if on cue, a major study was released four days after I published this, showing that a majority of American voters were misinformed about the issues they voted on in the recent mid-term elections. I discuss what that means here.

Journalism, capital J, is supposed to be about ideals such as “democracy” and “the public interest.” It’s probably important to be an informed voter, but this is a very shallow theory of why journalism is desirable. Most of what we see around us isn’t built on votes. It’s built on people imagining that some part of the world should be some other way, and then doing what it takes to accomplish that. Democracy is fine, but a real civic culture is far more participatory and empowering than elections. This requires not just information, but information tools.

Newspaper stories online and streaming video on a tablet are not those tools. They are transplantations of what was possible with paper and television. Much more is now possible, and I’m going to try to sketch the outlines of how newsroom products might better support the people who are actually changing the world.

What’s a journalism “product”?

This is an essay about product design, so I want to be clear on what I mean by a journalism “product.” Since journalists don’t make tangible objects, the product is defined by the user’s experience. It’s whatever the user interaction with the news is. It’s picking up the paper at breakfast, or watching CNN in bed, or waiting for your mobile app to update the headlines on the bus. And yes, the product includes the stories delivered by the medium, but those stories alone are not the product; they never were. The stories were packaged into a newspaper or a television show, and that was the product. Or more precisely, the newspaper and the television show as the user chose to use it was the product.

Much of the ongoing future-of-journalism discussion focusses on how reporting needs to change, and rightly so. But that analysis stops short of the user, and how journalism is actually used — or could be used.

Digital news product design has so far mostly been about emulation of previous media. Newspaper web sites and apps look like newspapers. “Multimedia” journalism has mostly been about clicking somewhere to get slideshows and videos. This is a little like the dawn of TV news, when anchors read wire copy on air. Digital media gives us an explosion of product design possibilities, but the envisioned interaction modes have so far stayed mostly the same.

This is not to say that the stories themselves don’t need to change. In fact, I think they do. But the question can’t be “how can we make better stories?” It must be “who are our users, what would we like to help them to do, and how can we build a system that helps them with that?”

Maybe the product has to include advertising too, and maybe it must be subsidized with celebrity gossip and movie listings. There are clearly business considerations in defining journalistic products, and that’s fine, but I’m going to focus on the user experience here. Lots of other people are already talking about the business of journalism. I’m trying to make something that actually fills a need in people’s lives. If you can do that you’ll have an audience, and then you can advertise to them or sell them subscriptions, or syndicate the product, or whatever.

The first thing I ask of an improved product is this: I want people to use it. Or rather, I want people to want to use it.

Is the news boring?

By boring I mean unengaging. Not something you want to spend a lot of time with. As Joshua Benton of the Nieman Journalism Lab put it,

You never hear from people “Man, I just got lost on that news site!” We don’t create experiences that people just want to live in for a while.

He was referring to the following fact: Americans apparently spend 12 minutes a month on the average news site, versus seven hours per month on Facebook. (I’ll use American numbers as illustrative, because they’re well studied. Please let me know of data for other countries!)

It’s not that people are paying less attention to the news than they used to. American spend about 70 minutes a day with all news media, the same as in 1996. (Most of that is television news, but online consumption is climbing.) Meanwhile, total consumption of media is at an all time high. This discrepancy in growth means that people aren’t choosing to spend their increasingly connected time learning about the world — or at least they’re not learning about it through professional journalism.

I think that this is because existing journalism formats are not very good at engaging curiosity.

I’m not talking about making the writing “sharper”, adding “multimedia”, or doing better page layout. Those might be good ideas, but it’s all rearranging the furniture. There’s no fundamental change in such things. When editors give smug interviews about their publication’s “fresh new style” I want to throw things. It’s still exactly the same model of interacting with the user. There are better ways.

Design for curiosity

I live surrounded by infophiles, and for almost all of them Wikipedia is a better resource than a news site when they want to learn about the world. People spend hours roaming Wikipedia; they don’t spend hours on bbc.co.uk or cnn.com or nytimes.com or news.yahoo.com (which actualy has a far bigger audience than any traditional news outlet.) Wikipedia also tends to take the top spot in Google results, which means that more people link to it than they do to any news site.

Why is this? If professional news products are supposed to be such a vital resource, why are users overwhelmingly choosing to satisfy their curiosity elsewhere?

I think the answers to this are starting to be understood. Standard news coverage is written as a series of incremental updates, which are useless if you’re not already following the story closely. When I returned to coverage of the oil spill after not checking on it for a couple of weeks, Wikipedia was far, far better at bringing me up to date on the story. If I’d gone to CNN I would have been forced to wade through a series of daily updates to learn what I wanted to know. This has become known as the “context” problem within the future-of-journalism community, and there’s no better introduction to it than Matt Thompson’s “An Antidote for Web Overload.”

The professional news media also have a serious problem with linking, or rather, not linking. Linking is vital for creating immersive experiences online. Links create the web, the greatest time waster ever invented. They’re also neccesary for any site that wishes to be indispensable. I don’t want the product with the best content overall, I want the product that is going to serve me up the best content every single time, regardless of whether or not it was created in-house. That means links. (And content syndication, but no walled garden will ever match the wild richness of the whole web.)

Links are also vital for transparency and depth; they provide the option of more serious investigation. The uses of links in journalism, and their relative paucity in professional work, is a topic I covered in great detail in a three part series for the Nieman Journalism Lab this summer, and I also wrote about the use of links for deep storytelling here. But to make it short: linking is not yet part of professional journalism culture, and this creates a serious problem with the product.

There are many other features that afford the satisfaction of curiosity. Take the humble “search” box. Almost every news site has one, and search is an obvious way for a user to answer their question. But news web sites almost always implement search badly, and many news apps don’t even have a search function. Or, you can only search content from last few days or within the current “issue.” I don’t care about the metaphor of “issues” when I have a specific desire to learn something.

And then the river of data stemming from search logs is poorly analyzed. There is an ongoing debate about exactly how much news organizations should respond to user demand. If “Paris Hilton” is the number one search, I don’t think that the newsroom should make Paris Hilton stories their number one priority, but ignoring what users want is folly. And it’s no way to engage my curiosity, which is about what I think, not what the editor thinks.

Don’t fear fragmentation

Which leads us to another deep problem with existing journalism products: they’re not designed to be personal.

Perhaps this stems from the classic “we’ll tell them what they need to know” mindset among editors. That makes sense for a paper product, where everybody reads exactly the same copy, and you can actually “finish” consuming the content when you run out of pages to read. It doesn’t make sense when pages are effectively infinite, and there’s no obvious reason that my reading should overlap with yours. Grizzled hacks love the idea of everyone talking about the same set of stories, just like the good old days, but I don’t believe that, today, it is a reasonable thing to expect or want.

NYU journalism profession Jay Rosen recently introduced me to a 20th century social critic named Raymond Williams. “There are no masses, “ Williams wrote in 1958, “there are only ways of seeing people as masses.” Lofty discussions of journalism use words like “public” and “citizen,” but of course those are ideas. There is no homogenous audience. And at last it is possible for our media to reflect this truth, because the internet is not a broadcast medium. A newspaper or a TV report is the same experience for everyone, while my usage of a web site or an app can be extremely personal.

Then we have the problem of information overload. One person can only track so many stories. If everyone follows the same few stories, lots of important things are going to be ignored. (Which is exactly what’s happening now, according to internet scholar Ethan Zuckerman.) Delivering the same information to everyone and expecting that everyone engage it equivalently is not only an insult to diversity, it ignores that fact that our civilization is built on specialization. Different people have different capabilities, positions, and passions, and so they’re going to need different news agendas. Empathetic journalists have always been aware of this, but it wasn’t previously possible to target individual stories to specific people.

This targeting is something that users could do for themselves to a large extent, given the right interface. Today, the personalization features of almost every digital news product amount to 1) showing me stories that happen in my zip code and 2) letting me pick from a small set of coarsely defined “sections.” Why can’t I subscribe to updates on particular stories? Why can’t I set up alerts for particular terms? Why can’t I tell my news app “no more updates on Lindsay Lohan, ever”? Etc.

Who needs to know?

If we’re going to try to deliver specific stories to specific people, then who should we target? First of all: anyone who wants to know. The broadest possible access is important, and happily, the internet has been very good at extending access.

Beyond that, I go back to the idea that journalism should be useful by those who are acting to change and build our societies. Then the answer is “whoever is going to do something about it.” The product should serve the people who choose to act.

Some people will have shaping society as part of their job description, such as teachers and civil servants. Other’s won’t, and this is truly a great era to be an interested amateur. It’s not that every individual is a motivated genius just waiting for the right call to action. Rather, let’s say that there is tremendous knowledge and capability scattered throughout society, untapped. Some people are highly trained specialists with spare weekends. Other are merely informed and interested.  Some problems call for unskilled mass participation, but many more call for the dedication of a small number of the right sort of people.

My favorite example of this is the The Guardian’s successful attempt to find someone among their audience who could untangle the mystery of Tony Blair’s tax returns. Eventually, some crusading geek of an accountant stepped forward with the solution (and bravo to you, sir.)

In fact, we know that most people won’t work on anything at all. This is the nut of the 90-9-1 rule which seems to hold across many different types of online social activities: 90% of people are just audience, 9% are involved in some way, and only 1% are the real creators. The numbers vary somewhat from platform to platform, but the upshot is always that only a very small number of people do the core work. There’s no reason to believe that offline social initiatives — everything from campaigning for gay rights to  getting potholes repaired – would typically attract a much broader base of core agents. And this is fine. This is normal, and nothing to be upset about. One percent of a population is still a huge pool of talent and labor.

In this framework, the purpose of journalism is to deliver each story to the right 1%, at the point when they need it. Saying that most people won’t read a story about Madagascar doesn’t get us very far; that’s expected, and that’s the only way it can be in an era of spectacular information overload.

I don’t know whose job it is to facilitate the creation and nurturing of communities around issues; maybe it’s fine to let Facebook have this role. What I do know is that journalism needs to concern itself explicitly with figuring out who its audience is — for each story, down to the level of individual people and groups. Where in the world are those 1% who have something to say or do about the coup in Madagascar, and how do we connect to them, and connect them to each other?

This probably requires that journalists listen a lot better to what’s already happening out there on the tubes. As Felix Salmon of Reuters puts it, we need to teach journalists to read.

Journalism designed for agency

I come from a culture of makers; my personal life and my chosen communities are steeped in the ethic of hackers, entrepreneurs, crafters and artists. All of these activities are the expression of a participatory instinct, an instinct that extends to trying to shape the society we live in. I want information systems that facilitate self-selected civic action and social entrepreneurship in all its forms. If this is the goal, then journalism needs a better understanding of how its products could actually be used to effect change.

News can no longer be (only) about the mass update. Stories need to be targeted to those who might be able to improve the situation. And journalism’s products — which are more than its stories — must be designed to facilitate this.

News needs to be built to engage curiosity about the world and the problems in it — and their solutions. People need to get lost in the news like they now get lost in Wikipedia and Facebook. There must be comprehensive stories that get the interested but uninformed up to speed quickly. Search and navigation must be improved to the point where satisfaction of curiosity is so easy it becomes a reflex. Destination news sites need to be more extensively hyperlinked than almost anything else (and not just insincere internal links for SEO, but links that are actually useful for the user.) The news experience needs to become intensely personal. It must be easy for users to find and follow exactly their interests, no matter how arcane. Journalists need to get proficient at finding and engaging the audience for each story.

And all of this has to work across all modes of delivery, so it’s always with us. Marketers understand this; it’s amazing to me that the news industry has been so slow to catch on to multi-modal engagement.

It sounds like a tall order, but there’s nothing here that requires exotic technology. Just real product design, in pursuit of concrete journalistic goals.

(thanks to @chanders for feedback on an early draft of this post)

Dutch journalist/coder Stijn Debrouwere has written a very thorough post describing the ways in which standard tags, like the ones on this blog or on Flickr, fall short when applied to news articles. There are lots of things we might like to know about a story, such as where and when it happened and who was involved. This additional information, sort of like the index to a book, is known as “metadata”, and there is within the online journalism community a great call for its use, including by Debrouwere:

Each story could function as part of a web of knowledge around a certain topic, but it doesn’t.

So here’s a well-intentioned idea you’ve heard before: journalists should start tagging. Jay Rosen insists that “Getting disciplined and strategic about tagging” may be one way professional journalism separates itself from the flood of cheap content online.” Tags can show how a news article relates to broader themes and topics. Just the ticket.

News metadata is a major topic, and many people have speculated deeply about the value of creating news metadata at the time of reporting, such as the ever-sarcastic Xark and the thoughtful Martin Belam who writes about why “linked data” is good for journalism. But I’m going to respond to Debrouwere because I read him today, because he has lovely diagrams that explain his good ideas, and because, in criticizing “tags” as a form of metadata, I think he misses some very important points.

And he’s not alone. My sense is that many of the coder-journalists of today have not learned from the mistakes of generations of technically-minded people who wished to talk about the world in more precise ways.

Moving forward from simple tagging, Debrouwere imagines more sophisticated annotation schemes that start to pick up on what the tags actually mean. For starters, the tags could be drawn from separate “vocabularies.” Does a tag refer to a person, or a place, or perhaps an event? Debrouwere uses the following picture, which I’m going to borrow here because it explains the idea so nicely:

But, he says, we can get even more sophisticated. What did the story actually say? If it mentioned a person, what did it say about them? Was it an interview? A profile? Did it criticize them? Here’s the diagram he draws:

He imagines using this information to perform chains of inferences, like so:

Barack Obama belongs to the Democratic Party and he’s from Chicago. If we tag an article with Barack Obama, it’s likely that the article also has something to do with the Democratic Party. If we’ve specified that the article is about Obama, and we’ve specified that Obama is part of the DP, the system now has all the necessary information to suggest our article about Obama as a possibly interesting related read on the topical page for the democratic party, even if we didn’t explicitly indicate that link.

First of all, note that this sort of thing is already possible, quite often, using tags as they exist today. Simple analysis of co-tagging information will tell us that Obama is related to the Democratic party, because many articles will be tagged with both. Which is not to say that encoding such relationships explicitly isn’t a good idea. We can do this sort of thing using “triples,” which are fundamental to the nascent evolution of the internet into a web of “linked data”:

• <Barack Obama> belongs-to-party <Democratic Party>

Here, “Barack Obama” is an object from, say, the “people” vocabulary, and “Democratic Party” is from, perhaps, the “political party” vocabulary, or maybe just from “groups.” Essentially, these are tags that have been pre-categorized. The relationship between the two is expressed by the “belongs-to-party” predicate.

But I argue that this is a rigged example. The world is normally much more messy.

“Are you now or have you ever been a member of the communist party?” was a killer question in its day, with complex answers like “I only attended one meeting.” And if parsing politician’s statements was easy, then Politifact wouldn’t devote entire articles to the question of whether a single sentence was true or false. Further, they distinguish between different “grades” of truth, like “mostly true.” Mathematical logic — which is what the sort of news inferences that Debrouwere and others discuss is based on — doesn’t deal with “mostly true.”

The problem is that the world is not neatly categorizable.

Don’t get me wrong — vocabularies and relationships (ala linked data triples) are surely a good idea. But they have some serious drawbacks that relate to very deep issues in knowledge representation.

Debrouwere says, “Events happen at a certain place and at a certain time.” Sometimes. For a house fire or a shooting, maybe, but how “long” were the post-election protests in Iran last summer? They continued at varying intensity for several days, then flared up weeks later. Was that one protest or two? And what about a Facebook protest that gathered supporters over the course of a week? “When” and “where” did that happen?

Or, take the example of describing what an article says about someone. How do we decide when a story “criticizes” someone? There will always be boundary cases — lots of them in professional reporting. How do we ensure inter-rater reliability? Can we extract any real data from analyses of this tag if we have no other reference points with which to interpret it?

Something is always lost in categorization. That is the point! To say that two things are like one another is to ignore their differences, for the purposes of the present discussion. Unfortunately, what can safely be ignored depends on the discussion. Simple date and place notations work for some purposes, and fail miserably for others. They are not very rich, and even worse, we don’t know exactly how much has been lost in each case. Knowledge of that error is sometimes critical, especially when trying to make chains of inferences, where errors multiply.

The reason we use text for reporting is that it’s good at representing these sorts of ambiguities. Strict adherence to the religion of finite relationship vocabularies leads one to believe that the world can be modeled in first-order logic (predicate logic), and this just isn’t true. Chains of automatic inference fail very quickly when applied even to very simple “real world” situations. The Artificial Intelligence research community went down that road for decades and found it really problematic, which is why we’re now seeing the rise of “statistical” AI techniques, such as statistical machine translation. This approach tries to find patterns in vast amounts of data rather than working out hard underlying rules; the categorization comes after you look at all available data, not before.

And therein lies the great virtue of tags: they are just about the simplest possible way of saying something, and don’t imply or require any particular inferential framework. They’re much harder to get wrong than more complex associations, and they make sense only in aggregate, and this makes them much more robust than predicate sentences. A tag says only, “there’s some association.” Full stop. I find this ambiguity a virtue. The meaning comes out of the relationships between the tags, articles, and users. Meaning is always relative, and tags force us to understand this, because there’s nothing else to go on.

Tags allow (or force) what we might call the “Google solution”: let humans describe it in a way that makes sense to them, then sort it all out later algorithmically. There are limits to this, of course, which is why metadata has value. But ultimately, computers serve humans, so the Google solution will always be a win when it is possible.

Linked data will be valuable because of the links. I predict that its main use will be as a sort of “super tagging” system: we still have “tags” in the linked data world,  it’s just that they’re now all “uniform resource identifiers” that are visible to everyone on the web. This means that tags can be shared between systems and maintained by communities, which only makes them more powerful. In fact, this is exactly what we’re already seeing, with the Wikipedia-derived DBPepdia at the center of all those linked data “bubble diagrams.”

Linked data also supports predicates that  say what the relationship between the tags is, like the “Barack Obama is a member of the Democratic Party” example. But I predict that these will  be much less useful, offering almost none of the “machine understanding” that’s supposed to come with the semantic web. I don’t know what “understanding” means if not the ability to draw inferences of some sort, and predicates are just too fragile, too subject to mis-categorization, too limited to capture the rich relationships of the real world. I do believe that we’ll see amazing new “artificial intelligence”-like applications built on top of linked data, but they’ll be built statistically: they’ll ignore the predicates or use them only in special cases, or only in aggregate.

Having said all this, I am fully in support of adding better metadata to news stories. I believe the  ”entity recognition” performed by OpenCalais is valuable, and that carefully managed tag vocabularies are essential. Often “location” will be a genuinely useful tag, and I can see the possibility for some wonderful news mashups.

But please, let’s not imagine that we can capture even the “essential” details of real journalism with any fixed vocabulary. And let’s not oversell the potential of machine reasoning or data-mining based on carefully-annotated news metadata.

We’re a very long way from understanding how to represent reality in machine-readable form.

For more on this topic, I recommend:

• “Ontology is overrated” by Clay Shirky
• “Metacrap” by Cory Doctorow
• “What is a knowledge representation?” by Davis et al. at MIT

Dowser.org is a new online news site that aims to cover solutions to social problems. It’s journalism, but focussed on what we are learning about what can be done.

A media theorist might say this is reporting from a “solution” frame. Whatever you call it, I support the idea.

They are actively soliciting feedback on their idea. This was mine:

Great idea.

I think solid information on who is solving which problems is really important. I have three broad suggestions.

1) Please consider including a set of “topic pages” as well as running news stories. Social problems are often complex and contextual, and we badly need continuously-updated primer articles that bring the newly curious up to speed on specific problems.

2) If you guys have any money at all, you should solicit contributions from freelance journalists for coverage and analysis of interesting global projects.

3) consider publishing in more than one language, or allowing hybrid computer-human translations using, e.g. the free World Wide Lexicon system.

I’m a freelance journalist and computer scientist living in Hong Kong, who specializes in applications of technology to social problems. @jonathanstray on twitter.

Would love to keep in touch.

There’s a great deal more I could say on why I made these specific recommendations. For now, I’ll leave it to the hyperlinks. For additional ideas on how to offer good translations of every page for near-free, see also Yeeyan.com, a successful volunteer news translation community with 100,000 users.

(hat tip: @NiemanLab for bringing them to my attention.)

Here’s a list of things that professional journalists do:

• decide what should be more broadly known
• decide what should be more deeply investigated
• collect information from sources both public and private
• check that information for factual accuracy
• construct narratives to make sense of that information
• produce content to convey those narratives
• publish and market that content

This list is by no means definitive or exhaustive. It’s just illustrative, a starting point for a thought experiment. Who could do each of these things best? And what tools to do they need to do it?

Having a network of people producing journalism around a newsroom is not a new idea. Jeff Jarvis has been discussing networked journalism since at least 2006, and naturally I think he’s on to something. In this essay I want concentrate on process and roles. If cheap networks make new types of collaboration possible, they also set the stage for new types of specialization. I think one of the problems of the traditional, mainstream media newsroom is that it it tries to handle the entire journalistic process internally, even the parts that it’s not actually very good at.

An example

On November 25, a video appeared on YouTube which appears to be the testimonial of a young woman recently fired from the credit card collections division of Bank of America. She had been allowing the bank’s most desperate customers to enroll in fixed-payment debt recovery schemes. Many of these customers are currently paying 30% interest as a result of recent rate hikes, so this was a great kindness. It was also against company policy.

The video is powerful. It’s an amazing first-person testimonial of the greed and heartlessness of large corporations.

So is this journalism?

Not quite yet, according to the traditional definitions. It’s a powerful story only if it’s true — and even if it’s true it may not be the whole story. I haven’t checked the video out myself, such as by trying to contact this woman, or calling Bank of America to see if anyone by that name ever worked there. And even if her description of events is accurate, I wonder what fraction of deeply indebted Bank of America credit card customers are denied access to debt relief programs, and on exactly what grounds. What about at other banks? What could be done that isn’t being done to address this situation, if it is a situation, and by whom?

A professional journalist on the story would answer these questions. They would make the calls, keep a notebook of what they found, select the most relevant points, and publish a full account of what they discovered as quickly as possible, or at least as full as time and word limits permitted. Good stuff nonetheless. And here the conversation with most pro journalists ends. “If paid professionals don’t do this, who will?” A pro might say that this video isn’t journalism at all.

The obvious problem with the “this isn’t journalism” line of thought it that denies the value of what this woman has done. She may not be a “journalist”, but she’s certainly participating in a journalistic process.

Focus on the economics

I’m going to approach the question of how journalism “should” be done by asking how to produce it as cheaply as possible.

In general I’m not convinced that efficiency is the goal. Markets fail too often for that. But journalism is currently very inefficiently produced in a rather obvious way: lots of reporters covering the same story. Even if only one reporter covered any given story, I still suspect that a traditional newsroom would be wildly inefficient, compared to what is possible in the age of the internet.

If solid journalism in the public interest is cheap to produce, we as a civilization can afford a lot of it. And I think we do want a lot of it. Transparency from those in power is important (and this applies both to governments and corporations) but it will always be possible to adhere to the letter of the disclosure law, rather than the spirit. Keeping track of what the hell is actually going on is always going to be a required function in a free society.

(I write this from corruption-laden Indonesia, where it’s alarmingly difficult to keep tabs on the powers that be.)

My suspicion is that the same shift that has destroyed the traditional, publishing monopoly-based business model for news organizations can also drop the cost of production quite dramatically. How? Cheap communication networks and specialization — the same factors that have been increasing productivity in all fields for the last several centuries. Let’s look at who might perform each of the journalistic functions I listed above.

Decide what should be more broadly known

This was previously the role of the editor in collaboration with the reporter. The editor assigns stories, or the reporter comes up with them. The decision of what to report on relies on “news judgement”, which has been described to me by one old wire-service hack as “tribal.” Different publications have different ideas about what counts as news. And this is great — Wired should cover different stories than the Wall Street Journal.

The problem is that the news media are not very good at consulting their audiences when developing news judgement. Letters to the editor, unsolicited tips to reporters, and the occasional marketing survey are all very narrow and unreliable back-channels.

This is not about tailoring a product to audience demand. It’s about service to the community, which news organizations need to engage with every tool at their disposal. It’s also about trust and authority, which works differently in the social media era. When Wikipedia goes through convulsions of public discussion about every major editorial change, will audiences really trust organizations that claim to decide the public interest behind (mostly) closed doors?

The possibilities for improvement are many, but there are blatantly obvious and simple things that could be done immediately. Facebook and other social media sites have well-publicized and lively forums where users debate the future of the product. News organizations do not.

Decide what should be more deeply investigated

Reporters are tasked with coming up with interesting questions, then getting them answered. The seasoned reporters I’ve had the pleasure of engaging are typically very knowledgeable folks, very bright and very widely read (or the equivalent contemporary expression — widely surfed?) But they are not experts in every field. They will always lack the context and detailed knowledge that allows them to perceive certain key questions. Similarly, who gets to decide whether a story is followed up or dropped?

Spot.us is an interesting attempt to let the audience vote with its wallet. Readers submit ideas for investigative stories, which freelance journalists then write pitches for, describing the work they propose to undertake asking for a specific amount of money. Other readers donate money to those pitches they want to see executed. The site has so far mostly been used for political issues in California.

HelpMeInvestigate.com takes the process a step further. Readers submit questions — a current example is “How much of the rent charged for University of Birmingham halls is actually spent directly on related costs?” — and then readers work together to answer them. The site acts as a clearinghouse for facts uncovered so far.

Professional reporters need to be embedded in systems like this. While it is true that many of the questions that people ask are going to be uninteresting — or easily answered from existing sources — not every idea is going to be bad. There’s been lots of talk and many products designed to help organizations track and manage their collective knowledge. For journalism, such systems need to extend outside the newsroom into the audience.

Collect information from sources both public and private

This is where traditional journalists are both strongest and weakest. A huge part of the value of a career reporter is the network of contacts and sources they build up over a lifetime. Simply put, it’s their job to cultivate relationships with knowledgeable and powerful people. This means the pro reporter has an irreplaceable investigative role to perform.

But the professional journalist is hardly the only person who can publish otherwise unavailable information. This is precisely the role that the woman in the video was playing: we didn’t have any reports of what had happened at Bank of America, now we do.

The fact that her report is unconfirmed does not mean that it is not valuable. Information of all grades is valuable.

On the internet, filtering comes after publishing, as Clay Shirky has noted. This concept is an inversion of traditional journalistic practice, but it is necessary because the journalist cannot be the filter for the entire web. Filtering must be collaborative to scale. Remember that there is no such thing as “automated” filtering: Google Search results may be returned by an algorithm, but that algorithm uses the manually placed links on the web to determine what content is relevant.

Filtering is also the key to finding facts in thick documents, of which there are now many. That buried government report is critical, especially as governments practice increasing data transparency. A blog post saying “hey, page 283 of this document is interesting” may not seem like a story to a reporter. Another blog post referring back to the first one even less so. In fact, both posts are extrmely valuable, because they are filtering mechanisms.

Publish first, ask questions later is the rule of the web. This applies to journalists too: facts or reports that aren’t immediately usable in a story should be considered for rough-and-ready publication anyway, such as through micro-blogging — or, better, by making the newsrooms files open wherever possible. This not only increases transparency, it allows users to build on the reporter’s work.

Check information for factual accuracy

Some facts can only be checked by making calls to highly-placed sources. Other facts can be confirmed by anyone with a telephone. And a great many others can be confirmed online. Only the first category of facts can only be checked by a career journalist.

Bloggers draw attention to unconfirmed reports or documents all the time with remarks like, “this is interesting. Can anyone confirm it?” I’d like to see reporters distributing more of their workload in this way, especially for material that isn’t immediately needed for a deadline.

Asking for help is one way that the work gets to the people who can do it best.

Saying that something needs to be fact checked is almost as valuable as checking it. It draws attention. It puts a pencilled-in question mark above an item that everyone else can see. Wikipedia, the greatest collaborative fact checking system of all time, recognizes this point with its famous “citation needed” tag.

We are entering the era of transparency in fact checking. It’s no longer enough to be right; the audience has to be able to understand why you are right. Compare the links and footnotes on the Wikipedia and New York Times articles for “Global Warming.” Meanwhile, Associated Press stories are still entirely plain text — no reference links at all.

Construct narratives to make sense of information

Everybody in the world wants to be part of narrative construction, if the number of active blogs is any indication. Career journalists have an advantage in that they are (hopefully) intimately familiar with the facts and history of a particular topic. But if smart readers are given deep access to those same facts (transparency!) I don’t see why the reporter’s narrative/interpretation is going to be any better than anyone else’s.

The days when a single publishing organization can set the agenda are over, though God knows Fox tries. Audience members who are broadly connected to each other in real time for free will negotiate the narrative among themselves, thank you very much. Taking part in this negotiation is not something a newsroom can charge for.

Refereeing the negotiations, providing the forum, or filtering the conversation might add value, if done properly. This is different than yelling your own point of view, however nobly constructed.

Of course, opinion columnists are popular. But they don’t seem to be profitable, or at least something that can be charged for. Both the New York Times and The Economist put their opinion outside their paywalls.

Produce content to convey those narratives

Demand Media Inc. pays an average of US $15 for an article of a few hundred words, and $20 for an original short video clip. They currently produce thousands of items each month, and aim to be producing a million items per month by next year. Article topics are assigned automatically by an algorithm that computes the expected search-engine ad value of all future hits to that page.

We are witnessing the beginning of industrial content production. Yes, it’s cheap, poor-quality stuff. That doesn’t matter, in exactly the same way that most of us now wear mass-manufactured clothing. My mass-produced clothes are actually pretty good these days, and in just this way the quality of industrially produced content is going to come up as producers figure out the efficiency issues.

High-quality, artisanal content — sparkling writing, slick video production — will always have higher value, but the market for it is in the process of collapsing. Polished is good, but is it necessary to the journalistic mission? As one blogger put it, “it’s casual Friday on the web.”

Publish and market content

It’s no longer useful to think of a news organizations as publishers, because their irreplaceable role has nothing to do with making information available to the public; that’s just a necessary sub-product, handled mostly by the telecommunications industry.

Marketing is the more interesting role. If a story has no value if it has no impact, then people need to know about the hot stories. Obviously social media offers unique opportunities here, and most online news sites have a decent array of “share this” buttons below each item.

This is a start. It is not enough. The possibility for personalized news is huge. I have yet to see a Facebook application that delivers me useful news and social recommendations for news. Both The Huffington Post’s Social News and The New York Times’ TimesPeople applications seem to be DOA due to a combination of technical and marketing mistakes, and besides they only filter stories on their own sites — which strikes me as incredibly presumptuous. TwitterTime.es, which aggregates the links that my friends have tweeted from all sources, is a lot closer to what I have in mind.

Social Journalism

I imagine a system where the traditional journalistic functions of a newsroom are distributed throughout a community consisting of newsroom staff and audience. The audience is going to be unpaid, let’s assume. (Paying your audience doesn’t seem like a sustainable business model.)

The audience can report information. The audience can check information. The audience sometimes even creates good content. Above all, the audience filters information for each other and for the journalist — not just the smattering of stories produced in the newsroom, but every story and piece of a story that they can find online or in their lives. And every scrap of information that the newsroom can give them access to, in its files and archives.

Salience is mostly decided by audience, not editor. I find it quite startling that when web audiences assemble their own news using sites such as Digg and Reddit, their is very little overlap with mainstream media. It’s even more surprising to me that in a recent ten-country survey, not even journalists ranked stories in agreement with the prominence they were given in the media. The current news agenda-generating process serves no one at all, apparently. As we say on the internet, ur doin it rong.

Instead, the editor needs to ask the audience to filter the ongoing discussion around the best use of society’s very scarce and expensive reporting resources.

The audience performs all of these roles — surfacing facts and checking them, filtering, setting agendas — because it’s in their interest, and because the newsroom makes it really easy for them. This requires software.

Social Software

Journalism needs its own killer social media application to organize all of this, and it hasn’t been invented yet. Social software is architecture and environment: different types of software are conducive to different types of behavior in its users. The resulting social system is the combination of the software and the community that is nurtured on top of it.

Social systems can produce things. Flickr produces a tagged database of photographs. YouTube produces videos. Facebook has produced the personal information of 350,000,000 people. Twitter produces global, real-time conversation. Wikipedia produces the most extensive encyclopedia in history, and makes it available free.

Wikipedia is worth examining very closely, because it may be the closest live example of how social journalism software could work. Users create topics, then edit them collaboratively. Less well known are the conflict-resolution procedures that Wikipedians use to resolve editorial disputes. These progress from discussion on each article’s talk page to informal mediation to binding arbitration. All of these activities are organized and staffed by volunteers.

The Wikimedia Foundation, of which Wikipedia is the flagship product, does not edit content or (usually) engage in dispute resolution. They provide the infrastructure. This is both the software itself, and the rules the govern the community. For example, Wikimedia is ultimately charged with maintaining the editorial policies that they believe will produce the best possible product. These policies are set in consultation with the user community, of course.

Wikimedia employs about 30 full-time staff. This represents an astounding amplification of effort and money. Imagine what an engaged community of readers could do around a single professional newsroom. For maximum amplification, the journalists in the newsroom must perform only those functions that no one else can do.

To accomplish this, the software has to be designed so that it can be used by a community to produce news without any professional journalists. A neighborhood, an organization, or a town should be able to use a social journalism software tool to track and inform itself, supplying any combination of paid and volunteer labor that it deems appropriate. Only when newsrooms give as much capability as possible to their audiences will they understand where the crucial gaps lie that professional journalists must fill.

When won’t this work?

There are many cases where the work of journalism is still going to look a lot like it always has.

First there is the issue of access. Stories that are primarily about the actions of elites who restrict access will need to be covered by accredited professionals — such as the White House press pool. Similarly, professional journalists are often allowed across police lines, into conferences, and in other restricted situations. Accreditation is necessary in such circumstances, and the current system of career journalists working for recognized institutions is probably a reasonable way of deciding how to apportion limited access.

Reporting across language and cultural barriers or from dangerous places will also require professionals. I spoke last week with a reporter who covered the massacres in the Philippines for the BBC. This required contacting authorities in another country, working with translators, and most of all having worked in Asia long enough to have the address book and understand how to operate. It’s not the sort of story that a community of interested followers can generate from the other side of the world. For this reason, I suspect that professional foreign correspondents are likely to be irreplaceable for some time.

Finally, the whole notion of networked journalism rests on the availability of the network. Here in Indonesia, internet access is still slow, expensive, and not widely used. That makes traditional centralized journalism both necessary and profitable, for the time being.

And there are doubtless other cases. Again, the fundamental shift that needs to take place is for professional journalists to try to do only those things that absolutely require their services.

So what does the newsroom do?

I would like to see the newsroom at the center of a system of social news production. The newsroom provides experienced journalists who have fat address books and access to elites. The newsroom designs, produces, and evolves a specialized social media application that allows its audience to self-organize to perform all of the functions that do not strictly require newsroom staff.

Jay Rosen refers to newsrooms as “closed” journalism and blogging as “open” journalism, and sees them as complementary processes that produce different things. I think the lines need to be a little more blurred.

It is true that newsrooms sometimes need to protect sources and keep certain facts private until the broader story is clear — or sometimes keep things secret forever. This is part of the game of getting people to talk. But whenever something does not have to be secret, it should be public. As much as possible, the newsroom should not have access to bigger files or better tools than the audience. Rather than the simple “inside” and “outside” that exists today, I imagine a journalistic source/article/fact/notes tracking system that has fine-grained privacy controls.

Using this shared system, journalists make their data, notes, and tools public whenever possible, so that the audience can help them. The newsroom now owns a sophisticated information tracking and filtering system which acts as a focal point for the aggregation of journalistically interesting material. The audience provides reports and facts. The audience checks facts. The audience interprets facts. Sometimes the audience creates content. The audience provides expert guidance. The audience assembles itself into communities around an issue, identity or topic. The audience constructs narratives and decides on its questions and its goals.

The editors no longer get to decide what goes on the front page. This not only matches the reality of how people consume information online, it’s implied by personalization. Besides, editorial curation of content won’t scale. A journalistic system has to be designed so that the audience — each of many audiences served by the same huge steam of content — can bring the most relevant content to the front. The audience filters, and from this filtering the newsroom also learns what is important to the audience.

I believe that the correct goal of the editor is not the production of stories but the management of the ecosystem of journalistic production, including reporters, community, software, files, and processes.

No doubt I am wrong in the details of how all of this has to work. No doubt there are ways in which this scenario is too optimistic. But I’ve yet to hear of anyone seriously attempting it.

From Google News just now:

In this case, there were 2,907 articles on the web covering the story of Iran freeing the Brits. Even accounting for the fact that most of these pages will be mirrors of the same text, we can see that Aljazeera, The Telegraph, the Associated Press, CNN, the New York Times, and many others covered the story. And they all did their own reporting, a horde of journalists from all over the world making phone calls and doing interviews.

Meanwhile, written-word reporters are getting laid off right and left, essentially due to the end of the print-based information distribution monopolies. Last year, 35,000 journalists lost their jobs in the US. This has made a number of quite clever people fret about the end of “accountability journalism,” the press that keeps people honest by serving as a watchdog.

This talk between Clay Shirky and Alex Jones is a great introduction to the argument that we’re “losing the news,” to use Jones’ phrase.

But what the Google numbers suggest to me is that, if we really are losing the kind of journalism that is essential for democracy, it’s not because we have too few people. It’s because they’re doing the wrong things. According to people like Shirky and the Knight Foundation, local news in particular is vastly under-served.

I discussed this with a journalist from the International Herald Tribune yesterday. He raised the point that we definitely need more than one organization covering each story. Competition is important, as is a plurality of viewpoints — I think we really do want to preserve the difference between the CNN and Al Jazeera.

So the right number of newsrooms on each story is greater than one. But I bet it’s less then 10 or 20, which is what we have now. Sadly, I think this means that there is a currently a tremendous duplication of effort in the global news-gathering system.

The world’s journalists need to get better organized and stop wasting their efforts. Markets are very good at maximizing efficiency for all sorts of things, but I think this is a case where just letting the market strip apart newspapers until they’re profitable again (if ever) is unlikely to give us the right answer.

Journalism is, arguably, a public good in the economic sense. This means that everyone benefits from it, but everyone also shares it — you will tell your friends the hot news, which makes it difficult to charge for. It’s been understood for at least a century that a competitive market tends to produce too little of a public good. The only reason we had so many newspapers before the internet is that they had a monopoly on distribution. Presses and paper are expensive.

I am not arguing that newspapers need to be run as non-profits or government subsidized, as some people have (and it’s worth noting that the world-wide BBC operation runs on UK government money.) Personally I favor hybrid “social-venture” models that subsidize news through other means. I find Jay Rosen’s list of ways that the news has been subsidized in the past to be particularly enlightening.

It seems likely to me that the bare market is going to under-produce journalism, at least the general-interest public service sort of journalism (financial journalism is currently quite profitable, thank you very much.). If we believe this, and we are going to start designing models and policies to ensure that we get more, then the question of when we have enough needs to be asked in a serious, empirical way.

Counting story duplication isn’t a very complete answer to “how do we know when we have enough journalists?” but it’s a start.

What I really don’t know how to address is the question of how many stories should have been written, if only someone had been there to cover them.

]]> http://jonathanstray.com/what-is-the-right-number-of-journalists/feed 1 http://jonathanstray.com/advertising-got-there-first http://jonathanstray.com/advertising-got-there-first#comments Fri, 20 Nov 2009 10:10:45 +0000 Jonathan Stray http://jonathanstray.com/?p=1177 Phantom 3D objects floating in the air, visible only through the portal of your phone? An urban game played with same? Mobile ad boutique The Hyper Factory seems to have got there first. Their recent ad campaign for Nike used image recognition of printed targets (on posters, in magazines, on the ground of a football field, etc.) to superimpose hovering shoes over the real world.

This is, without a doubt, creative. But looking at it strictly as a creative work, it is severely hamstrung by the fact that the objective is to sell shoes. My guess is that it will be games that push the aesthetic and technical boundaries of this technology. We’re going to see strange reality-fantasy hybrids that will make World of Warcraft and Second Life look old, boring, and flat. Then again, it might also make LARPing socially acceptable, and do we really want that?

And after the technology is ubiquitous and cheap, we’re going to use it to put deep labels on our environment in real time — this is already starting with a sort of Wikipedia for objects. If you’re one of those people who feel sorta blind without your smartphone, just wait until it’s built into your sunglasses.

Search is when the user asks the computer for a particular type of information, and the computer finds it.

News is when the computer has to figure out, by itself, what information a user wants in each moment.

This definition has useful consequences. For example, it says that accurately modeling the user and their needs is going to be absolutely essential for news, because the news problem doesn’t have a query to go on. All a news selection algorithm can know is what the user has done in the past. For this reason, I don’t believe that online news systems can truly be useful until they take into account everything of ourselves that we’ve put online, including Facebook profiles and emails, and viewing histories.

And yes, I do want my news engine to keep track of cool YouTube uploads and recommend videos to me. This in addition to telling me that Iran has a secret uranium enrichment facility. In the online era, “news” probably just means recently published useful information, of which journalistic reporting is clearly a very small segment.

It’s worth remembering that keyword web search wasn’t all that useful until Google debuted in 1998 with an early version of the now-classic PageRank algorithm.  I suspect that we have not yet seen the equivalent for news. In other words, the first killer news app has yet to be deployed. Because such an app will need to know a great deal about you, it will probably pull in data from Facebook and Gmail, at a minimum. But no one really knows yet how to turn a pile of emails into a filter that selects from the best of the web, blogosphere, Twitter, and mainstream media.

Classic journalism organizations are at a disadvantage in designing modern news apps, because broadcast media taught them bad habits. News organizations still think in terms of editors who select content for the audience. This one-size fits all attitude seems ridiculous in the internet era, a relic of the age when it would have been inconceivably expensive to print a different paper for each customer.

Of course, there are some serious potential problems with the logical end-goal of total customization. The loss of a socially shared narrative is one; the Daily Me effect where an individual is never challenged by anything outside of what they already believe is another. But shared narratives seem to emerge in social networks regardless of how we organize them — this is the core meaning of something “going viral.” And I believe the narcissism problem can be addressed through information maps. In fact, maps are so important that we should add another required feature to our hypothetical killer news app: it must in some way present a useful menu of the vast scope of available information. This is another function that existing search products have hardly begun to address.

Not that we have algorithms today that are as good as human editors as putting together a front page. But we will. Netflix’s recent million dollar award for a 10% improvement in their film recommendation system is a useful reminder of how seriously certain companies are taking the problem of predicting user preferences.

The explosion of blog, Twitter, and Wikipedia consumption demonstrates that classic news editors may not have been so good at giving us what we want, anyway.

A basic ranking system allows each user to add a vote to the items they like, then builds a “top rated” list by counting votes. The problem with this scheme is that users can only vote on items they’ve seen, and they are far more likely to see items near the top of the list. In fact, anything off the front page may get essentially no views at all — and therefore has virtually no chance of rising to top.

This is rather serious if the content being rated is serious. It’s fine for Digg to have weird positive-feedback popularity effects, but it’s not fine if we are trying to decide what goes on the front page of a news site. Potentially important stories might never make it to the top simply because they started a little lower in the rankings for whatever reason.

Slightly more sophisticated systems allow users to rate items on a scale, typically 1-5 stars.  This seems better, but still introduces weird biases. Adding up the stars assigned by all users to a single item doesn’t work, because users still have to see an item to vote on it. Averaging all the ratings assigned to a single item doesn’t work either, because it can push something permanently to the bottom of the list, if the first user to view it rates it only one star.

There are lots of subtle hacks that one can make to try to fix the system, but it turns out there might actually be a right way to do things.

If every item was rated by every user, there would be no problem with popularity feedback effects.

That’s completely impractical with thousands or even millions of items. But we can actually get close to the same result with much less work, if we take random samples. Like a telephone poll, the opinion of a small group of randomly selected people will be an accurate indicator, to within a few percent, of the result that we would get if we asked everyone.

In practice, this would mean adding a few select “sampling” stories to each front page served, different every time. Items can then by ranked simply their average rating, with no skewing due to who got to the front page first. (In fact, basic sampling math will tell us which items have the most uncertain ratings and need to be seen with the highest priority.) In effect, we are distributing the work of rating a huge body of items across a huge body of users — true collaborative filtering, using sampling methods to remove the “can’t see it can’t vote on it” bias.

This is not an end-all solution to the problem of distributed agenda-setting. User ratings are not necessarily the ideal criterion for measuring “relevance.” One problem is that not every user is going to take the trouble to assign a rating, so you will only be sampling from particularly motivated individuals. Other metrics such as length of time on page might be better — did this person read the whole thing?

Even more fundamentally, it’s not clear that popularity, however defined, is really the right way to set a news agenda in the public interest.

However, any attempt to use user polling for collaborative agenda setting needs to be aware of basic statistical bias issues. Sampling is a simple and very well-developed way to think about such problems.

The president did not lie. All of the proposed health care reform bills contain language excluding those residing illegally in the US from government-subsidized coverage. This single-sentence fact check was entirely absent from 50 of the 70 articles mentioning “wilson” and “lie” on the New York Times and Washington Post websites as of Monday night. Of the 20 which discussed actual policy, only nine articles mentioned it in the first two paragraphs. (Spreadsheet here.)

Wilson’s outburst will be forgotten long after millions of Americans are insured — or not — under Obama’s plan. It’s just noise and heat. Yet some of the most reputable newspapers in the world have lead with it for the last five days. In fact, the press has in some cases actively dodged the underlying issue. Consider this exchange from an online Q&A session with Dana Milbank of the Washington Post:

Cincinnati: Are you saying the President wasn’t lying when he said illegal immigrants won’t be covered? Why not look at the House bill and tell us whether or not it allows illegals to be covered? The Congressional Research service issued a report last week saying there was NOTHING in the House bill that excludes illegals from receiving government-run health care. In other words, be a REPORTER instead of a hack for Barack.

Dana Milbank:  Actually I wasn’t addressing the factual nature of Obama’s speech. The issue wasn’t that Wilson thought the president wasn’t telling the truth; part of the presidential job description calls for expertise in truth shading. The issue was shouting “you lie!” at the president on the House floor during an address to a joint session of Congress.

(For the record, the CRS report in question notes that HR 3200 says “Nothing in this subtitle shall allow Federal payments for affordability credits on behalf of individuals who are not lawfully present in the United States.” Which has, oddly, been spun as meaning that illegals would be subsidized!)

It should be no surprise that there is actually substance to the question of coverage for illegal immigrants. Only nine of the 70 pieces get into it: yes, a few undocumented workers could end up getting subsidized health care. No, it’s not worth taxpayer money to add an enforcement mechanism.

But even this is one level removed, and only one article grappled with the fundamental question: would it really be so bad if the poorest workers in America got a break? In fact we might even owe it to them. On average, migrant labor is thought to be a small net gain to the American economy.

I get that Wilson’s little moment is a great story, right up there with the guy who threw a shoe at Bush (who was imprisoned for his prank, with far less coverage.) And I do understand the logic of a populist press as the paper ship sinks. What cannot be excused is the omission of any mention of the substantive content of the debate from the majority of coverage — 50 out of 70 articles said nothing at all about anything that will last.

We are reporting on court theatrics while the citizens starve.