Archives

What is the Right Number of Journalists?

How many journalists does the world need to adequately serve the public? It’s a difficult question, and I’m going to argue that the market won’t tell us. But here’s a thought: for the first time in history, it’s trivial to check how many outlets covered any particular story.

From Google News just now:

TooMuchCoverage

In this case, there were 2,907 articles on the web covering the story of Iran freeing the Brits. Even accounting for the fact that most of these pages will be mirrors of the same text, we can see that Aljazeera, The Telegraph, the Associated Press, CNN, the New York Times, and many others covered the story. And they all did their own reporting, a horde of journalists from all over the world making phone calls and doing interviews.

Meanwhile, written-word reporters are getting laid off right and left, essentially due to the end of the print-based information distribution monopolies. Last year, 35,000 journalists lost their jobs in the US. This has made a number of quite clever people fret about the end of “accountability journalism,” the press that keeps people honest by serving as a watchdog.

This talk between Clay Shirky and Alex Jones is a great introduction to the argument that we’re “losing the news,” to use Jones’ phrase.

But what the Google numbers suggest to me is that, if we really are losing the kind of journalism that is essential for democracy, it’s not because we have too few people. It’s because they’re doing the wrong things. According to people like Shirky and the Knight Foundation, local news in particular is vastly under-served.

I discussed this with a journalist from the International Herald Tribune yesterday. He raised the point that we definitely need more than one organization covering each story. Competition is important, as is a plurality of viewpoints — I think we really do want to preserve the difference between the CNN and Al Jazeera.

So the right number of newsrooms on each story is greater than one. But I bet it’s less then 10 or 20, which is what we have now. Sadly, I think this means that there is a currently a tremendous duplication of effort in the global news-gathering system.

The world’s journalists need to get better organized and stop wasting their efforts. Markets are very good at maximizing efficiency for all sorts of things, but I think this is a case where just letting the market strip apart newspapers until they’re profitable again (if ever) is unlikely to give us the right answer.

Journalism is, arguably, a public good in the economic sense. This means that everyone benefits from it, but everyone also shares it — you will tell your friends the hot news, which makes it difficult to charge for. It’s been understood for at least a century that a competitive market tends to produce too little of a public good. The only reason we had so many newspapers before the internet is that they had a monopoly on distribution. Presses and paper are expensive.

I am not arguing that newspapers need to be run as non-profits or government subsidized, as some people have (and it’s worth noting that the world-wide BBC operation runs on UK government money.) Personally I favor hybrid “social-venture” models that subsidize news through other means. I find Jay Rosen’s list of ways that the news has been subsidized in the past to be particularly enlightening.

It seems likely to me that the bare market is going to under-produce journalism, at least the general-interest public service sort of journalism (financial journalism is currently quite profitable, thank you very much.). If we believe this, and we are going to start designing models and policies to ensure that we get more, then the question of when we have enough needs to be asked in a serious, empirical way.

Counting story duplication isn’t a very complete answer to “how do we know when we have enough journalists?” but it’s a start.

What I really don’t know how to address is the question of how many stories should have been written, if only someone had been there to cover them.

The News Doesn’t Matter if No One Cares

Gathering and delivering the news is the easy part, the part that the journalism industry already knows how to do. Getting the audience to care about the news is the hard part. This seems obvious if you define the mission of journalism as “an informed public,” but I fear that many journalism organizations aren’t thinking about what happens to the news after publication.

newsball

It’s long been understood that sports and celebrity gossip are far more popular among readers than actual news. In the paper era, everyone got serious factual reporting delivered to them anyway: the act of reading a physical newspaper forces you to see the headlines, no matter what your interests are. The web isn’t like this, and paper news consumption is low and falling among young readers, which means that the classic package of whether-you-knew-you-cared-or-not news is on the way out as a delivery format.

The following passage from Alex Jones’ Losing The News nails the problem:

News on the web is almost entirely chosen by the viewer. You have to seek it out, which means that you can easily miss important stories or avoid troubling ones. Print newspapers had offered a smorgasbord, but you had to cast your eye over a range of information that included some hard news. And both television and radio news game no choices other than to change the channel or station. On the Web, you get the news you want, and what Americans seem to want, when it comes to news, is a lot of Britney Spears and much less of the Iraq war. (p. 180)

Does this matter? The theory is that democracy will not function if people aren’t informed about the basic political, economic, and scientific happenings of the world. The dissolution of paper delivery, this line of reasoning goes, means the end of an informed citizenry.

I just think it means that journalism and journalists now have a new responsibility: they have to figure out how to get people to care. Inspiring curiosity is part of the job description.

So how do we make people more interested in knowing about the news? This is more than a question of format, or graphic design or packaging. This is not about snappier video editing and edgier prose. That sort of thing might be helpful in the short run, but these kinds of changes come from a fundamentally broadcast mentality. This is not the mindset of the creators of Google, Twitter, and Facebook. If news publishers seriously believe that they’re important players the public information game, then why is it the technology companies who have delivered all of the communication revolutions of the last decade?

Here are some innovative examples of public engagement that might offer clues:

  • Wikipedia proved that smart people will do good work for free, if they perceive it to be in the public interest.
  • The Guardian UK recently crowdsourced its review of 450,000 pages of suspect government expense reports.
  • The creator of the wildly successful programming advice site StackOverflow.com has a really good talk about how different web site designs create different types of social behavior.
  • Maps of the news might reward curiosity.

Mere publishing is no longer enough.

The Search Problem vs. The News Problem

I think I’ve found a useful distinction between the “search” and “news” problems. News organizations like to complain that search engines are taking their business, but that’s only because no one has yet built a passable news engine.

Search is when the user asks the computer for a particular type of information, and the computer finds it.

News is when the computer has to figure out, by itself, what information a user wants in each moment.

This definition has useful consequences. For example, it says that accurately modeling the user and their needs is going to be absolutely essential for news, because the news problem doesn’t have a query to go on. All a news selection algorithm can know is what the user has done in the past. For this reason, I don’t believe that online news systems can truly be useful until they take into account everything of ourselves that we’ve put online, including Facebook profiles and emails, and viewing histories.

And yes, I do want my news engine to keep track of cool YouTube uploads and recommend videos to me. This in addition to telling me that Iran has a secret uranium enrichment facility. In the online era, “news” probably just means recently published useful information, of which journalistic reporting is clearly a very small segment.

It’s worth remembering that keyword web search wasn’t all that useful until Google debuted in 1998 with an early version of the now-classic PageRank algorithm.  I suspect that we have not yet seen the equivalent for news. In other words, the first killer news app has yet to be deployed. Because such an app will need to know a great deal about you, it will probably pull in data from Facebook and Gmail, at a minimum. But no one really knows yet how to turn a pile of emails into a filter that selects from the best of the web, blogosphere, Twitter, and mainstream media.

Classic journalism organizations are at a disadvantage in designing modern news apps, because broadcast media taught them bad habits. News organizations still think in terms of editors who select content for the audience. This one-size fits all attitude seems ridiculous in the internet era, a relic of the age when it would have been inconceivably expensive to print a different paper for each customer.

Of course, there are some serious potential problems with the logical end-goal of total customization. The loss of a socially shared narrative is one; the Daily Me effect where an individual is never challenged by anything outside of what they already believe is another. But shared narratives seem to emerge in social networks regardless of how we organize them — this is the core meaning of something “going viral.” And I believe the narcissism problem can be addressed through information maps. In fact, maps are so important that we should add another required feature to our hypothetical killer news app: it must in some way present a useful menu of the vast scope of available information. This is another function that existing search products have hardly begun to address.

Not that we have algorithms today that are as good as human editors as putting together a front page. But we will. Netflix’s recent million dollar award for a 10% improvement in their film recommendation system is a useful reminder of how seriously certain companies are taking the problem of predicting user preferences.

The explosion of blog, Twitter, and Wikipedia consumption demonstrates that classic news editors may not have been so good at giving us what we want, anyway.

Advertisers Smoking Crack, and the Future of Journalism According to Leo Laporte

Leo Laporte of This Week in Tech gave a truly marvelous talk on Friday about how his online journalism model works. The first half of the talk is all about how TWIT moved from TV to podcasting and became profitable, and includes such gems as

Advertisers have been smoking the Google and Facebook crack. And they no longer want that shakeweed that the [TV] networks are offering.

The second half is in many ways even better, when Leo takes questions from the audience and discusses topics such as the future of printing news on dead trees

Maybe there will always be [paper] news, but it will be brought to you by your butler who has ironed it out carefully for you. It will be the realm of the rich person.

and the “holy calling” of being a journalist:

You reporters are really the monks of the information world. You labour in obscurity. You have to be driven by passion because  you’re paid nothing. And you sleep on rocks.

He goes on to discuss the necessity of bidirectional communication, Twitter as the “emerging nervous system” of the net, etc. — all the standard new media stuff, but put very succinctly by someone who has deep experience in both old and new media. Very information-dense and enlightening!

Mapping the Daily Me

If we deliver to each person only what they say they want to hear, maybe we end up with a society of narrow-minded individualists. It’s exciting to contemplate news sources that (successfully) predict the sorts of headlines that each user will want to read, but in the extreme case we are reduced to a journalism of the Daily Me: each person isolated inside their own little reflective bubble.

The good news is, specialized maps can show us what we are missing. That’s why I think they need to be standard on all information delivery systems.

For the first time in history, it is possible to map with some accuracy the information that free-range consumers choose for themselves. A famous example is the graph of political booksales produced by orgnet.com:

Social network graph of Amazon sales of political books, 2008

Here, two books are connected by a line if consumers tended to buy both. What we see is what we always suspected: a stark polarization. For the most part, each person reads either liberal or conservative books. Each of us lives in one information world but not the other. Despite the Enlightenment ideal of free debate, real-world data shows that we do not seek out contradictory viewpoints.

Which was fine, maybe, when the front page brought them to us. When information distribution was monopolized by a small number of newspapers and broadcasters, we had no choice but to be exposed to stories that we might not have picked for ourselves. Whatever charges one can press against biased editors of the past, most of them felt that they had a duty to diversity.

In the age of disaggregation, maybe the money is in giving people what they want. Unfortunately, there is a real possibility that we want is to have our existing opinions confirmed. You and I and everyone else are going to be far more likely to click through from a headline that confirms what we already believe than from one which challenges us. “I don’t need to read that,” we’ll say, “it’s clearly just biased crap.” The computers will see this, and any sort of recommendation algorithm will quickly end up as a mirror to our preconceptions.

It’s a positive feedback loop that will first split us along existing ideological cleavages, then finer and finer. In the extreme, each of us will be alone in a world that never presents information to the contrary.

We could try to design our systems to recommend a more diverse range of articles (an idea I explored previously) but the problem is, how? Any sort of agenda-setting system that relies on what our friends like will only amplify polarities, while anything based on global criteria is necessarily normative — it makes judgements on what everyone should be seeing. This gets us right back into all the classic problems of ideology and bias — how do we measure diversity of viewpoint? And even if we could agree on a definition of what a “healthy” range sources is, no one likes to be told what to read.

I think that maps are the way out. Instead of trying to decide what someone “should” see, just make clear to them what they could see.

An information consumption system — an RSS reader, online newspapers, Facebook — could include a map of the infosphere as a standard feature. There are many ways to draw such a map, but the visual metaphor is well-established: each node is an information item (an article, video, etc.) while the links between items indicate their “similarity” in terms of worldview.

iran_blogosphere_map

This is less abstract than it seems, and with good visual design these sorts of pictures can be immediately obvious. Popular nodes could be drawn larger; closely related nodes could be clustered. The links themselves could be generated from co-consumption data: when one user views two different items, the link between those items gets slightly stronger. There are other ways of classifying items as related — as belonging to similar worldviews — but co-consumption is probably as good a metric as any, and in fact co-purchasing data is at the core of Amazon’s successful recommendation system.

The concepts involved are hardly new, and many maps have been made at the site level where each node is an entire blog, such as the map of the Iranian blogosphere above. However, we have never had a map of individual news items, and never in real-time for everyone to see.

Each map also needs a “you are here” indicator.

This would be nothing more than some way of marking items that the user has personally viewed. Highlight them, center them on the map, and zoom in. But don’t zoom in too much. The whole purpose of the map is to show each of us how small, how narrow and unchallenging our information consumption patterns actually are. We will each discover that we live in a particular city-cluster of information sources, on a particular continent of language, ideology, or culture. A map literally lets you see this at a glance — and you can click on far-away nodes for instant travel to distant worldviews.

Giving people only what they like risks turning journalism into entertainment or narcissism. Forcing people to see things that they are not interested in is a losing strategy, and we there isn’t any obvious way to decide what we should see. Showing people a map of the broader world they live in is universally acceptable, and can only encourage curiosity.

Requiem For The Front Page

Oh Front Page, your days are clearly numbered. For generations all eyes were upon you; you set the public agenda, and advertisers loved you best. In the tumult of the world, your voice carried above all others, and we needed you. You told us when the war ended, and when The Beatles came to town.

But you are in your autumn now.

TheFrontPageisDead

We know that your children killed you, though they did not mean it. In the age of the scribe, it seemed that anyone could own a printing press. But now, Front Page, we talk online about the monopoly you once claimed. Some will pine for newsprint, but paper is just too expensive, too heavy and static.

But this is not about paper. This is about the way you lived your life, your insistence on a space that you and you alone controlled. You tried to move online, Front Page, but your model would not yield and your children ate your lunch. Google News chooses from the best, while Digg lets us choose for ourselves. There will always be reporters — those who assemble the narratives —  but there may not always be editors. Your stubborn insistence on one for all made us question your purpose.

We loved you and you ignored us! Advertisers deserted you first; they were very quick to understand that reader information could be leveraged into relevance. Google itself was built on this model. Meanwhile Amazon and iTunes grasped that efficiencies of delivery had moved the money to the infinite niche. But you admitted none of this, Front Page, and also you did not see that people live in networks, that our friends know what is important to us.

Why would you not give us what we wanted? No one questions your integrity, the standards of journalism you uphold. No one questions that we, the public, need to be told at least as much as we need to be listened to. But suddenly we could talk back, and you weren’t listening. You insisted that we go to you instead of just coming to us. Why did you not use our input to customize the agenda? You could have spawned Facebook applications and iPhone applications and even innovative social RSS readers that determined our interests and automatically delivered ten million personalized headlines! (And their ads.)

You had everything you needed, and this was your unforgivable sin. A hundred years ago you built the Associated Press to feed you, the prototype of distributed journalism. This could have been the beginning, if you had embraced more than the cream of international stories, if you had realized how cheap local reporting could be. Those long tail stories could be vastly cheaper, Front Page, if you embraced more sources, if you fought for transparency instead of access, if you taught citizens to be journalists instead of insisting that they can’t. You could have set the standards and franchised the platform. But instead of finding innovative ways to gather the news and innovative ways to deliver it to us, even now you fight hard to be seen less!

Instead of owning the aggregators and bringing to them the wisdom of an old hand, you scoffed at Digg, at Google, at Memeorandum. Why are there still so many news sites without a panel of  “Share This” links beneath each story? Why are we not allowed to speak to the New York Times with user ratings buttons? Your mannerisms are quaint as hoop-skirts, Front Page.

We know also that your less reputable cousin is only slightly younger, and the world will never listen to Television as their parents did. The internet will devour Broadcast too; in only a few more years bandwidth will be cheap enough for anyone to run their own station. We know that upcoming content analysis algorithms will soon make video search a reality, and we know that the RSS future will soon disaggregate Television News just as it only recently disaggregated you.

Front Page, your children are brash, but they are filled with the energy of youth. They have inherited a world you never foresaw, and they are hopeful in a way you are not. It is their world now. You must guide them, but you must let them have it.

Much as we loved you, your time has passed.

The New York Times Doesn’t Understand Twitter and Iran

In the editorial “New Tweets, Old Needs” experienced journalist Roger Cohen says that Twitter isn’t journalism, and that Iran “has gone opaque” without its mainstream media correspondents. He may be right about the recent paucity of good journalism out of Iran, but he misses some really crucial points about how information flows in the absence of a distribution monopoly (like a printing press.) In particular, he seems to assume that only professional journalists can be capable of producing professional journalism.

It is absolutely true that journalism is much more than random tweeting or blogging. I have been particularly inspired by the notion that “journalism is a discipline of verification,” and a tweet or a blog post neither requires nor endures the fact-checking and truthfulness standards that we expect of our more traditional news media. I also agree that search engines are simply not a substitute for being there. Someone must be a witness. Someone has to feed their experience into the maw of the internet at some point.

However, when Cohen says “the mainstream media — expelled, imprisoned, vilified — is missed” he is implicitly arguing that only the mainstream media can produce good journalism.  Traditionally, “journalist” was  a distinct, easily defined class: a journalist was someone who worked for a news organization. There weren’t many such organizations, because a distribution monopoly is an expensive thing. All this has changed with the advent of nearly free and truly democratic information distribution, and we are seeing a rapid erosion of the the distinction between professional and amateur or “citizen” journalists. The result is confusion, uncertainty and fear — especially on the part of those who have staked their careers or their fortunes on the clarity of this distinction.

But I see a big difference between journalists and journalism, and this is where Cohen and I part ways.

In my view the failure of journalism in Iran was not the failure of the mainstream media to hold their ground (or their funding, or their audiences) but rather the failure of the journalism profession to educate the public about what exactly it does, and how to do it. When Cohen asks questions such as

But who is there to investigate these deaths — or allegations of wholesale rape of hundreds of arrested men and women — and so shed light?

my answer is, the Iranians, of course!

Naturally, a young activist-turned-reporter does not have the experience or connections of an old-school foreign correspondent. But such a person is there, and they care enormously. What they lack is guidance. What is and is not journalism, exactly? What are the expected standards and daily, on-the-ground procedures of verification? Where can someone turn to for advice on covering the struggles they are immersed in? And what, actually, differentiates the New York Times from a blogger? We need clear answers, because the newspapers are no longer the only ones declaiming the news.

Perhaps the mainstream media couldn’t be in Iran, but they could have been mentoring and collaborating from afar, and yes, publishing the journalism of non-career journalists. And such a project needs to begin long before times of crisis, in every region, so that those who are there are ready.

If “citizen journalism” has so far been somewhat underwhelming, it is because we have not taught our citizens to be journalists.

American Press Covers Debate, Not Health Care

Representative Joe Wilson yelled “you lie!” at the president, and the papers loved it. Unfortunately, by a count of more than three to one, the major media articles covering the event did not bother to comment on the substance of issue of that provoked Wilson’s outburst: whether or not illegal immigrants would be provided health care under proposed reforms. There is no health care debate in the mainstream American press. There is only political drama.

The president did not lie. All of the proposed health care reform bills contain language excluding those residing illegally in the US from government-subsidized coverage. This single-sentence fact check was entirely absent from 50 of the 70 articles mentioning “wilson” and “lie” on the New York Times and Washington Post websites as of Monday night. Of the 20 which discussed actual policy, only nine articles mentioned it in the first two paragraphs. (Spreadsheet here.)

Wilson’s outburst will be forgotten long after millions of Americans are insured — or not — under Obama’s plan. It’s just noise and heat. Yet some of the most reputable newspapers in the world have lead with it for the last five days. In fact, the press has in some cases actively dodged the underlying issue. Consider this exchange from an online Q&A session with Dana Milbank of the Washington Post:

Cincinnati: Are you saying the President wasn’t lying when he said illegal immigrants won’t be covered? Why not look at the House bill and tell us whether or not it allows illegals to be covered? The Congressional Research service issued a report last week saying there was NOTHING in the House bill that excludes illegals from receiving government-run health care. In other words, be a REPORTER instead of a hack for Barack.

Dana Milbank:  Actually I wasn’t addressing the factual nature of Obama’s speech. The issue wasn’t that Wilson thought the president wasn’t telling the truth; part of the presidential job description calls for expertise in truth shading. The issue was shouting “you lie!” at the president on the House floor during an address to a joint session of Congress.

(For the record, the CRS report in question notes that HR 3200 says “Nothing in this subtitle shall allow Federal payments for affordability credits on behalf of individuals who are not lawfully present in the United States.” Which has, oddly, been spun as meaning that illegals would be subsidized!)

It should be no surprise that there is actually substance to the question of coverage for illegal immigrants. Only nine of the 70 pieces get into it: yes, a few undocumented workers could end up getting subsidized health care. No, it’s not worth taxpayer money to add an enforcement mechanism.

But even this is one level removed, and only one article grappled with the fundamental question: would it really be so bad if the poorest workers in America got a break? In fact we might even owe it to them. On average, migrant labor is thought to be a small net gain to the American economy.

I get that Wilson’s little moment is a great story, right up there with the guy who threw a shoe at Bush (who was imprisoned for his prank, with far less coverage.) And I do understand the logic of a populist press as the paper ship sinks. What cannot be excused is the omission of any mention of the substantive content of the debate from the majority of coverage — 50 out of 70 articles said nothing at all about anything that will last.

We are reporting on court theatrics while the citizens starve.

Twitter is Not Reality, Even in Guatemala

Guatemalans took to the streets in protest over the alleged murder of a prominent attorney by the country’s president, and an unrelated man was arrested for tweeting about it. The protests were reportedly organized on Facebook and other social networking sites, and streamed live to the world by laptop. Xeni Jardin of Boing Boing has been reporting from Guatemala directly for the past two weeks, and in an essay two days ago she calls this the “Twitter Revolution“. I love the story of new technology enabling mass social dissent and change, but I’m not at all sure it’s true. Sorely missing from Xeni’s narrative is the role of other communication networks — like good old fashioned word-of-mouth — and the demographics of internet access in a poor country.

The background: Attorney Rodrigo Rosenberg was shot while riding his bicycle on May 10th, just a few days after recording a video message which begins,

If you are watching this message, it is because I was assassinated by President Álvaro Colom.

The video implicates not only the president but the major state-owned bank, and indeed much of the current government, and there were mass protests in the capital city. Xeni has been covering the story from Guatemala since the 20th, and I can only commend her for actually being there. However, her coverage has focussed on the role of the internet in these protests.

Google is not reality and Twitter is not reality in exactly the same way that television is not reality. Part of the reason that Middle-Eastern peasants have such a warped view of America is that they too watch Desperate Housewives (via satellite or bootleg VCD), but never get the chance to actually meet some Americans. To them, all American women are blonde and slutty. There’s no reason to believe that we’re not getting a similarly warped view of other cultures when we watch their internet.

Continue reading Twitter is Not Reality, Even in Guatemala

Escaping the News Hall of Mirrors

We live in a cacaphony of news, but most of it is just echoes. Generating news is expensive; collecting it is not. This is the central insight of the news aggregator business model, be it a local paper that runs AP Wire and Reuters stories between ads, or web sites like Topix, Newser, and Memeorandum, or for that matter Google News. None of these sites actually pay reporters to research and write stories, and professional journalism is in financial crisis. Meanwhile there are more bloggers, but even more re-blogging. Is there more or less original information entering the web this year than last year? No one knows.

A computer could answer this question. A computer could trace the first, original source of any particular article or statement. The effect would be like donning special glasses in the hall of mirrors that is current news coverage, being able to spot the true sources without distraction from reflections. The required technology is nearly here.

This is more than geekery if you’re in a position of needing to know the truth of something. Last week I was researching a man named Michael D. Steele, after reading a newly leaked document containing his name. Steele gained fame as one of the stranded commanders in Black Hawk Down, but several of his soldiers later killed three unarmed Iraqi men. I rapidly discovered many news stories (1, 2, 3, 4, 5, 6, 7, etc.) claiming that Steele had ordered his men to “kill all military-age males.” This is a serious accusation, and widely reprinted — but no number of news articles, blog posts, and reblogs can make a false statement more true. I needed to know who first reported this statement, and its original source.

Continue reading Escaping the News Hall of Mirrors