The comments on the news are more revealing of a culture than the news itself. Journalism too often has a commitment to a sort of sanitized neutrality, and certainly tries for clarity, smoothing away complex disagreements. That has its uses, but the comments are a much messier, more divided, more personal look at a culture. They have the texture of life at street level.

First, America. On January 30 New York Times ran an article headlined “U.S. Suspends Haitian Airlift in Cost Dispute” which described how the US had stopped medical evacuations to Miami because of a state vs. federal arguments over who would pay for their medical care. The comments reveal a deeply divided country:

The richest country in the world bickering about who is going to pay before it treats patients who need critical care from the poorest country in the Western hemisphere! We Americans should be very proud of ourselves!

Another wave of third world, uneducated people of an alien culture is about to hit our shores, helped this time by the Obama administration’s desire to show compassion. Unfortunately, the tax-paying citizens of this country will have to pay, in more ways than one.

Meanwhile, Nigerians are talking about the Underwear Bomber. Global Voices has helpfully collected some of the blogger reactions. For America, the story was about fear and terrorism and security. For Nigeria, it was about reputation and identity:

Be honest, when you heard a Nigerian man tried to commit a terrorist act in America, how many of you immediately thought ‘Please don’t let him be [insert your ethnic group]?

There’s an Igbo proverb that says, “If one finger touches palm oil, it spreads to all the other fingers.” This is indicative of how Nigerians the world over felt when they heard the news of a young man who attempted to detonate a bomb on U.S. soil in the name of Al Qaeda. Many of us worried that the actions of this one finger would spread to cover the entire 150 million of us.

How does disowning him help Nigerians understand what role extreme Islamic ideology played in causing him to attempt detonating an explosive device on board a US-bound airliner? How does it help Nigerians understand the complex interplay of religious faith, access to extremist religious groups and ideological brainwashing?

Meanwile, the ever-wonderful ChinaSMACK took a break from pop-culture scandal (on the site right now: Hong Kong Girl Shows Off C Cup Breasts To Ex-Boyfriend) to translate online Chinese reactions to news of US sales of arms to Taiwan. And the Chinese, often portrayed as uniformly nationalist, are just as diverse and divided as any other country:

We don’t need to fear America selling arms to Taiwan, as soon as a war started these advanced weapons would be quickly consumed by our lower-quality but numerous weapons and many soldiers.

I only know that without America, the whole world would be chaotic.

As long as America exists, the world cannot be peaceful.

What I want to know is, where are the foreign voices in these conversations? Right now each culture is talking about the others like they’re not in the room. And they’re right. Our global conversation is fragmented and unmapped. It’s not a small world after all.

We live in a cacaphony of news, but most of it is just echoes. Generating news is expensive; collecting it is not. This is the central insight of the news aggregator business model, be it a local paper that runs AP Wire and Reuters stories between ads, or web sites like Topix, Newser, and Memeorandum, or for that matter Google News. None of these sites actually pay reporters to research and write stories, and professional journalism is in financial crisis. Meanwhile there are more bloggers, but even more re-blogging. Is there more or less original information entering the web this year than last year? No one knows.

A computer could answer this question. A computer could trace the first, original source of any particular article or statement. The effect would be like donning special glasses in the hall of mirrors that is current news coverage, being able to spot the true sources without distraction from reflections. The required technology is nearly here.

This is more than geekery if you’re in a position of needing to know the truth of something. Last week I was researching a man named Michael D. Steele, after reading a newly leaked document containing his name. Steele gained fame as one of the stranded commanders in Black Hawk Down, but several of his soldiers later killed three unarmed Iraqi men. I rapidly discovered many news stories (1, 2, 3, 4, 5, 6, 7, etc.) claiming that Steele had ordered his men to “kill all military-age males.” This is a serious accusation, and widely reprinted — but no number of news articles, blog posts, and reblogs can make a false statement more true. I needed to know who first reported this statement, and its original source.

Continue Reading »

How much overlap is there between the web in different languages, and what sites act as gateways for information between them? Many people have constructed partial maps of the web (such as the  blogosphere map by Matthew Hurst, above) but as far as I know, the entire web has never been systematically mapped in terms of language.

Of course, what I actually want to know is, how connected are the different cultures of the world, really? We live in an age where the world seems small, and in a strictly technological sense it is. I have at my command this very instant not one but several enormous international communications networks; I could email, IM, text message, or call someone in any country in the world. And yet I very rarely do.

Similarly, it’s easy to feel like we’re surrounded by all the international information we could possibly want, including direct access to foreign news services, but I can only read articles and watch reports in English. As a result, information is firewalled between cultures; there are questions that could very easily be answered by any one of tens or hundreds of millions of native speakers, yet are very difficult for me to answer personally. For example, what is the journalistic slant of al-Jazeera, the original one in Arabic, not the English version which is produced by a completely different staff?  Or, suppose I wanted to know what the average citizen of Indonesia thinks of the sweatshops there, or what is on the front page of the Shanghai Times today– and does such a newspaper even exist? What is written on the 70% of web pages that are not in English?

Continue Reading »

Wikipedia will save the world. Information is tolerance. When the internet succeeds and all humanity finally has egalitarian access to all information everywhere, a new era of enlightenment will dawn.

Oh really?

Continue Reading »

I’ve been reading StopTheACLU.com, because I want to get into their heads, because I want to avoid the classic mistake of intellectual isolation, and because I want to be challenged. Sure, they’re weirdos, but that doesn’t mean they don’t make sense. But there’s at least one thing in the StopTheACLU worldview that I find very hard to method-act: in their universe, global warming is a myth.

Okay, but how did I end up on this side and not that side?

Continue Reading »

The North Sage and the South Sage met at the crossroads. Or on, let’s say, a mountaintop. They began to discuss what they knew about the world, in the hopes of becoming wiser. Neither would call what they believed a religion.

Continue Reading »

But you saw it, right? Our man won the debates. Come on, it was totally obvious that the other guy simply does not understand what needs to be done. He’s such an imbecile, I just don’t understand anyone could believe such nonsense. As opposed to us– that man will take us into the future, I tell you. And, look, all the papers support us. The people want us. Just ask anyone you know.

Well, I certainly wouldn’t trust them. Not even worth listening to that crap.

Really?

Really. I read their web page once, and, man, it just enrages me. How could anyone think that way? Such people defy the plainest common sense. It’s not even worth listening to them. Here, read this book. It clearly shows that we’re right.

But how do you know?

Well. Isn’t it obvious? The arguments actually make sense, that’s the great thing about it. I mean, how many people have you met who didn’t agree? Yeah, okay, so there are some weirdos. Heh. That’s definitely true. Yeah, of course, research it if you want. Just remember that those reports are biased by their ideology. My god, who have you been listening to?

Talk to them?

Okay, but… you just can’t convince a person like that. It’s a waste of time with such people. Really. it’s been studied.

Do you ever wonder how they got that way?

What on earth do you mean?

You cannot read all of the news, every day. There is simply too much information for even a dedicated and specialized observer to consume it all, so someone or something has to make choices. Traditionally, we rely on some other person to tell us what to see: the editor of a newspaper decides what goes on the front page, the reviewer tells us what movies are worth it. Recently, we have been able to distribute this mediation process across wider communities: sites like Digg, StumbleUpon, or Slashdot all represent the collective opinions of thousands of people.

The next step is intelligent news agents. Google (search, news, reader, etc.) can already be configured to deliver to us only that information we think we might want to see. It’s not hard to imagine much more sophisticated agents that would scour the internet for items of interest.

In today’s context, it’s easy to see how such agents could actually be implemented. Sophisitacted customer preference engines are already capable of telling us what products we might like to consume — the best example is Amazon’s recommendation engine. It’s not a big leap to imagine using the same sort of algorithms to model the kinds of blog articles, web pages, youtube videos, etc. that we might enjoy consuming, and then deliver these things to us.

There is a serious problem with this. You’re going to get exactly what you ask for, and only that.

True, we all do this already. We read books and consume media which more or less confirm our existing opinions. This effect is visible as clustering in what we consume, as in this example of Amazon sales data for political books in 2008.

This image is from a beautiful analysis by orgnet.com. Basically, people buy either the red books or the blue books, but usually not both. The same sorts of patterns hold for movies, blogs, newspapers, ideologies, religions, and human beliefs of all kinds. This is a problem; but at least you can usually see the other color of books when you walk into Borders. If we end up relying on trainable agents for all of our information, we risk completely blacking out anything that disagrees with what we already believe.

I propose a simple solution. Automatic network analyses like the one above — of books, or articles, or web pages — could easily pinpoint the information sources that would expose me to the maximum novelty in the minimum time. If my goal is to gain a deep understanding of the entire scope of human discourse, rather than just the parts of it I already agree with, then it would be very simple to program my agent to bring to me exactly those things that would most rapidly give me insight into those regions of information space which are most vital and least known to me. I imagine some metric like “highest degree node most distant from the nodes I’ve already visited” would would work handily.

You can infer a lot about somewhat from the information they currently consume. If my agent noticed that I was a liberal, it could make me understand the conservative world-view, and vice-versa. If my agent detected that I was ignorant of certain crucial aspects of Chinese culture and politics, it could reccomend a primer article. Or it might deduce that I needed to understand just slightly more physics to participate meaningfully in the climate change debate, or decide (based on my movie viewing habits) that it was high time I review the influential films of Orson Welles. Of course, I might in turn decide that I actually, truly, don’t care about film at all; but the very act of excluding specific subjects or categories of thought would force us, consciously, to admit to the boundaries of our mental worlds.

We could program our information gathering systems to challenge us, concisely and effectively, if we so want. Intelligent agents could be mere sycophants, or they could be teachers.

Everybody thinks they know how their mind works, but they don’t. You can ask someone why they like their boyfriend, or why they chose a job, or whether a book changed their opinion of global warming, and they’ll think about it for a moment and happily give you an answer. But they’re making it up.

The experiments were done ages ago, and the research is still going, continuing to tease apart actual cause and psychological effect. We know now that what people tell us about their own mental processes is quite thoroughly inaccurate. We all believe that we have this magic thing called “introspection” that lets us see what is going on in our own minds, but in reality we don’t. It’s a fictional superpower.

The research on this point is really quite good. It’s not even a new finding, having been understood for at least the last fifty years. And yet this simple but important fact has never quite managed to make it into popular culture.

Perhaps no one wants to believe it.

Continue Reading »