The next step is intelligent news agents. Google (search, news, reader, etc.) can already be configured to deliver to us only that information we think we might want to see. It’s not hard to imagine much more sophisticated agents that would scour the internet for items of interest.

In today’s context, it’s easy to see how such agents could actually be implemented. Sophisitacted customer preference engines are already capable of telling us what products we might like to consume — the best example is Amazon’s recommendation engine. It’s not a big leap to imagine using the same sort of algorithms to model the kinds of blog articles, web pages, youtube videos, etc. that we might enjoy consuming, and then deliver these things to us.

There is a serious problem with this. You’re going to get exactly what you ask for, and only that.

True, we all do this already. We read books and consume media which more or less confirm our existing opinions. This effect is visible as clustering in what we consume, as in this example of Amazon sales data for political books in 2008.

This image is from a beautiful analysis by orgnet.com. Basically, people buy either the red books or the blue books, but usually not both. The same sorts of patterns hold for movies, blogs, newspapers, ideologies, religions, and human beliefs of all kinds. This is a problem; but at least you can usually see the other color of books when you walk into Borders. If we end up relying on trainable agents for all of our information, we risk completely blacking out anything that disagrees with what we already believe.

I propose a simple solution. Automatic network analyses like the one above — of books, or articles, or web pages — could easily pinpoint the information sources that would expose me to the maximum novelty in the minimum time. If my goal is to gain a deep understanding of the entire scope of human discourse, rather than just the parts of it I already agree with, then it would be very simple to program my agent to bring to me exactly those things that would most rapidly give me insight into those regions of information space which are most vital and least known to me. I imagine some metric like “highest degree node most distant from the nodes I’ve already visited” would would work handily.

You can infer a lot about somewhat from the information they currently consume. If my agent noticed that I was a liberal, it could make me understand the conservative world-view, and vice-versa. If my agent detected that I was ignorant of certain crucial aspects of Chinese culture and politics, it could reccomend a primer article. Or it might deduce that I needed to understand just slightly more physics to participate meaningfully in the climate change debate, or decide (based on my movie viewing habits) that it was high time I review the influential films of Orson Welles. Of course, I might in turn decide that I actually, truly, don’t care about film at all; but the very act of excluding specific subjects or categories of thought would force us, consciously, to admit to the boundaries of our mental worlds.

We could program our information gathering systems to challenge us, concisely and effectively, if we so want. Intelligent agents could be mere sycophants, or they could be teachers.

In truth, automated systems already rule the world, because there’s simply too much data for humans to do it on their own. Instant online loan applications necessarily employ software which embodies the policies of the lender. In 2002 there were estimated to be 500,000 surveillance cameras on private property in London, and those screens aren’t watched by humans. It is thought that in the next few years machine-to-machine internet traffic (such as automatic inventory control systems placing orders) will exceed all human generated traffic, and nobody crawls the web by hand. This is not even counting the current generation of “botnets“, dumb but virulent programs that attempt to silently steal CPU cycles on as many computers as possible, in order to do things like send spam or take specific machines off the internet via Distributed Denial of Service attacks. Such criminal behavior is worrying, but the situation would be intractably complex even if everyone on the internet was trying to play nice.

If you take a step back and apply the right metaphors, what you get is an ecosystem of self-directed software agents, which Suarez generically and broadly terms “bots”. These software agents are evolving, not because we’re anywhere near the type of self-replicating self-modifying programs that might be subject to bona-fide survival selection pressure — that’s a lot to ask from a little Perl script written by a day-job programmer at a bank — but because there’s often a strong incentive for us to make them just a little more efficient. Every fraction of a percent pattern that Google can tease out of our click-through habits translates into millions of dollars of revenue; thus, the bots are getting smarter, and there are already far too many of them out there for any one person to understand all their possible interactions.

The point isn’t that software runs the world. Post Y2K-bug, that should be obvious. Rather, Suarez wants us to think seriously about all the traditional machine-intelligence doomsday scenarios, because all of them are entirely possible without that cinematic “I’m sorry Dave” moment. It may be the concept of an alien mind that scares us, but Very Bad Things can happen within an enormous system of stupid but autonomous agents, most of them privately run and completely unknown and uninvestigated. “Consciousness” really isn’t the issue. Rather, we need to recognize that the digital environment has already become every bit as complex, fragile, and necessary as the biological and physical environment.

Suarez suggests the creation of a parallel “dark intenet” which is encrypted, authenticated, and humans only. The idea is that all bots allowed onto this network would have to be open-source and properly vetted. I like the idea, but I’m not sure it will work: individual programs can be vetted, but nobody can predict the behavior of the entire interacting system. I do support better security protocols for the internet, but I think humans are doomed to give up control to software on any new network for exactly the same reason we’ve already given it up on the internet: millions of tiny automated efficiencies end up enormously amplifying our power. What began as merely convenient must inevitably end as world-changing.