Depending on who you ask, machines taking over the world is either a good thing for humanity or a bad thing. The traditional SciFi script has advanced intelligences replicating through all the networks of the galaxy and having high-bandwidth intellectual conversations about things like the fundamental nature of physics and whether biological life deserves to continue to exist, since it’s such an out-dated evolutionary stage and all. But in his new novel Daemon, and in his talk last night at the Long Now Foundation‘s lecture series, Daniel Suarez argues that it’s not hyper-intelligence at all that we need to be wary of: humanity can lose control of the situation well before the appearance of consciousness on the internet. We’re already delegating our decision making to the machines, specifically the lowly “bots” we use now for a variety of practical online tasks.

In truth, automated systems already rule the world, because there’s simply too much data for humans to do it on their own. Instant online loan applications necessarily employ software which embodies the policies of the lender. In 2002 there were estimated to be 500,000 surveillance cameras on private property in London, and those screens aren’t watched by humans. It is thought that in the next few years machine-to-machine internet traffic (such as automatic inventory control systems placing orders) will exceed all human generated traffic, and nobody crawls the web by hand. This is not even counting the current generation of “botnets“, dumb but virulent programs that attempt to silently steal CPU cycles on as many computers as possible, in order to do things like send spam or take specific machines off the internet via Distributed Denial of Service attacks. Such criminal behavior is worrying, but the situation would be intractably complex even if everyone on the internet was trying to play nice.

If you take a step back and apply the right metaphors, what you get is an ecosystem of self-directed software agents, which Suarez generically and broadly terms “bots”. These software agents are evolving, not because we’re anywhere near the type of self-replicating self-modifying programs that might be subject to bona-fide survival selection pressure — that’s a lot to ask from a little Perl script written by a day-job programmer at a bank — but because there’s often a strong incentive for us to make them just a little more efficient. Every fraction of a percent pattern that Google can tease out of our click-through habits translates into millions of dollars of revenue; thus, the bots are getting smarter, and there are already far too many of them out there for any one person to understand all their possible interactions.

The point isn’t that software runs the world. Post Y2K-bug, that should be obvious. Rather, Suarez wants us to think seriously about all the traditional machine-intelligence doomsday scenarios, because all of them are entirely possible without that cinematic “I’m sorry Dave” moment. It may be the concept of an alien mind that scares us, but Very Bad Things can happen within an enormous system of stupid but autonomous agents, most of them privately run and completely unknown and uninvestigated. “Consciousness” really isn’t the issue. Rather, we need to recognize that the digital environment has already become every bit as complex, fragile, and necessary as the biological and physical environment.

Suarez suggests the creation of a parallel “dark intenet” which is encrypted, authenticated, and humans only. The idea is that all bots allowed onto this network would have to be open-source and properly vetted. I like the idea, but I’m not sure it will work: individual programs can be vetted, but nobody can predict the behavior of the entire interacting system. I do support better security protocols for the internet, but I think humans are doomed to give up control to software on any new network for exactly the same reason we’ve already given it up on the internet: millions of tiny automated efficiencies end up enormously amplifying our power. What began as merely convenient must inevitably end as world-changing.