We’ve all lived with Google for so long that most of us can’t even conceive of other methods of information retrieval. But as computer scientists and librarians will tell you, boolean keyword search is not the end-all. There are other classic search techniques, such as latent semantic analysis which tries to return results which are “conceptually similar” to the user’s query, even if the relevant documents don’t contain any of the search terms. I also believe that full-scale maps of the online world are important, I would like to know which web sites act as bridges between languages, and I want tools to track the source of statements made online. These sorts of applications might be a huge advance over keyword search, but large-scale search experiments are, at the moment, prohibitively expensive.

The problem is that the web is really big, and only a few companies have invested in the hardware and software required to index all of it. A full crawl of the web is expensive and valuable, and all of the companies who have one (Google, Yahoo, Bing, Ask, SEOmoz) have so far chosen to keep their databases private. Essentially, there is a natural monopoly here. We would like a thousand garage-scale search ventures to bloom in the best Silicon Valley tradition, but it’s just too expensive to get into the business.

DotBot is the only open web index project I am aware of. They are crawling the entire web and making the results available for download via BitTorrent, because

We believe the internet should be open to everyone. Currently, only a select few corporations have access to an index of the world wide web. Our intention is to change that.

Bravo! However, a web crawl is a truly enormous file. The first part of the DotBot index, with just 600,000 pages, clocks in at 3.2 gigabytes. Extrapolating to the more than 44 billion pages so far crawled, I estimate that they currently have 234 terabytes of data. At today’s storage technology prices of about $100 per terabyte, it would cost $24,000 just to store the file. Real-world use also requires backups, redundancy, and maintenance, all of which push data center costs to something closer to $1000 per terabyte. And this says nothing of trying to download a web crawl over the network — it turns out that sending hard drives in the mail is still the fastest and cheapest way to move big data.

Full web indices are just too big to play with casually; there will always be a very small number of them.

I think the solution to this is to turn web indices and other large quasi-public datasets into infrastructure: a few large companies collect the data and run the servers, other companies buy fine-grained access at market rates. We’ve had this model for years in the telecommunications industry, where big companies own the lines and lease access to anyone who is willing to pay.

The key to the whole proposition is a precise definition of access. Google’s keyword “access” is very narrow. Something like SQL queries would expand the space of expressible questions, but you still couldn’t run image comparison algorithms or do the computational linguistics processing necessary for true semantic search. The right way to extract the full potential of a database is to run arbitrary programs on it, and that means the data has to be local.

The only model for open search that works both technologically and financially is to store the web index on a cloud, let your users run their own software against it, and sell the compute cycles.

It is my hope that this is what DotBot is up to. The pieces are all in place already: Amazon and others sell cheap cloud-computing services, and the basic computer science of large-scale parallel data processing is now well understood. To be precise, I want an open search company that sells map-reduce access to their index. Map-reduce is a standard framework for breaking down large computational tasks into small pieces that can be distributed across hundreds or thousands of processors, and Google already uses it internally for all their own applications — but they don’t currently let anyone else run it on their data.

I really think there’s money to be made in providing open search infrastructure, because I really think there’s money to be made in better search. In fact I see an entire category of applications that hasn’t yet been explored outside of a few very well-funded labs (Google, Bellcore, the NSA): “information engineering,” the question of what you can do with all of the world’s data available for processing at high speed. Got an idea for better search? Want to ask new questions of the entire internet? Working on an investigative journalism story that requires specialized data-mining? Code the algorithm in map-reduce, and buy the compute time in tenth-of-a-second chunks on the web index cloud. Suddenly, experimentation is cheap — and anyone who can figure out something valuable to do with a web index can build a business out of it without massive prior investment.

The business landscape will change if web indices do become infrastructure. Most significantly, Google will lose its search monopoly. Competition will probably force them to open up access their web indices, and this is good. As Google knows, the world’s data is exceedingly valuable — too valuable to leave in the hands of a few large companies. There is an issue of public interest here. Fortunately, there is money to be made in selling open access. Just as energy drives change in physical systems, money drives changes in economic systems. I don’t know who is going to do it or when, but open search infrastructure is probably inevitable. If Google has any sense, they’ll enter the search infrastructure market long before they’re forced (say,  before Yahoo and Bing do it first.)

Let me know when it happens. There are some things I want to do with the internet.

Or at least help us to understand it. Climateprediction.net is a large-scale scientific computing experiment, relying on individual computer users who donate their computer time for the simulation of tens of thousands of global warming scenarios. This is important because, lacking other Earths to experiment with, computer simulations are really the only way we can validate our existing models of climate change — and then predict the future with models we think are accurate.

The climateprediction.net project comprises three separate experiments – one to explore the model we are using, the second to see how well the models replicate past climate and the third to finally produce a forecast for 21st century climate. Each model that we distribute will be used for all three experiments.

Built upon the BOINC scientific computing framework oriignally developed for the groundbreaking SETI@Home project, Climateprediction.net relies upon hundreds of thousands of volunteer users who donate their spare computer time. All of these machines together are effectively one of the largest supercomputers in the world, and this allows previously impossible scientific studies. The Climateprediction.net scientific team can run not just one or a few climate prediction simulations, but hundreds of thousands. One study performed this way was the Seasonal Attribution Experiment:

We focus on extreme weather events that occur on a seasonal timescale, and in our current project we focus specifically on the United Kingdom floods of Autumn 2000 which occurred during the wettest autumn ever recorded, causing widespread damage and an estimated insured loss of £ 1.3 billion.

Half of the climate model simulations we run are of the Autumn 2000 period, specifically including within them the effects of human-induced climate change caused by the emission of greenhouse gases. The other half will simulate a representation of the the Autumn 2000 climate had there not been any human-induced emissions of greenhouse gases over the last century. By then comparing the results of these two simulated climates, and recording the occurrence of floods like that of Autumn 2000 in each of them, we can determine how the frequency of occurrence (or ‘risk’) of such a flood has changed, and therefore how much risk is attributable to human-induced emissions of greenhouse gases over the last century.

So far, so good. This sort of modeling has been attempted before. Here’s the fun part: because so much computing power is available, a statistical study becomes possible. Rather than looking for floods in a handful of simulated case studies, the CPDN researchers are running 10,000 simulations each of the climate with and without post-industrial greenhouse gas emissions. Each simulation is started with slightly different initial conditions, which are then amplified by the classic butterfly effect into completely different simulated histories. Not every simulation will result in floods in Britain in the Autumn of 2000; in fact, because the flood was a once-in-a-century event in any case, most of them won’t. But by counting the number of floods with and without humanity’s greenhouse gas emissions, we can determine how much our industrial activity has amplified the risk of large flood.

This is akin to an epidemiologist determining how much smoking increases the risk of lung cancer, and the massive number of Climateprediction.net simulations are like to having not just ten but 10,000 case histories. It’s a breakthrough, and it’s only one of several different Climateprediction.net experiments, which have now cumulatively run almost 400,000 different climate simulations covering 41,000,000 years of hypothetical history. It made me very happy to learn about this project; it gives me hope that climate scientists have unlimited access to one of the largest supercomputers that has ever existed, given to them freely by the citizens of (it says on the map here) 132 different countries. You too can help.