Last month, my colleague Julian Burgess and I took a shot a peering into the Iraq War Logs by visualizing them in bulk, as opposed to using keyword searches in an attempt to figure out which of the 391,832 SIGACT reports we should be reading. Other people have created visualizations of this unique document set, such as plots of the incident locations on a map of Iraq, and graphs of monthly casualties. We wanted to go a step further, by designing a visualization based on the the richest part of each report: the free text summary, where a real human describes what happened, in jargon-inflected English.

Also, we wanted to investigate more general visualization techniques. At the Associated Press we get huge document dumps on a weekly or sometimes daily basis. It’s not unusual to get 10,000 pages from a FOIA request — emails, court records, meeting minutes, and many other types of documents, most of which don’t have latitude and longitude that can be plotted on a map. And all of us are increasingly flooded by large document sets released under government transparency initiatives. Such huge files are far too large to read, so they’re only as useful as our tools to access them. But how do you visualize a random bunch of documents?

We’ve found at least one technique that yields interesting results, a graph visualization where each document is node, and edges between them are weighted using cosine-similarity on TF-IDF vectors. I’ll explain exactly what that is and how to interpret it in a moment. But first, the journalism. We learned some things about the Iraq war. That’s one sense in which our experiment was a success; the other valuable lesson is that there are a boatload of research-grade visual analytics techniques just waiting to be applied to journalism.

click for super hi-res version

Interpreting the Iraq War, December 2006
This is a picture of the 11,616 SIGACT (“significant action”) reports from December 2006, the bloodiest month of the war. Each report is a dot. Each dot is labelled by the three most “characteristic” words in that report. Documents that are “similar” have edges drawn between them. The location of the dot is abstract, and has nothing to do with geography. Instead, dots with edges between them are pulled closer together. This produces a series of clusters, which are labelled by the words that are most “characteristic” of the reports in that cluster. I’ll explain precisely what “similar” and “characteristic” mean later, but that’s the intuition.

We colored each report/dot by the “incident type”, which is an existing field in the SIGACT, entered by military personnel. It’s important to note that the incident type field was not used to place the reports in the diagram — the placement depends only on the text of the document. This plots one one variable (color, which is incident type) against another (position, which depends on the summary text).

And it works. The central cluster is blue, the color for the “criminal event” type, and the documents within it all include the word “corpse.” There are a heartbreaking number of them, because this was the height of the Iraqi civil war. Sub-clusters include various modifiers such as “shot.” (Click any image for hi-res version.)

Above this, the blue murders merge into the green “enemy action” reports. At the interface we have “civ, killed, shot,” which are apparently reports of civilians wounded in battle. Enemy actions also have their own clusters  labelled with “mortar,” “female,” “officer,” and “injured.” We haven’t looked into the “female”/”enemy action” cluster yet, and I wonder if there’s a story there.

There is a red cluster off to the side. Red signifies that the military coded these reports as “explosive hazard,” and the documents here all include the words “tanker truck.” Sure enough, there are contemporaneous press reports of tankers being used as explosive weapons, and this cluster shows that there were at least several dozen such incidents throughout Iraq in Dec 2006 — though it doesn’t immediately distinguish between explosions and attempted or threatened explostions.

There’s another cluster of blue criminal action reports, labelled “blindfolded, feet, hands.” Bound feet and hands were common in sectarian violence at the time, and some reports include the word “torture.” There’s a nearby cluster of abductions.

It goes on. December 2006 was a vicious and disturbing and complicated time in Iraq, and the visualization has patterns at all scales, especially if you look at the hi-res image and read the tiny single-report labels. There are some dark green “friendly action” reports labelled “convoy,” and other “friendly actions” which mention the troublesome town of Hadithah (near bottom left). And there is the oil connection, a group of reports which include the word “pipeline.”

How we did it, and what we can and can’t learn from this picture
Visualization is metaphor. Certain details are thrown away, other are emphasized. The algorithms used to produce the visualization have their own sensitivities and blind spots. Without understanding these, a viewer will make false inferences. I’m going to explain in some detail about how this picture was produced, both so that others can replicate this research, and so that those looking at such visualizations can interpret them honestly.

We used standard text-analytics techniques, borrowed from information retrieval: the bag-of-words model, TF-IDF term weighting, and cosine similarity to compare documents. This is the stuff from which search engines are built, among other things. The geeky among us can learn as much as they could ever want to know from this wonderful free information retrieval textbook.

We start by turning each document into a fixed-length vector of numbers. There are as many numbers in this vector as their are words in the vocabulary of all the documents, over 17,000 distinct terms in the case of the Iraq War Logs. If “pipeline” appears three times in a report, we put a three in the count for “pipeline.” Of course the reports are much shorter than 17,000 words, usually just a couple hundred words, so most of the numbers in each document vector are zero.

We also don’t quite store the count of each word. Instead we store the frequency, that is, we divide the counts by the number of words in the document. If the document is 100 words long then “‘pipeline’ appeared three times” becomes  ”3% of the words in this document are ‘pipeline.’” This is “term frequency,” the TF part of TF-IDF.

Then we normalize again by how commonly the word appears across documents. It’s not enough to know that “pipeline” is common in a document.” We need to know that “pipeline” is unusually common in this document. So we count the fraction of documents where “pipeline” appears, and divide the term frequency by this document frequency. (Technically, by the logarithm). This has the effect of de-emphasizing terms which appear in almost every document, and it’s the “inverse document frequency” or IDF part of TF-IDF.

This is the sense in which that the labels on the documents and the clusters are “characteristic” words: they are words that occur frequently in those specific documents, but don’t appear at all in most other documents.

But by turning each document into a list of numbers, the order of the words is lost. Once we crunch the text in this way, “the insurgents fired on the civilians” and “the civilians fired on the insurgents” are indistinguishable. Both will appear in the same cluster. This is why a vector of TF-IDF numbers is called a “bag of words” model; it’s as if we cut out all the individual words and put them in a bag, losing their relationships before further processing. And so we get to:

• Important caveat #1: any visualization based on a bag-of-words model cannot show distinctions that depend on word order.

Once we have all the documents encoded as TF-IDF vectors, we compare every pair of documents to determine how similar they are. We call two documents similar if their characteristic words overlap, and we determine this by taking the dot product of the two document vectors. Why? The dot product multiplies the corresponding numbers at each position in the two vectors. If two documents both have a big number for “pipeline”, the dot product will be large. If one document has a big number for “pipeline” but zero for “abducted”, while the other has a large number for “abducted” but zero for “pipeline”, then the dot product will be zero. This is called the cosine similarity method of comparing documents, because of geometrical relationships between the cosine function and the dot product. Cosine similarity assigns a number to every pair of documents, from zero for “they are completely different” to one for “they are the same.” (At least, the same as far as the bag of words model is concerned.)

Each document is a dot in the visualization. To this we add edges, and the “weight” or strength of the edge — which shows up as line width in this visualization — is the cosine similarity. But we don’t put edges between every pair of documents, only those that are above some threshold of similarity. For this visualization, that threshold was 0.6.

And then we lay out the graph. We used Gephi, a free graph visualization tool. Generally, graph layout algorithms try to bring nodes with strong edges closer together. We found the Fruchterman-Reingold algorithm gave the clearest layout in this case, but the general idea is that points with strong ties gradually move closer as the algorithm runs. But there are conflicting demands; a node marked “corpse” and “abducted” may be pulled towards both clusters. Where a node ends up also depends a lot on where it started, and the nodes start in essentially random positions.

Cosine similarity-weighted graph layout is not the only way to view the relationships between thousands of documents in a 17,000-dimensional space. There are other techniques such as multi-dimensional scaling. But however the documents are visualized, we are trying to understand the structure of a something very complicated in only two dimensions, like trying to guess an object from its shadow. Depending on which angle you take, the shadow is going to be more or less revealing, and perhaps more or less misleading. This is:

• Important caveat #2: the positions of the dots are sort of arbitrary, though we hope that nearby dots actually represent similar documents.

In other words, quantitative measurements of distances on this visualization won’t mean much. Arguing that “these events are unrelated because they are on opposite sides of the image” is similarly fallacious.

What can we learn from this visualization technique? Clusters are fairly reliable structures. Using color to plot one type of information against another can reveal patterns. And we believe that this visualization captures some important macro-scale aspects of the War Logs. This picture isn’t a story in the usual sense, but we find it insightful nonetheless, and maybe it tells us where to look further. A search tool only can only answer the questions we ask, but a visualization tool lets us make maps.

Much more is possible
To begin with, we’d like to try coloring each dot according to the number of casualties, another field already available in the SIGACTs. We know that over 4000 U.S. forces and 100,000 civilians died in Iraq, but what were the circumstances of their deaths? Perhaps we can start to answer that question. We also want to find a way to animate this diagram through time, so we can see how the war changed as it progressed.

But there are plenty of other visualization techniques waiting to be applied to journalism, and plenty of other document sets to apply them to. It seems likely that TF-IDF and cosine similarity will be generally useful for full-text visualizations of a variety of document types, but it won’t always work. Threaded displays might be much more revealing for things like emails, where it’s important to identify and isolate conversations. In other contexts, entity-relationship diagrams can be insightful; theyrule.net is the granddaddy of this type of analysis, today being seriously pursued by Muckety.

Visualization is also only one part of the problem. This is a static image, but what we really need is an interactive system where a computer draws the pictures and a human directs the exploration. Visualization has to be combined with filtering and selection tools to allow an investigator to “zoom in” on only those documents of interest. Such complete systems exist in other fields, such as the Jigsaw visual analytics software, but there’s currently nothing that really works well for journalism. Performance is a huge issue when dealing with very large document sets, and data import and clean-up are often the real-world bottlenecks. Clean-up is often the most time consuming part of document set analysis, and new tools such as Google Refine give us hope that it can be streamlined.

The potential applications of an industrial-strength journalistic visual analytics system are far broader than document dumps. We got interested in visual analytics because we faced document sets that were so large that they were completely opaque without special tools. But a newsroom also has its archives, and the data and stories it generates every day. We’ve heard interest from historians, and at the other end of the immediacy scale are potential real-time monitoring applications, technologies that are being seriously pursued by organizations such as UN Global Pulse.

We see so much potential that we — the Associated Press in conjunction with several top-notch researchers — are embarking on a serious attempt to build an open-source system for journalistic visualization of very large document sets, be they document dumps, news archives, or the streams of data that now surround civilization. We have preliminary designs for a system called Overview, and we have applied for a Knight News Challenge grant to hire full-time developers to create it. I’ll soon post a more detailed description of the system we’d like to build. We’re going to need help from the journalist-programmer community.

Who is writing the major policies of the wars in Iraq and Afghanistan, and what is the Obama administration likely to do? There have been many analyses and news reports of individual policies and events, but it’s hard to wade into this flood of information, and besides, how would I know who to listen to? In an effort to get some perspective on at least one major aspect of American military strategy, I decided to plot out all the authors of (public) counterinsurgency policy over the last decade, and the relationships between them, as evidenced by co-authorship of articles and papers.

The resulting network shows that the Obama administration is relying heavily on the talents of a group called the Center for A New American Security (CNAS), which has close ties to the authors of the most recent US Army counterinsurgency manual. This means that Obama is unlikely to break with the current military strategies in Iraq and Afghanistan — but even if he wanted to, could he? Counterinsurgency is difficult, and many, many people die when you do it wrong; you can’t simply make this stuff up, so the choices are necessarily among existing clusters of people and policy.

The graph also suggests that the only quasi-independent body of COIN policy is centered around the RAND Corporation, who may not hold a terribly different opinion. If this analysis is correct, then Obama cannot rapidly change the military’s course in fighting these wars, because there simply do not exist credible alternative policies at this time. His only options for change in America’s handling of Iraq and Afghanistan lie outside of the scope of military strategy — perhaps through high level political or economic interventions.

Counterinsurgency Policy

American troops are shooting at someone in Iraq and Afghanistan, but the designated enemy is not another army. After the Taliban was decimated in Afghanistan and Saddam’s main forces were defeated in Iraq, dozens of armed groups stepped up to fill the power vacuum in both countries, ranging from militias to (of course) terrorists. Beset on all sides, the US military lashed out, conducting increasingly intrusive operations in the civilian population, such as house-by-house searches. The bad guys were no longer wearing uniforms, and worse, there was often popular sympathy for them. The US military ended up shooting at the people it had claimed to be liberating.

At the start of these wars, the US military was poorly prepared in counterinsurgency (COIN) tactics, a product of the Cold War strategies and the painful memory of Vietnam, which was also a counterinsurgency war. In fact, the standard COIN manuals of that time (which you can now read courtesy of Wikileaks) were stagnant for 25 years until 2006, when a major review was undertaken by Lieutenant General David Petraeus and others. The resulting revision of the FM 3-24 Counterinsurgency Manual was widely publicized, in contrast to previous secret revisions, with co-author Lieutenant Colonel John Nagl even appearing on The Daily Show to discuss it. Clearly, this revision was as much about building public support and confidence at home as it was about an actual change in strategy. The whole manual has been extensively discussed elsewhere, but the core of the new doctrine is the notion that an insurgency is as much a political as it is a military problem:

The integration of civilian and military efforts is crucial to successful COIN operations. All efforts focus on supporting the local populace and HN [host nation] government. Political, social, and economic programs are usually more valuable than conventional military operations in addressing the root causes of conflict and undermining an insurgency.

This is hardly a new idea, as FM 3-24 freely admits, but — as one interpretation of the new manual and the publicity surrounding it goes — it represents a fundamentally new role for the military, who are now faced with a problem that cannot be solved by force. Depending on who you believe, the decrease in violence in Iraq over the last two years may be due to these radically new policies, the surge, or other factors entirely.

Three years later the US is still in Iraq, and Afghanistan — if anything, an even messier place — is finally starting to return to public consciousness. Obama has to make some decisions about these military strategy in these wars. What will his answers be?

Social network analysis can help us answer this question because ideas always live among a community of minds; those who develop ideas together tend to share them. The clusters in a social network are therefore proxies for distinct worldviews, or possible answers to a question.

COIN Policy Social Network

Without further ado, here is the social graph of those writing public counterinsurgency policy over the last decade (click for a larger image, or pdf).

Each node is a person, and an edge is drawn between any two individuals who collaborated on a document, or worked in the same group — hopefully a reasonable proxy for policy similarity. (Although this is a “social” graph, merely having been in the same place at the same time does not count as an edge.) I have assigned colors to larger clusters: the Center for a New American Security (CNAS) think-tank is red, the authors of the revised FM 3-24 are green, and those from the RAND corporation are Cyan. CNAS and the FM 3-24 authors overlap in the person of John Nagl, in yellow.

Crucially, Obama (in blue) has selected Michèle Flournoy as his top Pentagon appointment, and Flournoy founded CNAS. There are other CNAS links: Colin Kahl was a military advistor to Obama during his campaign, and Lt. Nathaniel Fick spoke at the DNC in August in support of Obama. Fick and Nagl also recently co-authored a major policy paper on counterinsurgency in Afghanistan, which includes  an interview with Petraeus, another FM 3-24 author.

In other words: CNAS has adopted the military’s FM 3-24 strategy, and Obama has adopted CNAS. Therefore we should expect little change in the way that the military component of the American wars are currently prosecuted.

One of the advantages of CNAS is that it reprsents a unified, proflific, and highly visible body of strategic and policy thought. For Obama to choose another course, there has to be another course to choose. There are individual critiques of the Nagl/Fick position such as this by Afganistan social scientist Christian Bleuer, individual detractors such as the experienced but often wonky Edward Luttwak, and even the very sharp and compassionate Samantha Power, who was part of the Obama campaigin until (apparently) she referred to Hillary as “a monster.” But what Obama needs is a credible body of alternative policy.

Counterinsurgency as “Good Governance”

There is only one other major cluster on the graph. The  RAND Corporation (in cyan) is of course old-school defence establishment, but, unlike the FM 3-24 authors, they are not actually military. They have written a number of counterinsurgency documents since the start of the Iraq war, such as a long report by O’Connel and Pirnie which is summarized here. The RAND reports do not differ all that sharply from the Petraeus/Nagl policies, except that they see counterinsurgency as fundamentally more than a military operation:

Strategy should be developed at the highest level of government, by the President, his closest advisors, and his Cabinet offcials, with advice from the Director of  National Intelligence and regional experts, the Chairman of the Joint Chiefs of Stfff, and unified commanders. … Counterinsurgency is a political-military effrt that requires both good governance and military action. It follows that the entire U.S. government should conduct that effort.

This is exactly the sort of “nation building” that Bush had hoped to dispense with. If implemented thoroughly, it might also amount to little more than a classic colonial government. To this I can only say: what did you expect when invading a foreign country?

About Building the Graph

This graph represents four days of very manual web-surfing, and it is very much a work in progress. Starting with Fick and Nagl, each name was googled on the web, in the news, and in scholarly publications,  and the top 20 or so results in each category were read to determine co-authorship of policy papers and organizational affiliations. Two people were connected if they had ever co-authored an article together, or worked in the same group at the same time. Doubtless, there are connections that I have missed, such as some of the other authors on today’s New York Times blogs piece which I will have to add. Also, because this process was so time-consuming, I had to make many choices not to include individuals or follow links. It is therefore entirely possible that the graph I have drawn is actually embedded into a larger network in such a way that it invalidates my conclusions; or that there is a credible cluster of people working on alternative policy that I simply never found (but then again, Obama hasn’t found them either.)

The graph proper was built by collecting a text file of web references, and manually entering people and  connections in a .dot file for use with Graphviz. Again, this took days — I cannot stress how manual the process was. These difficulties highlight the dire necessity of better information visualization tools for journalism.

[Update: this work has come to the attention of the COIN community, in particular the folks at Abu Muqawama who were kind enough to discuss what it might mean. Aside from the fact that the original version contained not one but three misspelled names (calling Mr. Flick!) they have pointed out that many important people and links are missing. No doubt -- as I discuss above and in my response on AbuM. I'd like to stress that this is a work in progres, but I would also like to ask those in the know if they feel my conclusions on this restricted graph are substantially correct even so. -- js]

In his talk, and in his new book What Happened, McClellan describes a shift in Washington culture over the last few decades, where national interest has given way to “a game of who’s up and who’s down,” a culture of “permanent campaign,” where “governing becomes an offshoot of campaigning.” He describes Clinton and Bush Senior as sensitive to and constrained by the same pressure, and says that this culture of “permanent campaign” has gradually been getting worse over the past few decades, to the point where it is now a corrupting factor to all who work in Washington.

I don’t think any White House since Nixon has truly learned the lessons of Watergate. … It has become about which side can best manipulate the narrative to their advantage. … Deliberation and compromise get pushed further and further down on the list. People like Karl Rove didn’t create the permanent campaign. The permanent campaign created people like Karl Rove.

I’m not convinced that the “permanent campaign” is in any way a new feature of politics, given that the people in power seem to have been trying to stay that way throughout history. A 2007 Washington Post article takes the same position with respect to American politics specifically. However, these are surely strong words coming from someone whose job it was to shape our political narratives. Perhaps more illuminating were McClellan’s comments regarding the process of going to war in Iraq. Although it is by now clear that the administration was committed to an invasion long before there was any evidence to support the notion that Iraq had WMDs, McClellan’s description of the process of actually bringing the nation to war is the first direct testimony I’ve ever heard from someone who would know.

Was it deliberate to mislead? I don’t think the president’s top advisers were sitting in a room saying, ‘how do we mislead the American people?’ … But there was an effort to sell the war to the American people. It was sold like our education campaign, like tax cuts, like social security reform. It was ‘how do we make the strongest possible case?’ Forget about ‘how do we communicate the truth?’ … We took the nation to war by making it sound more urgent, more serious than it actually was. … It was all about shading.

He went on to describe President Bush’s role in the process,

The president is an instinctive leader. In some ways that’s admirable, in some ways that causes a lot of problems. … After the decision is made, he expects everyone else to follow in lock-step. … He had this belief, this idealistic and ambitious vision that we could go coercively into Iraw and create democracy and it would be the lynch-pin that would transform the Middle East … If he’d had a crystal ball, I don’t think he would have made that decision.

These are words from a man who publicly defended the administration’s decisions time and time again, a man who was formerly fiercely loyal to the president. McClellan’s own story is equally interesting. He says that people now forget that

Bush was a bipartisan and popular governor in Texas, who worked ‘across-the-aisle.’ This is why I was drawn to him. … [When I arrived in Washington] I got caught up in this destructive culture just like many others did. I am not saying this to remove blame from my doorstep. … Eventually I could no longer stand up and speak for the president because I kept getting undermined. … I trusted the administration. It was not until I began researching the book that I realized how misplaced my trust was. … It was hard to separate personal closeness with the president from his policy.

I find these remarks on power fascinating, and perhaps a valuable lesson in reality all on their own: the Bush administration is not stupid or evil. The problems appear to be ones of ideology, and a pronounced lack of commitment to candor and honesty. He finished by offering some suggestions, and his hopes for the future.

[The next press secretary] needs to have access to any meeting, any time. … I actually welcome a liberal media, if they’re fair, because they will be more skeptical of the government. And if they’re not fair, in this day and age it will be known. …I really want to see Washington change the way it governs forthe better. I want to see loyalty to the ideas of candor, transparency, and openness. Too often people think that loyalty to an individual office-holder overrides loyalty to the people. … We need to move beyond the philosophy that politics should be viewed as war.