Facebook just announced that they will remove humans from the production process for their “trending” news section (the list of stories on the right of the page, not the main news feed.) Previously, they’ve had to defend themselves from accusations of liberal bias in how these stories were selected. Removing humans is designed, in part, to address these concerns.
The reaction among technologically literate press and scholars (e.g. here, here, and here) has been skeptical. They point out that algorithms are not unbiased; they are created by humans and operate on human data.
I have to disagree with my colleagues here. I think this change does, or could, remove an important type of bias: a preference along the US liberal-conservative axis. Further, relying on algorithmic processes rather than human processes leads to a sort of procedural fairness. You know that every story is going to be considered for inclusion in the “trending” box in exactly the same way. (Actually, I don’t believe that Facebook’s trending topics were ever politically biased — the evidence was always thin — but this is as much about appearance and legitimacy as any actual wrongdoing.)
And yet, removing humans really can remove an important potential source of bias. The key is recognizing what type of bias Facebook’s critics are concerned about.
There are many ways to design a “trending topics” algorithm. You can just report which stories are most popular. But this might hide important news behind a wall of Kim Kardashian, so most trending algorithms also include a “velocity” component that responds to how fast a story is growing (e.g. Twitter.) Facebook’s trending topics are also location-specific and personalized. None of this is “objective.” These are choices about what it is important to see, just as an editor makes choices. And perhaps Facebook is making choices that make them the most money, rather than the supposedly neutral and public-service oriented choices of an editor, and that’s a type of bias too. It’s also true that algorithmic systems can be gamed by groups of users working together (which is either a feature or a bug, depending on what you feel deserves coverage.) Users can even work together to suppress topics entirely.
But none of this is left-right political bias, and that’s the kind of bias that everyone has been talking about. I can’t see anything in the design of these types of trend-spotting algorithms that would make them more favorable to one political orientation or another.
This doesn’t mean the results of the algorithm — the trending news stories themselves— are going to be politically neutral. The data that the algorithms operate on might be biased, and probably will be. Facebook monitors the articles that are being shared on their platform, and there is no guarantee that a) news sources produce and promote content in some “neutral” way and b) the users that share them are unbiased. If it turns out that more Facebook users are liberal, or liberal Facebook users are more active, then liberal-friendly articles will be more popular by definition.
However, this is a bias of the users, not Facebook itself. Every social software platform operates under a set of rules that are effectively a constitution. They define what can be said and how power is distributed. And some platform constitutions are more democratic than others: the administrators have power or the users have power in varying degrees over various things. Facebook has previously made other changes to reduce human judgment; this can be seen as a continual process of devolving control to the users, although it’s probably more to do with reducing costs through automation.
By removing humans entirely from the trending topics, Facebook is saying that the trending algorithm itself — which is very likely neutral with regard to the liberal/conservative axis — is the governing law of the system. The algorithm may not be “objective” in any deep way, but it is democratic a certain sense. We can expect the trending stories to mirror the political preferences of the users, rather than the political preferences of Facebook employees. This is exactly what both Facebook and its critics want.
Personally, I think that humans plus machines are terrific way to decide what is newsworthy. The former trends curators did important editorial work highlighting stories even when they weren’t popular: “People stopped caring about Syria … if it wasn’t trending on Facebook, it would make Facebook look bad.” This is exactly what a human editor should be doing. But Facebook just doesn’t want to be in this business.
It’s not easy to make social change with technology. There’s excitement around bringing “innovation” to social problems, which usually means bringing in ideas from the technology industry. But societies are more than software, and social enterprise doesn’t have the same economics as startups.
I knew all this going into my summer fellowship at Blue Ridge Labs, but my experience has given me a clearer idea of why. These are the themes that kept coming up for me after two months working with 16 other fellows on the problem of access to justice (A2J) for low-income New Yorkers.
You have to engage the incumbents
The culture of tech startups is not well adapted to taking on big systems. Startups have traditionally tried to enter the wide open spaces created by the new possibilities of technology, or use technical advantage to bypass incumbents. They generally try avoid engaging with major institutions, yet institutional reform is a key part of the “structural change” that so many of us want.
Uber does an end-run around the taxi system, but you can’t simply do an end run around the court system, the state Bar, or the local police.
Instead, tech startups who want to address social issues will need to work within very complex legacy systems. The first task is learning what’s already there. An issue like housing or immigration has a complex arrangement of parts around it: institutions, funding, practices, laws, incentives, and above all the people who work within the system.
Our first month was dedicated to a series of week-long “deep dives” into different areas. I think everyone agreed that there was no way that you could get deeply into a major social issue in a week. Longtime civic hacker Joshua Tauberer says that “until you’ve worked 5–10 years in government or advocacy, you can’t see what needs change.”
Technologists understand that mastering programming takes 10 years, so they should imagine that grappling with social issues also takes years, not months. I’ve worked on technology-enabled social efforts before (mostly around investigative journalism) but I’ve never worked on access to justice, which makes me a complete beginner in the space. After two months of hard work, I can make a very rough sketch of the ecosystem, and I might be able to list the major issues. I can barely see the outlines of what it is that I don’t know.
I don’t find any of this discouraging. If these problems were easy, they would have been solved already. There are people who have been working on them their whole lives. While fresh minds always have fresh insights, there’s also the real possibility that my best idea is ridiculously off the mark.
This doesn’t mean that you or I shouldn’t attempt a startup that aims to change a complex system. It just means we need someone on the team who really, really understands how to work within that system as it stands today, whether that’s a founder or merely a devoted advisor. This is where Blue Ridge Labs shines as an incubator: by virtue of being embedded in the Robin Hood Foundation, and because the fellowship included subject matter experts, we had phenomenal access to the players in this space. You say you want to talk with the woman who runs the A2J program at the New York State court system? How about Tuesday?
The complexity and inertia of the systems we are trying to change is a huge challenge, but it can also be an advantage. Startups traditionally run screaming from heavily regulated areas with entrenched incumbents, but during my research I ran into one founder who asks, “Why start a company in a regulated industry?” For him, “the answer is three-fold: 1) solving real problems, 2) solving hard problems, and 3) unlocking huge opportunities. A heavily regulated market is a clear signal for all three.”
If you are able to tame the systemic complexity in a given area, you will find yourself standing on the good side of a huge barrier to entry. “No one else wants to touch this” can be a very real competitive advantage.
If you want to work on poverty, at some point you have to have a conversation with someone who is poor.
Really, you need lots of deep conversations, and I had perhaps a dozen during my time at Blue Ridge. One of the big successes of the fellowship program is the Design Insight Group, essentially a database of people who have the types of problems we’re trying to solve. We met people in many different contexts, such as interviews, focus groups, and site visits. It was an absolutely essential part of the work, as user contact always is. Even so, it was sometimes uncomfortable for me. What do I say to a mother who has just told me about getting getting thrown out onto the street with her 4 year old son because she couldn’t afford rent? That sort of thing will probably never happen to me or my friends – which is precisely the point of talking to her.
These experiences made me realize how little my life crosses class boundaries. I have close friends of every race and gender identity, and from many different countries too, but I don’t really have low income friends. Fundamentally, I don’t understand poverty because I have very little occasion to talk to poor people.
And I suspect I’m not the only one with this blind spot. For whatever reason, the progressive politics of the moment center on discrimination around race and gender. Those are worthy problems, but ending discrimination will not end poverty (just ask poor white men.) Just as knowing a gay person makes straight people much more likely to support gay marriage, I fear that the problems of low income people will not get the attention they deserve until those who speak loudest spend more time with those who make less. Class segregation seems every bit as pernicious to me as racial segregation, and it’s getting stronger as inequality rises. This doesn’t even require any personal prejudice; the housing market efficiently sorts people of different incomes into different neighborhoods.
Blue Ridge Labs mediated my contact with people outside of my class, and meeting them was the highlight of the experience for me. Context matters hugely for honest conversations: I can’t simply ask someone about their credit card debt at a party. I can, and did, ask them during a private and anonymous interview, in a situation where they are paid for their time.
Which doesn’t mean I always knew how to ask. Different groups came up with wonderful tools for learning from people the people they talked to. Some teams asked people to use cards with titles like “received document” and “court appearance” to create the story of their legal journey. Another team intentionally spoke Spanish to a bewildered tester, so they could try out a translation product idea. I love these different interaction strategies, and we need more.
Even so, there were questions I didn’t get answered. Where are the boundaries of what it’s reasonable to ask? One team I was on was not comfortable asking what price someone would have paid to solve their legal issue, but another team had no problem asking people to price their potential product. While we should always to strive to make people comfortable when they’re talking to us, I don’t think we can or should protect people from all difficult conversations, if we believe that the conversation might lead to crucial insights to help others. This is a subtle issue of respect and ethics, and I could have used more guidance.
What’s a social enterprise anyway?
Zappos is famous for their approach to social responsibility, but any shoe company makes something that improves lives of millions of people. In that sense almost any successful company might be a social enterprise, which seems to make the term meaningless.
So here’s my definition: a social enterprise devotes itself to a mission, even if that mission isn’t the most profitable. “Mission over margin,” as one startup in incubation at Blue Ridge Labs put it.
Not every socially transformative idea is going to be wildly profitable. There’s every reason to believe that many worthwhile social enterprises aren’t going to be profitable at all, at least not through typical market strategies. If your customers can’t cover your costs in the long run, you will need funding elsewhere. The options boil down to various kinds of internal subsidy (e.g. Google’s 20% time), a complementary product (e.g. journalism and advertising), and philanthropy in one form or another.
This raises the whole for-profit vs. non-profit issue. My sense is that this distinction is widely misunderstood. Contrary to wide misconception, non-profits can charge money for services. Nor is there a definitive moral difference; in my work as a journalist I have seen plenty of scammy non-profits, and a solid number of commendable capitalists too. As one editor put it to me, “non-profit is just a tax status.” However, our user interviews revealed that “non-profit” can be hugely important for communication: it signals that the organization is mission-driven, and – rightly or wrongly – people generally trust non-profits more.
The more fundamental point is your sustainability plan, and the mix of market and subsidized revenue you plan to tap. You also need to decide how to measure the success of your mission. This is where metrics come in.
Impact metrics are not universally loved. We spoke to one city-funded credit counselor who asked, “why does my client’s credit score need to improve by 35 points before I can count them as someone I helped? Doesn’t a 34 point increase also move the needle?“ It’s an important question, but I don’t see this sort of arbitrariness as a problem with the idea of metrics in general.
You get to choose what you count as impact. Or perhaps your funders choose – whether your funders are social impact investors or straight philanthropists — but I would hope that funders will take you seriously when you tell them why you should count one thing and not another. But even the wisest metrics will not capture everything you care about. I prefer to think about evaluation rather than metrics. Ultimately, any social enterprise has to ask itself is this working? Counting something is a great way to compare alternatives, but only if you’re counting something that’s worth basing decisions on.
In short, non-profits and for-profits are both compromised, but in different ways: a non-profit might depend on arbitrary metrics, but a for-profit faces continual pressure to turn toward whatever grows the business fastest. The useful distinction is not the legal status or even where the money comes from, but what your definition of success is and how that influences your choices.
Impact metrics can also tell you where opportunity lies. Perhaps a social entrepreneur should be thinking about the number of people they might be able to help, and what that help is worth to those people. The Effective Altruism movement suggests that philanthropy should focus on doing the most good for a given amount of money. It’s an appealing moral idea, and focusses attention on the key concept of efficiency. Unfortunately this principle gets alarmingly complicated in practice – what is “good” and can it really be measured? Still, there is something attractive about sizing opportunities numerically.
For example, one team of Fellows investigated tools to help tenants organize to fight landlords who intend to illegally evict everyone so they can raise the rent. It’s a neat idea, but I found myself wondering how often this problem occurs. Suppose New York landlords wrongfully evict the tenants of 100 buildings every year, meaning perhaps a few thousand people would benefit from organizing. Is this a large number of people? Or is it a small number in a city of 8 million? It’s less than 1/10 of one percent. One of the best responses to this question is to ask what other problems you could you address with your time and funding, and how many people might be helped in each case.
Even if you do succeed as a social entrepreneur, you might not know it. It can take a long time for impact to become clear. In the fall of 2014, I scraped nearly two million Missouri court records for ProPublica, to help answer the question: who filed the most wage garnishment cases? The answer turned out to be a non-profit hospital. Reporter Paul Kiel visited the hospital and the patients in Missouri and wrote the story. Time passed and I moved on to another job. This was just one story of many. Then there was congressional inquiry, and nearly two years after my work on this project, the hospital stopped suing so many people.
But not every win has a straight line between the work and the outcome, and there usually isn’t a follow up story reporting it. The experience makes me wonder how much good I may have done that I will never know about. Something between lots and none at all – and maybe I’ve even harmed some people along the way. I’ve written about these difficulties before.
Imagine if a CEO only got intermittent, unreliable glimpses into revenue. That’s often the situation the mission-driven entrepreneur is in when they try to evaluate the success of their work. And yet, glimpses are better than no information at all – there’s no excuse for not trying to know our impact.
I’ve moved on from low-income access to justice work at Blue Ridge Labs, but I have high hopes for my fellow Fellows who are starting three exciting new projects. I believe tech has a very important role to play in addressing social problems. Obviously I do, or I wouldn’t be in the business of making software for investigative journalism.
But it seems we’re still thinking about the possible strategies in very limited terms. We’re imagining something that looks like a traditional VC-funded tech startup, or perhaps something that looks like a community-supported open-source tool. The reality of successful projects is going to be a lot more complicated. Critical institutional systems are resistant to startups, the economics of social change may look nothing like the economics of venture capital, and the people who can build technology might not even know any of the people who are supposed to benefit from it.
Update: A more recent version of this material appears in my book, The Curious Journalist’s Guide To Data.
I’m not convinced that journalists are always aware when they should be thinking about numbers. Usually, by training and habit, they are thinking about words. But there are deep relationships between words and numbers in our everyday language, if you stop to think about them.
A quantity is an amount, something that can be compared, measured or counted — in short, a number. It’s an ancient idea, so ancient that it is deeply embedded in every human language. Words like “less” and “every” are obviously quantitative, but so are more complex concepts like “trend” and “significant.” Quantitative thinking starts with recognizing when someone is talking about quantities.
In a country where a sitting congressman told a crowd that evolution and the Big Bang are “lies straight from the pit of hell,” where the chairman of a Senate environmental panel brought a snowball into the chamber as evidence that climate change is a hoax, where almost one in three citizens can’t name the vice president, it is beyond dispute that critical thinking has been abandoned as a cultural value.
This is pure cultural critique, and it can be interpreted many different ways. To start with, I don’t know of standard and precise meanings for “critical thinking” and “cultural value.” We could also read this paragraph as a rant, an exaggeration for effect, or an account of the author’s personal experience. Maybe it’s art. But journalism is traditionally understood as “non-fiction,” and there is an empirical and quantitative claim at the heart of this language.
“Critical thinking has been abandoned as a cultural value” is an empirical statement because it speaks about something that is happening in the world with observable consequences. It is, in principle, a statement that can be tested against history. This gives us a basis for saying whether it’s true or false.
It’s quantitative because the word “abandoned” speaks about comparing amounts at two different times: something that we never had cannot be abandoned. At each point in time we need to decide whether or not “critical thinking” is a “cultural value.” This is in principle a yes or no question. A more realistic answer might involve shades of gray based on the number of people and institutions who are embodying the value of critical thinking, or perhaps how many acts of critical thinking are occurring. Of course “critical thinking” is not an easy thing to pin down, but if we choose any definition at all we are literally deciding which things “count” as critical thinking.
One way or another, testing this claim demands that we count something at two different points in time, and look for a big drop in the number. Compare this with the evidence provided:
a sitting congressman told a crowd that evolution and the Big Bang are “lies straight from the pit of hell”
the chairman of a Senate environmental panel brought a snowball into the chamber as evidence that climate change is a hoax
almost one in three citizens can’t name the vice president
The first two pieces of evidence seem to me more anti-science than anti-critical thinking, but let’s suppose our definitions allow it. The real problem is that these are anecdotes – which is just a judgmental word for “examples.” Anecdotes make poor evidence when it’s just as easy to come up with examples on the other side. Yeah, someone brought a snowball into Congress to argue against climate change, but also the EPA decided to start regulating carbon dioxide as a pollutant. The issue is one of generalization: we can’t draw conclusions about the state of an entire culture from just a few specific examples. Generalization is tricky at the best of times, but it’s much easier when you can count or measure the entirety of something. Instead we have only scattered facts, and no information about whether these cases are representative of the whole.
Or, as in historian G. Kitson Clark’s famous advice about generalization:
Do not guess; try to count. And if you cannot count, admit that you are guessing.
The fact that “one in three citizens can’t name the vice president” is closer to the sort of evidence we need. Let’s leave aside, for a moment, whether being able to name the vice president is really a good indication that “critical thinking” is a “cultural value.” This statement is still stronger than the first two examples because it generalizes in a way that individual examples cannot: it makes a claim about all U.S. citizens. It doesn’t matter how many people I can name who know who the vice president is, because we know (by counting) that there are 100 million who cannot. But this still only addresses one point in time. Were things better before? Was there any point in history where more than two thirds of the population could name the vice-president? We don’t know.
In short, the evidence in this paragraph is fundamentally not the right type. The word “abandoned” has embedded quantitative concepts that are not being properly handled. We need something tested or measured or counted across the entire culture at two different points in time, and we don’t have that.
Very many words have quantitative aspects. Words like “all” “every” “none” and “some” are so explicitly quantitative that they’re called “quantifiers” in mathematics. Comparisons like “more” and “fewer” are explicitly about counting, but much richer words like “better” and “worse” also require counting or measuring at least two things. There are words that compare different points in time, like “trend” “progress” and “abandoned.” There are words that imply magnitudes such as “few” “gargantuan” and “scant.” A series of Greek philosophers, long before Christ, showed that the logic of “if” “then” “and” “or” and “not” could be captured symbolically. To be sure, all of these words have meanings and resonances far beyond the mathematical. But they lose their central meaning if the quantitative core is ignored.
The relation between words and numbers is of fundamental importance in journalism. It tells you when you need to get quantitative. It’s essential for planning data journalism work and for communicating the results. It’s the heart of the data journalist’s job, really. The first step is to become aware of when quantitative concepts are being used in everyday language.
The traditional goal of news is to say what just happened. That’s sort of what “news” means. But there are many more types of nonfiction information services, and many possibilities that few have yet explored.
I want to take two steps back from journalism, to see where it fits in the broader information landscape and try to imagine new things. First is the shift from content to product. A news source is more than the stories it produces; it’s also the process of deciding what to cover, the delivery system, and the user experience. Second, we need to include algorithms. Every time programmers write code to handle information, they are making editorial choices.
Imagine all the wildly different services you could deliver with a building full of writers and developers. It’s a category I’ve started calling editorial products.
In this frame, journalism is just one part of a broader information ecosystem that includes everything from wire services to Wikipedia to search engines. All of these products serve needs for factual information, and they all use some combination of professionals, participants, and software to produce and deliver it to users — the reporter plus the crowd and the algorithm. Here are six editorial products that journalists and others already produce, and six more that they could.
Some editorial products we already have
Record what just happened. This is the classic role of journalism. This is what the city reporter rushes out to cover, what the wire service specializes in, the role that a journalist plays in every breaking story. It’s the fundamental factual basis on which everything else depends. And my sense is we usually have enough of this. I know that people will disagree, saying there is much that is important that is not covered, but I want to distinguish between reporting a story and drawing attention to it. The next time you feel a story is being ignored, try doing a search in Google News. Almost always I find that some mainstream organization has covered it, even if it was never front-page. This is basic and valuable.
Locate pre-existing information. This is a traditional role of researchers and librarians, and now search engines. Even when the product is powered entirely by software, this is most definitely an editorial role, because the creation of an information retrieval algorithm requires careful judgement about what a “good” result is. All search engines are editorial products, as Google’s Matt Cutts has said: “In some sense when people come to Google, that’s exactly what they’re asking for — our editorial judgment. They’re expressed via algorithms.”
Filter the information tsunami. This is the act which produces your trusted information feed, whether that’s Facebook’s News Feed or Politico’s morning emails or Google News. It’s here that we can most productively complain that something “wasn’t covered.” Filtering depends upon aggregation and curation, because no one organization can produce original reporting on everything. Most filtering products also lean heavily on software, because human effort can’t match the scope of a web crawler, nor can a human editor prepare personalized headlines for millions of users. As with search engines, information filtering algorithms are both mathematical and editorial objects, and the best products use clever combinations of machines and people.
Give me background on this topic. This is also about locating pre-existing information, but in a summary or tutorial form. Because there are more complex issues than anyone can follow, most news is going to be about things that you don’t know much about. This has been called the context problem for news, and there have been many experiments in solving it. There are now entire sites devoted to explanatory journalism, such as Vox, but the 800-pound gorilla of getting up to speed is Wikipedia. So far, no other product can match Wikipedia’s scope, cost of production, or authority.
Expose wrongdoing. This is the classic role of investigative journalism, which fits within a whole ecosystem of accountability. Every government transparency initiative and every open data nonprofit aspires to support this goal, but transparency is not enough. Democracy needs people who are committed to exposing corruption, crime, and abuse. Sometimes this requires inside sources and secret documents, but accountability can also be about drawing attention to little-noted facts. But it is always about scandal, what has been called “the journalism of outrage.” This makes it powerless in the face of huge systemic issues without a clear locus of wrongdoing. Investigative journalism is vital, but only one part of the broad intersection between information and power.
Debunk rumors and lies. In this fairly new category, we have products like Politifact, which checks what politicians say, Emergent.info, which tracks the spread of rumors, and the venerable Snopes. It’s a little strange to me that the news media of old weren’t much into debunking, but I guess they thought “publish only true things” was sufficient. Clearly, truth-testing has since become a valuable public service, and journalists have learned to pay more attention.
Some editorial products that don’t exist yet
What can I do about it? More and more, this is the only beat I care to cover. Accurate news is essential to know the world, but reports of what just happened do not tell you what can be done about it, at a personal level. I don’t believe that citizens have become apathetic; I believe we are overwhelmed in the face of large and complex problems where it is hard to know where to start. We already know that stories that include solutions are more engaging. The main problem is one of plausible effectiveness: If you have ten dollars or ten hours to donate, where should you put your resources? Not every problem can be helped by large numbers of small actions — but some can. You could build a whole product around the question of what the reader could do.
A moderated place for difficult discussions. Traditionally, journalism has tried to present an objective truth that would be seen as legitimate by everyone. I’m not convinced that truth always works this way, and I’m sure that no institution today has this sort of argument-settling authority. But I do see a need for unifying narratives. Americans are more polarized than they’ve been in decades, and we fight online about everything from catcalls to tax rates. Perhaps there is a need for a safe place to talk, to know the other, with real human moderators gently tending the discussion and discouraging the trolls. When everyone can talk, the public sphere needs fewer authorities and more moderators. To me, seems a natural role for journalism.
Personalized news that isn’t sort of terrible. It seems obvious that different people need different news (if I do say so myself) and this requires algorithmic recommendation to scale, but the results have often been unimpressive — as anyone who has complained about the Facebook News Feed knows. I’ve spent a lot of time with recommendation algorithms and I’ve come to believe that this is fundamentally a user interface design challenge: How do you tell the computer what you want to see? Optimizing for clicks and likes inevitably degenerates into clickbait and likebait. Other systems require you to choose subjects in advance or people to follow, but none of these is really satisfying, and I still don’t have a “mute” button to tune out Kim Kardashian. I’m holding my breath for an interaction design breakthrough, some elegant way to create the perfect personal channel.
The online town hall. Democracy is supposed to be participatory; voting is not enough, but there is no scalable communication channel between citizens and government. So how does your voice get heard? And how do you hear the voices of other people — and how does a civil servant make sense of any of this deluge? There’s a hard problem here: We don’t have good models for a “conversation” that might include millions of people. I’m imagining something like a cross between Reddit and civic-listening platform PopVox. This too would require thoughtful moderation.
Systematic government coverage. Journalism has long looked for waste and corruption. But how many stories do you read about the Bureau of Land Management? Or the Office of Thrift Supervision, which should have been monitoring the financial industry before the crash? Sometimes it seems like journalists pull their subjects out of a hat. If we’re serious about the notion of an independent check on government, we need to get systematic about it. No one reports department by department, bureau by bureau, with robot reporters scrutinizing every single open data feed. Sound boring? It might be. But maybe that just means current accountability journalism is badly skewed by the demands of entertainment.
Choose-your-own-adventure reporting. Story creation could be interactive. There have been crowdfunding platforms such as Spot.us and Beacon, but nothing that operates on quite the level of granularity and speed envisioned by Jay Rosen’s explainthis.org, where users type in questions for journalists to answer. There are thousands of variations on the idea of having the users direct the reporting, everything from demand-driven production to a quiz after each story that says, “what should we report on next?” The point is to put journalists and users in an interactive loop. Good reporters listen anyway, but I want something stronger, a sort of contract with the audience where they know exactly how to be heard. For example: “Our reporter will investigate the top-voted question each week.”
What’s editorial, anyway?
I’ve used the word “editorial” to sidestep discussion of what “news” or “journalism” is. To ask that question misses the point of what it does. And there has been a strange lack of innovation here. Silicon Valley has never been afraid of wild ideas, but the tech world is allergic to any service which requires a lot of humans to deliver. That doesn’t scale, or so the thinking goes. Meanwhile, the journalism world has evolved and finally embraced software and new story forms. Yet the espoused goals of journalism — the fundamental services that journalists provide — seem virtually unchanged. That’s a pity, because there are so many different, useful things you can do by applying humans plus machines to nonfiction information production. We’ve barely scratched the surface.
Overview is an open-source document analysis and visualization system originally developed at the Associated Press for investigative journalists. It’s been used to report some of the biggest investigative stories of the last few years. We’re looking for a developer to extend the software to analyze millions of scraped syllabi for the Open Syllabus Project.
You will help us put 2 million scraped syllabi online, do natural language processing to extract citations from each syllabus, and build visualizations to do citation analysis. We want to see what people are actually teaching for each subject, and how this changes over time, and make this type of analysis widely available to researchers. We’re looking for someone to build out Overview to support this, growing our team from three to four people. This is an ideal job for a programmer with visualization, natural language processing, digital humanities or data journalism experience.
This is a six-month contract position to begin with. We hope to extend that, and we’d be especially excited to find someone who wants to grow into a larger role within our small team. We’re a distributed team based out of NYC, remote friendly, flexible hours.
What’s the best simple action you can take to address a particular social problem?
I wish there was somewhere that reviewed attempts to solve social problems, everything from activist campaigns to government programs. You’d go to this site, look up “homelessness” or “education” or “Asian tsunami” or “criminal justice reform” and get a recommendation for the most effective thing you could do right now, and if possible a button to do it or at least sign up to do it. The actions would be intentionally lightweight, like donating $10 or ten minutes of your time or pledging to vote a certain way. Think of a sort of Consumer Reports for social campaigns.
I’ve been calling this hypothetical civic information/action organization “How Can I Help?” because that’s the question it seeks to answer.
This is an ambitious idea, but there are working models to draw from. GiveWell is an organization whose sole purpose is to rate charities. They answer the question, “where should I give my small donation to do the maximum amount of good?” Of course this depends on what “good” means, but GiveWell has been careful in choosing and communicating their values. In fact the organization is a model of thoughtful and transparent research, with one of my very favorite blogs and a top-level “mistakes” section.
I want to expand on this idea. GiveWell makes only two or three recommendations, typically in the area of global health. But there’s also a good argument for taking care of the people and places where we live. What we should care about is a complicated question, and has no objective answer. But if you’re willing to say that something specific should change, then you can start to ask seriously how to get there.
How Can I Help wouldn’t need to dream up new solutions. Lots of organizations are already working on all sorts of problems, bringing forth a flood of reports and campaigns and proposals. But you and I don’t necessarily know about all of the proposals and attempts and actors around any particular issue. The difficult problem from the user’s point of view is not finding information about possible solutions, but selecting between them in an intelligent way. This follows a now-standard online media pattern: the aggregator. It’s the same logic that drives Yelp, Google News, and Rotten Tomatoes.
So: Imagine a site that continually reviewed the huge range of proposals from the huge range of actors who might seek to address any particular social ill, and maintained a few top recommendations for each cause.
Lightweight civic action
I think it’s important that the recommended actions be very lightweight. I’m inspired in this by two recent posts on civic engagement. Anthea Watson Strong focuses on the personal costs and benefits of any kind of political action. She argues that if we want a lot of people to be involved, we either have to make it very cheap and easy for them to engage or else give them some expectation of a big payoff. Ethan Zuckerman divides civic action along two axes: thin to thick, and symbolic to impactful. In that language, what I’m proposing is in the thin and impactful quadrant.
I’m focussing on “thin” or low cost actions because I feel the opportunities for “thick” engagement are well covered. The whole Internet is just dying to give you information on whatever concerns you, and significant causes often have dozens of organizations who would love you to get involved. And I’ve focussed on “impactful” because, again, the Internet is already really good at symbolic or expressive campaigns like turning your avatar green or sharing a video. Expressing oneself is important, and has consequences — it’s how we learn that we are not alone, and it can focus public attention or create new language (Occupy Wall Street “did” nothing, but now we all know what “the 1%” means.) But if you’re after more concrete results, it’s not enough that your time and money go to a good cause; we need resources flowing to efficient and effective causes. This is especially important if you believe that most efforts to help are ineffective, as I do.
Figuring out where an individual could actually do some good with a small donation of time or money is not easy. It requires asking questions about the effectiveness of many different possible solutions, of course, but you also have to ask who is already working in that space, what they’re doing, and what the bottlenecks are. Raising money won’t help if lack of money isn’t the thing holding back progress.
So the question that this hypothetical organization must answer is “What can anyone do right now that is both easy and plausibly effective?” This is not an easy thing to answer, nor is there a guaranteed solution: for any particular issue there may be no clear way for an individual to contribute without getting deeply involved, and that’s a valuable answer too. And merely knowing the answer isn’t enough: the organization must communicate it, show all of the analysis that led to it, and continually update that analysis as learning happens and as new opportunities arise. Producing good recommendations for civic action is a major ongoing research and communication task.
Don’t be authoritarian
Offering easy, vetted ways to contribute to social change is an ambitious goal. Maybe even an audacious goal. For a start, it’s not at all obvious what a “social problem” is. And why should anyone believe a self-appointed authority? These issues make me think that How Can I Help needs to be more plural and participatory than the typical media organization.
First of all, social problems don’t define themselves. There isn’t some obvious master list of what’s wrong with a community or a country, and not everyone will agree on whether any particular thing is a problem. In many cases, deciding on the problem is the problem.
Consider someone who comes to How Can I Help and searches for “taxes are too high.” Should they learn the name of the advocacy organization that most effectively lobbies for lower taxes? Or is the problem really something else? Perhaps this person is having money problems, so better wages or cheaper health insurance would be just as good to them. Maybe this person feels that the government is wasting public money, so what they really want is accountability and efficiency. Or maybe they have specific ideas about the scope of government, and the things that should and should not be collectivized. Similarly, do we understand the huge increase in America’s prison population to be about poverty, racist policing practices, or farcical drug laws?
It may not be possible to frame a problem definitively, but it should be possible to figure out what assumptions the answer depends on. In trying to solve any complex problem you’ll need answers to prior questions, many of which cannot be definitely decided. Some prior questions will be empirical questions where the necessary information is just not available, while others will be questions of values where there is no general agreement. Instead of arbitrarily picking one scenario, it should be possible to document how different answers to these unknowns lead to different courses of action.
But who should define these alternate framings, and who gets to influence the final recommendations? You could do How Can I Help as a traditional media organization, by hiring a small number of smart people to research and write. But involving users in the process has multiple benefits.
First, you might get better answers. Diversity is helpful for finding good solutions and even more important for defining problems. There is emerging evidence that crowds beat experts for political prediction problems — and recommending an effective course of action is most definitely a prediction problem. You’re predicting that the recommended action will have a better effect than all available alternatives.
Participation is also important for scaling. If this model of reviewing and recommending actions works, people might want not want to wait for paid staff to get around to their issue. Maybe the process that generates this type of knowledge could be made self-serve, something that a group could apply. I’ve written before on the need to leverage participation in journalism if we want a lot of journalism to be done, which means it’s not only an egalitarian concern but a business model issue. Unfortunately, participatory journalism has proved difficult to systematize.
But the best reason for participation is that it’s not enough to produce good answers. People have to trust that they are good. How Can I Help must generate not only knowledge but legitimacy, and there is no better legitimacy than the sense of ownership. There are places on the internet where the user community feels like it’s theirs, where the administrators tread lightly and take fairness seriously. We all know when someone is playing dictator; it’s important to get the platform’s embedded constitutional principles right.
There is a balance to be found here. GiveWell is a mostly closed institution and doesn’t have to suffer trolls, but it is vulnerable to the charge that it’s just a bunch of technocrats telling us what’s best. Reddit is a mostly open model with millions of users who love and defend the space they have created, but it doesn’t generally produce sophisticated research. There is a huge unexplored space of design variations between open and closed.
Connecting information to action
Like many people, I got into journalism because I wanted to make a difference in the world. It’s clearly important to have accurate, timely reports of what’s happening, but modern hopes for journalism go well beyond mere information. When we talk about trying to measure the effects of journalism or concepts like solution journalism, we’re talking about making change in society. But journalism isn’t really set up to do this. There is a huge taboo against becoming too specific or effective, lest a news organization be seen as “activist” or “taking sides.” (I’m not necessarily saying this is wrong, just that it structurally precludes certain things.)
Other endeavors are unabashedly all about change: activist and advocacy organizations, charities and philanthropies, the NGO and development worlds — not to mention government social programs. Many of these organizations also produce information. But they go beyond mere information because they try to get people to do some specific thing, whether that’s “sign a petition” or “donate money” or “show up and help us dig trenches.” Action is powerful. But these these sorts of organizations might not be the most trustworthy sources of information, because they’ve already committed to a solution. There’s a basic conflict of interest here. You know they’re trying to sell you something.
So journalists are charged with producing accurate information yet they are reluctant to recommend specific actions. Meanwhile, activists and advocates are all about action but the information they produce may not be trustworthy. There is a huge disconnect here! How do we connect trustworthy information to informed action?
The core concept of How Can I Help — an independent review of everyone who claims to be working on a problem and the effectiveness of their proposed solutions — is one possible response to this dilemma. I don’t know if it counts as journalism or advocacy or what, but I don’t particularly care (see also Adrian Holovaty’s classic answer to “is data journalism?”)
Now that I’ve started thinking of the interplay between information and action, many other problems with journalism seem clearer. I’ve always been frustrated at the way investigative journalists choose which stories to cover. News organizations tend to put resources into issues that are both little known and highly offensive. A journalist who goes deeply into climate change is not producing “news” because everybody already knows about the problem, while a great story that moves no one gets no attention, has no effect, and silently disappears. This model is broken. The dual frame of “our job is to bring new information to light” plus “it is someone else’s responsibility to decide what to do” prevents journalism from directly addressing the most obvious, pressing, ongoing issues.
Without information we cannot know what to do. Without action we do nothing. I am interested in connecting the two, and I think it may take new kinds of organizations to do that.
The organizers of Build Peace tell me it was the first conference specifically on peace and technology, and they should know. I don’t know the peace building field very well, but I could see that some of its leading lights were in attendance. I learned quite a bit, and I am very glad I went.
I have to start by saying I don’t think “technology for peace” is a sure win. My understanding is that peace building is incredibly difficult work, and rarely truly successful, and I don’t see why technology necessarily changes that. Yet I am also a technologist and I presented some of my own data-driven peace work at the conference. Clearly I believe it might be good for something.
There is a great need for conversations between capable conflict resolution workers and thoughtful technologists — hence this conference. Here are some of the things I think I learned.
Try existing social networking platforms first
In the 5-minute long ignite talks I watched speaker after speaker present their work on “online discussion platforms,” “spaces for dialog,” and “peaceful online interaction.” Increasingly, I was bothered by a simple question: what do existing social media platforms lack for peace-building uses?
On the assumption that cross-cultural dialogue is key to peace (more on that below) the Internet seems to hold infinite potential, if we can just get people talking to each other the right way. This simple logic drives the explosion of online experiments. Which is great. But I rarely heard anyone talking about what makes one platform better than another — and if we don’t know what a peaceful platform should look like, why not just use Facebook?
This is what The Peace Factory does. The concept began with a Facebook page called Israel Loves Iran and quickly spawned other “X loves Y” pages which have reached millions of people. It progressed to the Friend Me 4 PEACE program which encourages people to friend someone from “the other side.”
People ask me, “Why would I ‘friend’ someone from Ramallah? What would I say?” Nothing. But you’ll see their stupid selfies in your feed.
Will selfies bring peace? I don’t know. They do seem humanizing, which is probably important. Also there is a natural escalation channel on Facebook, towards greater interaction and engagement. But what I really like about this work is that the experiment is cheap and easy to replicate.
It has become a staple of the crisis mapping community that crowdsourced crisis response must rely on already-deployed technology, not on crisis-specific apps. No one is going to install your app when the network is down and they can’t find their family. Similarly, do you really want to be in the position of convincing people involved in a civil war that they should switch social networks? My sense is, let’s find out where Facebook etc. fall short as a peace platform, before we go attempting to build an alternative — and get masses of people to use it, which is even harder than building it!
Do No Harm
One of the most significant things I learned about is the existence of a Do No Harm movement within peace and conflict work. This seems like a basic principle when working in a dangerous area, but its articulation is surprisingly young. I was referred by multiple people to the 1999 book by Mary B. Anderson. The book has spawned a sub-field both academic and practical.
Huge ethical issues recording what people say in conflict area. Might get source killed & speaker has no control over data use. #buildpeace
I haven’t read the book, so I can’t claim to understand the details. But the powerful idea that well-intentioned peace builders might make matters worse will stay with me.
Online interaction done right
Waidehi Gilbert-Gokhale of Soliya gave one of the most impressive presentations at the conference. Like a lot of other projects, Soliya aims to build peace through online discussion. Unlike a lot of other projects, Soliya can articulate why conversation alone is not enough. In Gilbert-Gokhale’s words: “unmoderated chat polarizes.” Here she is referencing a wide body of work that shows that bringing people with conflicting opinions together to talk can actually reinforce pre-existing divisive beliefs, not moderate them.
Soliya sees their online cross-cultural interactions as a new form of “exchange” program and even calls their new platform Exchange 2.0. Online interactions typically take place in a school setting, which gives teachers the chance to moderate and guide the discussion.
Most interestingly, Soliya seems serious about knowing whether any of this works — aka evaluation. Gilbert-Gokhale said,
The biggest thing we have to do is run control groups. Without that we have no validity to our findings.
And I love her for saying that. To me, this emphasis on evaluation seems way ahead of everyone else doing dialog programs — even though Soliya’s evaluations to date don’t seem to include a control group. Soliya also has produced is a lengthy 2009 report “covering the past 60 years of research into the impact of media on attitudes and behavior.” Certainly worth checking out!
That report also includes some very interesting neuro-imaging studies of conflict by Emile Bruneua of MIT, who also spoke at the conference. Brunuea has shown that our brains react differently when considering the suffering of members of an in-group versus an out-group. This is remarkable; however I have not included the pretty brain scan images because I know that brain scan images are very persuasive, whereas this work is very young. It’s a promising line of research, but it has not been reproduced by other researchers and it’s not clear how you might use it in the field. Evaluation of peace work is never simple.
Measurement and evaluation are key
I suspect that most peace building efforts don’t end up helping very much, and all the experienced peace workers I’ve spoken to agree. If this seems harsh, consider that there are good reasons to believe that much international aid is ineffective, and quite plausibly that a wide range of non-profit work in general is ineffective. Preventing or resolving violent conflict is probably even harder than those things.
There seems to be very little solid evidence that conflict resolution work does any good at all — certainly not anything up to the standards of a controlled study, because you can’t really do a controlled study in conflict areas. You go in and try to stop the violence because not attempting to stop it would be unethical (assuming, of course, you Do No Harm.) Then the violence diminishes or it does not. But there is no counterfactual to compare against. That is, we don’t know what would have happened had we done nothing.
It was my favorite session, and well-attended too, though much of the younger, hipper set seemed to be elsewhere. That saddens me. If we can’t figure out what works and what doesn’t we have nothing at all. If we can figure out how to do good evaluation then we can learn.
I came away with several big ideas from the evaluation working group.
First, control groups might be nice but qualitative explanations count! Say you held a bunch of mediation sessions between community leaders in different communities. Then the conflict seemed to settle down. You theorize that it was your work organizing these meetings that caused things to get better. Are you right?
Sometimes people who practice ethnography and other qualitative research methods get into arguments with data people about what can be learned from only one specific case, only one historical experience. I experienced this at the conference in the conversations around metrics and data. Personally I believe that the well developed theory of causation says you can’t know the magnitude of a causal effect unless you have lots of cases, divided between control and non-control groups. But obviously being in the time and place of a conflict and trying to shape it can teach you something deep about what happened. The question is, what?
I learned during this session that there’s a whole body of knowledge about this kind of single-case causation analysis under the name of process tracing. For example, you need to test your proposed explanations against historical facts, and certain types of tests provide more evidence than others, and in fact there’s a whole theory of case study selection and inference. See also analysis of competing hypotheses, developed in intelligence work, which I now see as closely related. Process tracing won’t get you results equivalent to a large number of controlled cases, but you can get immensely valuable knowledge anyway, and it will be even better than statistical analysis in certain ways.
But suppose you really need the kind of evidence that only a controlled study can provide, such as estimates of the magnitude of your effect (also estimates of the uncertainty in your estimates, which can be just as important.) I learned that there are several different controlled designs that might work in peace building. Instead of comparing against doing nothing, you could compare against doing something else. You can do the same thing in different places (say, different villages) at different times, and look for a time correlation, which is called a stepped wedge design. Or you do the program only in places where some metric of need is above a certain threshold, which is called a regression discontinuity design.
Effectiveness is important for many reasons, one of which is that there are many more things that could be done than there is funding to do them. So someone has to make hard choices, and effectiveness has to be a key factor.
But the biggest idea I took from the measurement workshop is that even this seemingly airtight logic is suspect, because peace programs so very often end up doing something completely different from what they set out to do! The reality of conflict assures that. It does no good to charge in with an evaluation strategy that measures the wrong thing… and maybe you can only know if you’re measuring the right thing after you’ve started the project.
"There's design, monitoring, and evaluation. These three things could all be combined together." #buildpeace
In other words, learning on the ground might (and probably should) convince you that your goal should change. I was delighted to discover that this idea of questioning your goals as you move toward them has a name: double-loop learning.
Open and closed, the crowd and the authorities
There was a fundamental tension at the conference between open and closed approaches to peace. If you like, there were two narratives about how projects were constructed. Some speakers presented about explicitly open projects (“Peace is everyone’s business: Mass SMS to prevent violence.”) Other projects involved a small group of outsiders working with existing authorities (“Elections data for the people in transitioning MENA countries.”) And more than a few projects have chosen to keep their data completely private, to prevent their human sources from coming to harm.
Is the future of technology-enabled peace building open or closed? I think both. There is great potential for open, flattened, peer-to-peer projects because ultimately it is people who must be at peace, not their governments. But not all processes can include all people, for some very good reasons. Even a “consensus” processes almost always has to exclude someone, either for logistical reasons or to deal with spoilers. Quinn Norton’s scathing dissection of Occupy Wall Street’s General Assembly is a beautiful example of the failure of an open system.
Because the [General Assembly] had no way to reject force, over time it fell to force. Proposals won by intimidation; bullies carried the day. What began as a way to let people reform and remake themselves had no mechanism for dealing with them when they didn’t. It had no way to deal with parasites and predators.
Of course I’m not arguing that we are currently at ideal levels of openness, either for peace building projects or anything else. Just that the ideal is some careful hybrid.
I have done a little bit of work on data-driven ways to understand conflict, which stems from my interest in visualizing communities. It’s possible to see the political divisions of the U.S. population in many different types of data: political book sales, who talks to who on Twitter, geographical voting patterns, and more. My own contribution to this is an interactive visualization of the gun control debate on Twitter from one week in February 2013, published in The Atlantic. In that visualization, which shows “people who Tweeted this link also tweeted that link,” you can clearly see that there are two poles of thought on the matter, led by (for this particular week of Twitter conversation) the White House on one side and The Blaze on the other.
Here are the slides I showed at the beginning of the session, which became a very lively group discussion (yay!)
It’s striking to me that conflict dynamics show up so clearly in big data visualization… but I’m really not sure how helpful that is, if your concern is peace. Yes, plots like these could help in conflict analysis, but anyone who’s actually paying attention to a conflict already knows who the sides are. A more interesting possibility is a time-based analysis where you animate these association patterns through time, to see if anything is changing. This type of network analysis could also be used just as marketers use it, to identify influential people and groups for the purposes of media planning.
Various people including myself suggested that maybe peace builders should look at these networks to find people who bridge between the sides. But Ethan Zuckerman made a very interesting counter-suggestion: maybe we need to look outside of the conflict divisions entirely, to find completely unrelated identities that many people can agree on. He pointed to the Harry Potter Alliance, which was founded to address the conflict in Sudan.
Location data might help us to find bridging spaces, literally spaces in the physical world. Mobile phone companies have location history for each subscriber, so it should be possible to figure out a) which “side” each person is on by where they travel and who they associate with and b) where people from the two sides come together. What if we discover that otherwise mortal enemies drink in the same bar every Friday night? Is that useful? Does it violate privacy in that creepy location-data way if we just know the name of the bar, not the names of the people who go there? I have no answers.
I’ve been hanging around journalism innovation and crisis mapping and ICT4D and related things for six years now, which is (surprisingly?) long enough to see several generations of projects come and go. I feel like peace technology is currently making some of the classic mistakes: people are making without considering what already exists, technologists with no knowledge of peace building are going to suck at understanding user needs, and there’s not really any talk of tech project sustainability.
But I am also elated. This conference was a unique confluence of enthusiasm, expertise, and experiments. It has made me optimistic that if there there is a role for technology and technologists in peace building, we will find it. It will probably take a few more years for all of it to settle down into useful practices. I certainly came away with some things to try — and I’d go on that data scientist conflict zone exchange program in a heartbeat. Or at least back to Build Peace next year.
Several of my friends — actual, real life good friends — shared this story on Facebook in a, let’s say, somewhat triumphant mood. And I wasn’t sure what to think. This is what I wrote.
I’m having trouble understanding what all this signifies. Here’s what I come up with that I am sure about:
I’m having trouble understanding what all this signifies. Here’s what I come up with that I am sure about:
my friends do not like cops
clearly there are other people who do not like cops
people who do not like cops are either more common on Twitter or more vocal than those who like them
the NYPD sure have beaten up a lot of people
But, these are the questions I remain unable to answer:
I think we probably want a police force that engages with people on social media. How should they have engaged?
Were any of these beatings “proportionate?” This is horrible language, I know, but give it a pass for a moment.
Is any beating ever proportionate? How could we even know the answer to this in principle, let alone in specific cases?
What is the overall record of the NYPD? Is this a question that even has meaning given the multidimensional nature of the problem? Can the answer be anything other than “terrible” if there are incidents like these?
What would I do if I was king of the NYPD?
Will my friends perceive this post as “defending the cops”? Will there be social sanctions of some sort for expressing these ideas? Is my echo chamber just as pernicious as the echo chambers of those that belong to my perceived “other”?
We might be able to do better at conflict resolution — making peace in violent conflicts — with the help of good data analysis. There have long been data sets about war and violent conflict at the state level, but we now have much more.
There are now extraordinarily detailed, open-source event data streams that can be used for violence prediction. Conflict “microdata” from social media and communications records can be used to visualize the divisions in society. I also suggest a long term program of conflict data collection to learn, over many cases, what works in conflict resolution and what doesn’t.
We’re really just at the beginning of all of this. There are huge issues around data collection, interpretation, privacy, security, and politics. But the potential is too great to ignore.
Update: two excellent resources have come to my attention in the days since I gave this talk (which is, of course, part of why I give talks.)
First, see the International Peace Institute’s paper on Big Data for Conflict Prevention. This paper was co-authored by Patrick Meier, who has been deeply involved in the crisis mapping work I mentioned in my talk.
But even more awesome, Erica Chenoweth has done exactly the sort of data-driven case-control study I was contemplating in my talk, and shown that non-violent political resistance succeeds twice as often as armed resistance. Her data set, the Nonviolent and Violent Campaigns and Outcomes (NAVCO) Data Project, also shows that non-violence is much more likely to lead to good democracies five years later, and that a movement that can recruit 10% of the population is almost guaranteed to succeed.