It’s not easy to make social change with technology. There’s excitement around bringing “innovation” to social problems, which usually means bringing in ideas from the technology industry. But societies are more than software, and social enterprise doesn’t have the same economics as startups.

I knew all this going into my summer fellowship at Blue Ridge Labs, but my experience has given me a clearer idea of why. These are the themes that kept coming up for me after two months working with 16 other fellows on the problem of access to justice (A2J) for low-income New Yorkers.

You have to engage the incumbents

The culture of tech startups is not well adapted to taking on big systems. Startups have traditionally tried to enter the wide open spaces created by the new possibilities of technology, or use technical advantage to bypass incumbents. They generally try avoid engaging with major institutions, yet institutional reform is a key part of the “structural change” that so many of us want.

Uber does an end-run around the taxi system, but you can’t simply do an end run around the court system, the state Bar, or the local police.

Instead, tech startups who want to address social issues will need to work within very complex legacy systems. The first task is learning what’s already there. An issue like housing or immigration has a complex arrangement of parts around it: institutions, funding, practices, laws, incentives, and above all the people who work within the system.

Our first month was dedicated to a series of week-long “deep dives” into different areas. I think everyone agreed that there was no way that you could get deeply into a major social issue in a week. Longtime civic hacker Joshua Tauberer says that “until you’ve worked 5–10 years in government or advocacy, you can’t see what needs change.”

Technologists understand that mastering programming takes 10 years, so they should imagine that grappling with social issues also takes years, not months. I’ve worked on technology-enabled social efforts before (mostly around investigative journalism) but I’ve never worked on access to justice, which makes me a complete beginner in the space. After two months of hard work, I can make a very rough sketch of the ecosystem, and I might be able to list the major issues. I can barely see the outlines of what it is that I don’t know.

I don’t find any of this discouraging. If these problems were easy, they would have been solved already. There are people who have been working on them their whole lives. While fresh minds always have fresh insights, there’s also the real possibility that my best idea is ridiculously off the mark.

This doesn’t mean that you or I shouldn’t attempt a startup that aims to change a complex system. It just means we need someone on the team who really, really understands how to work within that system as it stands today, whether that’s a founder or merely a devoted advisor. This is where Blue Ridge Labs shines as an incubator: by virtue of being embedded in the Robin Hood Foundation, and because the fellowship included subject matter experts, we had phenomenal access to the players in this space. You say you want to talk with the woman who runs the A2J program at the New York State court system? How about Tuesday?

The complexity and inertia of the systems we are trying to change is a huge challenge, but it can also be an advantage. Startups traditionally run screaming from heavily regulated areas with entrenched incumbents, but during my research I ran into one founder who asks, “Why start a company in a regulated industry?” For him, “the answer is three-fold: 1) solving real problems, 2) solving hard problems, and 3) unlocking huge opportunities. A heavily regulated market is a clear signal for all three.”

If you are able to tame the systemic complexity in a given area, you will find yourself standing on the good side of a huge barrier to entry. “No one else wants to touch this” can be a very real competitive advantage.

Have you met the people you’re trying to help?

User-centered design. Build with, not for. Community engagement. Don’t solve other people’s problems. There are many ways to talk about the idea of collaborating with the people you are trying to help, but they all boil down to contact.

If you want to work on poverty, at some point you have to have a conversation with someone who is poor.

Really, you need lots of deep conversations, and I had perhaps a dozen during my time at Blue Ridge. One of the big successes of the fellowship program is the Design Insight Group, essentially a database of people who have the types of problems we’re trying to solve. We met people in many different contexts, such as interviews, focus groups, and site visits. It was an absolutely essential part of the work, as user contact always is. Even so, it was sometimes uncomfortable for me. What do I say to a mother who has just told me about getting getting thrown out onto the street with her 4 year old son because she couldn’t afford rent? That sort of thing will probably never happen to me or my friends – which is precisely the point of talking to her.

These experiences made me realize how little my life crosses class boundaries. I have close friends of every race and gender identity, and from many different countries too, but I don’t really have low income friends. Fundamentally, I don’t understand poverty because I have very little occasion to talk to poor people.

And I suspect I’m not the only one with this blind spot. For whatever reason, the progressive politics of the moment center on discrimination around race and gender. Those are worthy problems, but ending discrimination will not end poverty (just ask poor white men.) Just as knowing a gay person makes straight people much more likely to support gay marriage, I fear that the problems of low income people will not get the attention they deserve until those who speak loudest spend more time with those who make less. Class segregation seems every bit as pernicious to me as racial segregation, and it’s getting stronger as inequality rises. This doesn’t even require any personal prejudice; the housing market efficiently sorts people of different incomes into different neighborhoods.

Blue Ridge Labs mediated my contact with people outside of my class, and meeting them was the highlight of the experience for me. Context matters hugely for honest conversations: I can’t simply ask someone about their credit card debt at a party. I can, and did, ask them during a private and anonymous interview, in a situation where they are paid for their time.

Which doesn’t mean I always knew how to ask. Different groups came up with wonderful tools for learning from people the people they talked to. Some teams asked people to use cards with titles like “received document” and “court appearance” to create the story of their legal journey. Another team intentionally spoke Spanish to a bewildered tester, so they could try out a translation product idea. I love these different interaction strategies, and we need more.

Even so, there were questions I didn’t get answered. Where are the boundaries of what it’s reasonable to ask? One team I was on was not comfortable asking what price someone would have paid to solve their legal issue, but another team had no problem asking people to price their potential product. While we should always to strive to make people comfortable when they’re talking to us, I don’t think we can or should protect people from all difficult conversations, if we believe that the conversation might lead to crucial insights to help others. This is a subtle issue of respect and ethics, and I could have used more guidance.

What’s a social enterprise anyway?

Zappos is famous for their approach to social responsibility, but any shoe company makes something that improves lives of millions of people. In that sense almost any successful company might be a social enterprise, which seems to make the term meaningless.

So here’s my definition: a social enterprise devotes itself to a mission, even if that mission isn’t the most profitable. “Mission over margin,” as one startup in incubation at Blue Ridge Labs put it.

Not every socially transformative idea is going to be wildly profitable. There’s every reason to believe that many worthwhile social enterprises aren’t going to be profitable at all, at least not through typical market strategies. If your customers can’t cover your costs in the long run, you will need funding elsewhere. The options boil down to various kinds of internal subsidy (e.g. Google’s 20% time), a complementary product (e.g. journalism and advertising), and philanthropy in one form or another.

This raises the whole for-profit vs. non-profit issue. My sense is that this distinction is widely misunderstood. Contrary to wide misconception, non-profits can charge money for services. Nor is there a definitive moral difference; in my work as a journalist I have seen plenty of scammy non-profits, and a solid number of commendable capitalists too. As one editor put it to me, “non-profit is just a tax status.” However, our user interviews revealed that “non-profit” can be hugely important for communication: it signals that the organization is mission-driven, and – rightly or wrongly – people generally trust non-profits more.

The more fundamental point is your sustainability plan, and the mix of market and subsidized revenue you plan to tap. You also need to decide how to measure the success of your mission. This is where metrics come in.

Impact metrics are not universally loved. We spoke to one city-funded credit counselor who asked, “why does my client’s credit score need to improve by 35 points before I can count them as someone I helped? Doesn’t a 34 point increase also move the needle?“ It’s an important question, but I don’t see this sort of arbitrariness as a problem with the idea of metrics in general.

You get to choose what you count as impact. Or perhaps your funders choose – whether your funders are social impact investors or straight philanthropists — but I would hope that funders will take you seriously when you tell them why you should count one thing and not another. But even the wisest metrics will not capture everything you care about. I prefer to think about evaluation rather than metrics. Ultimately, any social enterprise has to ask itself is this working? Counting something is a great way to compare alternatives, but only if you’re counting something that’s worth basing decisions on.

In short, non-profits and for-profits are both compromised, but in different ways: a non-profit might depend on arbitrary metrics, but a for-profit faces continual pressure to turn toward whatever grows the business fastest. The useful distinction is not the legal status or even where the money comes from, but what your definition of success is and how that influences your choices.

Impact metrics can also tell you where opportunity lies. Perhaps a social entrepreneur should be thinking about the number of people they might be able to help, and what that help is worth to those people. The Effective Altruism movement suggests that philanthropy should focus on doing the most good for a given amount of money. It’s an appealing moral idea, and focusses attention on the key concept of efficiency. Unfortunately this principle gets alarmingly complicated in practice – what is “good” and can it really be measured? Still, there is something attractive about sizing opportunities numerically.

For example, one team of Fellows investigated tools to help tenants organize to fight landlords who intend to illegally evict everyone so they can raise the rent. It’s a neat idea, but I found myself wondering how often this problem occurs. Suppose New York landlords wrongfully evict the tenants of 100 buildings every year, meaning perhaps a few thousand people would benefit from organizing. Is this a large number of people? Or is it a small number in a city of 8 million? It’s less than 1/10 of one percent. One of the best responses to this question is to ask what other problems you could you address with your time and funding, and how many people might be helped in each case.

Even if you do succeed as a social entrepreneur, you might not know it. It can take a long time for impact to become clear. In the fall of 2014, I scraped nearly two million Missouri court records for ProPublica, to help answer the question: who filed the most wage garnishment cases? The answer turned out to be a non-profit hospital. Reporter Paul Kiel visited the hospital and the patients in Missouri and wrote the story. Time passed and I moved on to another job. This was just one story of many. Then there was congressional inquiry, and nearly two years after my work on this project, the hospital stopped suing so many people.

But not every win has a straight line between the work and the outcome, and there usually isn’t a follow up story reporting it. The experience makes me wonder how much good I may have done that I will never know about. Something between lots and none at all – and maybe I’ve even harmed some people along the way. I’ve written about these difficulties before.

Imagine if a CEO only got intermittent, unreliable glimpses into revenue. That’s often the situation the mission-driven entrepreneur is in when they try to evaluate the success of their work. And yet, glimpses are better than no information at all – there’s no excuse for not trying to know our impact.

Stubborn optimism

I’ve moved on from low-income access to justice work at Blue Ridge Labs, but I have high hopes for my fellow Fellows who are starting three exciting new projects. I believe tech has a very important role to play in addressing social problems. Obviously I do, or I wouldn’t be in the business of making software for investigative journalism.

But it seems we’re still thinking about the possible strategies in very limited terms. We’re imagining something that looks like a traditional VC-funded tech startup, or perhaps something that looks like a community-supported open-source tool. The reality of successful projects is going to be a lot more complicated. Critical institutional systems are resistant to startups, the economics of social change may look nothing like the economics of venture capital, and the people who can build technology might not even know any of the people who are supposed to benefit from it.