How our emphasis on measurement shapes civil society and weakens social trust
A Q&A with Stanford sociologist, Aaron Horvath
This week, we’re running our first Q&A feature with a Connective Tissue subscriber, Aaron Horvath. Our friend Aaron is a sociologist and Research Scholar at the Stanford Center on Philanthropy and Civil Society, where he is developing an initiative called the Project on Private Wealth and the Public Good. Aaron’s research examines how vague ideals like democracy and accountability become incorporated into organizational practices, and how these practices shape our ability to collectively imagine and pursue social progress. Beyond his research, he’s also a coffee roaster, woodworker, and former bike mechanic and punk rocker (like any good Oakland, CA resident).
My conversations with Aaron have helped put words to some of the tensions I experienced while running the Armed Services Arts Partnership and conducting academic research in recent years. Why must seemingly everything be quantified and measured? What is the relationship between this emphasis on measurement and social trust? And how does the act of measurement, itself, shape the nature of community groups and civil society?
We dig into these questions, in all of their messiness, in this week’s Q&A. I hope you enjoy this conversation with Aaron as much as I did. And if you’re interested in learning more about his work, you can email him at ahorvath@stanford.edu and follow him on Twitter (@horvathianisms).
- Sam
How would you describe your area of research in layman’s terms?
Broadly speaking, my research explores overlaps between philanthropy, civil society, and democracy. I have a particular interest in how practices from the world of for-profit business find their way into and potentially transform the public sphere. So I examine seemingly innocuous administrative practices and managerial fads—often things that are billed as scientific or rational—and how these things can have unexpected and deleterious effects on how we imagine or participate in democracy. Currently, I'm doing a lot of thinking around the idea of “impact,” which seems to mean everything and nothing at the same time. I'm curious how this ended up taking over our collective conceptions of civic life and participation in the public sphere.
This idea of “impact” relates to one of my favorite pieces you’ve written, Counting Alone. In it, you connect this concept of impact to what you describe as philanthropy’s “obsession with metrics.” How did this obsession come about?
I will start with a sociological account here. There's research that points to the social conditions under which things like audits, evaluations, and other formalized approaches to checking and verifying end up taking off. These practices are rationalized and quantified stand-ins for trust that might otherwise develop face-to-face. As social relationships grow more diffuse and we do things more from a distance, the less we trust each other and the more we end up looking to intermediaries (e.g., experts, studies, etc.) to convince us of someone else's or some organizations’ integrity.
These exact circumstances developed in the nonprofit sector around the turn of the 21st century. You can point to the declining rates of civic participation, as described in books like Bowling Alone or Diminished Democracy. You can point to the rise of the internet and the increased possibility and ease of doing things from a distance rather than face-to-face. There's also new entrants at that time into philanthropy who were bringing ideas from tech and finance for how to run organizations.
Then the coup de grâce of all this were the scandals in the late 90s and early 2000s, where directors of well-respected charities were convicted of fraud and embezzlement. This led to legislative hearings that attempted to redeem the nonprofit sector from what was perceived as widespread abuse of people's trust. And this is all occurring around the time of Sarbanes-Oxley, so there's already this broader trend of us moving toward demanding more audit and accountability, and the nonprofit sector in many ways follows along with that. This contributed to the emergence of all these charity rating agencies, like Charity Navigator, in the early 2000s. There’s an explosion of people talking about the best way to evaluate the impact of individual nonprofits and the entire nonprofit sector.
So to put it very reductively, but probably crisply, the decline of civil society fed this more technocratic approach to trying to grasp civil society. We started trusting each other less and trusting the numbers more.
What have been the implications of this emphasis on measurement? How has it affected nonprofits and the populations they serve? How has it changed the nature of civil society and civic life?
That's the trillion dollar question. A lot of proponents of this evaluative activity would like to think that evaluating is merely capturing the action on the field. The reality is that evaluation, itself, is actually on the field and shaping the state of play.
How does this affect organizations being evaluated? At the simplest level, demands for reporting and measurement take up time and resources for organizations that are often understaffed, overworked, and don't have the time and resources to be doing these things. I've talked to plenty of people who’ve shared how this distracts from accomplishing the things they’re trying to accomplish. I often think of the phrase, “if a tree falls in the forest, and there's no one there to hear it, did it make a sound?” I think in a similar way it’s like, “if a nonprofit had an impact, and there was no one there to measure it, did it make a sound?” You ask the nonprofit director, they'll probably say, “well, hell yes we did.” But if you ask the people funding them and demanding the evaluations, they'd be like, “nobody measured it.”
Evaluation also has the effect of forcing organizations to play by a new set of rules. Evaluation is a form of regulation. For example, by holding organizations to the overhead metric, it has forced organizations to change how they allocate funds. It forces them to not be able to invest in their staff or invest in developing their programs. Simply holding organizations to various criteria forces them to play along with what those criteria are demanding of them, and that has pernicious organizational effects.
Taking that further, when you subject organizations to a narrower set of evaluative criteria, all the stuff that's not on the list gets deemphasized. It's similar to the problem of teaching to the test in school. If it's not on the test, you just don't teach it. And what are the things that are hard to measure? Civic engagement, community development, relationships. Because those things are pretty hard to measure, they often get deemphasized. These evaluative demands treat civic organizations as vendors of social services or as production functions, where you're trying to maximize the amount of throughput or impact they have on some stated mission.
But rarely do these civic organizations do only what they say on the label—they often do all sorts of other stuff. I wonder, “what does it mean when we've come to know community organizations through quantitative abstractions?” It suggests to me that we don't know the face-to-face aspect anymore. There's something about local community engagement that really matters for the health of a democracy, and something is really getting lost there. This leads me to the question that I'm trying to figure out now: “When we institutionalize all these abstract and quantified forms of knowing and engaging in civil society, does that cause our civic or democratic muscles to atrophy?” The follow up question is: “How? And how do we get back from that?”
Are any of these findings translatable to the role of government? Does the government's approach to funding and evaluation contribute to similar consequences?
I tend to think of government contracts or grants to nonprofits as often emphasizing service provision. In the American context, we often have government provide funding to nonprofits that carry out the services of government. In this case, measurement is an effort to engage in democratic accountability. If we're using tax dollars directly to provide for something, we the taxpayer would expect our government agencies to show us what they've done, and this expectation gets passed on to nonprofits.
At the same time, very few nonprofits are merely service providers. If you take the example of a food bank, officially all they're doing is handing out food. But unofficially, they're also a point of contact for all sorts of service agencies. They are providing masks and educational information during COVID. They're doing all this stuff that isn't stated in the mission. And so I wonder how much flexibility you can actually build into these systems that try to create democratic accountability. Can you build a system to allow for community-serving flexibility and capture the civicness of an organization?
Last week, I heard someone say, “Government keeps asking why we don’t trust them. Maybe they should start asking themselves why they don’t trust us.” This immediately reminded me of your work. How would you describe the relationship between technocracy, democracy, and social trust?
The impetus for a lot of my research is that I'm deeply suspicious of technocracy, particularly the desire for self-appointed experts to dictate the character of what society should look like. I think that's bound to backfire, and I think it already is backfiring in a lot of ways.
Regardless of where you are on the right or the left, there is a deep distrust in government and institutions—which is the most throwaway line anyone ever said—but it's true. And I wonder: how much of that is the mass of the American public feeling that the government is tone deaf to their concerns and their needs? And having someone condescend to them or thumb their nose at them and say “we know best” is not going to work well. To me, that leads to fundamental questions of how we make government more participative. We need to figure out the role we expect of experts in our democracy, because right now, there's just no trust in expertise.
How do you think about the role of scale in all of this?
We have this pressing question: “how do you develop trust at scale?” More metrics are not going to do that. I think the answer is more local stuff. If you participate in a local organization and you participate in local government, you might develop a sense of political efficacy—a sense that you can understand your community and government and influence their direction. This stands in contrast to the way society feels right now, which is like me as an individual and then the national government, and we don’t feel embedded in any institutions in between. You can show me as many numbers you want about efficacy, I'm not going to feel any sense of trust. But if I have a deeper sense of how things work—because I'm engaged in the local organization or local community—that ends up allowing for the scale problem of trust to be altered.
There’s also a question of professionalization lingering in this discussion about expertise, participation, and social trust.
The more progressive, left-leaning approach to nonprofits has long been: “we believe in professionalism, so as long as there are professionals on the job, we're happy with how it works.” I often wonder if this undercuts the efficacy of a lot of these organizations. Sure, I'm happy that there are legal advocates working on climate change. But how much more progress would we make on climate change if people were actually interested in participating in a climate change organization? I don't feel like I have any real personal stake in the issue of climate change other than being a human who is flammable.
There is this paradoxical effect of professionalization, where it seems like the direct route to accomplishing a given goal is to have the person who knows how to do that specific thing on the job. But since so many of the social ills that plague our society are collective action problems, you actually need to enlist the help of people who are not experts and not just do things on their behalf. You need everyone to be part of the solution.
Where do we go from here?
Sometimes, I'll give talks on questions of evaluation and measurement. And almost invariably someone in the audience will be like, “you hate metrics.” And, I’ll be like, “actually, I don't.” Then the follow up question will be like, “you want us to go back to what you think are the good old days,” and they’ll name some period in American history that was incredibly racist and unequal. And I'll be like, “that’s not a question. And, actually, no, I don't.”
There's no doubt that quantification and metrics often codify biases and prejudices and hide them behind the patina of objectivity. But the so-called old-fashioned way, where everything happens in-person and through interpersonal relationships is rife with problems, too. If you were to take this hyper-relational approach to, say, philanthropic grantmaking, you'd end up giving to the people you know. And because we live in a world that is socially and geographically segregated by class and race, that'd probably end up looking like white men who went to business school giving to other white men who went to business school. So the non-quantified, interpersonal approach has all sorts of problems, too.
Where do we go from here? We need to think critically about how we use different mechanisms of trust. The quantification promise is that it can discipline our cognitive biases and our prejudices, and democratize who has access to things. What I'm ultimately trying to advocate for is an intelligent use of different approaches to building, maintaining, and communicating trust—whether quantified, relational, or whatever. We need to recognize that different approaches have unique costs and downsides. An emphasis on measurable impact might make sense in some circumstances. But it can have destructive effects in others. Prioritizing more hands-on, engaged forms of participation and accountability are essential for a vibrant civic life—but there are plenty of organizations for which that would be a burden.
There's definitely no one-size-fits-all solution when it comes to civil society. So where we go from here is not a novel solution, per se. It's a critical awareness of the problems inherent in different approaches to building trust.
A lot of what we’re talking about is rebalancing. It’s a recognition that things have gone far in one direction, and asking, “how do we hold these a bit more in tension with one another?” In my case, I’m not an uber communitarian, I just think we need to hold liberty and obligation in more tension than we have been. Because things have gotten out of whack and now we’re in this state of hyperindividualism.
I totally agree. I often feel like an uber communitarian just because I'm trying to pull against this thing that goes the other way. And, it’s just the blind spot in a lot of the shibboleths that have guided civil society over the last 30 years. We forgot about the important community and democracy stuff because we got so obsessed with using nonprofits to get stuff done. There needs to be some corrective effort, where we actually find ways to fund and develop the meaningful civic infrastructure that upholds democracy. And the stuff that we've been doing for the last 30 years just isn’t that.