Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy, by Cathy O’Neil


Five stars, read in December 2017.

I’ve read a lot of nonfiction this year that I consider to be important, even essential to the goal of social justice. There was Bertrand Russell, Christopher Hitchens, Susan Jacoby, and Sam Harris on the damaging nature of religion and anti-intellectualism; Angela Davis, James Baldwin, Audre Lorde, Toni Morrison, Martin Luther King, Jr., John Lewis, and Ta-Nehisi Coates on systematic white terrorism against black Americans; Roxane Gay, Lindy West, and Jaclyn Friedman on how utterly barbaric society is toward fat people and women’s bodies in general; Mark Schatzker’s The Dorito Effect and Jane Mayer’s Dark Money about the ways that powerful people with exclusively their own wealth in mind are controlling enormous facets of our lives in ways we can’t even see.

So I’ve said this before, and I’m sure I’ll say it again, because I make a point of seeking out books like this. But if there’s one book I could get everyone to read right now, it would be this one. Even next to Ta-Nehisi Coates’s We Were Eight Years in Power, which I’m reading concurrently and which is staggering in its importance, this book feels urgent—because what it explains is how our new economy of Big Data is codifying, concealing, and amplifying inequality of every kind, on an enormous scale. Every injustice I read about in Coates’s book is made worse by what I read here.

We think of math as inherently neutral, the way we think of technology and logic and science. But these things are like any tool, dependent on the person using them, capable of being used for good or evil. (Remember the way science was abused throughout American history, fabricating the entire concept of biological race to provide justification for white humans to enslave black ones.) When we start using math to make decisions instead of people, assuming that this will decrease human prejudice and make things more fair, what we can actually end up doing is obscuring that prejudice and magnifying the scale of its effect.

Cathy O’Neil was a mathematician working in the finance industry in 2008. She left after a few years, disgusted with the way the system was clearly rigged, and created the concept of Weapons of Math Destruction to address a problem: mathematical models that affect large numbers of people, in a huge variety of ways, based on incomplete or biased human input. In this book, O’Neil takes us through several WMDs, showing how they’re used in different areas of our lives.

For example, there was the disastrous school reform attempt in Washington D.C., which tried to evaluate teachers in a way math simply cannot be used, firing hundreds based on scores no one in the district could explain or defend. A class of thirty students is not a statistically sound data set, nor is it possible to measure a person’s impact on another person through test scores. And no one knew what the scores were measuring: O’Neil gives the example of a teacher who received the horrifyingly low score of 6 one year, then 96 the next year, without doing anything differently (he was tenured, which protected him from the first bad score, though a second would have changed things). But for . . . some reason . . . these factors were ignored, along with the fact that very good teachers were being fired—and the only reasoning seemed to be, “because math said so.”

Statisticians count on large numbers to balance out exceptions and anomalies. And WMDs, as we’ll see, often punish individuals who happen to be the exception.

In other areas of the job market, there’s the way many employers now use credit scores to screen applications, assuming that making regular payments means you’ll be a dependable employee, ignoring the facts that (1) responsible people experience financial difficulty, too and (2) systematically preventing people with bad credit from getting jobs is only going to make the situation worse.

There’s the way U.S. News & World Report helped cause an arms race when it decided to start ranking universities on everything except their cost, driving up tuition over 500 percent (almost four times the rate of inflation) in just the 30-ish years that I’ve been alive. This was surely not U.S. News‘s intent, but something that happened because of what data they did and didn’t include in their formula.

There’s social media, the section of the book which explains why I finally deactivated my Facebook account over the summer. It had been driving me crazy for a long time, and I just couldn’t stand anymore the palpable sense that Facebook was pulling strings I couldn’t see, manipulating my relationships with people, choosing whose stuff I got to see and who could see what I posted. If you’re under the impression that you post something and anyone who’s friends with you will see it in their news feed, Facebook hasn’t worked like that in a long time. To illustrate, O’Neil describes what would happen if she decided to circulate a petition among her network.

As soon as I hit send, that petition belongs to Facebook, and the social network’s algorithm makes a judgment about how best to use it. It calculates the odds that it will appeal to each of my friends. Some of them, it knows, often sign petitions, and perhaps share them with their own networks. Others tend to scroll right past . . . The Facebook algorithm takes all of this into account as it decides who will see my petition. For many of my friends, it will be buried so low on their news feed that they’ll never see it.

Especially because any time you click on something to see more than the headline, the news feed might reset itself, almost as if it’s actually trying to prevent you ever reaching those posts at the “bottom” (which in fact does not exist; unless you glitch, there’s no point at which it’s going to tell you, “The end! Congratulations, you finished your news feed”).

Then there’s predatory advertising like that done by for-profit colleges, which specifically targets vulnerable people’s “pain points” looking for those who are “stuck,” who have “few people in their lives who care about them,” who have “low self-esteem” and are “unable to see and plan well for the future” (these quotes, shared in the book by O’Neil, come from the California Attorney General’s office when it was investigating one of these colleges). And if you’re wondering how they would know this about people, the point is that they know this about all of us now. They know what we type into search engines; just try to imagine what someone could learn about you from seeing everything you do online.

If it was true during the early dot-com days that “nobody knows you’re a dog,” it’s the exact opposite today. We are ranked, categorized, and scored in hundreds of models, on the basis of our revealed preferences and patterns. This establishes a powerful basis for legitimate ad campaigns, but it also fuels their predatory cousins: ads that pinpoint people in great need and sell them false or overpriced promises. They find inequality and feast on it . . . When it comes to WMDs, predatory ads practically define the genre. They zero in on the most desperate among us at enormous scale.

O’Neil goes into more detail about this than I can bear to do here.

Political advertising is individually targeted, too, of course, to the extent that even people who support the same candidate, even people visiting the candidate’s website, could see totally different things based on the profile the campaign has created for them.

As this happens, it will become harder to access the political messages our neighbors are seeing—and as a result, to understand why they believe what they do, often passionately . . . It is not enough to simply visit the candidate’s web page, because they, too, automatically profile and target each visitor, weighing everything from their zip codes to the links they click on the page, even the photos they appear to look at.

It’s just another way in which public discourse is being sabotaged, because how can we discuss things if we’re all working from our own personalized set of facts? How can a democratic republic possibly function under these conditions?

Most important among this list (although they’re all so interconnected that distinctions can get murky), there are the models that judges use in sentencing, trying to gauge from prisoners’ background information—where they grew up, whether their friends and family have criminal records; information which would be inadmissible in court—how likely they are to return to prison after being released, and how long their sentence should be.

You might think that computerized risk models fed by data would reduce the role of prejudice in sentencing and contribute to more even-handed treatment . . . But embedded within these models are a host of assumptions, some of them prejudicial . . . As the questions continue, delving deeper into the person’s life, it’s easy to imagine how inmates from a privileged background would answer one way and those from tough inner-city streets another. Ask a criminal who grew up in comfortable suburbs about “the first time you were ever involved with the police,” and he might not have a single incident to report other than the one that brought him to prison. Young black males, by contrast, are likely to have been stopped by police dozens of times, even when they’ve done nothing wrong . . .

A person who scores as “high risk” is likely to be unemployed and to come from a neighborhood where many of his friends and family have had run-ins with the law. Thanks in part to the resulting high score on the evaluation, he gets a longer sentence, locking him away for more years in a prison where he’s surrounded by fellow criminals—which raises the likelihood that he’ll return to prison. He is finally released into the same poor neighborhood, this time with a criminal record, which makes it that much harder to find a job. If he commits another crime, the recidivism model can claim another success. But in fact the model itself contributes to a toxic cycle and helps to sustain it. That’s a signature quality of a WMD.

The worst and best part of this is that it makes clear how biased our law enforcement is—in the system as a whole, aside from (or in addition to) the prejudices of any individual officer. The bias expands even wider, not just within law enforcement but throughout Western societies—the belief that crime is only criminal when committed by the right (or wrong) people. In the United States, this means that although white and black people use drugs at essentially the same rate, black people are three to four times more likely to get arrested for it, and almost six times more likely to go to prison. It means that because crime is statistically higher in poor neighborhoods, police devote more attention to those neighborhoods, and their increased presence increases the likelihood that they’ll find things to arrest people for. It means, among other things, that we criminalize poverty.

In the 2000s, the kings of finance threw themselves a lavish party. They lied, they bet billions against their own customers, they committed fraud and paid off rating agencies. Enormous crimes were committed there, and the result devastated the global economy for the best part of five years. Millions of people lost their homes, jobs, and health care. We have every reason to believe that more such crimes are occurring right now . . . Just imagine if police enforced their zero-tolerance strategy in finance. They would arrest people for even the slightest infraction, whether it was chiseling investors on 401ks, providing misleading guidance, or committing petty frauds. Perhaps SWAT teams would descend on Greenwich, Connecticut. They’d go undercover in the taverns around Chicago’s Mercantile Exchange . . .

My point is that police make choices about where they direct their attention. Today they focus almost exclusively on the poor.

And now, with mathematical models to help, this focus is magnified. “If black men were overrepresented among drug dealers and absentee dads of the world, it was directly related to their being underrepresented among the Bernie Madoffs and Kenneth Lays of the world,” Ta-Nehisi Coates explained in “My President Was Black,” an essay included in We Were Eight Years in Power (I told you these books overlap). “Power was what mattered, and what characterized the differences between black and white America was not a difference in work ethic, but a system engineered to place one on top of the other.” That system includes WMDs, and I’ve hit on only a fraction of them here.

Examples like [Michigan, in which a system called Midas falsely accused more than 20,000 people of unemployment fraud, ruining their lives by charging them fines of up to $100,000 just when they needed money the most] demonstrate how critical an issue accountability standards are becoming. When algorithmic systems like the Midas system contain fatal flaws, whether intentional or not, they end up being worse than the human systems they’ve replaced. And, as we’ve seen repeatedly throughout the book, the resulting pain is not distributed equally, but is rather borne by society’s most vulnerable citizens. The very people that cannot afford to hire fancy lawyers have to go head-to-head with the machine. It’s not a fair fight, and examples like this make a clear case for placing the burden of proof on those designing and implementing the algorithms.

This is such an important point, one that surprises me in how rarely I hear it. The last book I read which addressed it was Why We Can’t Wait, by Martin Luther King, Jr. “In effect,” he explains, “the most impoverished Americans, facing powerfully equipped adversaries, are required to finance and conduct complex litigation that may involve tens of thousands of dollars . . . To be forced to accumulate resources for legal actions imposes intolerable hardships on the already overburdened.” Like conservatives’ insistence that issues of poverty and homelessness be addressed by the charity of individuals rather than nationwide policy, and the tying of health insurance to a person’s ability to find a full-time job, this practice leaves the already-disadvantaged with even fewer options.

It might seem like the logical response is to disarm these weapons, one by one. The problem is that they’re feeding on each other. Poor people are more likely to have bad credit and live in high-crime neighborhoods, surrounded by other poor people. Once the dark universe of WMDs digests that data, it showers them with predatory ads for subprime loans or for-profit schools. It sends more police to arrest them, and when they’re convicted it sentences them to longer terms. This data feeds into other WMDs, which score the same people as high-risk or easy targets and proceed to block them from jobs, while jacking up their rates for mortgages, car loans, and every kind of insurance imaginable. This drives their credit rating down further, creating nothing less than a death spiral of modeling. Being poor in a world of WMDs is getting more and more dangerous and expensive.

An acutely important thing to notice is that:

The same WMDs that abuse the poor also place the comfortable classes of society in their own marketing silos . . . For many of them it can feel as though the world is getting smarter and easier. [Mathematical] models highlight bargains on prosciutto and chianti, recommend a great movie on Amazon Prime, or lead them, turn by turn, to a cafe in what used to be a “sketchy” neighborhood. The quiet and personal nature of this targeting keeps society’s winners from seeing how the very same models are destroying lives, sometimes just a few blocks away.

Though we’re all affected by them in ways we can’t see, right now the majority of the damage is being done to the same groups who are always hurt by flaws in the system—minorities and the poor. These same models could easily be used to help people instead of prey on them, but that won’t happen in the glorious free market of capitalism. We need regulation and oversight in the data industry. We need accountability.

When statistics itself, and the public’s trust in statistics, is being actively undermined by politicians across the globe, how can we possibly expect the Big Data industry to clarify rather than contribute to the noise? We can because we must. Right now, mammoth companies like Google, Amazon, and Facebook exert incredible control over society because they control the data. They reap enormous profits while somehow offloading fact-checking responsibilities to others. It’s not a coincidence that, even as he works to undermine the public’s trust in science and in scientific fact, Steve Bannon sits on the board of Cambridge Analytica, a political data company that has claimed credit for Trump’s victory while bragging about secret “voter suppression” campaigns. It’s part of a more general trend in which data is privately owned and privately used to private ends of profit and influence, while the public is shut out of the process and told to behave well and trust the algorithms. It’s time to start misbehaving. Algorithms are only going to become more ubiquitous in the coming years. We must demand that systems that hold algorithms accountable become ubiquitous as well.

In the United States, we like to boast about freedom while structuring our country so that the powerful can exploit, cheat, and abuse the unpowerful, generally with no consequences. It’d be nice if someday, perhaps led by O’Neil and Coates and others like them, we decide to stop doing that. WMDs are going to be an important place to start.

One Comment Add yours

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.