O'Neil 2016
O'Neil, Cathy. Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. Crown, 2016.
Introduction
Page 3 · Location 136 By 2010 or so, mathematics was asserting itself as never before in human affairs, and the public largely welcomed it.
Page 7 · Location 205 In WMDs, many poisonous assumptions are camouflaged by math and go largely untested and unquestioned.
Page 8 · Location 209 The privileged, we’ll see time and again, are processed more by people, the masses by machines.
Page 10 · Location 248 The human victims of WMDs, we’ll see time and again, are held to a far higher standard of evidence than the algorithms themselves.
Page 12 · Location 266 Ill-conceived mathematical models now micromanage the economy, from advertising to prisons.
Chapter 1: Bomb Parts: What Is a Model?
Page 17 · Location 325 the folks building WMDs routinely lack data for the behaviors they’re most interested in. So they substitute stand-in data, or proxies. They draw statistical correlations between a person’s zip code or language patterns and her potential to pay back a loan or handle a job. These correlations are discriminatory, and some of them are illegal.
Page 18 · Location 333 That’s how trustworthy models operate. They maintain a constant back-and-forth with whatever in the world they’re trying to understand or predict. Conditions change, and so must the model.
Page 21 · Location 373 A model’s blind spots reflect the judgments and priorities of its creators.
Page 21 · Location 378 models, despite their reputation for impartiality, reflect goals and ideology.
Page 21 · Location 380 Models are opinions embedded in mathematics.
Page 22 · Location 397 Racism, at the individual level, can be seen as a predictive model whirring away in billions of human minds around the world. It is built from faulty, incomplete, or generalized data. Whether it comes from experience or hearsay, the data indicates that certain types of people have behaved badly. That generates a binary prediction that all people of that race will behave that same way.
Page 29 · Location 493 WMDs are, by design, inscrutable black boxes.
Page 31 · Location 517 these are the three elements of a WMD: Opacity, Scale, and Damage.
Chapter 2: Shell Shocked: My Journey of Disillusionment
Page 44 · Location 704 I was forced to confront the ugly truth: people had deliberately wielded formulas to impress rather than clarify.
Chapter 3: Arms Race: Going to College
Page 55 · Location 842 proxies, it is far simpler for people to game it. This is because proxies are easier to manipulate than the complicated reality they represent.
Page 61 · Location 930 As colleges position themselves to move up the U.S. News charts, they manage their student populations almost like an investment portfolio.
Page 64 · Location 988 Each college’s admissions model is derived, at least in part, from the U.S. News model, and each one is a mini-WMD.
Page 65 · Location 992 vast majority of Americans, the poor and middle-class families who don’t have thousands of dollars to spent on courses and consultants. They miss out on precious insider knowledge. The result is an education system that favors the privileged.
Page 65 · Location 997 They don’t master important skills by jumping through many more hoops or writing meticulously targeted college essays under the watchful eye of professional tutors. Others scrounge online for cut-rate versions of those tutors. All of them, from the rich to the working class, are simply being trained to fit into an enormous machine—
Page 65 · Location 999 to satisfy a WMD. And at the end of the ordeal, many of them will be saddled with debt that will take decades to pay off. They’re pawns in an arms race, and it’s a particularly nasty one.
Chapter 4: Propaganda Machine: Online Advertising
Page 81 · Location 1224 The presidents of the leading for-profit universities make millions of dollars every year. For example, Gregory W. Cappelli, CEO of Apollo Education Group, the parent company of the University of Phoenix, took home $ 25.1 million in total compensation in 2011.
Chapter 10: The Targeted Citizen: Civic Life
Page 195 · Location 2814 The political marketers maintain deep dossiers on us, feed us a trickle of information, and measure how we respond to it. But we’re kept in the dark about what our neighbors are being fed.
Page 197 · Location 2840 With political messaging, as with most WMDs, the heart of the problem is almost always the objective. Change that objective from leeching off people to helping them, and a WMD is disarmed—and can even become a force for good.
Conclusion
Page 204 · Location 2920 Big Data processes codify the past. They do not invent the future. Doing that requires moral imagination, and that’s something only humans can provide. We have to explicitly embed better values into our algorithms, creating Big Data models that follow our ethical lead. Sometimes that will mean putting fairness ahead of profit.
Page 208 · Location 2976 We’d begin by treating the WMD as a black box that takes in data and spits out conclusions. This person has a medium risk of committing another crime, this one has a 73 percent chance of voting Republican, this teacher ranks in the lowest decile. By studying these outputs, we could piece together the assumptions behind the model and score them for fairness.
Afterword
Page 199 · Location 3212 Let’s reframe the question of fairness: instead of fighting over which single metric we should use to determine the fairness of an algorithm, we should instead try to identify the stakeholders and weigh their relative harms.