Moneyball by Michael Lewis explains big data Concepts

Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy

by Cathy O'Neil · 5 Sep 2016 · 252pp · 72,473 words

growing tyranny of an arrogant establishment.” —Ralph Nader, author of Unsafe at Any Speed “Next time you hear someone gushing uncritically about the wonders of Big Data, show them Weapons of Math Destruction. It’ll be salutary.” —Felix Salmon, Fusion “From getting a job to finding a spouse, predictive algorithms are silently

…

a trademark of Penguin Random House LLC. Library of Congress Cataloging-in-Publication Data Name: O’Neil, Cathy, author. Title: Weapons of math destruction: how big data increases inequality and threatens democracy / Cathy O’Neil Description: First edition. | New York: Crown Publishers [2016] Identifiers: LCCN 2016003900 (print) | LCCN 2016016487 (ebook) | ISBN 9780553418811

…

(hardcover) | ISBN 9780553418835 (pbk.) | ISBN 9780553418828 (ebook) Subjects: LCSH: Big data—Social aspects—United States. | Big data—Political aspects—United States. | Social indicators—Mathematical models—Moral and ethical aspects. | Democracy—United States. | United States—Social conditions—21st century. Classification: LCC

…

Journey of Disillusionment CHAPTER 3 ARMS RACE: Going to College CHAPTER 4 PROPAGANDA MACHINE: Online Advertising CHAPTER 5 CIVILIAN CASUALTIES: Justice in the Age of Big Data CHAPTER 6 INELIGIBLE TO SERVE: Getting a Job CHAPTER 7 SWEATING BULLETS: On the Job CHAPTER 8 COLLATERAL DAMAGE: Landing Credit CHAPTER 9 NO SAFE

…

were studying our desires, movements, and spending power. They were predicting our trustworthiness and calculating our potential as students, workers, lovers, criminals. This was the Big Data economy, and it promised spectacular gains. A computer program could speed through thousands of résumés or loan applications in a second or two and sort

…

of that gap is due to her teacher? It’s hard to know, and Mathematica’s models have only a few numbers to compare. At Big Data companies like Google, by contrast, researchers run constant tests and monitor thousands of variables. They can change the font on a single advertisement from blue

…

Pandora, the ideal job on LinkedIn, or perhaps the love of their life on Match.com. Think of the astounding scale, and ignore the imperfections. Big Data has plenty of evangelists, but I’m not one of them. This book will focus sharply in the other direction, on the damage inflicted by

…

finding and holding a job. All of these life domains are increasingly controlled by secret models wielding arbitrary punishments. Welcome to the dark side of Big Data. It was a hot August afternoon in 1946. Lou Boudreau, the player-manager of the Cleveland Indians, was having a miserable day. In the first

…

get him out? And how will that affect their overall odds of winning? Baseball is an ideal home for predictive mathematical modeling. As Michael Lewis wrote in his 2003 bestseller, Moneyball, the sport has attracted data nerds throughout its history. In decades past, fans would pore over the stats on the back of

…

that, rather than the movement of markets, I was now predicting people’s clicks. In fact, I saw all kinds of parallels between finance and Big Data. Both industries gobble up the same pool of talent, much of it from elite universities like MIT, Princeton, or Stanford. These new hires are ravenous

…

success, and growing feedback loops. Those who objected were regarded as nostalgic Luddites. I wondered what the analogue to the credit crisis might be in Big Data. Instead of a bust, I saw a growing dystopia, with inequality rising. The algorithms would make sure that those deemed losers would remain that way

…

economy, raking in outrageous fortunes and convincing themselves all the while that they deserved it. After a couple of years working and learning in the Big Data space, my journey to disillusionment was more or less complete, and the misuse of mathematics was accelerating. In spite of blogging almost daily, I could

…

to get the same or better policing out of a smaller force. So in 2013 he invested in crime prediction software made by PredPol, a Big Data start-up based in Santa Cruz, California. The program processed historical crime data and calculated, hour by hour, where crimes were most likely to occur

…

includes risk terrain analysis, which incorporates certain features, such as ATMs or convenience stores, that might attract crimes. Like those in the rest of the Big Data industry, the developers of crime prediction software are hurrying to incorporate any information that can boost the accuracy of their models. If you think about

…

crime. That’s where it is, they say, pointing to the highlighted ghetto on the map. And now they have cutting-edge technology (powered by Big Data) reinforcing their position there, while adding precision and “science” to the process. The result is that we criminalize poverty, believing all the while that our

…

’t incorporate information about how the candidate would actually perform at the company. That’s in the future, and therefore unknown. So like many other Big Data programs, they settle for proxies. And as we’ve seen, proxies are bound to be inexact and often unfair. In fact, the Supreme Court ruled

…

on the hunt for employees who think creatively and work well in teams. So the modelers’ challenge is to pinpoint, in the vast world of Big Data, the bits of information that correlate with originality and social skills. Résumés alone certainly don’t cut it. Most of the items listed there—the

…

faith in the science of phrenology. Phrenology was a model that relied on pseudoscientific nonsense to make authoritative pronouncements, and for decades it went untested. Big Data can fall into the same trap. Models like the ones that red-lighted Kyle Behm and blackballed foreign medical students at St. George’s can

…

hire a part-timer to help with the Saturday crush. These changes add a bit of intelligence to the dumb and inflexible status quo. With Big Data, that college freshman is replaced by legions of PhDs with powerful computers in tow. Businesses can now analyze customer traffic to calculate exactly how many

…

Microsoft use in-house programs to do just this. It’s very similar to a dating algorithm (and often, no doubt, has similarly spotty results). Big Data has also been used to study the productivity of call center workers. A few years ago, MIT researchers analyzed the behavior of call center employees

…

. Investors double down on scientific systems that can place thousands of people into what appear to be the correct buckets. It’s the triumph of Big Data. And what about the person who is misunderstood and placed in the wrong bucket? That happens. And there’s no feedback to set the system

…

still make a mountain of money. That was Douglas Merrill’s idea. A former chief operating officer at Google, Merrill believed that he could use Big Data to calculate risk and offer payday loans at a discount. In 2009, he founded a start-up called ZestFinance. On the company web page, Merrill

…

whether they’ve been in an accident—and not by their consumer patterns or those of their friends or neighbors. Yet in the age of Big Data, urging insurers to judge us by how we drive means something entirely new. Insurance companies now have manifold ways to study drivers’ behavior in exquisite

…

the same politician, one vowing to protect wilderness and the other stressing law and order. Direct mail was microtargeting on training wheels. The convergence of Big Data and consumer marketing now provides politicians with far more powerful tools. They can target microgroups of citizens for both votes and money and appeal to

…

be heading up the data team for Obama’s campaign. In his previous position, at Accenture Labs in Chicago, Ghani had developed consumer applications for Big Data, and he trusted that he could apply his skills to politics. The goal for the Obama campaign was to create tribes of like-minded voters

…

adapt, we change, and so do our processes. Automated systems, by contrast, stay stuck in time until engineers dive in to change them. If a Big Data college application model had established itself in the early 1960s, we still wouldn’t have many women going to college, because it would have been

…

men, the people paid by rich patrons to create art. The University of Alabama’s football team, needless to say, would still be lily white. Big Data processes codify the past. They do not invent the future. Doing that requires moral imagination, and that’s something only humans can provide. We have

…

to explicitly embed better values into our algorithms, creating Big Data models that follow our ethical lead. Sometimes that will mean putting fairness ahead of profit. In a sense, our society is struggling with a new

…

important decisions about the people we trust to teach our children. That’s a job that requires subtlety and context. Even in the age of Big Data, it remains a problem for humans to solve. Of course, the human analysts, whether the principal or administrators, should consider lots of data, including the

…

needs an update. The bill currently prohibits medical exams as part of an employment screening. But we need to update it to take into account Big Data personality tests, health scores, and reputation scores. They all sneak around the law, and they shouldn’t be able to. One possibility already under discussion

…

and Accountability Act (HIPAA), which protects our medical information, in order to cover the medical data currently being collected by employers, health apps, and other Big Data companies. Any health-related data collected by brokers, such as Google searches for medical treatments, must also be protected. If we want to bring out

…

Section 8 was put up, an extremely awkward and brief conversation took place. Someone demanded the slide be taken down. The party line prevailed. While Big Data, when managed wisely, can provide important insights, many of them will be disruptive. After all, it aims to find patterns that are invisible to human

…

York Times, May 12, 2014, www.nytimes.com/2014/05/13/sports/baseball/whos-on-third-in-baseballs-shifting-defenses-maybe-nobody.html?_r=0. Moneyball: Michael Lewis, Moneyball: The Art of Winning an Unfair Game (New York: W. W. Norton, 2003). In 1997, a convicted murderer: Manny Fernandez, “Texas Execution Stayed Based on

…

/data-mining-moves-to-human-resources. MIT researchers analyzed the behavior of call center employees: Joshua Rothman, “Big Data Comes to the Office,” New Yorker, June 3, 2014, www.newyorker.com/books/joshua-rothman/big-data-comes-to-the-office. A Nation at Risk: National Commission on Excellence in Education, A Nation at Risk

…

/01/31/your-money/credit-and-debit-cards/31money.html. Douglas Merrill’s idea: Steve Lohr, “Big Data Underwriting for Payday Loans,” New York Times, January 19, 2015, http://bits.blogs.nytimes.com/2015/01/19/big-data-underwriting-for-payday-loans/. On the company web page: Website ZestFinance.com, accessed January 9, 2016

…

, www.zestfinance.com/. A typical $500 loan: Lohr, “Big Data Underwriting.” ten thousand data points: Michael Carney, “Flush with $20M from Peter Thiel

…

, ZestFinance Is Measuring Credit Risk Through Non-traditional Big Data,” Pando, July 31, 2013, https://pando.com/2013/07/31/flush-with-20m-from-peter

…

-thiel-zestfinance-is-measuring-credit-risk-through-non-traditional-big-data/. one of the first peer-to-peer exchanges, Lending Club: Richard MacManus, “Facebook App, Lending Club, Passes Half a Million Dollars in Loans,” Readwrite, July

…

.1014098. each campaign developed profiles of American voters: Sasha Issenberg, “How President Obama’s Campaign Used Big Data to Rally Individual Voters,” Technology Review, December 19, 2012, www.technologyreview.com/featuredstory/509026/how-obamas-team-used-big-data-to-rally-voters/. Four years later: Adam Pasick and Tim FernHolz, “The Stealthy, Eric Schmidt-Backed

Big Data: A Revolution That Will Transform How We Live, Work, and Think

by Viktor Mayer-Schonberger and Kenneth Cukier · 5 Mar 2013 · 304pp · 82,395 words

1986 as its first undergrad to major in computer science. From his perch at the University of Washington, he started a slew of big-data companies before the term “big data” became known. He helped build one of the Web’s first search engines, MetaCrawler, which was launched in 1994 and snapped up

…

world where probability and correlation are paramount. In the movie Moneyball, baseball scouts were upstaged by statisticians when gut instinct gave way to sophisticated analytics. Similarly, subject-matter specialists will not go away, but they will have to contend with what the big-data analysis says. This will force an adjustment to traditional

…

base pairs. But the absolute number of data points alone, the size of the dataset, is not what makes these examples of big data. What classifies them as big data is that instead of using the shortcut of a random sample, both Flu Trends and Steve Jobs’s doctors used as much of

…

they are likely to change over time. To start, let’s examine each category—data holder, data specialist, and big-data mindset—in turn. The big-data value chain The primary substance of big data is the information itself. So it makes sense to look first at the data holders. They may not have done

…

numerous acquisitions in the big-data business. For example, in 2006 Microsoft rewarded Etzioni’s big-data mindset by buying Farecast for around $110 million. But two years later Google paid $700 million to acquire Farecast’s data supplier, ITA Software. The demise of the expert In the movie Moneyball, about how the Oakland

…

t much different. Similar empty reasoning is employed from Manhattan boardrooms to the Oval Office to coffee shops and kitchen tables everywhere else. Moneyball, based on the book by Michael Lewis, tells the true story of Billy Beane, the Oakland A’s general manager who threw out the century-old rulebook on how

…

find they no longer need to attain a threshold in size. Instead, they can remain small and still flourish (or be acquired by a big-data giant). Big data squeezes the middle of an industry, pushing firms to be very large, or small and quick, or dead. Many traditional sectors will eventually be

…

firms in all sectors, but it will certainly place pressure on companies in industries that are vulnerable to being shaken up by the power of big data. Big data is poised to disrupt the competitive advantages of states as well. At a time when manufacturing has been largely lost to developing countries and innovation

…

the office and going online to learn how to protect themselves, not because the searchers are ill themselves. The dark side of big data As we have seen, big data allows for more surveillance of our lives while it makes some of the legal means for protecting privacy largely obsolete. It also renders

…

—and one fraught with risk for the rest of us. It is obviously impossible to foretell how a technology will develop; even big data can’t predict how big data will evolve. Regulators will need to strike a balance between acting cautiously and boldly—and the history of antitrust law points to one

…

inherently fallible. This doesn’t mean they’re wrong, only that they are always incomplete. It doesn’t negate the insights that big data offers, but it puts big data in its place—as a tool that doesn’t offer ultimate answers, just good-enough ones to help us now until better methods

…

at, [>]–[>], [>] data-reuse by, [>], [>] and e-books, [>]–[>] Amazonia (Marcus), [>] ancient world: record-keeping in, [>]–[>] Anderson, Chris: on “end of theory,” [>]–[>] anonymization: big data defeats, [>]–[>], [>] of data, [>], [>]–[>] privacy and, [>]–[>] antitrust regulation: big data and, [>]–[>] AOL: fails to understand data-reuse, [>] releases personal data, [>]–[>] Apple, [>], [>] and cell phone data, [>] Arabic numerals, [>]–[>] Arnold, Thelma, [>]–[>] artificial intelligence

…

: big data and, [>]–[>] at Google, [>] Asthmapolis, [>] astronomy: big data in, [>] automobiles: anti-theft systems, [>] data-gathering by, [>]–[>], [>]–[>], [>], [>], [>]–[>] automobiles, electric: big data and, [>]–[>] IBM and, [>]–[>] automobiles, self-driving, [>], [>], [>], [>] Aviva, [>]–[>] Ayres, Ian: Super Crunchers, [>] Bacon, Francis, [>] Banko, Michele, [>], [>] Barabási,

…

Albert-László, [>]–[>] Barnes & Noble, [>]–[>] Basis, [>] Beane, Billy, [>]–[>] Being Digital (Negroponte), [>] Bell Labs, [>] Berners-Lee, Tim, [>] Bezos, Jeff, [>], [>], [>], [>] big data. See also data; information; open data and antitrust regulation, [>]–[>] and artificial intelligence, [>]–[>] in astronomy, [>] as based on theory, [>]–[>] and business models, [>], [>]–[>], [>], [>]–[>] and calculation of inflation

…

, [>]–[>], [>], [>]–[>] privacy and, [>]–[>], [>], [>], [>] psychological effects, [>]–[>] replaces statistical sampling, [>]–[>], [>], [>]–[>], [>]–[>] role of subject-area expertise in, [>]–[>] social & economic effects of, [>]–[>], [>]–[>], [>]–[>], [>]–[>], [>], [>], [>], [>]–[>], [>]–[>] as source of competitive advantage, [>]–[>] value chain, [>], [>], [>]–[>], [>]–[>] “big-data companies,” [>]–[>], [>] Billion Prices Project, [>]–[>] Bing, [>] Binney, William, [>] births, premature: McGregor and, [>]–[>], [>], [>] Bloomberg, Michael, [>]–[>] bookkeeping, double-entry: history of, [>]–[>] Pacioli and, [>]–[>] books. See also e-books

…

in, [>]–[>], [>], [>] of sales data, [>] vs. scientific method, [>]–[>] and subprime mortgage scandal (2009), [>] of text, [>]–[>] in video game design, [>]–[>] Coursera, [>], [>] Craigslist, [>] Crawford, Kate, [>] credit card fraud: big data and, [>]–[>], [>]–[>] Kunze on, [>] credit scores: correlation analysis and, [>] datafication and, [>] credit transactions: analysis of, [>] crime prevention: predictive policing and, [>]–[>] Crosby, Alfred, [>], [>] Cross, Bradford, [>]–[>] “culturomics,” [>]–[>]

…

data. See also big data; information; open data aggregation of, [>], [>], [>], [>], [>]–[>], [>]–[>], [>], [>], [>], [>], [>], [>]–[>], [>], [>]–[>] anonymization of, [>], [>]–[>] brokering, [>] compared to energy, [>] decision-making driven by, [>]–[>] depreciating value of, [>]–[>] “dictatorship” of, [>], [>]–[>], [>]–[>] economic value of reusing, [>]–[>], [>]–[>], [>]–[>], [>]–[>], [>], [>]

…

[>] DataSift, [>] Davenport, Thomas, [>], [>] Decide.com, [>]–[>], [>] decision-making: driven by data, [>]–[>], [>] Delano, Robert, [>] Deloitte Consulting, [>] Derawi Biometrics, [>] Derwent Capital, [>] digitization: vs. datafication, [>]–[>], [>]–[>] revolution in, [>], [>], [>] DNA sequencing: big data in, [>] cost of, [>] Steve Jobs and, [>]–[>], [>] Domesday Book, [>]–[>] Dostert, Leon, [>] Duhigg, Charles: The Power of Habit, [>]–[>] Eagle, Nathan, [>]–[>] eBay, [>], [>] e-books. See also books Amazon

…

and, [>]–[>] and datafication, [>]–[>] and data-reuse, [>]–[>], [>]–[>] e-commerce: big data in, [>]–[>] economic development: big data in, [>]–[>] education: misuse of data in, [>] online, [>] edX, [>] Eisenstein, Elizabeth, [>] Elbaz, Gil, [>] election of 2008: data-gathering in, [>] electrical meters: data-gathering by, [>]–[>]

…

airline fare pricing patterns, [>]–[>], [>], [>], [>], [>], [>], [>], [>], [>] Euclid, [>] European Union: open data in, [>] Evans, Philip, [>] exactitude. See also imprecision and big data, [>]–[>], [>], [>], [>], [>] in database design, [>]–[>], [>] and measurement, [>]–[>], [>] necessary in sampling, [>], [>]–[>] Excite, [>] Experian, [>], [>], [>], [>], [>] expertise, subject-area: role in big data, [>]–[>] explainability: big data and, [>]–[>] Facebook, [>], [>], [>]–[>], [>]–[>], [>], [>], [>], [>] data processing by, [>] datafication by, [>], [>] IPO by, [>]–[>] market valuation of, [>]–[>] uses “data exhaust,” [>] Factual, [>]

…

, [>]–[>] insurance industry uses data, [>] UPS uses data, [>]–[>] Germany, East: as police state, [>], [>], [>] Global Positioning System (GPS) satellites, [>]–[>], [>], [>], [>] Gnip, [>] Goldblum, Anthony, [>] Google, [>], [>], [>], [>], [>], [>], [>], [>] artificial intelligence at, [>] as big-data company, [>] Books project, [>]–[>] data processing by, [>] data-reuse by, [>]–[>], [>], [>] Flu Trends, [>], [>], [>], [>], [>], [>] gathers GPS data, [>], [>], [>] Gmail, [>], [>] Google Docs, [>] and language translation, [>]–[>], [>], [>], [>], [>] MapReduce, [>], [>] maps, [>] PageRank, [>] page

…

[>]–[>] search-term analytics by, [>], [>], [>], [>], [>], [>] speech-recognition at, [>]–[>] spell-checking system, [>]–[>] Street View vehicles, [>], [>]–[>], [>], [>] uses “data exhaust,” [>]–[>] uses mathematical models, [>]–[>], [>] government: and open data, [>]–[>] regulation and big data, [>]–[>], [>] surveillance by, [>]–[>], [>]–[>] Graunt, John: and sampling, [>] Great Britain: open data in, [>] guilt by association: profiling and, [>]–[>] Gutenberg, Johannes, [>] Hadoop, [>], [>] Hammerbacher, Jeff, [>] Harcourt, Bernard, [>] health care

…

information, [>]–[>] Hitwise, [>], [>] Hollerith, Herman: and punch cards, [>], [>] Hollywood films: profits predicted, [>]–[>] Honda, [>] Huberman, Bernardo: and social networking analysis, [>] human behavior: datafication and, [>]–[>], [>]–[>] human perceptions: big data changes, [>] IBM, [>] and electric automobiles, [>]–[>] founded, [>] and language translation, [>]–[>], [>] Project Candide, [>]–[>] ID3, [>] “If You Have Too Much Data, Then ‘Good Enough’ Is Good Enough” (Hellend

…

), [>] Import.io, [>] imprecision. See also exactitude in data-processing, [>]–[>] nature of, [>]–[>] as positive feature of big data, [>]–[>], [>]–[>], [>]–[>], [>], [>], [>] and scale, [>], [>], [>], [>], [>] and truth, [>] In Retrospect (McNamara), [>] inflation: big data and calculation of, [>]–[>] information. See also big data; data; open data analysis of, [>]–[>], [>] as basis of the universe, [>]–[>] growth in amount of, [>]–[>], [>], [>]–[>], [>], [>], [>] Hilbert attempts to

…

: analyzes ergonomic data, [>], [>], [>], [>]–[>] Kunze, John: on credit card fraud, [>] Laney, Doug, [>], [>], [>] Large Synoptic Survey Telescope, [>] laws: against misuse of big data, [>], [>]–[>] protecting privacy, [>], [>] for use of information, [>] Leavitt, Stephen: Freakonomics, [>]–[>] Levis, Jack, [>]–[>] Lewis, Michael: Moneyball, [>] lexicology, computational, [>] Linden, Greg, [>]–[>] LinkedIn, [>], [>], [>], [>], [>] Luther, Martin, [>] Lytro camera, [>]–[>] machine learning, [>], [>] machine translation. See translation, language manhole

…

datafication, [>]–[>] metric system, [>] Microsoft, [>], [>], [>] Amalga software, [>]–[>], [>] and data-valuation, [>] and language translation, [>] Word spell-checking system, [>]–[>] Minority Report [film], [>]–[>], [>] Moneyball [film], [>], [>]–[>], [>], [>] Moneyball (Lewis), [>] Moore’s Law, [>] Mydex, [>] nanotechnology: and qualitative changes, [>] Nash, Bruce, [>] nations: big data and competitive advantage among, [>]–[>] natural language processing, [>] navigation, marine: correlation analysis in, [>]–[>] Maury revolutionizes, [>]–[>], [>], [>], [>], [>], [>], [>], [>], [>], [>] Negroponte, Nicholas: Being Digital

…

, [>] police: use predictive analytics, [>], [>]–[>], [>] police state: East Germany as, [>], [>], [>] Power of Habit, The (Duhigg), [>]–[>] precision. See exactitude predictive analytics, [>], [>]. See also correlation analysis; data analysis big data and, [>]–[>], [>], [>]–[>] Department of Homeland Security uses, [>] vs. free will, [>], [>], [>], [>]–[>] in health care, [>]–[>], [>] in insurance industry, [>]–[>] in Iraq War, [>] in mechanical & structural failure, [>], [>]–[>], [>], [>], [>] parole boards

…

UPS, [>] predictive policing, [>] and crime prevention, [>]–[>] price-prediction: for consumer products, [>]–[>], [>] PriceStats, [>] printing press: socioeconomic effects of, [>], [>], [>]–[>] Prismatic: analyzes online media, [>]–[>] privacy: and anonymization, [>]–[>] and big data, [>]–[>], [>], [>], [>] and cell phone data, [>], [>] Google and, [>]–[>] and Internet, [>]–[>] laws protecting, [>], [>] and notice & consent, [>], [>], [>]–[>] Ohm on, [>] and opting out, [>], [>] and personal data, [>]–[>], [>]–[>], [>], [>], [>] profiling: and guilt

…

[>] Roadnet Technologies, [>] Rolls-Royce, [>] Roman numerals, [>]–[>] Rudin, Cynthia, [>], [>] Rudin, Ken, [>] sabermetrics, [>] Saddam Hussein: trial of, [>] Salathé, Marcel, [>]–[>] sales data: analysis of, [>], [>], [>], [>] Salesforce.com, [>] sampling, statistical: big data replaces, [>]–[>], [>], [>]–[>], [>]–[>] exactitude necessary in, [>], [>]–[>] Graunt and, [>] limitations inherent in, [>]–[>], [>], [>] Neyman on, [>] in quality control, [>] randomness needed in, [>]–[>] scale in, [>] Silver on, [>] scale: in data, [>]–[>] imprecision

…

, [>] traffic-pattern analysis: by Inrix, [>]–[>], [>] translation, language, [>] Google and, [>]–[>], [>], [>], [>] IBM and, [>]–[>], [>] Microsoft and, [>] transparency: of algorithms, [>] truth: data as, [>], [>] imprecision and, [>] 23andMe, [>] Twitter, [>], [>], [>]–[>], [>] as big-data company, [>], [>]–[>] data processing by, [>] datafication by, [>]–[>] message analysis by, [>] Udacity, [>] Universal Transverse Mercator (UTM) system, [>] universe: information as basis of, [>]–[>] “Unreasonable Effectiveness of Data, The

…

, [>] uses predictive analytics, [>] U.S. National Security Agency (NSA): data-gathering by, [>]–[>] U.S. President’s Council of Advisors on Science and Technology, [>] value, economic: big data and creation of, [>], [>], [>], [>], [>]–[>], [>]–[>], [>]–[>], [>]–[>] of reusing data, [>]–[>], [>]–[>], [>]–[>], [>]–[>], [>], [>] Varian, Hal, [>] video game design: correlation analysis in, [>]–[>] Vietnam War: data misused in, [>], [>]–[>] Visa, [>] von Ahn, Luis: invents

Reinventing Capitalism in the Age of Big Data

by Viktor Mayer-Schönberger and Thomas Ramge · 27 Feb 2018 · 267pp · 72,552 words

FEEDBACK EFFECTS 9 UNBUNDLING WORK 10 HUMAN CHOICE ACKNOWLEDGMENTS ABOUT THE AUTHOR ALSO BY VIKTOR MAYER-SCHÖNBERGER PRAISE FOR REINVENTING CAPITALISM IN THE AGE OF BIG DATA NOTES INDEX – 1 – REINVENTING CAPITALISM IT SHOULD HAVE BEEN A VICTORY CELEBRATION. BY THE time eBay’s new CEO, Devin Wenig, climbed the stage for

…

its preference-matching algorithm and search for the products we are most likely to purchase. Amazon’s strategy isn’t unique; it’s representative of Big Data, an approach to data analysis that aims to capture data comprehensively about a particular phenomenon, looking for complex patterns embedded in the data. By concentrating

…

, it differs from conventional statistics that have been focused on condensing data to its essence, from calculating averages to running regressions. A feature of many Big Data approaches is that the pattern one is looking for isn’t defined from the outset; rather, it emerges as huge amounts of training data are

…

information intermediaries and did well. More followed, and today a growing number of highly specialized boutique firms utilize the latest in digital technologies, working with big data analytics firms such as Contix and Kensho and utilizing machine learning systems to do what investment banking originally did: offer information rich with insight about

…

clever exploitation of our informational surplus, and data-rich markets are the mechanisms and the places where we can achieve this. When artificial intelligence and Big Data meet the social reality of human coordination, we can become more sustainable. Spurred by “smart meter” technology, for example, energy markets will become data-rich

…

, we’ll be able to dial up (or down) the amount of help we’d like. We’ll choose to choose. Experts have warned that Big Data and artificial intelligence may endanger human volition by making decisions not only about what we buy, but with whom we coordinate. The fear is that

…

why we call the shift from money to data a revival of the market instead of the rise of artificial intelligence or the advent of Big Data. Without the market, neither data nor technology will protect—let alone advance—humankind and help people work together. Hence, in this book, the market has

…

for their patience. VIKTOR MAYER-SCHÖNBERGER (left) is a professor at the University of Oxford and the coauthor, with Kenneth Cukier, of the best-selling Big Data. THOMAS RAMGE is the technology correspondent of the business magazine brand eins and writes for the Economist. ALSO BY VIKTOR MAYER-SCHÖNBERGER

…

Big Data: A Revolution That Will Transform How We Live, Work, and Think (with Kenneth Cukier) Learning with Big Data (with Kenneth Cukier) Delete: The Virtue of Forgetting in the Digital Age Governance and Information Technology: From

…

Electronic Government to Information Government (with David Lazer) Praise for REINVENTING CAPITALISM IN THE AGE OF BIG DATA “Digitalization is challenging us to re-think the future of our economy. This thought-provoking book provides excellent insights and guidance.” —Henning Kagermann, former CEO

…

New Economics of Multisided Platforms (Cambridge: Harvard Business Review Press, 2016). algorithm predicted which team would win: Tim Adams, “Job Hunting Is a Matter of Big Data, Not How You Perform at an Interview,” Observer, May 10, 2014, https://www.theguardian.com/technology/2014/may/10/job-hunting

…

-big-data-interview-algorithms-employees; Sue Tabbitt, “Forget Myers-Briggs: Algorithms Can Better Predict Team Chemistry,” Guardian, May 27, 2016, https://www.theguardian.com/small-business-network/

…

://uk.businessinsider.com/simple-questions-like-do-you-like-horror-films-can-predict-whether-a-startup-will-implode-2015-12. representative of Big Data: Viktor Mayer-Schönberger and Kenneth N. Cukier, Big Data: A Revolution That Will Transform How We Live, Work, and Think (New York: Houghton Mifflin Harcourt, 2013). going beyond its initial

…

training: For those who want to learn more about machine-learning methods (rather than Big Data more generally) in an easily accessible way, see Ethem Alpaydin, Machine Learning (Cambridge: MIT Press, 2016). Tesla’s semiautonomous driving system: Dana Hull, “The Tesla

…

, https://hbr.org/2010/12/robert-s-mcnamara-and-the-evolution-of-modern-management. simplifying data to make it more digestible: Mayer-Schönberger and Cukier, Big Data, 164–165, 168. tools to shape the flow of information: Ludwig Siegele and Joachim Zepelin, Matrix der Welt: SAP und der neue globale Kapitalismus (Frankfurt

…

. See also Daniel Kahneman, Thinking, Fast and Slow (New York: Farrar, Straus and Giroux, 2011); on how Kahneman and Tversky achieved their breakthrough insights, see Michael Lewis, The Undoing Project: A Friendship That Changed Our Minds (New York: W. W. Norton, 2016). confirmation bias: Yoram Bar-Tal and Maria Jarymowicz, “The Effect

…

E. Stucke, Virtual Competition: The Promise and Perils of the Algorithm-Driven Economy (Cambridge: Harvard University Press, 2016); see also Maurice Stucke and Allen Grunes, Big Data and Competition Policy (New York: Oxford University Press, 2016). “open up” their algorithms: For a critical view of algorithmic transparency, see Joshua A. Kroll et

…

–705, https://www.pennlawreview.com/print/165-U-Pa-L-Rev-633.pdf. economists… offer an intriguing idea: Jens Prüfer and Christoph Schottmüller, “Competing with Big Data,” February 16, 2017, TILEC Discussion Paper 2017-006, available at http://dx.doi.org/10.2139/ssrn.2918726; this extends an idea originally suggested in

…

–118. shift in focus from collection to use: See, e.g., Fred H. Cate and Viktor Mayer-Schönberger, “Notice and Consent in a World of Big Data,” International Data Privacy Law 3 (2013), 67–73; Kirsten E. Martin, “Transaction Costs, Privacy, and Trust: The Laudable Goals and Ultimate Failure of Notice and

…

Bauer, Florian, 55 Bear Stearns, 155 Beer, Staffors, 175–176 Bethlehem Steel, 95 Betterment, 151, 153 Bezos, Jeff, 68, 88, 89, 96, 106, 107, 130 Big Data, 77, 213, 219, 222 See also data-rich markets Bitcoin, 48, 147 BlaBlaCar, 3, 9, 65 blockchain, 147, 148 BMW, 120 book value, 172 bookkeeping

When McKinsey Comes to Town: The Hidden Influence of the World's Most Powerful Consulting Firm

by Walt Bogdanich and Michael Forsythe · 3 Oct 2022 · 689pp · 134,457 words

’t win, you lose. The Quarterly did not mention that the Astros were also a McKinsey client. Luhnow told the interviewer where baseball was headed: Big data combined with artificial intelligence is the next big wave in baseball, and I think we’re just starting to scratch the surface. It’s an

…

.” GO TO NOTE REFERENCE IN TEXT “McKinsey didn’t just cash the checks”: McDonald, Firm, 242. GO TO NOTE REFERENCE IN TEXT a book called Moneyball: Michael Lewis, Moneyball: The Art of Winning an Unfair Game (New York: Norton, 2003). GO TO NOTE REFERENCE IN TEXT sports and gambling: For years the major sports

The Metropolitan Revolution: How Cities and Metros Are Fixing Our Broken Politics and Fragile Economy

by Bruce Katz and Jennifer Bradley · 10 Jun 2013

Authority in downtown Brooklyn. The campus will be known as the Center for Urban Science and Progress (CUSP). “We are about applying the technologies of big data to urban problems and urban systems,” said Steven Koonin, CUSP’s founding director. The center’s 02-2151-2 ch2.indd 27 5/20/13

…

are grounded in evidence, developed through the accumulation of relevant data and information, accompanied by smart analysis, experience, and intuition. This is, in part, Moneyball for metros. Moneyball—Michael Lewis’s popular book and a subsequent movie—documents the unique metrics developed by the Oakland Athletics’ general manager Billy Beane and his staff to

…

), p. 3. 2. Jacobellis v. Ohio, 378 U.S. 184 (1964) (www.law.cornell.edu/supct/html/ historics/USSC_CR_0378_0184_ZS.html). 3. Michael Lewis, Moneyball: The Art of Winning an Unfair Game (New York: W. W. Norton, 2003). 4. Bruce worked with Secretary Cisneros for four years and remembers him

The Signal and the Noise: Why So Many Predictions Fail-But Some Don't

by Nate Silver · 31 Aug 2012 · 829pp · 186,976 words

. Many things that seem predictable over the long run foil our best-laid plans in the meanwhile. The Promise and Pitfalls of “Big Data” The fashionable term now is “Big Data.” IBM estimates that we are generating 2.5 quintillion bytes of data each day, more than 90 percent of which was created in

…

and books, and traveling everywhere from Las Vegas to Copenhagen in pursuit of my investigation, I came to realize that prediction in the era of Big Data was not going very well. I had been lucky on a few levels: first, in having achieved success despite having made many of the mistakes

…

Ioannidis’s hypothesis. They could not replicate about two-thirds of the positive findings claimed in medical journals when they attempted the experiments themselves.40 Big Data will produce progress—eventually. How quickly it does, and whether we regress in the meantime, will depend on us. Why the Future Shocks Us Biologically

…

Red Sox infielder Tim Naehring was one of the best players in the game.) My interest peaked, however, in 2002. At the time Michael Lewis was busy writing Moneyball, the soon-to-be national bestseller that chronicled the rise of the Oakland Athletics and their statistically savvy general manager Billy Beane. Bill James

…

paradigms that each group had adopted to evaluate player performance (statistics, of course, for the statheads, and “tools” for the scouts). In 2003, when Moneyball was published, Michael Lewis’s readers would not have been wrong to pick up on some animosity between the two groups. (The book itself probably contributed to some

…

. “Pedroia made strengths out of things that would be weaknesses for other players.” The Real Lessons of Moneyball “As Michael Lewis said, the debate is over,” Billy Beane declared when we were discussing Moneyball. For a time, Moneyball was very threatening to people in the game; it seemed to imply that their jobs and livelihoods

…

understand all the intricacies of the economy to accurately read those gauges.”55 This kind of statement is becoming more common in the age of Big Data.56 Who needs theory when you have so much information? But this is categorically the wrong attitude to take toward forecasting, especially in a field

…

discoveries. Most are not really contributing much to generating knowledge.” This is why our predictions may be more prone to failure in the era of Big Data. As there is an exponential increase in the amount of available information, there is likewise an exponential increase in the number of hypotheses to investigate

…

, it is hardly a golden road to statistical perfection if you are not using it in a sensible way. As Ioannidis noted, the era of Big Data only seems to be worsening the problems of false positive findings in the research literature. Nor is the frequentist method particularly objective, either in theory

…

2001: A Space Odyssey, which decided it had no more use for the astronauts and tried to suffocate them. As we enter the era of Big Data, with information and processing power increasing at exponential rates, it may be time to develop a healthier attitude toward computers and what they might accomplish

…

that Deep Blue’s programmers used: trial and error. This is at the core of business strategy for the company we most commonly associate with Big Data today. When Trial and Error Works Visit the Googleplex in Mountain View, California, as I did in late 2009, and it isn’t always clear

…

with the crowd; the second for someone who enjoys being an iconoclast. It may be no coincidence that many of the successful investors profiled in Michael Lewis’s The Big Short, who made money betting against mortgage-backed securities and other bubbly investments of the late 2000s, were social misfits to one

…

, if never all its secrets. And yet if science and technology are the heroes of this book, there is the risk in the age of Big Data about becoming too starry-eyed about what they might accomplish. There is no reason to conclude that the affairs of men are becoming more predictable

…

forecasts any time we are presented with new information. A less literal version of this idea is simply trial and error. Companies that really “get” Big Data, like Google, aren’t spending a lot of time in model land.* They’re running thousands of experiments every year and testing their ideas on

…

U.S.-based inventors only as the U.S. Patent and Trade Office also processes many patent applications that originate from abroad. 36. “What Is Big Data?,” IBM. http://www-01.ibm.com/software/data/bigdata/. 37. Chris Anderson, “The End of Theory: The Data Deluge Makes the Scientific Method Obsolete,” Wired

…

, 197–99, 200 of scouts, 91–93, 102 of statheads, 91–93 of weather forecasts, 134–38 Bible, 2 Wicked, 3, 13 Biden, Joseph, 48 Big Data, 9–12, 197, 249–50, 253, 264, 289, 447, 452 Big Short, The (Lewis), 355 Billings, Darse, 324 Bill James Baseball Abstract, The, 77, 78

…

economic growth, 6, 6, 186n economic progress, 7, 112, 243 economics, predictions in, 33, 176–77, 230 actual GDP vs., 191–93, 192, 193, 194 Big Data and, 197 computers in, 289 consensus vs. individual, 197–98, 335 context ignored in, 43 an ever-changing economy, 189–93 economics, predictions in (Cont

…

(Hough), 157 prediction, 1, 16 computers and, 292 consensus, 66–67, 331–32, 335–36 definition of, 452n Enlightenment debates about, 112 in era of big data, 9, 10, 197, 250 fatalism and, 5 feedback on, 183 forecasting vs., 5, 149 by foxes, see foxes of future returns of stocks, 330–31

On the Edge: The Art of Risking Everything

by Nate Silver · 12 Aug 2024 · 848pp · 227,015 words

there,” said Ryder. This surprised a lot of machine learning researchers—and it surprised me. In The Signal and the Noise, I expressed skepticism toward “big data’’ approaches because my experience was that you needed to give models a helping hand, imparting some domain knowledge and some wisdom from our broader understanding

…

of the world. And indeed, when it comes to my specialties like building election models, that objection still holds. Election forecasting is emphatically not a “big data” problem; just the opposite—there’s only one election every four years, so the data is exceptionally sparse. In cases like these, you need to

…

-ask spread: The typically narrow gap between the price at which you’re offered to sell a stock (the “bid”) and buy it (the “ask”). Big data: Very large data sets (e.g., hundreds of millions of customer records) that may be suitable for machine learning. Circa 2010–2016, the term was

…

IN TEXT worth $26.5 billion: “Sam Bankman-Fried,” Forbes, forbes.com/profile/sam-bankman-fried. GO TO NOTE REFERENCE IN TEXT the Met Gala: Michael Lewis, Going Infinite: The Rise and Fall of a New Tycoon, Kindle ed. (New York: W. W. Norton & Company, 2023), 19–20. GO TO NOTE REFERENCE

…

Bernoulli, Nicolaus, 498 Betancourt, Johnny, 332–33 bet sizing, 396, 479 Bezos, Jeff, 277, 410 Bid-ask spread, 444–46, 479 Biden, Joe, 269, 375 big data, 432–33, 479 Billions, 112 Bitcoin bubble in, 6, 306, 307, 307, 310, 312 creation of, 322–23, 496 vs. Ethereum, 324, 326–27 as

Don't Trust Your Gut: Using Data to Get What You Really Want in LIfe

by Seth Stephens-Davidowitz · 9 May 2022 · 287pp · 69,655 words

Index About the Author Also by Seth Stephens-Davidowitz Copyright About the Publisher Introduction: Self-Help for Data Geeks You can make better life decisions. Big Data can help you. We are living through a quiet revolution in our understanding of the most important areas of human life—thanks to the internet

…

others suggest. You dress better. You whiten your teeth. You get a pricey new haircut. But still. The dates, they’re not coming. Insights from Big Data might help. The mathematician and author Christian Rudder studied tens of millions of preferences on OkCupid to learn the qualities of the site’s most

…

traits that the best neighborhoods for raising kids tend to share; in the process, they have upended much conventional wisdom about child-rearing. Thanks to Big Data, we are finally able to tell parents what really matters for raising a successful kid (hint: adult role models) and what matters a lot less

…

friends. You revise your pieces again and again and again. But nothing seems to work. You can’t figure out what you are doing wrong. Big Data has uncovered a likely mistake. A recent study of the career trajectories of hundreds of thousands of painters, led by Samuel P. Fraiberger, has uncovered

…

. Shaving your head or dyeing your hair blue to get more dates is an infield shift of life. Here’s another one, uncovered in the Big Data of sales. Suppose you are trying to sell something. This is an increasingly common experience. As the author Daniel Pink put it in his book

…

words, really like sentences that include the word “you.” Hence, the first paragraph of Don’t Trust Your Gut: “You can make better life decisions. Big Data can help you.” That was a data-driven, not a gut-driven, first paragraph. It was delivered to you in a book written to help

…

that basic facts about the world are hidden from us. There are secrets about who gets what they want in life that are uncovered by Big Data. Take this secret: who is rich? Clearly, knowing this would help any person who wants to earn more money. But knowing this is complicated by

…

“listening to yourself” sounds, frankly, dangerous after reading the latest issue of Psychological Review or Wikipedia’s wonderful article, “List of cognitive biases.” Finally, the Big Data revolution offers us an alternative to listening to ourselves. While our intuitions—and the counsel of our fellow human beings—may have seemed to the

…

data-driven list discussed in this chapter. Recall that I had previously discussed the qualities that make people most desirable as romantic partners, according to Big Data from online dating sites. It turns out that that list—the qualities that are most valued in the dating market, according to

…

Big Data from online dating sites—almost perfectly overlaps with the list of traits in a partner that don’t correlate with long-term relationship happiness, according

…

in their relationship. If I had to sum up, in one sentence, the most important finding in the field of relationship science, thanks to these Big Data studies, it would be something like as follows (call it the First Law of Love): In the dating market, people compete ferociously for mates with

…

, had success by focusing their attention on players, like Youkilis, who lacked the shiny traits that excited teams that didn’t know the data. As Michael Lewis put it, “The human mind played tricks on itself when it relied exclusively on what it saw, and every trick it played was a financial

…

expertise, where your lack of experience may give you a leg up against insiders who are so “constrained” by their “home field”? No. Once again, Big Data definitively rejects this idea. AJKM, in addition to studying the age of entrepreneurs, studied the employment history of entrepreneurs. In particular, over their entire sample

…

patiently builds skills in a narrow field—and then strikes on their own. But, let’s be honest, luck plays a large role in success. Big Data, including deep dives into the sales of hundreds of thousands of artists, can tell us how luck works. And you can use data-driven insights

…

artists who did not travel widely failed to find one of these break-making shows. As I was learning of Fraiberger and his team’s Big Data studies of artists, I was watching Bruce Springsteen’s show Springsteen on Broadway. Springsteen described his experience at the age of twenty-one, when he

…

in bars in his hometown on the Jersey Shore. At this young age, Springsteen intuitively discovered the lesson that Fraiberger and others found in the Big Data: that talent wasn’t enough and that he would have to hustle his way to be discovered. Here’s how Springsteen diagnosed his problem: I

…

Ostrowski and so many other successful artists, made his own luck. Even if you are not an artist, the lessons of artists, as uncovered by Big Data, carry over to many other arenas. If your field is a complete meritocracy, traveling widely to find your break may be unnecessary. The top football

…

work for you. If you haven’t yet landed that no-brainer, life-altering opportunity, you should not be like the unsuccessful artists uncovered by Big Data. You should not stay in a job with unconnected higher-ups who fail to recognize your talent. You should avoid places where talented people stagnate

…

? How happy do you feel, on a scale of 1 to 100? So, did this project succeed in bringing happiness research into the age of Big Data? You betcha. After some years, the Mappiness team had built a dataset that contained more than 3 million happiness measures from more than 60,000

…

as less painful. So, what have we learned from data available from dating sites, tax records, Wikipedia entries, Google searches, and other sources of Big Data? Well, Big Data shows us that we often have notions about the way the world works that are different from the way it actually works. Sometimes the data

…

from modern happiness research into one sentence. We might call such a sentence “the data-driven answer to life.” How might we sum up what Big Data tells us about life’s most important question? What do millions of pings only available thanks to smartphones reveal about the answer to the mystery

…

. Fraiberger et al., “Quantifying reputation and success in art,” Science 362(6416) (2018): 825–29. Oakland A’s: Michael Lewis, Moneyball: The Art of Winning an Unfair Game (New York: Norton, 2004). “more Moneyball than the Moneyball A’s themselves”: Jared Diamond, “How to succeed in baseball without spending money,” Wall Street Journal, October 1

Simple Rules: How to Thrive in a Complex World

by Donald Sull and Kathleen M. Eisenhardt · 20 Apr 2015 · 294pp · 82,438 words

may tell us little about what the future holds. Throwing more data and computing horsepower into the mix doesn’t necessarily resolve this problem, because big data mixed with little theory is a recipe for overfitting. IBM recently released a study, based on a hundred years of data, showing that the increases

…

throw more pitches overall, and disciplined hitting does not erode much with age. These and other insights are at the heart of what author Michael Lewis famously described as moneyball. Moneyball, the book and movie, is the ultimate sports fairy tale, with the A’s playing the role of Cinderella. But unlike Cinderella, the

…

Netflix obtain House of Cards before other buyers were ready to commit. It’s an intriguing rule as well because it exploits Netflix’s unique “big data” capabilities, making it a less risky rule than it would be for a conventional television network. By exploiting its unique capabilities, Netflix proactively redefined the

…

Fit: The Rise, the Fall and the Renaissance of Liz Claiborne,” Academy of Management Journal, 44, no. 4 (2001): 838–57. [>] Alderson, a former Marine: Michael Lewis, Moneyball: The Art of Winning an Unfair Game (New York: W. W. Norton, 2004). [>] These and other insights: Ibid. [>] Enter Farhan Zaidi: Susan Slusser, “A Beautiful

…

/sloan-analytics-farhan-zaidi-on-as-analytics/print/. [>] At Zaidi’s urging: Slusser, “A Beautiful Mind.” The five tools are described more fully in Michael Lewis’s book Moneyball. [>] One was a how-to rule: Alexander Smith, “Billy Beane’s Finest Work Yet: How the Oakland A’s Won the AL West,” BleacherReport

…

, “Giving Viewers What They Want,” New York Times, February 25, 2013, http://www.nytimes.com/2013/02/25/business/media/for-house-of-cards-using-big-data-to-guarantee-its-popularity.html. [>] Its analytics, for instance: Ibid. [>] The company had data: Ibid. [>] Netflix has, however: Robert I. Sutton and Hayagreeva Rao, Scaling

Prediction Machines: The Simple Economics of Artificial Intelligence

by Ajay Agrawal, Joshua Gans and Avi Goldfarb · 16 Apr 2018 · 345pp · 75,660 words

over. More data, better models, and enhanced computers have enabled recent developments in machine learning to improve prediction. Improvements in the collection and storage of big data have provided feedstock for new machine learning algorithms. Compared to their older statistical counterparts, and facilitated by the invention of more suitable processors, the new

…

right data investment decisions, you must understand how prediction machines use data. Prediction Requires Data Before the recent enthusiasm over AI, there was excitement about big data. The variety, quantity, and quality of data have increased substantially over the last twenty years. Images and text are now in digital form, so machines

…

is a way of predicting using a formula instead of a human, the formula should be considered seriously. Poor expert prediction was the centerpiece of Michael Lewis’s Moneyball.4 The Oakland Athletics baseball team faced a problem when, after three of their best players left, the team did not have the financial

…

Summer Research Project on Artificial Intelligence.” 4. Ian Hacking, The Taming of Chance (Cambridge, UK: Cambridge University Press, 1990). Chapter 5 1. Hal Varian, “Beyond Big Data,” lecture, National Association of Business Economists, San Francisco, September 10, 2013. 2. Ngai-yin Chan and Chi-chung Choy, “Screening for Atrial Fibrillation in 13

…

, Thinking, Fast and Slow (New York: Farrar, Strauss and Giroux, 2011); and Dan Ariely, Predictably Irrational (New York: HarperCollins, 2009). 4. Michael Lewis, Moneyball (New York: Norton, 2003). 5. Of course, while Moneyball was based on the use of traditional statistics, it should be no surprise that teams are now looking to machine-learning