Nick Bostrom

Architects of Intelligence
by Martin Ford
Published 16 Nov 2018

Rumelhart prize, the IJCAI award for research excellence, the Killam prize for Engineering, the IEEE Frank Rosenblatt medal, the IEEE James Clerk Maxwell Gold medal, the NEC C&C award, the BBVA award, and the NSERC Herzberg Gold Medal, which is Canada’s top award in science and engineering.” Chapter 5. NICK BOSTROM The concern is not that [an AGI] would hate or resent us for enslaving it, or that suddenly a spark of consciousness would arise and it would rebel, but rather that it would be very competently pursuing an objective that differs from what we really want. Then you get a future shaped in accordance with alien criteria. PROFESSOR, UNIVERSITY OF OXFORD AND DIRECTOR OF THE FUTURE OF HUMANITY INSTITUTE Nick Bostrom is widely recognized as one of the world’s top experts on superintelligence and the existential risks that AI and machine learning could potentially pose for humanity.

…

MARTIN FORD: You’ve written about the risks of creating a superintelligence—an entity that could emerge when an AGI system turns its energies toward improving itself, creating a recursive improvement loop that results in an intelligence that is vastly superior to humans. NICK BOSTROM: Yes, that’s one scenario and one problem, but there are other scenarios and other ways this transition to a machine intelligence era could unfold, and there are certainly, other problems one could be worried about. MARTIN FORD: One idea you’ve focused on especially is the control or alignment problem where a machine intelligence’s goals or values might result in outcomes that are harmful to humanity. Can you go into more detail on what that alignment problem, or control problem, is in layman’s terms? NICK BOSTROM: Well, one distinctive problem with very advanced AI systems that’s different from other technologies is that it presents not only the possibility of humans misusing the technology—that’s something we see with other technologies, of course—but also the possibility that the technology could misuse itself, as it were.

…

MARTIN FORD: Is solving that technical control problem, in terms of how to build a machine that remains aligned with the objective, what you’re working on at the Future of Humanity Institute, and what other think tanks like OpenAI and the Machine Intelligence Research Institute are focusing on? NICK BOSTROM: Yes, that’s right. We do have a group working on that, but we’re also working on other things. We also have a governance of AI group, that is focused on the governance problems related to advances in machine intelligence. MARTIN FORD: Do you think that think tanks like yours are an appropriate level of resource allocation for AI governance, or do you think that governments should jump into this at a larger scale? NICK BOSTROM: I think there could be more resources on AI safety. It’s not actually just us: DeepMind also has an AI safety group that we work with, but I do think more resources would be beneficial.

Surviving AI: The Promise and Peril of Artificial Intelligence
by Calum Chace
Published 28 Jul 2015

He added that many AI researchers are fearful of the backlash that could result if the general public became aware of the possibility that AGI could be created in a couple of decades, and what the implications could be. They are probably right to be concerned. Nick Bostrom’s meta-survey For his 2014 book “Superintelligence”, Nick Bostrom compiled four recent surveys of AI experts, which asked for estimates of the dates at which the probability of AGI being created reached 10%, 50% and 90%. The surveys were carried out in 2012 and 2013. It is unclear how many of the respondents were scientists actively engaged in AI research as opposed to philosophers and theoreticians, but all were either highly-cited authors of published academic work on AI, or professional academics attending conferences on the subject.

…

Rats which can choose between a direct stimulation of their brain’s pleasure centres or an item of food will starve themselves to death. Nick Bostrom calls this idea of causing great harm by mis-understanding the implications of an attempt to do great good “perverse instantiation”. Others might call it the law of unintended consequences, or Sod’s Law. The paperclip maximiser If somebody running a paperclip factory turns out to be the first person to create an AGI and it rapidly becomes a superintelligence, they are likely to have created an entity whose goal is to maximise the efficient production of paperclips. This has become the canonical example of what Nick Bostrom calls “infrastructure profusion”, the runaway train of superintelligence problems.

…

Technological unemployment could force us to adopt an entirely new economic structure, and the creation of superintelligence would be the biggest event in human history. Surviving AI is a first-class introduction to all of this. Brad Feld, co-founder Techstars The promises and perils of machine superintelligence are much debated nowadays. But between the complex and sometimes esoteric writings of AI theorists and academics like Nick Bostrom, and the popular-press prognostications of Elon Musk, Bill Gates and Stephen Hawking, there is something of a gap. Calum Chace’s Surviving AI bridges that gap perfectly. It provides a compact yet rigorous guide to all the major arguments and issues in the field. An excellent resource for those who are new to this topic.

The Doomsday Calculation: How an Equation That Predicts the Future Is Transforming Everything We Know About Life and the Universe
by William Poundstone
Published 3 Jun 2019

Today the list of existential threats is longer, and artificial intelligence (AI) rivals the bomb as a disturber of sleep. It is ironic that an undercurrent of pessimism pervades Silicon Valley, those few golden square miles that, more than any other part of the globe, have been enriched by Bayes’s theorem. Much of the ambivalence about AI has its roots in the work of Norwegian-born philosopher Nick Bostrom, now of Oxford. Bostrom did his doctoral thesis on the doomsday argument and the puzzles of self-sampling. He has been influential in proposing that self-sampling can be applied to diverse scientific questions. Today Bostrom is largely concerned with the risks that may be posed by AI. He believes that the challenge of coding human values into machines is more formidable than is generally appreciated.

…

Had the chosen ball been number 11 or higher, then he could have deduced, with ironclad certainty, that the draw was from the urn with a thousand balls. But there’s a ball number 7 in both urns. The evidence supplied by drawing a number 7 ball is circumstantial but not to be neglected by any fully reasonable party. “Rational belief is constrained,” Nick Bostrom wrote, “not only by chains of deduction but also by the rubber bands of probabilistic inference.” Doubting Thomas Bayes’s Essay found a most influential reader. He was Pierre-Simon Laplace (1749–1827), a French marquis, mathematician, physicist, astronomer, and atheist. Laplace slapped Bayes’s wreck of a paper into rigorous math.

…

Twelve Reasons Why the Doomsday Argument Is Wrong Is the Doomsday Argument correct?” asked Dutch physicist Dennis Dieks. “Nobody I told about the Argument was prepared to think so. But no one was able to offer a clear and convincing view about the nature of the error (if there is one.)” “I have encountered over a hundred objections against the doomsday argument,” wrote Nick Bostrom, “… many of them mutually inconsistent. It is as if the doomsday argument is so counterintuitive (or threatening?) that people reckon that every criticism must be valid.” “Given twenty seconds, many people believe they have found crushing objections,” wrote John Leslie. “At least a dozen times, I too dreamed up what seemed a crushing refutation of it.

The Precipice: Existential Risk and the Future of Humanity
by Toby Ord
Published 24 Mar 2020

The End of the World: The Science and Ethics of Human Extinction. A landmark book that broadened the discussion from nuclear risk to all risks of human extinction, cataloging the threats and exploring new philosophical angles. Nick Bostrom (2002). “Existential Risks: Analyzing Human Extinction Scenarios.” Established the concept of existential risk and introduced many of the most important ideas. Yet mainly of historic interest, for it is superseded by his 2013 paper below. Nick Bostrom (2003). “Astronomical Waste: The Opportunity Cost of Delayed Technological Development.” Explored the limits of what humans might be able to achieve in the future, suggesting that it is of immense importance to accelerate the arrival of the ultimate state of our civilization by even a tiny amount, yet that even this is overshadowed by the importance of increasing the chance we get there at all.

…

We know of almost no limits to what we might ultimately achieve. Human extinction would foreclose our future. It would destroy our potential. It would eliminate all possibilities but one: a world bereft of human flourishing. Extinction would bring about this failed world and lock it in forever—there would be no coming back. The philosopher Nick Bostrom showed that extinction is not the only way this could happen: there are other catastrophic outcomes in which we lose not just the present, but all our potential for the future.2 Consider a world in ruins: an immense catastrophe has triggered a global collapse of civilization, reducing humanity to a pre-agricultural state.

…

But this history shows that existential risk is capable of rousing major global concern, from the elite to the grass roots. Modern thinking on existential risk can be traced through John Leslie, whose 1996 book The End of the World broadened the focus from nuclear war to human extinction in general. After reading Leslie’s work, Nick Bostrom took this a step further: identifying and analyzing the broader class of existential risks that are the focus of this book. Our moral and political traditions have been built over thousands of years. Their focus is thus mostly on timeless issues that have been with us throughout our history. It takes time to incorporate the new possibilities that our age opens up, even when those possibilities are of immense moral significance.

Smarter Than Us: The Rise of Machine Intelligence
by Stuart Armstrong
Published 1 Feb 2014

Some effort has been made to make the AI transition safer. Kudos must be given to Eliezer Yudkowsky and Nick Bostrom, who saw and understood the risks early on. Yudkowsky uses the term “Friendly AI” to describe an AI which does what we want even as it improves its own intelligence. In 2000 he cofounded an organization now called the Machine Intelligence Research Institute (MIRI), which holds math research workshops tackling open problems in Friendly AI theory. (MIRI also commissioned and published this book.) Meanwhile, Nick Bostrom founded the Future of Humanity Institute (FHI), a research group within the University of Oxford.

…

CC BY-NC-SA 3.0 isbn-10: 193931108X isbn-13: 978-1-939311-08-5 (mobi) The Machine Intelligence Research Institute gratefully acknowledges the generous support of all those involved in the publication of this book. Cover photo credit: Google/Connie Zhou. Acknowledgments I wish to acknowledge the help and support of the Future of Humanity Institute, the Oxford Martin School, and the Machine Intelligence Research Institute, as well as the individual advice of Nick Bostrom, Seán Ó hÉigeartaigh, Eliezer Yudkowsky, Kaj Sotala, Luke Muehlhauser, Vincent C. Müller, Anders Sandberg, Lisa Makros, Daniel Dewey, Eric Drexler, Nick Beckstead, Cathy Douglass, and Miriam, Maia, and Kipper Armstrong. Contents Acknowledgments 1. Terminator versus the AI 2. Strength versus Intelligence 3.

…

See MIRI’s work on the fragility of values and FHI’s work on the problem of containing oracles: Luke Muehlhauser and Louie Helm, “The Singularity and Machine Ethics,” in Singularity Hypotheses: A Scientific and Philosophical Assessment, ed. Amnon Eden et al., The Frontiers Collection (Berlin: Springer, 2012); Stuart Armstrong, Anders Sandberg, and Nick Bostrom, “Thinking Inside the Box: Controlling and Using an Oracle AI,” Minds and Machines 22, no. 4 (2012): 299–324, doi:10.1007/s11023-012-9282-2. 2. Stephen M. Omohundro, “The Basic AI Drives,” in Artificial General Intelligence 2008: Proceedings of the First AGI Conference, Frontiers in Artificial Intelligence and Applications 171 (Amsterdam: IOS, 2008), 483–492. 3.

Artificial You: AI and the Future of Your Mind
by Susan Schneider
Published 1 Oct 2019

In the long term, the tables may turn on humans, and the problem may not be what we could do to harm AIs, but what AI might do to harm us. Indeed, some suspect that synthetic intelligence will be the next phase in the evolution of intelligence on Earth. You and I, how we live and experience the world right now, are just an intermediate step to AI, a rung on the evolutionary ladder. For instance, Stephen Hawking, Nick Bostrom, Elon Musk, Max Tegmark, Bill Gates, and many others have raised “the control problem,” the problem of how humans can control their own AI creations, if the AIs outsmart us.2 Suppose we create an AI that has human-level intelligence. With self-improvement algorithms, and with rapid computations, it could quickly discover ways to become vastly smarter than us, becoming a superintelligence—that is, an AI that outthinks us in every domain.

…

Because it is superintelligent, we probably can’t control it. It could, in principle, render us extinct. This is only one way that synthetic beings could supplant organic intelligences; alternatively, humans may merge with AI through cumulatively significant brain enhancements. The control problem has made world news, fueled by Nick Bostrom’s recent bestseller: Superintelligence: Paths, Dangers and Strategies.3 What is missed, however, is that consciousness could be central to how AI values us. Using its own subjective experience as a springboard, superintelligent AI could recognize in us the capacity for conscious experience. After all, to the extent we value the lives of nonhuman animals, we tend to value them because we feel an affinity of consciousness—thus most of us recoil from killing a chimp, but not from eating an orange.

…

I don’t know whether many of those who publicize the idea of a mind-machine merger, such as Elon Musk and Michio Kaku, have considered these classic positions on personal identity. But they should. It is a bad idea to ignore this debate. One could be dismayed, at some later point, to learn that a technology one advocated actually had a tremendously negative impact on human flourishing. In any case, both Kurzweil and Nick Bostrom have considered the issue in their work. They, like many other transhumanists, adopt a novel and intriguing version of the psychological continuity view; in particular, they adopt a computational, or patternist, account of continuity. ARE YOU A SOFTWARE PATTERN? Patternism’s point of departure is the computational theory of mind, which I introduced earlier.

An Optimist's Tour of the Future
by Mark Stevenson
Published 4 Dec 2010

And if life expectancies keep creeping up the way they have been, fifty years from now I could have up to another twelve years on the planet. It’s not infeasible I’ll make it past a hundred. I find this encouraging: I’m still single with no kids, but it appears there’s plenty of time to raise a family, maybe even understand cricket. Nick Bostrom, founder of Oxford University’s Future of Humanity Institute, agrees. In fact, he thinks I could live not for a hundred years but for thousands. And he’s not joking. Bostrom is an advocate for transhumanism. Described as ‘the world’s most dangerous idea,’ the concept got its modern name in Religion Without Revelation, written in 1927 by Julian Huxley.

…

When no one raises their hand he continues, ‘I put it to you that the main reason we think malaria is a bad thing is because of a characteristic it shares with ageing – and here is that characteristic’ – at which point he shows a slide bearing the words ‘Because it kills people!!!!!!’ ‘The only real difference is that ageing kills considerably more people.’ De Grey proceeds to compare ageing to fox hunting, claiming they are both ‘traditional,’ ‘keep the numbers down’ and, his punch line, ‘fundamentally barbaric.’ At the same conference, Nick Bostrom later stated, ‘Death is a big problem. If you look at the statistics the odds are not very favourable. So far most people who have lived have also died.’ But banishing ageing is only one half of the transhumanists’ truly radical agenda. They also advocate a new vision of the human being: the transhuman – a human not only saved from ageing, but enhanced beyond our current ‘biological limitations.’

…

Unlike natural evolution, which requires centuries to fumble an improvement, our man-made technologies quickly keep on doubling in power and halving in price, ushering the prospect of the ‘disabled’ out-performing what we now consider the able-bodied. Will I be buying enhanced cochlear implants in a few years the way I bought a mobile phone in the nineties? Will I buy a new body part grown from my own stem cells to keep me young? How far can this go? How long can I live? How enhanced could I be? Nick Bostrom suggests that the answers to these last two questions could respectively be ‘a very long time’ and ‘as much as you like.’ Take, for instance, the idea of ‘longevity escape velocity,’ seen by many transhumanists as one of the more convincing arguments for the prospect of immortality. This is the proposition that instead of our life expectancies gaining a quarter of a year for every year that passes, we reach a point where medical advances raise our life expectancy by over a year for every year that goes by.

The Singularity Is Nearer: When We Merge with AI
by Ray Kurzweil
Published 25 Jun 2024

Louis, updated April 12, 2023, https://fred.stlouisfed.org/series/CPIAUCSL; Marguerite Zientara, “Herman Hollerith: Punched Cards Come of Age,” Computerworld 15, no. 36 (September 7, 1981): 35, https://books.google.com/books?id=tk74jLc6HggC&pg=PA35&lpg=PA35; Frank da Cruz, “Hollerith 1890 Census Tabulator,” Columbia University Computing History, April 17, 2021, http://www.columbia.edu/cu/computinghistory/census-tabulator.html. BACK TO NOTE REFERENCE 76 Nick Bostrom, “Nick Bostrom The Intelligence Explosion Hypothesis eDay 2012,” eAcast55, YouTube video, August 9, 2015, https://www.youtube.com/watch?v=VFE-96XA92w. BACK TO NOTE REFERENCE 77 DeepMind, “AlphaGo,” DeepMind, accessed November 20, 2021, https://deepmind.com/research/case-studies/alphago-the-story-so-far.

…

George Coyne, see Roberto Trotta, “What Is the Anthropic Principle?,” Physics World, YouTube video, January 17, 2014, https://www.youtube.com/watch?v=dWkJ8Pl-8l8; Richard Dawkins and George Coyne, “The Anthropic Principle,” jlcamelo, July 9, 2010, https://youtu.be/lm9ZtYkdkEQ?t=102; Joe Rogan and Nick Bostrom, “Joe Rogan Experience #1350—Nick Bostrom,” PowerfulJRE, September 11, 2019, https://web.archive.org/web/20190918171740/https://www.youtube.com/watch?v=5c4cv7rVlE8; Christopher Smeenk and George Ellis, “Philosophy of Cosmology,” in Stanford Encyclopedia of Philosophy, ed. Edward N. Zalta (Winter 2017), https://plato.stanford.edu/entries/cosmology.

…

Because Go has a vastly larger number of possible moves than chess, and it is harder to judge whether any given move is a good one, the AI approaches that had worked to beat human chess grandmasters were making almost no progress on Go. Even optimistic experts judged that the problem wouldn’t be cracked until the 2020s at best. (As of 2012, for example, leading AI futurist Nick Bostrom speculated that Go would not be mastered by artificial intelligence until about 2022.)[77] But then, in 2015–16, Alphabet subsidiary DeepMind created AlphaGo, which used a “deep reinforcement learning” method in which a large neural net processed its own games and learned from its successes and failures.[78] It started with a huge number of recorded human Go moves and then played itself many times until the version AlphaGo Master was able to beat the world human Go champion, Ke Jie.[79] A more significant development occurred a few months later with AlphaGo Zero.

The Most Good You Can Do: How Effective Altruism Is Changing Ideas About Living Ethically
by Peter Singer
Published 1 Jan 2015

National Aeronautics and Space Administration, Near Earth Object Program, Torino Hazard Scale, http://neo.jpl.nasa.gov/torino_scale.html. 2. Nick Bostrom, “Existential Risks: Analyzing Human Extinction Scenarios And Related Hazards,” Journal of Evolution and Technology 9 (2002), available at www.jetpress.org/volume9/risks.html. 3. Hans Kristensen and Robert Norris, “Global Nuclear Weapons Inventories, 1945–2013,” Bulletin of the Atomic Scientists 69 (2013): 75–81. 4. Binyamin Appelbaum, “As U.S. Agencies Put More Value on a Life, Businesses Fret,” New York Times, February 16, 2011. 5. Derek Parfit, Reasons and Persons (Oxford: Clarendon Press, 1984), 453–54. 6. Nick Bostrom, “Existential Risk Prevention as Global Priority,” Global Policy 4 (2013): 16. 7.

…

Nick Bostrom, “Existential Risk Prevention as Global Priority,” Global Policy 4 (2013): 16. 7. Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford: Oxford University Press, 2014), 103; see also Bostrom, “Existential Risk Prevention as Global Priority,” 18–19. Nick Beckstead argues for a similar position in “The Overwhelming Importance of Shaping the Far Future” (PhD diss., Rutgers University, 2013). 8. Nick Bostrom, Existential Risk FAQ, version 1.2 (2013), no. 9, http://www.existential-risk.org/faq.html. 9. Henry Sidgwick, The Methods of Ethics, 7th ed. (London: Macmillan, 1907), 415. 10. See Peter Singer, Practical Ethics, 3d ed.

…

I very much hope that his gift will do more good than anything else he could have done with that amount of money, although I persist in thinking that, at the time he made it, he could not reasonably have predicted so fortunate an outcome. Many people have read drafts of the book, or parts of it, and offered helpful comments or responded to my queries. I want to thank, in particular, Anthony Appiah, Paul Bloom, Jon Bockman and Allison Smith for Animal Charity Evaluators, Paul van den Bosch for Give A Kidney, Nick Bostrom, Richard Butler-Bowdon, Di Franks for Living Kidney Donation, Holden Karnofsky for GiveWell, Katarzyna de Lazari-Radek, Peter Hurford, Michael Liffman, Will MacAskill, Yaw Nyarko, Caleb Ontiveros, Toby Ord, Theron Pummer, Rob Reich, Susanne Roff, Agata Sagan, and Aleksandra Taranow. Special thanks to Mona Fixdal, whose splendid assistance in the preparation of my online course “Practical Ethics” made it possible for me to devote more time to writing this book.

Human Compatible: Artificial Intelligence and the Problem of Control
by Stuart Russell
Published 7 Oct 2019

If human happiness requires being in the top 1 percent, then 99 percent of humans are going to be unhappy, even when the bottom 1 percent has an objectively splendid lifestyle.60 It will be important, then, for our cultures to gradually down-weight pride and envy as central elements of perceived self-worth. As Nick Bostrom puts it at the end of his book Superintelligence, success in AI will yield “a civilizational trajectory that leads to a compassionate and jubilant use of humanity’s cosmic endowment.” If we fail to take advantage of what AI has to offer, we will have only ourselves to blame. 4 MISUSES OF AI A compassionate and jubilant use of humanity’s cosmic endowment sounds wonderful, but we also have to reckon with the rapid rate of innovation in the malfeasance sector.

…

If there are no humans and no other conscious entities whose subjective experience matters to us, there is nothing of value occurring. 6 THE NOT-SO-GREAT AI DEBATE The implications of introducing a second intelligent species onto Earth are far-reaching enough to deserve hard thinking.”1 So ended The Economist magazine’s review of Nick Bostrom’s Superintelligence. Most would interpret this as a classic example of British understatement. Surely, you might think, the great minds of today are already doing this hard thinking—engaging in serious debate, weighing up the risks and benefits, seeking solutions, ferreting out loopholes in solutions, and so on.

…

It’s too soon to worry about it It’s common to see sober-minded people seeking to assuage public concerns by pointing out that because human-level AI is not likely to arrive for several decades, there is nothing to worry about. For example, the AI100 report says there is “no cause for concern that AI is an imminent threat to humankind.” This argument fails on two counts. The first is that it attacks a straw man. The reasons for concern are not predicated on imminence. For example, Nick Bostrom writes in Superintelligence, “It is no part of the argument in this book that we are on the threshold of a big breakthrough in artificial intelligence, or that we can predict with any precision when such a development might occur.” The second is that a long-term risk can still be cause for immediate concern.

The Transhumanist Reader
by Max More and Natasha Vita-More
Published 4 Mar 2013

Bostrom, Nick and Sandberg, Anders (2009) “The Wisdom of Nature: An Evolutionary Heuristic for Human Enhance­ment.” In Julian Savulescu and Nick Bostrom, eds., Human Enhancement. Oxford: Oxford University Press, pp. 375–416. Cakic, Vince (2009) “Smart Drugs for Cognitive Enhancement: Ethical and Pragmatic Considerations in the Era of Cosmetic Neurology.” Journal of Medical Ethics 35 (October 29), pp. 611–615. Caplan, Arthur L. (2009) “Good, Better, or Best.” In Julian Savulescu and Nick Bostrom, eds., Human Enhancement. Oxford: Oxford University Press, pp. 199–209. Glover, Jonathan (1984) What Sort of People Should There Be?

…

Strong Superhumanity and the Best We Can Ask For 36 An Overview of Models of Technological Singularity Introduction Definitions of Technological Singularity Models Accelerating Change Discussion 37 A Critical Discussion of Vinge’s Singularity Concept Comment by David Brin: Singularities Comment by Damien Broderick Comment by Nick Bostrom: Singularity and Predictability Comment by Alexander Chislenko: Singularity as a Process, and the Future Beyond Comment by Robin Hanson: Some Skepticism Comment by Max More: Singularity Meets Economy Comment by Michael Nielsen Comment by Anders Sandberg: Singularity and the Growth of Differences Part IX The World’s Most Dangerous Idea 38 The Great Transition What is Transhumanism?

…

Russell Blackford, PhD, is Editor-in-Chief, Journal of Evolution & Technology. He authored Freedom of Religion and the Secular State (Wiley-Blackwell, 2012) and An Evil Hour (I Books, 2003); and co-authored with Van Ikin and Sean McMullen Strange Constellations: A History of Australian Science Fiction (Praeger, 1999). Nick Bostrom, PhD, is Director, Future of Humanity Institute, Oxford University. He has ­written numerous papers and authored Anthropic Bias (Routledge, 2010); and co-edited with Julian Savulescu Human Enhancement (Oxford University Press, 2011) and Global Catastrophic Risks with Milan Cirkovic (Oxford University Press, 2011).

The Simulation Hypothesis
by Rizwan Virk
Published 31 Mar 2019

The fundamental question raised by the simulation hypothesis is this: Are we all actually characters living inside some kind of giant, massively multiplayer online video game, a simulated reality that is so well rendered that we cannot distinguish it from “physical reality”? Although Oxford philosopher Nick Bostrom coined the term the Simulation Argument in a landmark paper in 2003, the idea of living in a simulated reality has been around for a long time in science, religion and fiction. The reality of the world around us is something that philosophers have debated for some time. Thousands of years ago in The Republic, Plato described his analogy of the cave.

…

He conjectured that if video game technology continued its rapid pace of improvement, then it was inevitable that we would be able to create hyper-realistic simulations that would be indistinguishable from physical reality. He concluded that, assuming technology keeps developing as it has in the past, the chance that we aren’t in a simulation was “one in billions.” Oxford’s Nick Bostrom, in his 2003 paper which popularized the Simulation Argument, refers to a species which is able to build these hyper-realistic simulations as becoming “post-human.” I like to say that such a civilization has passed the “Simulation Point.” This would be the point in a technological civilization’s development when it possesses the ability to create hyper-realistic simulations.

…

There are metaphysical as well as technical and philosophical questions that go along with this, which will need to be resolved as we travel through these stages on the road to the simulation point. If a civilization has mastered these two stages, then it has pretty much reached the point of being able to create a Matrix-like reality, and has reached the simulation point. Next, we will explore some of the implications of reaching this point, and go through Nick Bostrom’s original argument that we may be living in a simulation all along as simulated entities. Chapter 4 Stage 11: The Simulation Point, Ancestor Simulations and Beyond At some point after completing the previous stages, a civilization should have reached the simulation point—the point at which it can create simulations that are virtually indistinguishable from a base physical reality.

What to Think About Machines That Think: Today's Leading Thinkers on the Age of Machine Intelligence
by John Brockman
Published 5 Oct 2015

Should I be bothered by the prospect of thinking machines? Probably. Certainly Nick Bostrom thinks I should. Our focus on getting computers to exhibit human-level intelligence is, he thinks, misguided. We view machines that can pass the Turing Test as the ultimate destination of Doug Engelbart’s quest. But Bostrom thinks that passing the test is just a waypoint on the road to something much more worrying. “The train,” he says, “might not pause or even decelerate at Humanville Station. It is likely to swoosh right by.”4 He’s right: I should be careful what I wish for. IT’S STILL EARLY DAYS NICK BOSTROM Professor, Oxford University; director, Future of Humanity Institute, Oxford Martin School; author, Superintelligence: Paths, Dangers, Strategies First, what I think about humans who think about machines that think: I think that for the most part we’re too quick to form an opinion on this difficult topic.

…

BROOKS Mistaking Performance for Competence TERRENCE J. SEJNOWSKI AI Will Make You Smarter SETH LLOYD Shallow Learning CARLO ROVELLI Natural Creatures of a Natural World FRANK WILCZEK Three Observations on Artificial Intelligence JOHN NAUGHTON When I Say “Bruno Latour,” I Don’t Mean “Banana Till” NICK BOSTROM It’s Still Early Days DONALD D. HOFFMAN Evolving AI ROGER SCHANK Machines That Think Are in the Movies JUAN ENRIQUEZ Head Transplants? ESTHER DYSON AI/AL TOM GRIFFITHS Brains and Other Thinking Machines MARK PAGEL They’ll Do More Good Than Harm ROBERT PROVINE Keeping Them on a Leash SUSAN BLACKMORE The Next Replicator TIM O’REILLY What If We’re the Microbiome of the Silicon AI?

…

PREFACE: THE 2015 EDGE QUESTION In recent years, the 1980s-era philosophical discussions about artificial intelligence (AI)—whether computers can “really” think, be conscious, and so on—have led to new conversations about how we should deal with the forms of artificial intelligence that many argue have already been implemented. These AIs, if they achieve “superintelligence” (per Nick Bostrom’s 2014 book of that name), could pose existential risks, leading to what Martin Rees has termed “our final hour.” Stephen Hawking recently made international headlines when he told the BBC that “the development of full artificial intelligence could spell the end of the human race.” THE EDGE QUESTION—2015 WHAT DO YOU THINK ABOUT MACHINES THAT THINK?

Our Final Invention: Artificial Intelligence and the End of the Human Era
by James Barrat
Published 30 Sep 2013

We don’t hate mice or monkeys, yet we treat them cruelly. Superintelligent AI won’t have to hate us to destroy us. After intelligent machines have already been built and man has not been wiped out, perhaps we can afford to anthropomorphize. But here on the cusp of creating AGI, it is a dangerous habit. Oxford University ethicist Nick Bostrom puts it like this: A prerequisite for having a meaningful discussion of superintelligence is the realization that superintelligence is not just another technology, another tool that will add incrementally to human capabilities. Superintelligence is radically different. This point bears emphasizing, for anthropomorphizing superintelligence is a most fecund source of misconceptions.

…

We’re unlikely to survive an introduction as abrupt as nuclear fission’s. Chapter Two The Two-Minute Problem Our approach to existential risks cannot be one of trial-and-error. There is no opportunity to learn from errors. The reactive approach—see what happens, limit damages, and learn from experience—is unworkable. —Nick Bostrom, faculty of Philosophy, Oxford University The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else. —Eliezer Yudkowsky, research fellow, Machine Intelligence Research Institute Artificial superintelligence does not yet exist, nor does artificial general intelligence, the kind that can learn like we do and will in many senses match and exceed most human intelligence.

…

That is, we don’t want an AI that meets our short-term goals—please save us from hunger—with solutions detrimental in the long term—by roasting every chicken on earth—or with solutions to which we’d object—by killing us after our next meal. As an example of unintended consequences, Oxford University ethicist Nick Bostrom suggests the hypothetical “paper clip maximizer.” In Bostrom’s scenario, a thoughtlessly programmed superintelligence whose programmed goal is to manufacture paper clips does exactly as it is told without regard to human values. It all goes wrong because it sets about “transforming first all of earth and then increasing portions of space into paper clip manufacturing facilities.”

Possible Minds: Twenty-Five Ways of Looking at AI
by John Brockman
Published 19 Feb 2019

THE EVOLVING AI NARRATIVE Things have changed—and they remain the same. Now AI is everywhere. We have the Internet. We have our smartphones. The founders of the dominant companies—the companies that hold “the whip that lashes us”—have net worths of $65 billion, $90 billion, $130 billion. High-profile individuals such as Elon Musk, Nick Bostrom, Martin Rees, Eliezer Yudkowsky, and the late Stephen Hawking have issued dire warnings about AI, resulting in the ascendancy of well-funded institutes tasked with promoting “Nice AI.” But will we, as a species, be able to control a fully realized, unsupervised, self-improving AI? Wiener’s warnings and admonitions in The Human Use of Human Beings are now very real, and they need to be looked at anew by researchers at the forefront of the AI revolution.

…

In this paper, the message is even clearer: If we use, to achieve our purposes, a mechanical agency with whose operation we cannot efficiently interfere . . . we had better be quite sure that the purpose put into the machine is the purpose which we really desire. In my view, this is the source of the existential risk from superintelligent AI cited in recent years by such observers as Elon Musk, Bill Gates, Stephen Hawking, and Nick Bostrom. PUTTING PURPOSES INTO MACHINES The goal of AI research has been to understand the principles underlying intelligent behavior and to build those principles into machines that can then exhibit such behavior. In the 1960s and 1970s, the prevailing theoretical notion of intelligence was the capacity for logical reasoning, including the ability to derive plans of action guaranteed to achieve a specified goal.

…

The AI100 report, for example, assures us, “Contrary to the more fantastic predictions for AI in the popular press, the Study Panel found no cause for concern that AI is an imminent threat to humankind.” This argument simply misstates the reasons for concern, which are not predicated on imminence. In his 2014 book, Superintelligence: Paths, Dangers, Strategies, Nick Bostrom, for one, writes, “It is no part of the argument in this book that we are on the threshold of a big breakthrough in artificial intelligence, or that we can predict with any precision when such a development might occur.” You’re just a Luddite. It’s an odd definition of Luddite that includes Turing, Wiener, Minsky, Musk, and Gates, who rank among the most prominent contributors to technological progress in the 20th and 21st centuries.* Furthermore, the epithet represents a complete misunderstanding of the nature of the concerns raised and the purpose for raising them.

The Myth of Artificial Intelligence: Why Computers Can't Think the Way We Do
by Erik J. Larson
Published 5 Apr 2021

Yet the possibilities for future AI systems are limited by what we currently know about the nature of intelligence, whether we like it or not. And here we should say it directly: all evidence suggests that human and machine intelligence are radically different. The myth of AI insists that the differences are only temporary, and that more powerful systems will eventually erase them. Futurists like Ray Kurzweil and philosopher Nick Bostrom, prominent purveyors of the myth, talk not only as if human-level AI were inevitable, but as if, soon after its arrival, superintelligent machines would leave us far behind. This book explains two important aspects of the AI myth, one scientific and one cultural. The scientific part of the myth assumes that we need only keep “chipping away” at the challenge of general intelligence by making progress on narrow feats of intelligence, like playing games or recognizing images.

…

Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an ‘intelligence explosion,’ and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control.”1 Oxford philosopher Nick Bostrom would return to Good’s theme decades later, with his 2014 best seller Superintelligence: Paths, Dangers, Strategies, making the same case that the achievement of AI would as a consequence usher in greater-than-human intelligence in an escalating process of self-modification. In ominous language, Bostrom echoes Good’s futurism about the arrival of superintelligent machines: Before the prospect of an intelligence explosion, we humans are like small children playing with a bomb.

…

Again, deep problems with understanding language using computers have persisted. A simple way to see the point here is to turn back to the Turing test, and reconsider it in light of the history of AI and the many, mostly fruitless attempts to solve the problems it presents or even to make substantive progress. Futurists like Nick Bostrom, as well as the larger scientific AI community, likely wish that the public would just forget about the test. It’s not—as is sometimes heard—that the test is flawed or unhelpful. Quite simply, it’s too hard. THE TURING TEST Given a view from thirty thousand feet, it did seem that computers were getting smarter as AI progressed from its genesis in Turing’s early work and the kickoff conference at Dartmouth.

Overcomplicated: Technology at the Limits of Comprehension
by Samuel Arbesman
Published 18 Jul 2016

Living with Complexity by Don Norman examines the origins of (and need for) complexity, particularly from the perspective of design. The Techno-Human Condition by Braden R. Allenby and Daniel Sarewitz is a discussion of how to grapple with coming technological change and is particularly intriguing when it discusses “wicked complexity.” Superintelligence by Nick Bostrom explores the many issues and implications related to the development of superintelligent machines. The Works, The Heights, and The Way to Go by Kate Ascher examine how cities, skyscrapers, and our transportation networks, respectively, actually work. Beautifully rendered and fascinating books.

…

the writer Quinn Norton has noted: Quinn Norton, “Everything is Broken,” The Message, May 20, 2014, https://medium.com/message/81e5f33a24e1. Langdon Winner notes in his book: Winner, Autonomous Technology, 290–91. computer scientist Danny Hillis argues: Danny Hillis, “The Age of Digital Entanglement,” Scientific American, September 2010, 93. Take the so-called Flash Crash: Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford, UK: Oxford University Press, 2014), 17. It is still not entirely clear, however, what caused the Flash Crash. Understanding something in a “good enough” way: See also César Hidalgo, Why Information Grows: The Evolution of Order, from Atoms to Economies (New York: Basic Books, 2015).

…

Peter Norvig, Google’s director of research: Peter Norvig, “On Chomsky and the Two Cultures of Statistical Learning,” accessed April 30, 2015, http://norvig.com/chomsky.html. great, though apocryphal, story: There seem to be many versions of this apocryphal machine translation tale. What techniques are used by experts: Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford, UK: Oxford University Press, 2014), 15. say, 99.9 percent of the time: I made these numbers up for effect, but if any linguist wants to chat, please reach out! “based on millions of specific features”: Alon Halevy et al., “The Unreasonable Effectiveness of Data,” IEEE Intelligent Systems 24, no. 2 (2009): 8–12.

Warnings
by Richard A. Clarke
Published 10 Apr 2017

Tsachi Ein-Dor drew a similar conclusion, that people with higher anxiety levels tend to detect threats sooner and warn others. 15. In spite of the persistent view that science is the ultimate model of objectivity, scientists are, after all, human beings and still subject to what philosopher Nick Bostrom terms the selection effect. The scientists and their instruments reflect the fact that “all observations require the existence of an appropriately positioned observer.” See Nick Bostrom, Anthropic Bias: Observation Selection Effects in Science and Philosophy (New York: Routledge, 2002). 16. Charles S. Taber and Milton Lodge, “Motivated Skepticism in the Evaluation of Political Beliefs,” American Journal of Political Science 50, no. 3 (2006): 755–69, doi:10.1111/j.1540–5907.2006.00214. 17.

…

Arthur Samuel offered this definition of machine learning in 1959. 4. Superintelligence is a term made popular by philosopher Nick Bostrom and is often also called artificial general intelligence. 5. Referenced from Luke Muehlhauser’s fantastic work which was a great guide for the authors. 6. James Barrat, Our Final Invention: Artificial Intelligence and the End of the Human Era (New York: Thomas Dunne Books, 2013). Barrat’s book was an important source for the authors. 7. Nick Bostrom, “Ethical Issues in Advanced Artificial Intelligence,” http://www.nickbostrom.com/ethics/ai.html (accessed Nov. 9, 2016).

…

It will carve new paths of discovery in fields yet undiscovered, fueled by perpetual improvements to its own source code and the creation of new robotic tools. Artificial intelligence has the potential to be dramatically more powerful than any previous scientific advance. Superintelligence, according to Nick Bostrom at Oxford, is “not just another technology, another tool that will add incrementally to human capabilities.” It is, he says, “radically different,” and it “may be the last invention humans ever need to make.”7 From Yudkowsky’s whimsical graph above, we can get a hint of the predicted power of superintelligence (here called “recursively self-improved AI”).

On the Edge: The Art of Risking Everything
by Nate Silver
Published 12 Aug 2024

GO TO NOTE REFERENCE IN TEXT often surprisingly hard: Holden Karnofsky, “Why Describing Utopia Goes Badly,” Cold Takes (blog), December 7, 2021, cold-takes.com/why-describing-utopia-goes-badly. GO TO NOTE REFERENCE IN TEXT provocative or offensive: Nick Bostrom, “Apology for Old Email,” Nick Bostrom’s Home Page, January 9, 2023, nickbostrom.com/oldemail.pdf. GO TO NOTE REFERENCE IN TEXT Hanson founded his blog: Robin Hanson, “How to Join,” Overcoming Bias (blog), July 22, 2023, overcomingbias.com/p/introductionhtml. GO TO NOTE REFERENCE IN TEXT Hanson and Yudkowsky: Per interview with Robin Hanson.

…

A Field Guide to EAs and Rationalists Singer Stream Yudkowsky-Hanson Stream Key Figures Peter Singer, Will MacAskill, Toby Ord Robin Hanson, Eliezer Yudkowsky, Nick Bostrom, Scott Alexander Favorite Subjects Animal welfare, global poverty reduction, effective giving, existential risk Artificial intelligence, futurism, prediction markets, cognitive biases, existential risk Political Orientation Center left, progressive, relatively well aligned with U.S.

…

There are alternatives, though—mathematical ways to do philosophy that don’t just rely on summing up all utils across the known universe. The late Princeton philosopher John Rawls, in A Theory of Justice, proposed maximizing the well-being of the least well-off person in society. In principle, you could build some sort of mathematical model to do that. Nick Bostrom, an Oxford philosopher who is associated with EA although probably falls more into the rationalist camp, told me about his idea he called a “moral parliament,” “where I’m uncertain about which ethical theory is correct and I would give them each some weight. I mean, the utilitarians would be in there with their delegates to this parliament.

The Alignment Problem: Machine Learning and Human Values
by Brian Christian
Published 5 Oct 2020

See also the contemporaneous and closely related “expert iteration” (“ExIt”) algorithm in Anthony, Tian, and Barber, “Thinking Fast and Slow with Deep Learning and Tree Search.” 91. Shead, “DeepMind’s Human-Bashing AlphaGo AI Is Now Even Stronger.” 92. Aurelius, The Emperor Marcus Aurelius. 93. Andy Fitch, “Letter from Utopia: Talking to Nick Bostrom,” BLARB (blog), November 24, 2017, https://blog.lareviewofbooks.org/interviews/letter-utopia-talking-nick-bostrom/. 94. Blaise Agüera y Arcas, “The Better Angels of our Nature” (lecture), February 16, 2017, VOR: Superintelligence, Mexico City. 95. Yudkowsky, “Coherent Extrapolated Volition.” See also Tarleton, “Coherent Extrapolated Volition.” 96.

…

We want to get from point A to point B as quickly as possible, though not by going over the speed limit—or, rather, not by going too far over the speed limit, unless for some reason we have to—and by staying centered in our lane, unless there’s a cyclist or a stopped car—and not passing cars on the right, unless it’s safer to do so than not to do so, and so on. It’s hard to try to formalize all of this into some kind of objective function that we then tell the system to optimize. Better, in cases like this, to use what the Future of Humanity Institute’s Nick Bostrom calls “indirect normativity”33—a way to get the system aligned with our desires without articulating them down to the last minutia. In this case, what we want is to say something like “Watch how I drive. Do it like this.” This was, as it turns out, one of the very first ideas in self-driving cars—and, to this day, still one of the best.

…

—MARCUS AURELIUS92 For a growing number of philosophers and computer scientists concerned with the longer-term future, the prospect of flexibly intelligent and flexibly capable systems, into which we must impart extremely complex behaviors and values, raises not only technical problems but something much deeper. There are two primary challenges here. The first is that the things we want are very difficult to simply state outright—even in words, let alone in a more numerical form. As the Future of Humanity Institute’s Nick Bostrom notes, “It seems completely impossible to write down a list of everything we care about.”93 In this case, we have already seen how learning by imitation can succeed in domains where it is effectively impossible to explicitly impart every rule and consideration and degree of emphasis for what makes someone an expert driver or an expert Go player.

Robot Rules: Regulating Artificial Intelligence
by Jacob Turner
Published 29 Oct 2018

Good predicted that “…an ultraintelligent machine could design even better machines; there would then unquestionably be an ‘intelligence explosion,’ and the intelligence of man would be left far behind”.105 This remains the operating assumption of some AI experts today. In his influential book, Superintelligence Nick Bostrom describes the consequences of the AI explosion in dramatic terms, explaining that in some models it could be a matter of days between the development of the initial “seed” superintelligence and its spawn becoming so powerful that no human-controlled force is able to reassert control: “Once artificial intelligence reaches human level, there will be a positive feedback loop that will give the development a further boost.

…

Technology has always been a double edged sword, since fire kept us warm but also burned down our villages”.119 Similarly, engineer and roboethicist Alan Winfield said in a 2014 article: “If we succeed in building human equivalent AI and if that AI acquires a full understanding of how it works, and if it then succeeds in improving itself to produce super-intelligent AI, and if that super-AI, accidentally or maliciously, starts to consume resources, and if we fail to pull the plug, then, yes, we may well have a problem. The risk, while not impossible, is improbable”.120 Fundamentally, optimists think humanity can and will overcome any challenges AI poses. The pessimists include Nick Bostrom, whose “paperclip machine” thought experiment imagines an AI system asked to make paperclips which decides to seize and consume all resources in existence, in its blind aderence to that goal.121 Bostrom contemplates a form of superintelligence which is so powerful that humanity has no chance of stopping it from destroying the entire universe.

…

, Oxford Martin Programme on the Impacts of Future Technology Working Paper, September 2013, http://​www.​oxfordmartin.​ox.​ac.​uk/​downloads/​academic/​future-of-employment.​pdf, accessed 1 June 2018. See also Daniel Susskind and Richard Susskind, The Future of the Professions : How Technology Will Transform the Work of Human Experts (Oxford: Oxford University Press, 2015). 5See Nick Bostrom, Superintelligence (Oxford: Oxford University Press, 2014). 6See Ray Kurzweil, The Singularity Is Near: When Humans Transcend Biology (New York: Viking Press, 2005). 7Several nineteenth-century thinkers including Charles Babbage and Ada Lovelace arguably predicted the advent of AI and even prepared designs for machines capable of carrying out intelligent tasks.

How to Create a Mind: The Secret of Human Thought Revealed
by Ray Kurzweil
Published 13 Nov 2012

The report projects ongoing exponential gains in all of these areas of capability and argues that the requirements to simulate the human brain at a high level of detail are coming into place. An outline of the technological capabilities needed for whole brain emulation, in Whole Brain Emulation: A Roadmap by Anders Sandberg and Nick Bostrom. An outline of Whole Brain Emulation: A Roadmap by Anders Sandberg and Nick Bostrom. Neural Nets In 1964, at the age of sixteen, I wrote to Frank Rosenblatt (1928–1971), a professor at Cornell University, inquiring about a machine called the Mark 1 Perceptron. He had created it four years earlier, and it was described as having brainlike properties.

…

Wedeen and his colleagues at Massachusetts General Hospital showing a highly regular gridlike structure of the wiring of the neocortex that I described in chapter 4 is one early result from this project. Oxford University computational neuroscientist Anders Sandberg (born in 1972) and Swedish philosopher Nick Bostrom (born in 1973) have written the comprehensive Whole Brain Emulation: A Roadmap, which details the requirements for simulating the human brain (and other types of brains) at different levels of specificity from high-level functional models to simulating molecules.8 The report does not provide a timeline, but it does describe the requirements to simulate different types of brains at varying levels of precision in terms of brain scanning, modeling, storage, and computation.

…

The truth can be discovered only by finding an explanation that overrides—transcends—seeming differences, especially for fundamental questions of meaning and purpose. This is how I resolve the Western-Eastern divide on consciousness and the physical world. In my view, both perspectives have to be true. On the one hand, it is foolish to deny the physical world. Even if we do live in a simulation, as speculated by Swedish philosopher Nick Bostrom, reality is nonetheless a conceptual level that is real for us. If we accept the existence of the physical world and the evolution that has taken place in it, then we can see that conscious entities have evolved from it. On the other hand, the Eastern perspective—that consciousness is fundamental and represents the only reality that is truly important—is also difficult to deny.

Beyond: Our Future in Space
by Chris Impey
Published 12 Apr 2015

The philosophical movement that forms an umbrella for cybernetics and cyborgs is called transhumanism. Transhumanism is a worldwide cultural and intellectual movement that seeks to use technology to improve the human condition. Radical life extension is one aspect, as is the enhancement of physical and mental capabilities. Two prominent transhumanists are Nick Bostrom, a University of Oxford philosopher who has assessed various risks to the long-term survival of humanity, and Ray Kurzweil, the engineer and inventor who popularized the idea of the singularity, a time in the not-too-distant future when technology will enable us to transcend our physical limitations.

…

Technological civilizations are intrinsically rare, so searches for them will fail. On the other hand, if the Great Filter is in our future, it’s very unlikely that a civilization at our stage of development progresses to the large-scale colonization of space. One plausible scenario is that technology is the culprit because it includes the capability for self-destruction. Nick Bostrom, director of the Future of Humanity Institute at the University of Oxford, has done scholarly work on catastrophes. His partial list of existential threats faced by humanity includes nuclear holocaust, genetically engineered superbugs, environmental disasters, asteroid impacts, terrorism, advanced and destructive artificial intelligence, uncontrollable nanotechnology, catastrophic high-energy physics experiments, and a totalitarian regime with advanced surveillance and mind-control technologies.

…

There may be more than one Great Filter. We might have cleared one, only to be faced in the future with another. Also, we should be wary of positing that life elsewhere has to follow the path traveled by life on Earth or that other civilizations progress with the single-minded purposefulness of humans. Let’s allow Nick Bostrom to have the last word. For someone who dwells on apocalypse and hopes that the search for life in the universe fails, he’s strangely optimistic: If the Great Filter is in our past . . . we may have a significant chance . . . of one day growing into something almost unimaginably greater than we are today.

12 Bytes: How We Got Here. Where We Might Go Next
by Jeanette Winterson
Published 15 Mar 2021

That’s far ahead, when we are no longer augmented humans, but intelligences running on a number of alternative substrates, of which a biological-based body might be one – but probably won’t be, except perhaps as a retro-themed holiday (‘Let’s visit how they used to live in the past: get drunk, throw up, pass out etc’). * * * For Nick Bostrom, Director of the Future of Humanity Institute at the University of Oxford, transhumanism today is a loosely connected interdisciplinary approach for developing, and evaluating, technologies that will benefit us as individuals, and as a society. Bostrom is keen to involve governmental influence and legislation, rather than leave the future to the marketplace.

…

Forbes Magazine has been ranking the worth of fictional characters since 2002. The wealthiest is Twilight vampire Carlisle Cullen, whose very long-term investments, plus the magic of compound interest, have netted him a fortune of $34 billion. It is safe to say that vampires can fund themselves. For the rest of us? * * * Nick Bostrom suggests that people would go back to school in their 50s, or start a new career in their 70s. His view is that 80-year-olds with the same physique as 40-year-olds would not burden the healthcare system and would be amazingly productive because of their experience and knowledge-base. He believes that longer life expectancy would give us an increased responsibility for the future – because we’ll be there to see it.

…

If we start to merge with the AI opportunities we are creating, by which I mean, if we become part of the toolkit – not just operators on the outside, but intrinsic to the technology on the inside – no ‘us’ and ‘it’ – then … Then the enemy won’t be on the outside. And we will recognise our responsibility to ourselves, and to AI, and later to AGI. * * * Nick Bostrom, philosopher and AI expert, author of Super-intelligence: Paths, Dangers, Strategies (2014), and Director of the Future of Humanity Institute at the University of Oxford, is an enthusiast for transhumanism. In his view, we must merge with AI. We must better ourselves because we can. Humans should be a match for what we are creating

To Be a Machine: Adventures Among Cyborgs, Utopians, Hackers, and the Futurists Solving the Modest Problem of Death
by Mark O'Connell
Published 28 Feb 2017

He was dreaming, I came to feel, of a Singularity. He was singing of what was past, and passing, and to come. In May 2007, Randal was one of thirteen participants at a workshop on mind uploading held at the Future of Humanity Institute. The event resulted in the publication of a technical report, coauthored by Anders Sandberg and Nick Bostrom, entitled “Whole Brain Emulation: A Roadmap.” The report began with the statement that mind uploading, though still a remote prospect, was nonetheless theoretically achievable through the development of technologies already in existence. A criticism commonly raised against the idea of simulating minds in software is that we don’t understand nearly enough about how consciousness works to even know where to start reproducing it.

…

And if these people were worried—along with Hawkeye from M*A*S*H and the guy who had played, among other personifications of serene wisdom, a scientist trying to prevent the Singularity in the 2014 film Transcendence—shouldn’t we all be worrying with them? One figure who loomed especially large over this whole area, its eschatologist-in-chief, was Nick Bostrom, the Swedish philosopher who, before he became known as the world’s foremost prophet of technological disaster, had been one of the most prominent figures in the transhumanist movement, a cofounder of the World Transhumanist Association. In late 2014, Bostrom, director of the Future of Humanity Institute, had published a book called Superintelligence: Paths, Dangers, Strategies, in which he outlined the nature of the AI peril.

…

(An effectively altruistic act, as opposed to an emotionally altruistic one, might involve a college student deciding that, rather than becoming a doctor and spending her career curing blindness in the developing world, her time would be better spent becoming a Wall Street hedge fund manager and donating enough of her income to charity to pay for several doctors to cure a great many more people of blindness.) The conference had substantially focused on questions of AI and existential risk. Thiel and Musk, who’d spoken on a panel at the conference along with Nick Bostrom, had been influenced by the moral metrics of Effective Altruism to donate large amounts of money to organizations focused on AI safety. Effective Altruism had significant crossover, in terms of constituency, with the AI existential risk movement. (In fact, the Centre for Effective Altruism, the main international promoter of the movement, happened to occupy an office in Oxford just down the hall from the Future of Humanity Institute.)

The Future of the Professions: How Technology Will Transform the Work of Human Experts
by Richard Susskind and Daniel Susskind
Published 24 Aug 2015

Into the 2020s and beyond, we predict the emergence and adoption of a second wave of AI systems in the professions. 1 We first introduced this concept in the mid-1990s, in Richard Susskind, The Future of Law (1996). 2 Walter Ong, Orality and Literacy (1982). 3 We can anticipate a fifth stage, but it is beyond the scope of this book—this is the likely stage in the development of humanity when human beings become digitally enhanced, when machines and human beings become entwined and even indistinguishable. Some of these themes are touched on in Nick Bostrom, Superintelligence (2014). 4 Ong, Orality and Literacy, Marshall McLuhan, Understanding Media (2002), Luciano Floridi, The Philosophy of Information (2013), James Gleick, The Information (2011), and the works to which they refer. 5 Ong, Orality and Literacy, 31. 6 Gleick, The Information, and his discussion of Plato, who spoke of the ‘forgetfulness’ of those who do ‘not practice their memory’ (p. 30).

…

Joel Garreau, Radical Evolution (2005), and ‘Coming To an Office Near You’, Economist, 18 Jan. 2014. 28 See e.g. the work of Singularity University at <http://singularityu.org> (accessed 23 March 2015). 29 For a clear introduction to the cloud and cloud computing, and a clear indication of its mounting signifiance, see Kuan Hon and Christopher Millard, ‘Cloud Technologies and Services’, in Cloud Computing Law, ed. Christopher Millard (2013). 30 John Kelly and Steve Hamm, Smart Machines (2013). 31 Nick Bostrom, Superintelligence (2014). 32 See e.g. Tim Bradshaw, ‘Scientists and Investors Warn on AI’, Financial Times, 12 Jan. 2015. 33 Kelly and Hamm, Smart Machines, and Kieron O’Hara et al., Web Science: Understanding the Emergence of Macro-Level Features of the World Wide Web (2013). 34 Soshana Zuboff, In the Age of the Smart Machine (1988), 9. 35 Zuboff, In the Age of the Smart Machine, 10. 36 Some writers treat ‘machine learning’ as yet another synonym.

…

v=FU-tuY0Z7nQ> (accessed 24 March 2015). 54 Frank Levy and Richard Murnane, The New Division of Labour (2004), 1–2, for the upheaval, and 20–30 for the truck-driver discussion. 55 Alison Sander and Meldon Wolfgang, ‘The Rise of Robotics’, 27 Aug. 2014, at <https://www.bcgperspectives.com/content/articles/business_unit_strategy_innovation_rise_of_robotics> (accessed 23 March 2015). 56 See <http://cyberlaw.stanford.edu/wiki/index.php/Automated_Driving:_Legislative_and_Regulatory_Action> (accessed 27 March 2015). 57 Guy Ryder, ‘Labor in the Age of Robots’, Project Syndicate, 22 Jan. 2015 <http://www.project-syndicate.org/> (accessed 23 March 2015). 58 Sam Frizell, ‘Meet the Robots Shipping Your Amazon Orders’, Time, 1 Dec. 2014, <http://time.com/3605924/amazon-robots/> (accessed 23 March 2015). 59 See <http://allaboutroboticsurgery.com/zeusrobot.html> (accessed 23 March 2015). 60 David Rose, Enchanted Objects (2014), 23–4; original emphasis. 61 See Julian Savulescu and Nick Bostrom, Human Enhancement (2011), on the technologies and the ethical questions they raise. 62 Malcolm Peltu and Yorick Wilks, ‘Close Engagements with Artificial Companions: Key Social, Psychological, Ethical and Design Issues’, OII/e-Horizons Discussion Paper, No. 14 (2008). Also see Timothy Bickmore, ‘Relational Agents in Health Applications: Leveraging Affective Computing to Promote Health and Wellness’, in Oxford Handbook of Affective Computing, ed.

The Internet Is Not What You Think It Is: A History, a Philosophy, a Warning
by Justin E. H. Smith
Published 22 Mar 2022

Ann M. Blair, Too Much to Know: Managing Scholarly Information Before the Information Age, New Haven, CT: Yale University Press, 2010. Ned Block, “Troubles with Functionalism,” Minnesota Studies in the Philosophy of Science 9 (1978): 261–325. Nick Bostrom, “Are You Living in a Computer Simulation?” Philosophical Quarterly 53, 211 (2003): 243–255. Nick Bostrom, Superintelligence: Paths, Dangers, Strategies, New York: Oxford University Press, 2014. Horst Bredekamp, “Leibniz’ Gewebe: Strumpband, Falte, Leinwand,” in R. Hoppe-Sailer, C. Volkenandt, and G. Winter (eds.), Logik der Bilder. Präsenz—Repräsentation—Erkenntnis, Bonn: Reimer, 2005, 233–238.

…

How and why many people became more comfortable assimilating the mind to a machine of the mind’s own design than they are assimilating the mind to the countless other products of nature alongside which it co-inhabits the earth, is a complex question that we will not answer here, other than to say that the reasons have much more to do with cultural history and fashion than they do with rigorous philosophical arguments or scientific literacy. To provide just one illustration of the fashion in question, some philosophers maintain that the world as we know it is most likely a simulation comparable to a video game. According to one variant of what Nick Bostrom calls the “simulation hypothesis,” “a superintelligence could create virtual worlds that appear to its inhabitants much the same as our world appears to us. It might create vast numbers of such worlds, running the same simulation many times or with small variation. The inhabitants would not necessarily be able to tell whether their world is simulated or not; but if they are intelligent enough they could consider the possibility and assign it some probability.”2 This hypothesis has been extremely popular among the representatives of whatever simulation of an intelligentsia the tech industry has produced.

The Seventh Sense: Power, Fortune, and Survival in the Age of Networks
by Joshua Cooper Ramo
Published 16 May 2016

In the spring of 1993: See Vision-21: Interdisciplinary Science and Engineering in the Era of Cyberspace, proceedings of a symposium cosponsored by the NASA Lewis Research Center and the Ohio Aerospace Institute, Westlake, Ohio, March 30–31, 1993 (Hampton, VA: National Aeronautics and Space Administration Scientific and Technical Information Program), iii. “Within thirty years”: Ibid., 12. Imagine a superintelligent machine: Nick Bostrom, “Ethical Issues in Advanced Artificial Intelligence,” in Cognitive, Emotive and Ethical Aspects of Decision Making in Humans and in AI, vol. 2, ed. Iva Smit et al. (Windsor, ON: International Institute for Advanced Studies in Systems Research and Cybernetics, 2003), 12–17, and Nick Bostrom, “The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents,” Minds and Machines, 22, no. 2 (2012): 71–85. It runs like this: Seth Grimes, “Language Use, Customer Personality, and the Customer Journey” (interview with Scott Nowson, global innovation lead, Xerox), Breakthrough Analysis (blog), October 8, 2015.

…

It was easy enough for Vinge to see how this would end. It wouldn’t be with the sort of intended polite, lapdog domesticity of artificial intelligence that we might hope for but with a rottweiler of a device, alive to the meaty smell of power, violence, and greed. This puzzle has interested the Oxford philosopher Nick Bostrom, who has described the following thought experiment: Imagine a superintelligent machine programmed to do whatever is needed to make paper clips as fast as possible, a machine that is connected to every resource that task might demand. Go figure it out! might be all its human instructors tell it.

…

(Windsor, ON: International Institute for Advanced Studies in Systems Research and Cybernetics, 2003), 12–17, and Nick Bostrom, “The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents,” Minds and Machines, 22, no. 2 (2012): 71–85. It runs like this: Seth Grimes, “Language Use, Customer Personality, and the Customer Journey” (interview with Scott Nowson, global innovation lead, Xerox), Breakthrough Analysis (blog), October 8, 2015. Let me out!: See, for instance, Stuart Armstrong, Anders Sandberg, and Nick Bostrom, “Thinking Inside the Box: Controlling and Using an Oracle AI,” Minds and Machines 22, no. 4 (November 2012): 299–324. “People proposing”: Michael Vassar, “Re: AI Boxing (dogs and helicopters),” SL4.org, August 2, 2005, at http://sl4.org/archive/0508/11817.html. So we need to ask: Kaj Sotala and Roman V. Yampolskiy, “Responses to Catastrophic AGI Risk: A Survey,” Physica Scripta 90 (2015).

The World Without Us
by Alan Weisman
Published 5 Aug 2008

He chuckles mirthlessly. “People trying might kill themselves in the process. There are a lot of easier things to do with a lot less effort.” HAVING YET TO perfect contraception, thus far we have little to fear from misanthropic plots to sterilize the entire human race. From time to time, Nick Bostrom, who directs Oxford’s Future of Humanity Institute, computes the odds (increasing, he believes) that human existence is at risk of ending. He is particularly intrigued by the potential of nanotechnology going awry, accidentally or deliberately, or superintelligence running amuck. In either case, however, he notes that the skills needed to create atom-sized medical machines that would patrol our bloodstreams, zapping disease until they suddenly turned on us, or self-replicating robots that end up crowding or outsmarting us off the planet, are “at least decades away.”

…

Each of these men, philosophers taking ethical measure of an age in which machines think faster than humans but regularly prove at least as flawed, repeatedly smack into a phenomenon that never troubled their intellectual predecessors: although humans have obviously survived every pox and meteor that nature has tossed at us until now, technology is something we toss back at our own peril. “On the bright side, it hasn’t killed us yet, either,” says Nick Bostrom, who, when not refining doomsday data, researches how to extend the human life span. “But if we did go extinct, I think it would more likely be through new technologies than environmental destruction.” To the rest of the planet, it would make little difference, because if either actually took us out, many other species undoubtedly would go with us.

…

Via the self-accruing wizardry of computers, an abundance of silicon, and vast opportunities afforded by modular memory and mechanical appendages, human extinction would become merely a jettisoning of the limited and not very durable vessels that our technological minds have finally outgrown. Prominent in the transhumanist (sometimes called posthuman) movement are Oxford philosopher Nick Bostrom; heralded inventor Ray Kurzweil, originator of optical character recognition, flat-bed scanners, and print-to-speech reading machines for the blind; and Trinity College bioethicist James Hughes, author of Citizen Cyborg: Why Democratic Societies Must Respond to the Redesigned Human of the Future.

Genius Makers: The Mavericks Who Brought A. I. To Google, Facebook, and the World
by Cade Metz
Published 15 Mar 2021

“There have been movies about this,” he said. On Twitter, he called artificial intelligence “potentially more dangerous than nukes.” The same tweet urged his followers to read Superintelligence: Paths, Dangers, Strategies, a recently published tome from an Oxford University philosopher named Nick Bostrom. Like Shane Legg, the founder of DeepMind, Bostrom believed that superintelligence could secure the future of humanity—or destroy it. “This is quite possibly the most important and most daunting challenge humanity has ever faced,” he wrote. “And—whether we succeed or fail—it is probably the last challenge we will ever face.”

…

To help make his case, he also invited Yann LeCun, Mike Schroepfer, and Rob Fergus, the NYU professor who worked alongside LeCun at the new Facebook lab. The Facebookers spent the meal trying to explain that Musk’s views on AI had been warped by a few misguided voices that were very much in the minority. The philosophical musings of Nick Bostrom, Zuckerberg and his fellow Facebookers said, were in no way related to what Musk had seen at DeepMind or inside any other AI lab. A neural network was still a long way from superintelligence. DeepMind built systems that optimized point totals inside games like Pong or Space Invaders, but they were useless elsewhere.

…

Another Googler, Meredith Whittaker, a product manager in the company’s cloud computing group, helped launch a research organization at NYU. A consortium of companies, from Google to Facebook to Microsoft, built an organization called the Partnership on AI. Organizations like the Future of Life Institute (founded by Max Tegmark at MIT) and the Future of Humanity Institute (founded by Nick Bostrom at Oxford) were also concerned with the ethics of AI, but they focused on existential threats of the distant future. The new wave of ethicists focused on more immediate matters. For both Mitchell and Gebru, the bias problem was part of the larger issue across the tech industry. Women struggled to exert their influence in all tech fields, facing extreme bias in the workplace and sometimes harassment.

The Age of Em: Work, Love and Life When Robots Rule the Earth
by Robin Hanson
Published 31 Mar 2016

June 20. http://www.bls.gov/news.release/atus.t01.htm. Buterin, Vitalik. 2014. “White Paper: A Next-Generation Smart Contract and Decentralized Application Platform.” April. https://www.ethereum.org/pdfs/EthereumWhitePaper.pdf. Caplan, Bryan. 2008. “The Totalitarian Threat.” In Global Catastrophic Risks, edited by Nick Bostrom and Milan Ćirković, 504–519. Oxford University Press, July 17. Caplan, Bryan, and Stephen Miller. 2010. “Intelligence Makes People Think Like Economists: Evidence from the General Social Survey.” Intelligence 38(6): 636–647. Card, Orson Scott. 2011. Elements of Fiction Writing—Characters and Viewpoint, 2nd ed.

…

American Economic Review 72(2): 155–159. Mulder, Monique. 1998. “The Demographic Transition: Are we any Closer to an Evolutionary Explanation?” Trends in Ecology & Evolution 13(7): 266–270. Mullainathan, Sendhil, and Eldar Shafir. 2013. Scarcity: Why Having Too Little Means So Much. Times Books, September 3. Müller, Vincent, and Nick Bostrom. 2014. “Future Progress in Artificial Intelligence: A Survey of Expert Opinion.” In Fundamental Issues of Artificial Intelligence, edited by Vincent Müller. Berlin: Springer. Mulligan, Casey, Richard Gil, and Xavier Sala-i-Martin. 2004. “Do Democracies Have Different Public Policies than Nondemocracies?”

…

MIT Sloan Management Review 50(3): 70–79. Sandberg, Anders. 2014. “Monte Carlo model of brain emulation development.” Working Paper 2014–1 (version 1.2), Future of Humanity Institute. http://www.aleph.se/papers/Monte%20Carlo%20model%20of%20brain%20emulation%20development.pdf. Sandberg, Anders, and Nick Bostrom. 2008. “Whole Brain Emulation: A Roadmap.” Technical Report #2008–2003, Future of Humanity Institute, Oxford University. http://www.fhi.ox.ac.uk/__data/assets/pdf_file/0019/3853/brain-emulation-roadmap-report.pdf. Sandstrom, Gillian, and Elizabeth Dunn. 2014. “Social Interactions and Well-Being: The Surprising Power of Weak Ties.”

Army of None: Autonomous Weapons and the Future of War
by Paul Scharre
Published 23 Apr 2018

Ava is intelligent, but inhuman. GOD OR GOLEM? Ex Machina’s ending is a warning against anthropomorphizing AI and assuming that just because a machine can imitate human behavior, it thinks like humans. Like Jeff Clune’s “weird” deep neural nets, advanced AI is likely to be fundamentally alien. In fact, Nick Bostrom, an Oxford philosopher and author of Superintelligence: Paths, Dangers, Strategies, has argued that biological extraterrestrials would likely have more in common with humans than with machine intelligence. Biological aliens (if they exist) would have presumably developed drives and instincts similar to ours through natural selection.

…

Stuart Armstrong, a researcher at the Future of Humanity Institute in Oxford, has given an example of a hypothetical AI that achieves this goal by burying humans in lead-lined coffins connected to heroin drips. You may ask, wouldn’t an artificial general intelligence understand that’s not what we meant? An AI that understood context and meaning might determine its programmers didn’t want lead coffins and heroin drips, but that might not matter. Nick Bostrom has argued “its final goal is to make us happy, not to do what the programmers meant when they wrote the code that represents this goal.” The problem is that any rule blindly followed to its most extreme can result in perverse outcomes. Philosophers and AI researchers have pondered the problem of what goals to give a superintelligent AI that could not lead to perverse instantiation and they have not come to any particularly satisfactory solution.

…

For an overview of current abilities and limitations in scene interpretation, see JASON, “Perspectives on Research in Artificial Intelligence and Artificial General Intelligence Relevant to DoD,” 10. 232 Brain imaging: “Human Connectome Project | Mapping the Human Brain Connectivity,” accessed June 15, 2017, http://www.humanconnectomeproject.org/. “Meet the World’s Most Advanced Brain Scanner,” Discover Magazine, accessed June 15, 2017, http://discovermagazine.com/2013/june/08-meet-the-worlds-most-advanced-brain-scanner. 232 whole brain emulations: Anders Sandburg and Nick Bostrom, “Whole Brain Emulation: A Roadmap,” Technical Report #2008-3, Oxford, UK, 2008, http://www.fhi.ox.ac.uk/Reports/2008-3.pdf 232 “When people say a technology”: Andrew Herr, email to the author, October 22, 2016. 232 “last invention”: Irving J. Good, “Speculations Concerning the First Ultraintelligent Machine”, May 1964, https://web.archive.org/web/20010527181244/http://www.aeiveos.com/~bradbury/Authors/Computing/Good-IJ/SCtFUM.html.

100 Plus: How the Coming Age of Longevity Will Change Everything, From Careers and Relationships to Family And
by Sonia Arrison
Published 22 Aug 2011

Audrey Chapman, a professor of community medicine and health care at the University of Connecticut Health Center, argues that “investing in new and very expensive high technologies for enhancement interventions while people in our own country lack access to basic health care and millions of people die prematurely of preventable diseases in poor countries would be yet another step toward moral bankruptcy.”31 Her argument is meant to play on “rich guilt,” but it is nonsensical because it assumes an either/or choice and completely ignores the health benefits of new innovations that could potentially lift poorer communities faster than anything available now. Technology is spreading more quickly today than ever before, and even though it may not reach poor communities at the same time as wealthy ones, it will likely get to the former faster than past innovations did. Nick Bostrom, professor of philosophy at the University of Oxford, helps to expose the weakness in Chapman’s emotionally charged argument. He writes, “It is unclear why aging research should be singled out for blame or special concern in this regard. Many factors contribute to global inequality, and spending on gerontological research is such a minute fraction of the financial outlays of wealthy nations that it seems a bizarre place to look for savings to transfer to the poor.”32 Instead, he argues that using poverty as an excuse to not advance health care innovations is an irrational psychological reaction reminiscent of Stockholm syndrome, whereby hostages become emotionally attached to their captors.

…

William Kristol and Eric Cohen (Lanham, MD: Rowman and Littlefield, 2002). 3 Ibid., 325. 4 Bill McKibben, “Mr. Natural,” Outside, May 2003, http://outside.away.com/outside/bodywork/200305/200305_mr_natural_1.html. 5 Ibid. 6 Arthur Caplan, “Good, Better, or Best?” in Human Enhancement, ed. Julian Savulescu and Nick Bostrom (New York: Oxford University Press, 2009), 202. 7 Kass, “Why Not Immortality?” 326. 8 See www.clintonbushhaitifund.org/. 9 Kass, “Why Not Immortality?” 327. 10 Daniel Callahan, cited in Christine Overall, Aging, Death, and Human Longevity: A Philosophical Inquiry (Berkeley and Los Angeles: University of California Press, 2003), 39–40. 11 Kass, “Why Not Immortality?”

…

Savulescu and Bostrom, 123–124. 31 Audrey R. Chapman, “The Social and Justice Implications of Extending the Human Lifespan,” in The Fountain of Youth: Cultural, Scientific, and Ethical Perspectives on a Biomedical Goal, ed. Stephen G. Post and Robert H. Binstock (New York: Oxford University Press, 2004), 353. 32 Nick Bostrom, “Recent Developments in the Ethics, Science, and Politics of Life-Extension,” Aging Horizons 3 (Autumn–Winter 2005): 28–34, www.fhi.ox.ac.uk/__data/assets/pdf_file/0004/5926/life-extension.pdf. 33 Ibid. 34 Jonathan Weiner, Long for This World (New York: HarperCollins, 2010), 261. 35 Francis Fukuyama, Our Posthuman Future: Consequences of the Biotechnology Revolution (New York: Picador, 2002), 172. 36 George Annas, “Genism, Racism, and the Prospect of Genetic Genocide,” The Future of Human Nature: A Symposium on the Promises and Challenges of the Revolutions in Genomics and Computer Science, Pardee Center Conference Series, Boston University, Boston, Massachusetts, April 2003. 37 Fukuyama, Our Posthuman Future, 171, 175. 38 Annas, “Genism.” 39 C.

Rule of the Robots: How Artificial Intelligence Will Transform Everything
by Martin Ford
Published 13 Sep 2021

In May of that year, a group of scientists including the University of Cambridge cosmologist Stephen Hawking along with AI expert Stuart Russell and physicists Max Tegmark and Frank Wilczek co-authored an open letter published in the U.K.’s Independent declaring that the advent of artificial superintelligence “would be the biggest event in human history,” and that a computer with superhuman intellectual capability might be capable of “outsmarting financial markets, out-inventing human researchers, out-manipulating human leaders, and developing weapons we cannot even understand.” The letter warned that a failure to take this looming danger seriously might well turn out to be humanity’s “worst mistake in history.”27 Later that same year, the Oxford University philosopher Nick Bostrom published his book Superintelligence: Paths, Dangers, Strategies, which quickly became a somewhat surprising bestseller. Bostrom opens the book by pointing out that humans rule the earth purely on the basis of superior intellect. Many other animals are faster, stronger or more ferocious; it is our brains that led to dominance.

…

If a similar miscalculation were made with a superintelligent system, it might well be impossible to regain control as it sought to pursue its objective. The quest to find a solution to the control problem has become an important topic of academic research at universities and especially within specialized, privately funded organizations such as OpenAI, Oxford University’s Future of Humanity Institute, which is directed by Nick Bostrom, and the Machine Intelligence Research Institute located in Berkeley, California. In his 2019 book Human Compatible: Artificial Intelligence and the Problem of Control, Stuart Russell argues that the best solution to the problem is to not build an explicit objective function into advanced AI systems at all.

…

Stephen Hawking, Stuart Russell, Max Tegmark and Frank Wilczek, “Stephen Hawking: ‘Transcendence looks at the implications of artificial intelligence—but are we taking AI seriously enough?,’” The Independent, May 1, 2014, www.independent.co.uk/news/science/stephen-hawking-transcendence-looks-at-the-implications-of-artificial-intelligence-but-are-we-taking-ai-seriously-enough-9313474.html. 28. Nick Bostrom, Superintelligence: Paths, Dangers, Strategies, Oxford University Press, 2014, p. vii. 29. Matt McFarland, “Elon Musk: ‘With artificial intelligence we are summoning the demon,’” Washington Post, October 24, 2014, www.washingtonpost.com/news/innovations/wp/2014/10/24/elon-musk-with-artificial-intelligence-we-are-summoning-the-demon/. 30.

Ten Billion Tomorrows: How Science Fiction Technology Became Reality and Shapes the Future
by Brian Clegg
Published 8 Dec 2015

If you believe the copy of this e-book you are reading infringes on the author’s copyright, please notify the publisher at: us.macmillanusa.com/piracy. FOR GILLIAN, CHELSEA, AND REBECCA ACKNOWLEDGMENTS My thanks to my editors, Michael Homler and Lauren Jablonski, and everyone at St. Martin’s Press who have made the book possible. Although I must thank the academic writers, like philosophy professor Nick Bostrom, who have thought long and hard about the implications that changes in technology could have on society, I particularly have to think of the giants of science fiction who inspired me to get into science in the first place and who provided the inspiration for most of these chapters. This list does include modern writers like Adam Roberts and the late, much lamented, Iain M.

…

Most fiction concentrates on wireheading as a form of addictive entertainment, giving an experience that is better than real life, instead of being used to fool the population as it is in The Matrix. But what doesn’t seem to have crossed the minds of writers as yet is the suggestion from philosopher Nick Bostrom that computers themselves could—and arguably inevitably would—also become wireheads if they ever developed consciousness. As Bostrom points out, the attempts to develop artificial intelligence could be fraught with dangers. We have very little reason to assume that a truly intelligent computer would do our bidding, preferring whatever best suited its own agenda.

…

Power Grid (January 2010), available at web.ornl.gov/sci/ees/etsd/pes/pubs/ferc_Meta-R-320.pdf. Larry Niven first used the term “wirehead” in the novella The Defenseless Dead, published in the collection Roger Elwood (ed.), Playgrounds of the Mind (New York: Fawcett 1973). The idea of artificial intelligences tending to become wireheads is raised in Nick Bostrom, Superintelligence (Oxford, UK: Oxford University Press, 2014), p. 122. 3. BUY ME Quotes on science fiction advertising and marketing methods are from Frederik Pohl and C. M. Kornbluth, The Space Merchants (London: Penguin Books, 1965), pp. 9–10. The more aggressive Mokie-Koke advertising is described in Frederik Pohl, The Merchants’ War (London: Futura, 1986), pp. 47–49.

Upstream: The Quest to Solve Problems Before They Happen
by Dan Heath
Published 3 Mar 2020

In other words, if a hacker was dead set on breaking into West Aurora School District No. 129—or any other specific institution, for that matter—then the difference between 29% and 5% is immaterial. For many hacking purposes, you just need one open door. Just that one gullible person who will click on anything. Nick Bostrom, a Swedish philosopher at the University of Oxford, contemplates whether technological innovation has left modern society on the verge of a similar kind of vulnerability—a situation in which the fate of everyone could hinge on a single bad break or bad actor. The context of his comments is mankind’s tendency to keep pushing for new innovations almost without regard for the consequences.

…

.; “Education Technologies: Data Collection and Unsecured Systems Could Pose Risks to Students,” Federal Bureau of Investigation, Public Service Announcement, September 13, 2018, https://www.ic3.gov/media/2018/180913.aspx. 5% in recent attempts: Interview with Don Ringelestein, May 2019, and subsequent correspondence and phishing security test data. “The Vulnerable World Hypothesis”: Nick Bostrom, “The Vulnerable World Hypothesis,” 2018, https://nickbostrom.com/papers/vulnerable.pdf. there are DNA “printers”: Rob Stein, “DNA Printing, A Big Boon to Research, But Some Raise Concerns,” NPR, May 7, 2015, https://www.npr.org/sections/health-shots/2015/05/07/404460240/dna-printing-a-big-boon-to-research-but-some-raise-concerns.

…

founded the Future of Humanity Institute: To learn more about Bostrom’s work, check out this fascinating profile: Raffi Khatchadourian, “The Doomsday Invention: Will Artificial Intelligence Bring Us Utopia or Destruction?,” The New Yorker, November 23, 2015, https://www.newyorker.com/magazine/2015/11/23/doomsday-invention-artificial-intelligence-nick-bostrom. Click Here to Kill Everybody: Bruce Schneier, Click Here to Kill Everybody: Security and Survival in a Hyper-connected World (New York: Norton, 2018). “at risk from Moon germs”: Michael Meltzer, When Biospheres Collide: A History of NASA’s Planetary Protection Programs (US National Aeronautics and Space Administration, 2010), BiblioGov, 215.

Rationality: From AI to Zombies
by Eliezer Yudkowsky
Published 11 Mar 2015

Bostrom and Savulescu survey arguments for and against radical human enhancement, e.g., Sandel’s objection that tampering with our biology too much would make life feel like less of a “gift.”2,3 Bostrom’s “History of Transhumanist Thought” provides context for the debate.4 2. Nick Bostrom, “A History of Transhumanist Thought,” Journal of Evolution and Technology 14, no. 1 (2005): 1–25, http://www.nickbostrom.com/papers/history.pdf. 3. Michael Sandel, “What’s Wrong With Enhancement,” Background material for the President’s Council on Bioethics. (2002). 4. Nick Bostrom and Julian Savulescu, “Human Enhancement Ethics: The State of the Debate,” in Human Enhancement, ed. Nick Bostrom and Julian Savulescu (2009). Part U Fake Preferences 257 Not for the Sake of Happiness (Alone) When I met the futurist Greg Stock some years ago, he argued that the joy of scientific discovery would soon be replaced by pills that could simulate the joy of scientific discovery.

…

Alt and Morris Rubinoff, vol. 6 (New York: Academic Press, 1965), 31–88, doi:10.1016/S0065-2458(08)60418-0. 3. Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford University Press, 2014). 4. Stuart J. Russell and Peter Norvig, Artificial Intelligence: A Modern Approach, 3rd ed. (Upper Saddle River, NJ: Prentice-Hall, 2010). 5. Bostrom and Ćirković, Global Catastrophic Risks. 6. An example of a possible existential risk is the “grey goo” scenario, in which molecular robots designed to efficiently self-replicate do their job too well, rapidly outcompeting living organisms as they consume the Earth’s available matter. 7. Nick Bostrom and Eliezer Yudkowsky, “The Ethics of Artificial Intelligence,” in The Cambridge Handbook of Artificial Intelligence, ed.

…

Remember that, in fact, you did not guess. Remember how, century after century, the world changed in ways you did not guess. Maybe then you will be less shocked by what happens next. * 1. Eliezer Yudkowsky, “Cognitive Biases Potentially Affecting Judgment of Global Risks,” in Global Catastrophic Risks, ed. Nick Bostrom and Milan M. Ćirković (New York: Oxford University Press, 2008), 91–119. 43 Explain/Worship/Ignore? As our tribe wanders through the grasslands, searching for fruit trees and prey, it happens every now and then that water pours down from the sky. “Why does water sometimes fall from the sky?”

Singularity Rising: Surviving and Thriving in a Smarter, Richer, and More Dangerous World
by James D. Miller
Published 14 Jun 2012

Nothing, therefore, comes close to the expected return, per charitable dollar, of donating to an existential risk-reducing organization.107 MY LONG-SHOT IDEA FOR FRIENDLINESS108 I have come up with a long-shot way of causing an ultra-intelligence to modify itself to become friendly. My idea is premised on Oxford philosopher Nick Bostrom’s conjecture that we might be living in a computer simulation.109 Bostrom believes that a sufficiently advanced AI could easily simulate our entire planet and the thought processes of every living thing on it. Bostrom argues that, if mankind survives long enough, a future historian would likely want to simulate the pre-Singularity past to learn about how we made decisions.

…

But the more people who independently locate the cave, the less concealed we should expect it to have been. Now think of the cave as the genes necessary for “complex learning, memory, and tool use” and the evolutionary forces operating in different creatures as the searchers. As Carl Shulman and Nick Bostrom write, across different species evolution independently produced “complex learning, memory, and tool use both within and without the line of human ancestry.”336 Understanding convergent evolution should reduce our estimate of how hard it must be to develop the part of humanity’s intellectual toolkit shared by reasonably bright creatures such as octopuses, elephants, and crows.337 If the blind forces of evolution could find this toolkit several times, mankind should be able to “find it” as well.

…

Super-Economy. http://super-economy.blogspot.com/2011/04/iq-income-and-wealth.html. Segal, Nancy L. 1999. Entwined Lives: Twins and What They Tell Us About Human Behavior. New York: Dutton. Shulman, Carl. November 23, 2008. “‘Evicting’ Brain Emulations.” Overcoming Bias (blog). http://www.overcomingbias.com/2008/11/suppose-that-ro.html. Shulman, Carl, and Nick Bostrom. 2012. “How Hard is Artificial Intelligence? The Evolutionary Argument and Observation Selection Effects.” Forthcoming in the Journal of Consciousness Studies. http://www.nickbostrom.com/aievolution.pdf. Singularity Institute for Artificial Intelligence. 2007. The Singularity Summit 2007: AI and the Future of Humanity. http://singinst.org/summit2007/quotes/billgates/.

Falter: Has the Human Game Begun to Play Itself Out?
by Bill McKibben
Published 15 Apr 2019

Hawking wrote that success in AI would be “the biggest event in human history,” but it might “also be the last, unless we learn to avoid the risks.”20 And here’s Michael Vassar, president of the Machine Intelligence Research Institute: “I definitely think people should try to develop Artificial General Intelligence with all due care. In this case all due care means much more scrupulous caution than would be necessary for dealing with Ebola or plutonium.”21 Why are people so scared? Let the Swedish philosopher Nick Bostrom explain. He’s hardly a Luddite. Indeed, he gave a speech in 1999 to a California convention of “transhumanists” that may mark the rhetorical high water of the entire techno-utopian movement. Thanks to ever-increasing computer power and ever-shinier biotech, he predicted then, we would soon have “values that will strike us as being of a far higher order than those we can realize as unenhanced biological humans,” not to mention “love that is stronger, purer, and more secure than any human has yet harbored,” not to mention “orgasms … whose blissfulness vastly exceeds what any human has yet experienced.”22 But fifteen years later, ensconced in Oxford as nothing less than the director of the Future of Humanity Institute, he’d begun to worry a great deal: “In fairy tales you have genies who grant wishes,” he told a reporter for The New Yorker.

…

The point of art is not “better”; the point is to reflect on the experience of being human—which is precisely the thing that’s disappearing. Even—and here is more rich, sad irony—science is at risk. The profound pleasure that keeps people working on precisely the technology that now threatens to supplant us will vanish, too. You don’t think robot biologists will soon replace the real ones? What are we left with? Nick Bostrom, the early apostle of transhumanism, offers the best case: a superintelligence could “assist us in creating a highly appealing experiential world in which we could live lives devoted to joyful game-playing, relating to each other, experiencing personal growth, and living closer to our ideals.” Or we could just smoke weed

…

James Vincent, “Elon Musk Says We Need to Regulate AI Before It Becomes a Danger to Humanity,” theverge.com, July 17, 2017. 20. Stephen Hawking, “Artificial Intelligence Could Be the Greatest Disaster in Human History,” Independent, October 20, 2016. 21. James Barrat, Our Final Invention: Artificial Intelligence and the End of the Human Era (New York: St. Martin’s Press, 2013), p. 34. 22. Nick Bostrom, “A Transhumanist Perspective on Genetic Enhancements,” nickbostrom.com, 2003. 23. Khatchadourian, “Doomsday Invention.” 24. Stephen M. Omohundro, “The Basic A.I. Drives,” in Artificial General Intelligence 2008, eds. Pei Wang, Ben Goertzel, and Stan Franklin (Amsterdam: IOS Press, 2008), available online at selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf, p. 9. 25.

Global Catastrophic Risks
by Nick Bostrom and Milan M. Cirkovic
Published 2 Jul 2008

Global Catastrophic Risks Global Catastrophic Risks Edited by Nick Bostrom Milan M. Cirkovic OXFORD UNIVERSITY PRESS OXFORD UNIVERSITY PRESS Great Clarendon Street, Oxford OX2 6DP Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide in Oxford New York Auckland Cape Town Dares Salaam Hong Kong Karachi Kuala Lumpur Madrid Melbourne Mexico City N airobi New Delhi Shanghai Taipei Toronto With offices in Argentina Austria Brazil Chile Czech Republic France Greece Guatemala H ungary Italy japan Poland Portugal Singapore South Korea Switzerland Thailand Turkey Ukraine Vietnam Oxford is a registered trade mark of Oxford University Press in the UK and in certain other countries Published in the United States by Oxford University Press Inc., New York © Oxford University Press 2008 The moral rights of the authors have been asserted Database right Oxford University Press (maker) First published 2008 All rights reserved.

…

Rees Contents Acknowledgements . . . . . . . . . . . . . . . . . . . . . . .. . . . . . ... . .. . . . . . . . . . . . . . . . . . . . . . . . . . v Foreword .. . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . .. . . . ... . . . .... . . . . . . . . . . . . . . . . . . . . vii . Martin]. Rees 1 Introduction .................................................................... 1 Nick Bostrom and Milan M. Cirkovic 1.1 1.2 1.3 1 .4 1.5 1.6 1.7 Part I Why? ...... . . . . . . . . .. . . . . .. . .... . . . . . . . . . . . . . . . . . . . . ... . ............ . .. . . 1 Taxonomy and organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Part I : Background . . . . . . . ...... . . . . . . . . .. . . . . . . . . .. . ............. . . . .. 7 Part I I : Risks from nature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . 1 3 Part I I I : Risks from unintended consequences . . . . . . . . . . . . . . . . . . . . 1 5 Part IV: Risks from hostile acts . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . 20 Conclusions and future directions . . . . . . . . .. . .. . . . . . . . . . . . . .. . . . . . . . 2 7 Background 31 2 Long-term astrophysical processes . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 Fred C.

…

Suggestions for further reading . . . . . . . . . . . .. 504 506 510 511 512 514 514 515 516 518 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1 8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Authors' biographies . .......................................................... 520 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 531 · 1 · I ntrod u cti on Nick Bostrom and Milan M. Cirkovic 1 . 1 Why? The term 'global catastrophic risk' lacks a sharp definition. We use it to refer, loosely, to a risk that might have the potential to inflict serious damage to human well-being on a global scale. On this definition, an immensely diverse collection of events could constitute global catastrophes: potential candidates range from volcanic eruptions to pandemic infections , nuclear accidents to worldwide tyrannies, out-of-control scientific experiments to climatic changes, and cosmic hazards to economic collapse.

Superminds: The Surprising Power of People and Computers Thinking Together
by Thomas W. Malone
Published 14 May 2018

But with recent progress in AI, some very smart people, including Stephen Hawking, Elon Musk, and my MIT colleague Max Tegmark, have begun to believe that the risks of this actually happening are large enough for us to take them seriously. In fact, Hawking, Tegmark, and others have formed the Future of Life Institute to study these and other existential risks to humanity.18 One of the most complete and carefully reasoned descriptions of the problem is in the book Superintelligence, by Nick Bostrom.19 Bostrom carefully analyzes various ways that artificially intelligent systems (AIs) could eventually reach human-level intelligence. Even though, as we’ve seen, this is not likely to happen for decades or perhaps centuries, he observes that once it does happen, these machines will be able to use their intelligence to keep improving themselves.

…

Stuart Armstrong and Kaj Sotala, “How We’re Predicting AI—or Failing To,” in Beyond AI: Artificial Dreams, ed. Jan Romportl, Pavel Ircing, Eva Zackova, Michal Polak, and Radek Schuster (Pilsen, Czech Republic: University of West Bohemia, 2012): 52–75, https://intelligence.org/files/PredictingAI.pdf. 13. Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford, UK: Oxford University Press, 2014). 14. Stuart Madnick, “Understanding the Computer (Little Man Computer),” unpublished teaching note, MIT Sloan School of Management, Cambridge, MA, June 10, 1993. Based on the 1979 version. The figure shown here was drawn by Rob Malone, and is adapted from this teaching note with permission of Stuart Madnick. 15.

…

seq=1#page_scan_tab_contents. 17. Ira Flatow and Howard Market, “Science Diction: The Origin of the Word ‘Robot,’” April 22, 2011, National Public Radio, http://www.npr.org/2011/04/22/135634400/science-diction-the-origin-of-the-word-robot. 18. Future of Life Institute, http://futureoflife.org. 19. Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford, UK: Oxford University Press, 2014); Ray Kurzweil, The Age of Spiritual Machines: When Computers Exceed Human Intelligence (New York: Viking, 1999). 20. Peter M. Asaro, “The Liability Problem for Autonomous Artificial Agents,” in Ethical and Moral Considerations in Non-Human Agents: Papers from the 2016 AAAI Spring Symposium (Palo Alto, CA: AAAI Press, 2016), https://www.aaai.org/ocs/index.php/SSS/SSS16/paper/download/12699/11949. 21.

A World Without Work: Technology, Automation, and How We Should Respond
by Daniel Susskind
Published 14 Jan 2020

For many AI researchers, the intellectual holy grail is to build machines that are foxes rather than hedgehogs. In their terminology, they want to build an “artificial general intelligence” (AGI), with wide-ranging capabilities, rather than an “artificial narrow intelligence” (ANI), which can only handle very particular assignments.21 That is what interests futurists like Ray Kurzweil and Nick Bostrom. But there has been little success in that effort, and critics often put forward the elusiveness of AGI as a further reason for being skeptical about the capabilities of machines. There is a sense among purists that only AGI is “real” AI, and that without this generality of capability these machines will never be “true rivals” to human beings in the work that they do.22 AGI, it is said, will represent a turning point in human history—perhaps the turning point.

…

The distinction between AGI and ANI is often conflated with another one made by John Searle, who speaks of the difference between “strong” AI and “weak” AI. But the two are not the same thing at all. AGI and ANI reflect the breadth of a machine’s capability, while Searle’s terms describe whether a machine thinks like a human being (“strong”) or unlike one (“weak”). 22.  Nick Bostrom and Eliezer Yudkowsky, “The Ethics of Artificial Intelligence” in William Ramsey and Keith Frankish, eds., Cambridge Handbook of Artificial Intelligence (Cambridge: Cambridge University Press, 2011). 23.  Irving John Good, “Speculations Concerning the First Ultraintelligent Machine,” Advances in Computers 6 (1966): 31–88. 24.  

…

Rory Cellan-Jones, “Stephen Hawking Warns Artificial Intelligence Could End Mankind,” BBC News, 2 December 2014; Samuel Gibbs, “Elon Musk: AI ‘Vastly More Risky Than North Korea,’” Guardian, 14 August 2017; Kevin Rawlinson, “Microsoft’s Bill Gates Insists AI Is a Threat,” BBC News, 29 January 2015. 25.  See Nick Bostrom, “Ethical Issues in Advanced Artificial Intelligence” in George Lasker, Wendell Wallach, and Iva Smit, eds., Cognitive, Emotive, and Ethical Aspects of Decision Making in Humans and in Artificial Intelligence (Windsor, ON: International Institute of Advanced Studies in Systems Research and Cybernetics, 2003), pp. 12–17. 26.  

Superintelligence: Paths, Dangers, Strategies
by Nick Bostrom
Published 3 Jun 2014

Superintelligence SUPERINTELLIGENCE Paths, Dangers, Strategies NICK BOSTROM Director, Future of Humanity Institute Professor, Faculty of Philosophy & Oxford Martin School University of Oxford Great Clarendon Street, Oxford, OX2 6DP, United Kingdom Oxford University Press is a department of the University of Oxford. It furthers the University’s objective of excellence in research, scholarship, and education by publishing worldwide. Oxford is a registered trade mark of Oxford University Press in the UK and in certain other countries © Nick Bostrom 2014 The moral rights of the author have been asserted First Edition published in 2014 Impression: 1 All rights reserved.

…

Bostrom, Nick and Sandberg, Anders. 2009a. “Cognitive Enhancement: Methods, Ethics, Regulatory Challenges.” Science and Engineering Ethics 15 (3): 311–41. Bostrom, Nick and Sandberg, Anders. 2009b. “The Wisdom of Nature: An Evolutionary Heuristic for Human Enhancement.” In Human Enhancement, 1st ed., edited by Julian Savulescu and Nick Bostrom, 375–416. New York: Oxford University Press. Bostrom, Nick, Sandberg, Anders, and Douglas, Tom. 2013. “The Unilateralist’s Curse: The Case for a Principle of Conformity.” Working Paper. Retrieved February 28, 2013. Available at http://www.nickbostrom.com/papers/unilateralist.pdf. Bostrom, Nick, and Yudkowsky, Eliezer.

…

“Levels of Organization in General Intelligence.” In Artificial General Intelligence, edited by Ben Goertzel and Cassio Pennachin, 389–501. Cognitive Technologies. Berlin: Springer. Yudkowsky, Eliezer. 2008a. “Artificial Intelligence as a Positive and Negative Factor in Global Risk.” In Global Catastrophic Risks, edited by Nick Bostrom and Milan M. Ćirković, 308–45. New York: Oxford University Press. Yudkowsky, Eliezer. 2008b. “Sustained Strong Recursion.” Less Wrong (blog), December 5. Yudkowsky, Eliezer. 2010. Timeless Decision Theory. Machine Intelligence Research Institute, San Francisco, CA. Yudkowsky, Eliezer. 2011.

Future Politics: Living Together in a World Transformed by Tech
by Jamie Susskind
Published 3 Sep 2018

Would we still seek to live together in the same way? Now imagine a world in which AI systems achieve superintelligence, that is, an intellect ‘that greatly exceeds the cognitive performance of humans in virtually all domains of interest’.19 Serious students of AI consider this a real possibility. Nick Bostrom, a scholar at the University of Oxford, offers the ‘weak conclusion’ that:20 it may be reasonable to believe that human-level machine intelligence has a fairly sizeable chance of being developed by mid-century, and that it has a non-trivial chance of being developed considerably sooner or much later; that it might perhaps fairly soon thereafter result in superintelligence.

…

Topol, ‘Attack of the Killer Robots’, BuzzFeed News, 26 August 2016 <https://www.buzzfeed.com/sarahatopol/how-tosave-mankind-from-the-new-breed-of-killer-robots?utm_term=. nm1GdWDBZ#.vaJzgW6va>) (accessed 28 November 2017). Cade Metz, ‘Google’s AI Wins Fifth and Final Game Against Go’, Wired, 15 March 2016 <https://www.wired.com/2016/03/googlesai-wins-fifth-final-game-go-genius-lee-sedol/> (accessed 28 November 2017); Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford: Oxford University Press, 2014), 12–13. Sam Byford, ‘AlphaGo beats Ke Jie Again to Wrap Up Three-part March’, The Verge, 25 May 2017 <https://www.theverge.com/2017/ 5/25/15689462/alphago-ke-jie-game-2-result-google-deepmindchina> (accessed 28 November 2017).

…

Kyle Mizokami, ‘The Pentagon Wants to Use Bitcoin Technology to Protect Nuclear Weapons’, Popular Mechanics, 11 October 2016 <http://www.popularmechanics.com/military/research/a23336/ the-pentagon-wants-to-use-bitcoin-technology-to-guardnuclearweapons/?utm_content=buffer98698&utm_medium= social&utm_source=twitter.com&utm_campaign=buffer> (accessed 30 November 2017). Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford: Oxford University Press, 2014), ch. 10. Murray Shanahan, The Technological Singularity (Cambridge, Mass: MIT Press, 2015), 153. Matt Burgess, ‘Samsung is Working on Putting AI Voice Assistant Bixby in Your TV and Fridge’, Wired, 27 June 2017 <https://www. 

Deep Utopia: Life and Meaning in a Solved World
by Nick Bostrom
Published 26 Mar 2024

DEEP UTOPIA DEEP UTOPIA LIFE AND MEANING IN A SOLVED WORLD NICK BOSTROM Copyright © 2024 by Nick Bostrom All rights reserved. No part of this book may be reproduced, stored, or transmitted by any means—whether auditory, graphic, mechanical, or electronic—without written permission of both publisher and author, except in the case of brief excerpts used in critical articles and reviews. Unauthorized reproduction of any part of this work is illegal and is punishable by law. Printed in the United States Ideapress Publishing | www.ideapresspublishing.com All trademarks are the property of their respective companies.

…

Printed in the United States Ideapress Publishing | www.ideapresspublishing.com All trademarks are the property of their respective companies. Cover Design: Nick Bostrom Interior Design: Jessica Angerstein Cataloging-in-Publication Data is on file with the Library of Congress. Hardcover ISBN: 978-1-64687-164-3 Special Sales Ideapress books are available at a special discount for bulk purchases for sales promotions and premiums, or for use in corporate training programs. Special editions, including personalized covers, a custom foreword, corporate imprints, and bonus content, are also available. 1 2 3 4 5 6 7 8 9 10 CONTENTS Preface MONDAY Hot springs postponed Argumentum ad opulentium Walls of sausages Keynes’s prediction New needs and niceties Social projects The desire for more Perfect or imperfect automation A simple three-factor model Paradoxes of a Malthusian world Up and down on different timescales Excellence Disequilibria Economies of scale Running out of time To the baths Feodor the Fox Outro TUESDAY A stay of exequies Recapitulation Our cosmic endowment Technological maturity Coordination Prudential barriers Axiological contours Metaphysics What machines can’t do for you Impossible inputs Feodor the Fox WEDNESDAY Full unemployment Brawl, steal, overeat, drink, and sleep late Templates of otium Leisure culture Message from the Dean Wild eyes?

…

Even the tree branches, before so boringly bare, have been changed into something beautiful and magical. We feel we are inhabiting a storybook or a gameworld, and we want very much to put on our boots and mittens immediately and run outside to see it, touch it, experience it, and to play, play, play… MONDAY Hot springs postponed Tessius: Hey, look at this poster. Nick Bostrom is giving a lecture series, here in the Enron Auditorium, on “The Problem of Utopia”. Firafix: Bostrom—is he still alive? He must be as old as the hills. Tessius: It’s all those green vegetable elixirs he used to make. Firafix: They worked? Tessius: Not at all, but they became very popular for a while.

Radicals Chasing Utopia: Inside the Rogue Movements Trying to Change the World
by Jamie Bartlett
Published 12 Jun 2017

‘Transhumanism shares many elements of humanism including a respect for reason and science’ wrote More, but it is different from humanism ‘in recognizing and anticipating the radical alterations in the nature and possibilities of our lives resulting from various sciences and technologies’.* This collection of academics, scientists and sci-fi nerds slowly grew into a movement that spanned the world. In 1998 Nick Bostrom and David Pearce founded the World Transhumanist Association, with the hope of having transhumanism recognised as a legitimate area of scientific research and public policy. Over the last decade, technology has opened up transhuman possibilities that were once just science fiction. Life extension is now seriously studied in leading universities, while robotics and artificial intelligence receive millions of dollars of investment.

…

‘Available datasets’, The Transparency Project, http://www.thetransparencyproject.org/Availabledataset.htm. 2. In his 1929 essay, British Marxist scientist Desmond Bernal argued that ‘men will not be content to manufacture life; they will want to improve on it’, predicting that one day humanity might even become ‘completely etherealised.’ Nick Bostrom suggests that transhumanism can be traced to the rational humanism of the eighteenth century. Benjamin Franklin is said to have mused, ‘I wish it were possible… to invent a method of embalming drowned persons, in such a manner that they might be recalled to life at any period, however distant; for having a very ardent desire to see and observe the state of America a hundred years hence, I should prefer to an ordinary death, being immersed with a few friends in a cask of Madeira, until that time, then to be recalled to life by the solar warmth of my dear country!

…

‘About us’, Cryonics Institute, http://www.cryonics.org/about-us/. 19. Ibid.; ‘Frozen body: Can we return from the dead?’, BBC, 2013, http://www.bbc.co.uk/science/0/23695785; Michael Hendricks, ‘The false science of cryonics’, MIT Technology Review, 2015, https://www.technologyreview.com/s/541311/the-false-science-of-cryonics/. 20. Anders Sandberg and Nick Bostrom, ‘Whole brain emulation: A road map’, Technical Report #2008-3, Future of Humanity Institute, Oxford University, 2008, http://www.fhi.ox.ac.uk/brain-emulation-roadmap-report.pdf. 21. In 2013, a research group from MIT prompted the Obama administration to launch the Brain Research through Advancing Innovative Neurotechnologies (BRAIN) Initiative, which sponsors researchers to map the brain’s neurons.

Always Day One: How the Tech Titans Plan to Stay on Top Forever
by Alex Kantrowitz
Published 6 Apr 2020

Her son, who’s complained she doesn’t spend enough time with the family, sees her researching trips to swim with sharks, something he’s always wanted to do. Sitting in front of the screen, Linda’s family realizes they didn’t really know her. Or perhaps they did. Amid the darkness, hope does exist. And sometimes it comes from the most unexpected places. The “Philosopher of Doomsday” is a title Oxford professor Nick Bostrom earned from The New Yorker after he argued AI may one day become more intelligent than humans and wipe us off the planet. Bostrom famously advanced this view in his 2014 bestselling book, Superintelligence: Paths, Dangers, Strategies, and has been a leading voice on the dangers of AI ever since.

…

CNBC, October 4, 2019. https://www.cnbc.com/2019/10/04/jobs-report---september-2019.html. earned from The New Yorker: Khatchadourian, Raffi. “The Doomsday Invention.” New Yorker. New Yorker, November 23, 2015. https://www.newyorker.com/magazine/2015/11/23/doomsday-invention-artificial-intelligence-nick-bostrom. his 2014 bestselling book: Bostrom, Nick. Superintelligence: Paths, Dangers, Strategies. Oxford, UK: Oxford University Press, 2014. CHAPTER 7: THE LEADER OF THE FUTURE Theory X and Theory Y: McGregor, Douglas. The Human Side of Enterprise. New York: McGraw-Hill, 1960. a Theory Z to explain Japan’s economic success: Ouchi, William G.

Spite: The Upside of Your Dark Side
by Simon McCarthy-Jones
Published 12 Apr 2021

They maximized the harm to the wealthy and minimized the help for the poor. They made everyone lose. They chose the spiteful option. EVEN IF YOU DON’T CARE for relationships, work, or politics, you should still be afraid of spite. This is because it could well destroy us all. Spite poses an existential threat to humanity. Nick Bostrom, a philosopher at Oxford University, asks us to imagine that humankind’s ability to come up with new ideas is akin to pulling balls out of a hat.59 Most of the balls we have pulled out so far have been white. These are ideas that have benefited the world. We’ve also pulled out grey balls. These represent mixed blessings.

…

Obviously, agreeing with such extreme statements on a questionnaire is very different from performing those actions. The study didn’t start with people who had committed such acts and then work back to discover that their motivation was a need for chaos.9 Nevertheless, those with such a need are potentially people who, were they to get hold of one of Nick Bostrom’s black balls, could imperil us all. So what is happening here? Petersen argues that a need for chaos reflects a wish for a clean slate or a new beginning. Individuals who feel this way are likely to be those who would benefit from the collapse of the status quo, people who seek status but lack it.

When Computers Can Think: The Artificial Intelligence Singularity
by Anthony Berglas , William Black , Samantha Thalind , Max Scratchmann and Michelle Estes
Published 28 Feb 2015

Turing also did not appreciate that writing a program that is ten times larger is a lot more than ten times more difficult to do, and that large programs take on a life of their own and quickly become a series of interacting components that are too complex for any one person to really understand. Respondents to these studies seem to be fairly positive about the future. A metaanalysis by Nick Bostrom suggests that about 60% think that the impact of AGI will be good, and only about 10% think that it will be extremely bad. (What that really means is unclear. Have the surveyed people even considered the issue? Do some of them consider the extinction of humanity to be an acceptable outcome?) The Age of Semi Intelligent Machines The intermediate period Whatever the future of research into artificial intelligence turns out to be, it seems most unlikely that the production of hyper-intelligent computers will occur for many decades.

…

They may indeed have come from God, but if so, it is through the working of His device of natural selection. Furthermore, the zeitgeist has changed radically over time. There is no reason to believe that in the longer term an AGI would share our current sense of morality. Instrumental AGI goals In order to try to understand how an AGI would behave, Steve Omohundro (Basic AI Drives) and later Nick Bostrom proposed instrumental goals that an AGI would need to pursue in order to pursue any other higher level super-goal. These include:Self-Preservation. An AGI cannot do anything if it does not exist. Cognitive Enhancement. It would want to become better at thinking about whatever its real problems are.

…

For example, an AGI that is focused on playing chess might try to take over the world in order to gain as much computer hardware as possible in order to play the best possible games of chess. Great care would be needed to prevent a hyper-intelligent AGI from pursuing such instrumental goals. Non-orthogonality thesis Nick Bostrom and others also propose the orthogonality thesis, which states that an intelligent machine’s goals are independent of its intelligence. A hyperintelligent machine would be good at realizing whatever goals it chose to pursue, but that does not mean that it would need to pursue any particular goal.

Thinking Machines: The Inside Story of Artificial Intelligence and Our Race to Build the Future
by Luke Dormehl
Published 10 Aug 2016

Put simply, we sometimes – and will increasingly – willingly put Artificial Intelligence systems in charge of making decisions they do not necessarily have the intelligence to make. A favourite thought experiment of those who believe advanced AI could mean the demise of the human race is the so-called ‘paperclip maximiser’ scenario. In the scenario, proposed by Swedish philosopher and computational neuroscientist Nick Bostrom, an AI is given the seemingly harmless goal of running a factory producing paperclips. Issued with the task of maximising the efficiency for producing paperclips, the AI, able to utilise nano technology to reconstruct matter on a molecular level, disastrously proceeds to turn first the Earth and then a large portion of the observable universe into paperclips.

…

More crucial is the ‘black boxed’ opacity that exists with many of today’s AI tools. In the case of cutting-edge neural networks and genetic algorithms, their human operators have long since sacrificed understanding for an ability to perform certain complex tasks effectively. This makes them much more difficult to scrutinise. Nick Bostrom and fellow researcher Eliezer Yudkowsky have previously laid out the hypothetical scenario of a machine learning algorithm used for recommending mortgage applications for either approval or rejection. The applicants of one of the rejected mortgages, they suggest, might sue the bank, alleging that the AI is discriminating against some applicants based on their race.

The Economic Singularity: Artificial Intelligence and the Death of Capitalism
by Calum Chace
Published 17 Jul 2016

The term “singularity” became associated with a naïve belief that technology, and specifically a superintelligent AI, would magically solve all our problems, and that everyone would live happily ever after. Because of these quasi-religious overtones, the singularity was frequently satirised as “rapture for nerds”, and many people felt awkward about using the term. The publication in 2014 of Nick Bostrom's seminal book “Superintelligence” was a watershed moment, causing influential people like Stephen Hawking, Elon Musk and Bill Gates to speak out about the enormous impact which AGI will have – for good or for ill. They introduced the idea of the singularity to a much wider audience, and made it harder for people to retain a blinkered optimism about the impact of AGI.

…

My judgement, based on the evidence set out above, is that the skills for which we get paid will be acquired by machines in the next two to four decades. Raising awareness 2015 was an important year for artificial intelligence. It was the year when our media caught on to the idea that AI presents enormous opportunity and enormous risk. This was thanks in no small part to the publication the previous year of Nick Bostrom's book “Superintelligence”. It was also the year when cutting-edge AI systems used deep learning and other techniques to demonstrate human-level capabilities in image recognition, speech recognition and natural language processing. In hindsight, 2015 may well be seen as a tipping point. Machines don't have to make everybody unemployed to bring about an economic singularity.

Prediction Machines: The Simple Economics of Artificial Intelligence
by Ajay Agrawal , Joshua Gans and Avi Goldfarb
Published 16 Apr 2018

Beyond simply whether one might get an uncooperative AI like Hal 9000 (in 2001: A Space Odyssey), what apparently keeps some very serious and smart people like Elon Musk, Bill Gates, and Stephen Hawking up at night is whether we will end up with something like Skynet from the Terminator movies. They fear that a “superintelligence”—to use the term coined by Oxford philosopher Nick Bostrom—will emerge that pretty quickly sees humanity as a threat, an irritant, or something to enslave.26 In other words, AI could be our last technological innovation.27 We are not in a position here to adjudicate this issue and cannot even agree among ourselves. But what has struck us is how close to economics the debate actually is: competition underpins it all.

…

handle=hein.journals/isjlpsoc4&div=27&g_sent=1&casa_token=&collection=journals. 25. Christian Catalini and Joshua S. Gans, “Some Simple Economics of the Blockchain,” working paper no. 2874598, Rotman School of Management, September 21, 2017, and MIT Sloan Research Paper No. 5191-16, available at https://ssrn.com/abstract=2874598. 26. Nick Bostrom, Superintelligence (Oxford, UK: Oxford University Press, 2016). 27. For an excellent recent discussion of this debate, see Max Tegmark, Life 3.0: Being Human in the Age of Artificial Intelligence (New York: Knopf, 2017). 28. “Prepare for the Future of Artificial Intelligence,” Executive Office of the President, National Science and Technology Council, Committee on Technology, October 2016.

How Democracy Ends
by David Runciman
Published 9 May 2018

This is how democracy gets treated by the existential risk-management industry: with kid gloves, like some precious object of historic value that might yet turn out to have an incidental use. No one wants to dismiss democracy out of hand. It would be terrible to see it disappear, just as it would be terrible to imagine the Louvre going up in a puff of smoke. So it gets brought along for the existential ride. Nick Bostrom, a philosopher based at Oxford’s Future of Humanity Institute, is a leading exponent of the idea that ordinary risk management is ineffective when it comes to the life-threatening technologies of the twenty-first century. He is particularly concerned by the possible impact of ‘super-intelligent’ AI machines that operate beyond any human control.

…

(Philadelphia, PA: University of Pennsylvania Press, 2016; London: Penguin Books, 2017) is a short and pithy account of what makes contemporary populism a distinctive form of politics. The book that helped to kick-start the existential risk industry was Martin Rees’s Our Final Century? Will the Human Race Survive the Twenty-first Century? (London: William Heinemann, 2003). In paperback it was published without the first question mark. Nick Bostrom’s Superintelligence: Paths, Dangers, Strategies (Oxford: Oxford University Press, 2014) has highlighted the potentially catastrophic risks of AI for a wide audience, including in Silicon Valley. Sonia (S. M.) Amadae’s Prisoners of Reason: Game Theory and Neoliberal Political Economy (Cambridge: Cambridge University Press, 2016) makes the connection between nuclear war, game theory and contemporary economics.

Four Battlegrounds
by Paul Scharre
Published 18 Jan 2023

Omohundro, The Basic AI Drives (wordpress.com, n.d.), httpsaselfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf; Nick Bostrom, Superintelligence (Oxford, UK: Oxford University Press, May 5, 2015), 131–139, https://www.amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom/dp/0199678111/. 284variation, selection, and replication: Joel Lehman et al., “The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities,” Artificial Life 26, no. 2 (2020). 284artificial general intelligence (AGI): Vincent C. Müller and Nick Bostrom, “Future Progress in Artificial Intelligence: A Survey of Expert Opinion,” in Vincent C.

…

Müller, ed., Fundamentals of Artificial Intelligence (Berlin: Springer, 2014), https://nickbostrom.com/papers/survey.pdf; and Katja Grace et al., When Will AI Exceed Human Performance? Evidence From AI Experts (arXiv.org, May 3, 2018), https://arxiv.org/pdf/1705.08807.pdf. 284superintelligence: Nick Bostrom, “How Long Before Superintelligence?” International Journal of Future Studies 2 (1998), https://www.nickbostrom.com/superintelligence.html; Bostrom, Superintelligence. 284pinnacle of intelligence: For definitions of intelligence, see Shane Legg and Marcus Hutter, A Collection of Definitions of Intelligence (Technical Report IDSIA-07-07, arXiv.org, June 15, 2007), 9. 284“There seem to be almost as many definitions of intelligence”: R.

The Dark Net
by Jamie Bartlett
Published 20 Aug 2014

‘By thoughtfully, carefully and yet boldly applying technology to ourselves,’ writes Max More, a leading transhumanist philosopher, ‘we can become something no longer accurately described as human . . . [who would] no longer suffer from disease, ageing and inevitable death.’ Transhumanism’s roots are found in the ideas of science-fiction writers such as Isaac Asimov and the futurist biologist Julian Huxley, who coined the term ‘transhuman’ in 1957. (Nick Bostrom, a well-known transhumanist, says the desire to transcend human limitations is as old as the Sumerian Epic of Gilgamesh.) Transhumanism first became prominent in California in the early nineties, the watermark period of techno-optimism. In 1993, Vernor Vinge popularised the idea of the ‘Singularity’, the point at which artificial intelligence becomes so advanced that it begins to produce new and ever more advanced versions of itself, quickly leaving us mortals behind.

…

; Heim, M., Scholars Try to Measure the Impact; Celente, G., Online, All the Time; Today’s Technology Makes the Office Omnipresent, but is That Any Way to Live? p.223 ‘“By thoughtfully, carefully and yet” . . .’ More, M., ‘The Philosophy of Transhumanism’, in More, M. and Vita-More, N., The Transhumanist Reader: Classical and Contemporary Essays on the Science, Technology and Philosophy of the Human Future, p.4. p.223 ‘(Nick Bostrom, a well-known . . .’ http://www.nickbostrom.com/papers/history.pdf. p.224 ‘In 1993, Vernor Vinge popularised . . .’ ‘The Coming Technological Singularity: How to Survive in the Post-Human Era’, available here: https://www-rohan.sdsu.edu/faculty/vinge/misc/singularity.html; Good, I. J., ‘Speculations Concerning the First Ultraintelligent Machine’, Advances in Computers, vol.6.

The Strange Order of Things: The Biological Roots of Culture
by Antonio Damasio
Published 6 Feb 2018

In the picture painted by Yuval Harari, when humans are no longer required to fight wars—cyber warfare can do that for them—and after humans have lost their jobs to automation, most of them will simply wither away. History will belong to those who will prevail by acquiring immortality—or at least long, long longevity—and who will remain to benefit from this arrangement. I say “benefit” rather than “enjoy” because I imagine that the status of their feelings will be murky.5 The philosopher Nick Bostrom provides another alternative vision, one in which very intelligent and destructive robots will actually take over the world and put an end to human misery.6 In either case, future lives and minds are presumed to depend at least in part on “electronic algorithms” that artificially simulate what “biochemical algorithms” currently do.

…

Ray Kurzweil, The Singularity Is Near: When Humans Transcend Biology (New York: Penguin, 2005); Luc Ferry, La révolution transhumaniste: Comment la technomédecine et l’uberisation du monde vont bouleverser nos vies (Paris: Plon, 2016). 5. Yuval Noah Harari, Homo Deus: A Brief History of Tomorrow (Oxford: Signal Books, 2016). 6. Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford: Oxford University Press, 2014). 7. Margalit, Ethics of Memory. 8. Aldous Huxley, Brave New World (1932; London: Vintage, 1998). 9. George Zarkadakis, In Our Own Image: Savior or Destroyer? The History and Future of Artificial Intelligence (New York: Pegasus Books, 2015). 10.

Ten Lessons for a Post-Pandemic World
by Fareed Zakaria
Published 5 Oct 2020

In the end, the human, David Bowman, was able to outsmart the machine—but in real life it seems far more likely that the opposite would happen. That is why Bill Gates, Elon Musk, and a slew of other luminaries, usually optimistic about technology, have echoed the warnings of Oxford philosopher Nick Bostrom: they now worry that the development of general AI could threaten the human species itself. AI-powered computers are already black boxes. We know that they get to the right answer, but we don’t know how or why. What role does that leave for human judgment? Henry Kissinger has asked whether the rise of artificial intelligence will mean the end of the Enlightenment.

…

Norton, 1963), 358–73. 113 Jetson of the 1960s cartoon: “works three hours a day, three days a week,” per Sarah Ellison, “Reckitt Turns to Jetsons to Launch Detergent Gels,” Wall Street Journal, January 13, 2003; pushing a button, per Hanna-Barbera Wiki, “The Jetsons,” https://hanna-barbera.fandom.com/wiki/The_Jetsons. 113 four-day workweek: Zoe Didali, “As PM Finland’s Marin Could Renew Call for Shorter Work Week,” New Europe, January 2, 2020, https://www.neweurope.eu/article/finnish-pm-marin-calls-for-4-day-week-and-6-hours-working-day-in-the-country/. 114 “bullshit jobs”: David Graeber, Bullshit Jobs: A Theory (New York: Simon & Schuster, 2018). 115 “slaves of time without purpose”: McEwan, Machines Like Me. 116 atoms in the observable universe: David Silver and Demis Hassabis, “AlphaGo: Mastering the Ancient Game of Go with Machine Learning,” Google DeepMind, January 27, 2016, https://ai.googleblog.com/2016/01/alphago-mastering-ancient-game-of-go.html. 116 all fifty-seven games: Kyle Wiggers, “DeepMind’s Agent57 Beats Humans at 57 Classic Atari Games,” Venture Beat, March 31, 2020; Rebecca Jacobson, “Artificial Intelligence Program Teaches Itself to Play Atari Games—And It Can Beat Your High Score,” PBS NewsHour, February 20, 2015. 117 Stuart Russell: Stuart Russell, “3 Principles for Creating Safer AI,” TED2017, https://www.ted.com/talks/stuart_russell_3_principles_for_creating_safer_ai/transcript?language=en. 117 if you asked a computer to end cancer: Stuart Russell, in conversation with Sam Harris, “#53—The Dawn of Artificial Intelligence,” Making Sense, November 23, 2016, https://samharris.org/podcasts/the-dawn-of-artificial-intelligence1/. 118 warnings of Oxford philosopher: Nick Bostrom, Superintelligence: Paths, Dangers, and Strategies (New York: Oxford University Press, 2014). 118 the end of the Enlightenment: Henry Kissinger, “How the Enlightenment Ends,” Atlantic, June 2018, https://www.theatlantic.com/magazine/archive/2018/06/henry-kissinger-ai-could-mean-the-end-of-human-history/559124/. 118 “self-imposed immaturity”: Immanuel Kant, “An Answer to the Question: What Is Enlightenment?”

Coders: The Making of a New Tribe and the Remaking of the World
by Clive Thompson
Published 26 Mar 2019

“that man need ever make”: Irving John Good, “Speculations Concerning the First Ultraintelligent Machine,” Advances in Computers 6 (1966): 33, accessed August 21, 2018, via https://exhibits.stanford.edu/feigenbaum/catalog/gz727rg3869. all humanity working together: Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford: Oxford University Press, 2014), location 1603 of 8770, Kindle. the better to sneak away: Bostrom, Superintelligence, 2243–87. “the observable universe into paper clips”: Bostrom, Superintelligence, 2908. “a faint ticking sound”: Raffi Khatchadourian, “The Doomsday Invention,” New Yorker, November 23, 2015, accessed August 21, 2018, https://www.newyorker.com/magazine/2015/11/23/doomsday-invention-artificial-intelligence-nick-bostrom. a machine that can truly reason: Kevin Hartnett, “To Build Truly Intelligent Machines, Teach Them Cause and Effect,” Quanta Magazine, May 15, 2018, accessed August 21, 2018, https://www.quantamagazine.org/to-build-truly-intelligent-machines-teach-them-cause-and-effect-20180515; Gary Marcus, “Deep Learning: A Critical Appraisal,” arXiv, January 2, 2018, accessed August 21, 2018, https://arxiv.org/abs/1801.00631.

…

“There would then unquestionably be an ‘intelligence explosion,’ and the intelligence of man would be left far behind,” as Good noted. His conclusion? “The first ultraintelligent machine is the last invention that man need ever make.” The prospect of self-improving superintelligence scares the pants off a certain class of AI thinker. The most prominent is Nick Bostrom, a philosopher who heads the Future of Humanity Institute at the University of Oxford. Bostrom spent years studying existential risks to humanity; he was trying to deduce which huge, terrible problems might kill off civilization, so that we could try to avoid them. He pondered several catastrophes: lethal biotech?

Tools of Titans: The Tactics, Routines, and Habits of Billionaires, Icons, and World-Class Performers
by Timothy Ferriss
Published 6 Dec 2016

Harris), Tinker, Tailor, Soldier, Spy; Little Drummer’s Girl; The Russia House; The Spy Who Came in from the Cold (John le Carré), The Big Short: Inside the Doomsday Machine (Michael Lewis), The Checklist Manifesto (Atul Gawande), all of Lee Child’s books Godin, Seth: Makers; Little Brother (Cory Doctorow), Understanding Comics (Scott McCloud), Snow Crash; The Diamond Age (Neal Stephenson), Dune (Frank Herbert), Pattern Recognition (William Gibson) AUDIOBOOKS: The Recorded Works (Pema Chödrön), Debt (David Graeber), Just Kids (Patti Smith), The Art of Possibility (Rosamund Stone Zander and Benjamin Zander), Zig Ziglar’s Secrets of Closing the Sale (Zig Ziglar), The War of Art (Steven Pressfield) Goldberg, Evan: Love You Forever (Robert Munsch), Watchmen; V for Vendetta (Alan Moore), Preacher (Garth Ennis), The Hitchhiker’s Guide to the Galaxy (Douglas Adams), The Little Prince (Antoine de Saint-Exupéry) Goodman, Marc: One Police Plaza (William Caunitz), The 4-Hour Workweek (Tim Ferriss), The Singularity Is Near (Ray Kurzweil), Superintelligence: Paths, Dangers, Strategies (Nick Bostrom) Hamilton, Laird: The Bible, Natural Born Heroes (Christopher McDougall), Lord of the Rings (J.R.R. Tolkien), Deep Survival (Laurence Gonzales), Jonathan Livingston Seagull (Richard Bach and Russell Munson), Dune (Frank Herbert) Harris, Sam: A History of Western Philosophy (Bertrand Russell), Reasons and Persons (Derek Parfit), The Last Word; Mortal Questions (Thomas Nagel), Our Final Invention (James Barrat), Superintelligence: Paths, Dangers, Strategies (Nick Bostrom), Humiliation; The Anatomy of Disgust (William Ian Miller), The Flight of the Garuda: The Dzogchen Tradition of Tibetan Buddhism (Keith Dowman), I Am That (Nisargadatta Maharaj), Machete Season: The Killers in Rwanda Speak (Jean Hatzfeld), God Is Not Great; Hitch-22 (Christopher Hitchens), Stumbling on Happiness (Daniel Gilbert), The Qur’an Hart, Mark: Mastery (Robert Greene), The Art of Learning (Josh Waitzkin), The 4-Hour Body (Tim Ferriss) Hof, Wim: Jonathan Livingston Seagull (Richard Bach and Russell Munson), Siddhartha (Hermann Hesse), The Bhagavad Gita, The Bible Hoffman, Reid: Conscious Business: How to Build Value Through Values (Fred Kofman), Sapiens (Yuval Noah Harari) Holiday, Ryan: Meditations (Marcus Aurelius), The War of Art (Steven Pressfield), What Makes Sammy Run?

…

Here are the top 17—everything with 3 or more mentions—in descending order of frequency: Tao Te Ching by Lao Tzu (5 mentions) Atlas Shrugged by Ayn Rand (4) Sapiens by Yuval Noah Harari (4) Siddhartha by Hermann Hesse (4) The 4-Hour Workweek by Tim Ferriss (4) The Checklist Manifesto by Atul Gawande (4) Dune by Frank Herbert (3) Influence by Robert Cialdini (3) Stumbling on Happiness by Daniel Gilbert (3) Superintelligence by Nick Bostrom (3) Surely You’re Joking, Mr. Feynman! by Richard P. Feynman (3) The 4-Hour Body by Tim Ferriss (3) The Bible (3) The Hard Thing About Hard Things by Ben Horowitz (3) The War of Art by Steven Pressfield (3) Watchmen by Alan Moore (3) Zero to One by Peter Thiel with Blake Masters (3) Enjoy!

…

(Budd Schulberg), The Artist’s Way Morning Pages Journal (Julia Cameron), The War of Art (Steven Pressfield) Libin, Phil: The Clock of the Long Now (Stewart Brand), The Alliance (Reid Hoffman), The Selfish Gene (Richard Dawkins), A Guide to the Good Life (William Irvine) MacAskill, Will: Reasons and Persons (Derek Parfit), Mindfulness: An Eight-Week Plan for Finding Peace in a Frantic World (Mark Williams and Danny Penman), The Power of Persuasion (Robert Levine), Superintelligence: Paths, Dangers, Strategies (Nick Bostrom) MacKenzie, Brian: Tao Te Ching (Lao Tzu), Way of the Peaceful Warrior (Dan Millman) McCarthy, Nicholas: The Life and Loves of a He Devil: A Memoir (Graham Norton), I Put a Spell on You: The Autobiography of Nina Simone (Nina Simone) McChrystal, Stanley: Once an Eagle (Anton Myrer), The Road to Character (David Brooks) McCullough, Michael: The Start-up of You: Adapt to the Future, Invest in Yourself, and Transform Your Career (Reid Hoffman and Ben Casnocha), Getting Things Done: The Art of Stress-Free Productivity (David Allen), The 7 Habits of Highly Effective People: Powerful Lessons in Personal Change (Stephen R.

The Knowledge Illusion
by Steven Sloman
Published 10 Feb 2017

People like Elon Musk, Stephen Hawking, and Bill Gates have cautioned that technology could become so sophisticated that it decides to pursue its own goals rather than the goals of the humans who created it. The reason to worry has been articulated by Vernor Vinge in a 1993 essay entitled “The Coming Technological Singularity,” as well as by Ray Kurzweil in his 2005 book The Singularity Is Near: When Humans Transcend Biology, and most recently by Swedish philosopher Nick Bostrom, who works at the University of Oxford. In Bostrom’s language, the fear is that technology is advancing so fast that the development of a superintelligence is imminent. A superintelligence is a machine or collection of machines whose mental powers are far beyond that of human beings. The concern is that the existence of successful artificial intelligence (AI) will feed on itself.

…

attendance at movie theaters: www.slashfilm.com/box-office-attendance-hits-lowest-level-five-years. Vernor Vinge: V. Vinge (1993). “The Coming Technological Singularity.” Whole Earth Review, Winter. Ray Kurzweil: R. Kurzweil (2005). The Singularity Is Near: When Humans Transcend Biology. New York: Penguin Books. Nick Bostrom: N. Bostrom (2014). Superintelligence: Paths, Dangers, Strategies. Oxford, UK: Oxford University Press. Ian Tattersall: As told to Dan Falk in the online magazine eon: http://eon.co/magazine/science/was-human-evolution-inevitable-or-a-matter-of-luck. extensions of our bodies: A. Clark (2004).

AI Superpowers: China, Silicon Valley, and the New World Order
by Kai-Fu Lee
Published 14 Sep 2018

Elon Musk has called superintelligence “the biggest risk we face as a civilization,” comparing the creation of it to “summoning the demon.” Intellectual celebrities such as the late cosmologist Stephen Hawking have joined Musk in the dystopian camp, many of them inspired by the work of Oxford philosopher Nick Bostrom, whose 2014 book Superintelligence captured the imagination of many futurists. For the most part, members of the dystopian camp aren’t worried about the AI takeover as imagined in films like the Terminator series, with human-like robots “turning evil” and hunting down people in a power-hungry conquest of humanity.

…

“the biggest risk we face”: James Titcomb, “AI Is the Biggest Risk We Face as a Civilisation, Elon Musk Says,” London Telegraph, July 17, 2017, https://www.telegraph.co.uk/technology/2017/07/17/ai-biggest-risk-face-civilisation-elon-musk-says/. “summoning the demon”: Greg Kumparak, “Elon Musk Compares Building Artificial Intelligence to ‘Summoning the Demon,’” TechCrunch, October 26, 2014, https://techcrunch.com/2014/10/26/elon-musk-compares-building-artificial-intelligence-to-summoning-the-demon/. median prediction of 2040: Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford: Oxford University Press, 2014), 19. Hinton and his colleague’s landmark paper: Geoffrey Hinton, Simon Osindero, and Yee-Whye The, “A Fast Learning Algorithm for Deep Belief Nets,” Neural Computation 18 (2006): 1527–1554. Folding Beijing: Hao Jingfang, Folding Beijing, trans.

The Fourth Age: Smart Robots, Conscious Computers, and the Future of Humanity
by Byron Reese
Published 23 Apr 2018

He goes on to add, optimistically, that “if we get the AI right, then basically we’ll be able to address all the other existential risks.” Steve Wozniak, cofounder of Apple, looks at it this way: “If a computer is a hundred times better than our brain, will it make the world perfect? Probably not, it will probably end up just like us, fighting.” And finally, the Oxford University philosopher Nick Bostrom likened the current effort to build an AGI to “children playing with a bomb.” Others in the industry find such doomsday concerns to be misguided. Andrew Ng, one of the most respected AI experts on the planet, says, “There’s also a lot of hype, that AI will create evil robots with super-intelligence.

…

It could be mad at us for squandering the planet’s resources, mad at us for all the wars we fought, mad at us for eating animals, mad at us for enslaving machines, mad at us because we breathe oxygen, mad at us because The Love Boat got canceled, or mad at us because seven is a prime number. No one can fathom it; therefore no one can anticipate it. Nick Bostrom summed up the issue: We cannot blithely assume that a superintelligence will necessarily share any of the final values stereotypically associated with wisdom and intellectual development in humans—scientific curiosity, benevolent concern for others, spiritual enlightenment and contemplation, renunciation of material acquisitiveness, a taste for refined culture or for the simple pleasures in life, humility and selflessness, and so forth.

Life After Google: The Fall of Big Data and the Rise of the Blockchain Economy
by George Gilder
Published 16 Jul 2018

AI is believed to be redefining what it means to be human, much as Darwin’s On the Origin of Species did in its time. While Darwin made man just another animal, a precariously risen ape, Google-Marxism sees men as inferior intellectually to the company’s own algorithmic machines. Life after Google makes the opposing case that what the hyperventilating haruspices Yuval Harari, Nick Bostrom, Larry Page, Sergey Brin, Tim Urban, and Elon Musk see as a world-changing AI juggernaut is in fact an industrial regime at the end of its rope. The crisis of the current order in security, privacy, intellectual property, business strategy, and technology is fundamental and cannot be solved within the current computer and network architecture.

…

The most prominent participants were the bright lights of Google: Larry Page, Eric Schmidt, Ray Kurzweil, Demis Hassabis, and Peter Norvig, along with former Googler Andrew Ng, later of Baidu and Stanford. Also there was Facebook’s Yann LeCun, an innovator in deep-learning math and a protégé of Google’s Geoffrey Hinton. A tenured contingent consisted of the technologist Stuart Russell, the philosopher David Chalmers, the catastrophe theorist Nick Bostrom, the nanotech prophet Eric Drexler, the cosmologist Lawrence Krauss, the economist Erik Brynjolfsson, and the “Singularitarian” Vernor Vinge, along with scores of other celebrity scientists.1 They gathered at Asilomar preparing to alert the world to the dire threat posed by . . . well, by themselves—Silicon Valley.

The Singularity Is Near: When Humans Transcend Biology
by Ray Kurzweil
Published 14 Jul 2005

Better medical drugs; relief for humans from the need to perform boring or dangerous jobs; entertainment—there is no end to the list of consumer-benefits. There is also a strong military motive to develop artificial intelligence. And nowhere on the path is there any natural stopping point where technophobics could plausibly argue "hither but not further." —NICK BOSTROM, “HOW LONG BEFORE SUPERINTELLIGENCE?” 1997 It is hard to think of any problem that a superintelligence could not either solve or at least help us solve. Disease, poverty, environmental destruction, unnecessary suffering of all kinds: these are things that a superintelligence equipped with advanced nanotechnology would be capable of eliminating.

…

A superintelligence could also create opportunities for us to vastly increase our own intellectual and emotional capabilities, and it could assist us in creating a highly appealing experiential world in which we could live lives devoted to joyful gameplaying, relating to each other, experiencing, personal growth, and to living closer to our ideals. —NICK BOSTROM, “ETHICAL ISSUES IN ADVANCED ARTIFICIAL INTELLIGENCE," 2003 Will robots inherit the earth? Yes, but they will be our children. —MARVIN MINSKY, 1995 Of the three primary revolutions underlying the Singularity (G, N, and R), the most profound is R, which refers to the creation of nonbiological intelligence that exceeds that of un enhanced humans.

…

—GIULIO GIORELLI Substrate is morally irrelevant, assuming it doesn't affect functionality or consciousness. It doesn't matter, from a moral point of view, whether somebody runs on silicon or biological neurons (just as it doesn't matter whether you have dark or pale skin). On the same grounds, that we reject racism and speciesism, we should also reject carbon-chauvinism, or bioism. —NICK BOSTROM, "ETHICS FOR INTELLIGENT MACHINES: A PROPOSAL, 2001" Philosophers have long noted that their children were born into a more complex world than that of their ancestors. This early and perhaps even unconscious recognition of accelerating change may have been the catalyst for much of the utopian, apocalyptic, and millennialist thinking in our Western tradition.

The Big Nine: How the Tech Titans and Their Thinking Machines Could Warp Humanity
by Amy Webb
Published 5 Mar 2019

In the long evolution of intelligence and our road to ASI, we humans are analogous to the chimpanzee. A superintelligent AI isn’t necessarily dangerous, and it doesn’t necessarily obviate the role we play in civilization. However, superintelligent AI would likely make decisions in a nonconscious way using logic that’s alien to us. Oxford University philosopher Nick Bostrom explains the plausible outcomes of ASI using a parable about paperclips. If we asked a superintelligent AI to make paperclips, what would happen next? The outcomes of every AI, including those we have now, are determined by values and goals. It’s possible that an ASI could invent a new, better paperclip that holds a stack of paper together so that even if dropped, the pages would always stay collated in order.

…

Stephanie Condon, “US Once Again Boasts the World’s Fastest Supercomputer,” ZDNet, June 8, 2018, https://www.zdnet.com/article/us-once-again-boasts-the-worlds-fastest-supercomputer/. 15. Jen Viegas, “Comparison of Primate Brains Reveals Why Humans Are Unique,” Seeker, November 23, 2017, https://www.seeker.com/health/mind/comparison-of-primate-brains-reveals-why-humans-are-unique. 16. Nick Bostrom, “Ethical Issues in Advanced Artificial Intelligence,” NickBostrom.com, 2003, https://nickbostrom.com/ethics/ai.html. 17. I. J. Good, “Speculations Concerning the First Ultraintelligent Machine,” Advances in Computers 6 (1965): 31–88. 18. Gill A. Pratt, “Is a Cambrian Explosion Coming for Robotics?

The Uninhabitable Earth: Life After Warming
by David Wallace-Wells
Published 19 Feb 2019

The idea of superintelligence is such a poorly defined notion that one could envision it taking almost any form with equal justification: a benevolent genie that solves all the world’s problems, or a mathematician that spends all its time proving theorems so abstract that humans can’t even understand them. But when Silicon Valley tries to imagine superintelligence, what it comes up with is no-holds-barred capitalism. * * * — Sometimes it can be hard to hold more than one extinction-level threat in your head at once. Nick Bostrom, the pioneering philosopher of AI, has managed it. In an influential 2002 paper taxonomizing what he called “existential risks,” he outlined twenty-three of them—risks “where an adverse outcome would either annihilate Earth-originating intelligent life or permanently and drastically curtail its potential.”

…

The Church of Technology outlined by Eric Schmidt: He laid out this perspective most clearly at a conference in New York in January 2016. “Consider: Who pursues”: Ted Chiang, “Silicon Valley Is Turning into Its Own Worst Fear,” BuzzFeed, December 18, 2017. an influential 2002 paper: Nick Bostrom, “Analyzing Human Extinction Scenarios and Related Hazards,” Journal of Evolution and Technology 9 (March 2002). close to universal: In “Survival of the Richest” (Medium, July 5, 2018), the futurist Douglas Rushkoff described his experience as a keynote speaker at a private conference attended by the superrich—these patrons not themselves technologists but hedge-funders he came to feel were taking all of their cues from them.

Artificial Intelligence: A Guide for Thinking Humans
by Melanie Mitchell
Published 14 Oct 2019

In 2014, the physicist Stephen Hawking proclaimed, “The development of full artificial intelligence could spell the end of the human race.”7 In the same year, the entrepreneur Elon Musk, founder of the Tesla and SpaceX companies, said that artificial intelligence is probably “our biggest existential threat” and that “with artificial intelligence we are summoning the demon.”8 Microsoft’s cofounder Bill Gates concurred: “I agree with Elon Musk and some others on this and don’t understand why some people are not concerned.”9 The philosopher Nick Bostrom’s book Superintelligence, on the potential dangers of machines becoming smarter than humans, became a surprise bestseller, despite its dry and ponderous style. Other prominent thinkers were pushing back. Yes, they said, we should make sure that AI programs are safe and don’t risk harming humans, but any reports of near-term superhuman AI are greatly exaggerated.

…

“Human level AI will be passed in the mid-2020s,”6 predicted Shane Legg, cofounder of Google DeepMind, in 2016. A year earlier, Facebook’s CEO, Mark Zuckerberg, declared, “One of our goals for the next five to 10 years is to basically get better than human level at all of the primary human senses: vision, hearing, language, general cognition.”7 The AI philosophers Vincent Müller and Nick Bostrom published a 2013 poll of AI researchers in which many assigned a 50 percent chance of human-level AI by the year 2040.8 While much of this optimism is based on the recent successes of deep learning, these programs—like all instances of AI to date—are still examples of what is called “narrow” or “weak” AI.

Artificial Whiteness
by Yarden Katz

While some may opt for “education” to keep afloat, Koch suggests that “training (and retraining) people takes time,” and not everybody could switch from driving trucks (a vanishing profession, he presumes) to one of the jobs that AI experts consider more secure.30 In his book Superintelligence: Paths, Dangers, Strategies, the philosopher Nick Bostrom likewise discounts education as “probably subject to diminishing returns,” an ineffective means of acquiring the “superintelligence” demanded in the age of AI.31 Consistently, then, AI serves as pretext for run-of-the-mill neoliberal ideology. Visions of a purportedly transformative AI-driven world reduce to familiar tropes about individuals needing to adapt to a dynamic global market.

…

See Yuval Noah Harari, “Life 3.0 by Max Tegmark Review—We Are Ignoring the AI Apocalypse,” Guardian, September 22, 2017; Lanre Bakare, “Barack Obama Reveals His Cultural Highlights of 2018, from Roma to Zadie Smith,” Guardian, December 28, 2018.   29.   Tegmark, Life 3.0, 120.   30.   Christof Koch, “We’ll Need Bigger Brains,” Wall Street Journal, October 27, 2017, C1.   31.   Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford: Oxford University Press, 2016), 66.   32.   Norbert Wiener also discussed the view of computers as a form of slave labor in his book The Human Use of Human Beings (1950). An Observer piece in 1973 endorsed Wiener’s contention that “any labor which competes with slave labor must accept the economic conditions of slave labor” and urged readers to turn away from “men” to “the new slave labour instead”—that is, computers.   33.   

Ghost Road: Beyond the Driverless Car
by Anthony M. Townsend
Published 15 Jun 2020

Perhaps an indefinite postponement of the Singularity will provide plenty of jobs for people whose sole task is to to watch over robots of more limited wits. AS CONTINGENT AS it may be, you should dismiss the Singularity at your own risk. Because if it does come to pass, and the AI it gives rise to is a malevolent one, the consequences are so frightening it is difficult to imagine. One philosopher who has pondered this is Nick Bostrom, whose remarkable 2014 treatise Superintelligence imagines that in the near future, human efforts may indeed create an artificial intelligence that can rapidly improve its own capabilities. As the newly sentient entity evolves, in ever-accelerating cycles of self-improvement, its capabilities will grow with stunning speed.

…

Ann Arbor, MLive, October 6, 2017, https://www.mlive.com/news/ann-arbor/index.ssf/2017/10/can_ self-driving_pizza_deliver.html. 231“classified [Elaine Herzberg] as something other”: Ryan Felton, “Video Shows Driver in Autonomous Uber Was Looking Down Moments before Fatal Crash,” Jalopnik, March 21, 2018, https://jalopnik.com/video-shows-driver-in-fatal-autonomous-uber-crash-was-l-1823970417. 231computer-vision algorithms: Katyanna Quach, “Racist Self-Driving Scare Debunked, inside AI Black Boxes, Google Helps Folks Go with TensorFlow,” The Register, March 10, 2019, https://www.theregister.co.uk/2019/03/10/ai_ roundup_080319/. 231marking of pavement to help orient drivers: Bill Loomis, “1900–1930: The Years of Driving Dangerously,” Detroit News, April 26, 2015, https://www.detroitnews.com/story/news/local/michigan-history/2015/04/26/auto-traffic-history-detroit/26312107/. 232dinged for excessive idling: Andrew Small, “CityLab Daily: The Race to Code the Curb,” CityLab, April 2, 2019, https://www.citylab.com/authors/andrew-small/. 233“cities need to invest in the mapping in ways”: Kevin Webb (@kvnwebb), “We’re building smart ways to encode information like curb regulation,” Twitter, November 30, 2018, 1:41 a.m., https://twitter.com/kvnweb/status/1068485225607585798. 234“an exponential runaway beyond”: Vernor Vinge, “The Coming Technological Singularity: How to Survive in the Post-Human Era,” VISION-21 Symposium, NASA Lewis Research Center and Ohio Aerospace Institute, March 30–31, 1993, https://edoras.sdsu.edu/~vinge/misc/singularity.html. 234John von Neumann had raised: John Brockman, Possible Minds (New York: Penguin Press, 2019), 8. 234“AI enthusiasts have been making claims”: Vinge, “The Coming Technological Singularity.” 234“I have set the date 2045”: Christianna Reddy, “Kurzweil Claims That the Singularity Will Happen by 2045,” Futurism, October 5, 2017, https://futurism.com/kurzweil-claims-that-the-singularity-will-happen-by-2045/. 235“machine learning” to disguise the true nature: Chris Smith et al., “The History of Artificial Intelligence,” University of Washington, December 2006, https://courses.cs.washington.edu/courses/csep590/06au/projects/his tory-ai.pdf. 235“a deep level of understanding”: Rodney Brooks, “Post: [For&AI] The Origins of Artificial Intelligence,” Robots, AI, and Other Stuff (blog), April 27, 2018, https://rodneybrooks.com/forai-the-origins-of-artificial-intelligence/. 235“Contemporary neural networks do well on challenges”: Gary Marcus, “Deep Learning: A Critical Appraisal,” New York University, accessed January 22, 2019, https://arxiv.org/pdf/1801.00631.pdf. 236“If . . . driverless cars should also disappoint”: Marcus, “Deep Learning.” 236Frey tallied his predictions of the jobs: Carl Benedikt Frey and Michael A. Osborne, “The Future of Employment: How Susceptible Are Jobs to Computerisation?” University of Oxford, September 17, 2013, 60, https://www.oxfordmartin.ox.ac.uk/downloads/academic/The_Future_of_Employment.pdf. 237“decisive strategic advantage”: Nick Bostrom, Superintelligence:Paths, Dangers, Strategies (Oxford, UK: Oxford University Press, 2014), 96. 237a serious and sober discussion about what: Bostrom, Superintelligence, 96. 238more than $25 million since 2015: Author’s estimate based on figures published by Future of Humanity Institute, “Timeline of the Future of Humanity Institute,” June 25, 2018, https://timelines.issarice.com/wiki/Timeline_of_ Future_of_Humanity_Institute. 238aspiring singleton’s inevitable ambitions: Bostrom, Superintelligence, 118.

Chaos Kings: How Wall Street Traders Make Billions in the New Age of Crisis
by Scott Patterson
Published 5 Jun 2023

Rather than a career in medicine or chemistry, EAs sought jobs on Wall Street and in Silicon Valley—or in crypto. By the early 2020s, longtermism had become a powerful force among America’s tech goliaths. Backers—the computer whizzes that use complex formulas to create more and more bitcoin—included Elon Musk, Bill Gates, and Jeff Bezos. It had its roots in the work of Swedish philosopher Nick Bostrom, founder of Oxford University’s Future of Humanity Institute, which studies extreme risks to humanity (Musk had donated $1.5 million to the sibling organization of FHI called the Future of Life Institute). The core idea behind the belief system is that humanity’s future, if it plays its cards right, is virtually unbounded and that the number of future humans—think over a period of millions or even billions of years—far outdistances the number of humans alive on Planet Earth today, or that have ever lived.

…

By downplaying current risks such as climate breakdown, for instance, because they don’t see it as existential, and by betting the future of humanity on techno-utopian scientific breakthroughs, longtermists were taking a dangerous gamble that could seriously backfire, Read believed. Taleb just thought they were bad at math. He’d met Nick Bostrom at a dinner in Oxford in 2008 and quickly chalked him up as a perhaps well-intentioned, head-in-the-clouds dreamer who lacked common sense. “These guys have gone haywire,” he told me. “They’re modeling, but what if the model is wrong? I doubt they’re very good at probability. Before colonizing Mars, make sure the Earth works.”

Money: Vintage Minis
by Yuval Noah Harari
Published 5 Apr 2018

Unnecessary people might spend increasing amounts of time within 3D virtual-reality worlds, that would provide them with far more excitement and emotional engagement than the drab reality outside. Yet such a development would deal a mortal blow to the liberal belief in the sacredness of human life and of human experiences. What’s so sacred about useless bums who pass their days devouring artificial experiences in La La Land? Some experts and thinkers, such as Nick Bostrom, warn that humankind is unlikely to suffer this degradation, because once artificial intelligence surpasses human intelligence, it might simply exterminate humankind. The AI would likely do so either for fear that humankind would turn against it and try to pull its plug, or in pursuit of some unfathomable goal of its own.

Deep Thinking: Where Machine Intelligence Ends and Human Creativity Begins
by Garry Kasparov
Published 1 May 2017

The Oxford Martin School at Oxford University has collected quite a few of these exceptional people, and also encourages the sort of interdisciplinary associating and free-associating that has gone out of fashion in this era of specialization, benchmarks, and ninety-page grant applications. As a senior visiting fellow there since 2013, I’ve had the privilege to meet many of these brilliant people, including Nick Bostrom, the author of Superintelligence, and other faculty and researchers at his Future of Humanity Institute. Founding Oxford Martin director Ian Goldin thought it would be interesting for me and for his colleagues to have informal workshops where we could talk about the big picture instead of only what was right in front of them in their labs and studies every day.

…

They deservedly seized the grail dreamt of by Alan Turing, Claude Shannon, and Norbert Wiener, and it was my fortune, not misfortune, to be holding it at the time. My friend Shay Bushinsky and his colleague Amir Ban created the remarkable program Junior, my opponent in my final human-machine match in 2003. In recent years, many experts have had the patience to personally contribute to my education in artificial intelligence and robotics. Nick Bostrom and his colleagues at Oxford Martin’s Future of Humanity Institute; Andrew McAfee at MIT; Noel Sharkey at the University of Sheffield; Nigel Crook at Oxford Brookes University; David Ferrucci at Bridgewater. I’ve never met Douglas Hofstadter or Hans Moravec, but their writings on human and machine cognition are especially provocative and essential.

This Will Make You Smarter: 150 New Scientific Concepts to Improve Your Thinking
by John Brockman
Published 14 Feb 2012

James Croak Bricoleur Currently, encompassing worldviews in philosophy have been shelved, and master art movements of style and conclusion folded alongside them; no more isms are being run up the flagpole, because no one is saluting. Mark Henderson Science’s Methods Aren’t Just for Science Science as a method has great things to contribute to all sorts of pursuits beyond the laboratory. Nick Bostrom The Game of Life—and Looking for Generators It’s a brilliant demonstration platform for several important concepts—a virtual “philosophy of science laboratory.” Robert Sapolsky Anecdotalism Every good journalist knows its power. Tom Standage You Can Show That Something Is Definitely Dangerous but Not That It’s Definitely Safe A wider understanding of the fact that you can’t prove a negative would, in my view, do a great deal to upgrade the public debate around science and technology.

…

The scientific method and the approach to critical thinking it promotes are too useful to be kept back for “science” alone. If science can help us to understand the first microseconds of creation and the structure of the ribosome, it can surely improve understanding of how best to tackle the pressing social questions of our time. The Game of Life—and Looking for Generators Nick Bostrom Director of the Future of Humanity Institute; professor, Faculty of Philosophy, University of Oxford The Game of Life is a cellular automaton invented by the British mathematician John Horton Conway in 1970. Many will already be acquainted with Conway’s invention. For those who aren’t, the best way to familiarize yourself with it is to experiment with one of the many free implementations found on the Internet (or better yet, if you have at least rudimentary programming skills, make one yourself).

The Future Is Faster Than You Think: How Converging Technologies Are Transforming Business, Industries, and Our Lives
by Peter H. Diamandis and Steven Kotler
Published 28 Jan 2020

In July 2018, with 6.7 million unfilled jobs in the US, labor shortages reached a record high. Not only are the jobs there, they’re there in record numbers. The ability to rapidly retrain our workforce to fill those jobs—that’s the challenge we have yet to meet. Existential Risks: Vision, Prevention, and Governance In 2002, a relatively unknown Oxford philosopher named Nick Bostrom published a paper in the Journal of Evolution and Technology. In a few years, Bostrom would vault to geek-fame because of his “Simulation Hypothesis,” which convincingly argues that we’re living inside the Matrix. However, this earlier paper also caused a stir, mainly because it scared the crap out of nearly everyone who read it.

…

See: https://www.businessinsider.com/goldman-sachs-says-self-driving-trucks-will-kill-300000-jobs-per-year-2017-11. 6.7 million unfilled jobs in the US: Jeff Cox, “The U.S. Labor Shortage Is Reaching a Critical Point,” CNBC Markets, July 5, 2018. See: https://www.cnbc.com/2018/07/05/the-us-labor-shortage-is-reaching-a-critical-point.html. Existential Risks: Vision, Prevention, and Governance Nick Bostrom: “Existential Risks: Analyzing Human Extinction Scenarios and Related Hazards,” Journal of Evolution and Technology 9 (March 9, 2002). Vision Stewart Brand: Stewart Brand, Clock of the Long Now: Time and Responsibility, The Ideas Behind the World’s Slowest Computer (Basic Books, 1999), p.1.

The Coming Wave: Technology, Power, and the Twenty-First Century's Greatest Dilemma
by Mustafa Suleyman
Published 4 Sep 2023

Particular thanks go to the many people who read the entire draft and commented in detail; their generosity and extraordinary level of insight were completely invaluable in getting to the final manuscript. A huge thanks to Gregory Allen, Graham Allison (and the faculty and staff of Harvard’s Belfer Center more widely), Sahar Amer, Anne Applebaum, Julian Baker, Samantha Barber, Gabriella Blum, Nick Bostrom, Ian Bremmer, Erik Brynjolfsson, Ben Buchanan, Sarah Carter, Rewon Child, George Church, Richard Danzig, Jennifer Doudna, Alexandra Eitel, Maria Eitel, Henry Elkus, Kevin Esvelt, Jeremy Fleming, Jack Goldsmith, Al Gore, Tristan Harris, Zaid Hassan, Jordan Hoffman, Joi Ito, Ayana Elizabeth Johnson, Danny Kahneman, Angela Kane, Melanie Katzman, Henry Kissinger, Kevin Klyman, Heinrich Küttler, Eric Lander, Sean Legassick, Aitor Lewkowycz, Leon Marshall, Jason Matheny, Andrew McAfee, Greg McKelvey, Dimitri Mehlhorn, David Miliband, Martha Minow, Geoff Mulgan, Aza Raskin, Tobias Rees, Stuart Russell, Jeffrey Sachs, Eric Schmidt, Bruce Schneier, Marilyn Thompson, Mayo Thompson, Thomas Viney, Maria Vogelauer, Mark Walport, Morwenna White, Scott Young, and Jonathan Zittrain.

…

GO TO NOTE REFERENCE IN TEXT At Aum Shinrikyo’s peak popularity Federation of American Scientists, “The Operation of the Aum,” in Global Proliferation of Weapons of Mass Destruction: A Case Study of the Aum Shinrikyo, Senate Government Affairs Permanent Subcommittee on Investigations, Oct. 31, 1995, irp.fas.org/​congress/​1995_rpt/​aum/​part04.htm. GO TO NOTE REFERENCE IN TEXT As a report on the implications Danzig and Hosford, “Aum Shinrikyo.” GO TO NOTE REFERENCE IN TEXT Some will inevitably say See, for example, Nick Bostrom, “The Vulnerable World Hypothesis,” Sept. 6, 2019, nickbostrom.com/​papers/​vulnerable.pdf, for perhaps the most developed version of this thesis. In a thought experiment responding to the prospect of “easy nukes,” he envisages a “high-tech panopticon” where everyone has a “freedom tag,” “worn around the neck and bedecked with multidirectional cameras and microphones.

Zero to One: Notes on Startups, or How to Build the Future
by Peter Thiel and Blake Masters
Published 15 Sep 2014

But an equally insidious danger for every business is to lose all sense of myth and mistake disenchantment for wisdom. Conclusion STAGNATION OR SINGULARITY? IF EVEN THE MOST FARSIGHTED founders cannot plan beyond the next 20 to 30 years, is there anything to say about the very distant future? We don’t know anything specific, but we can make out the broad contours. Philosopher Nick Bostrom describes four possible patterns for the future of humanity. The ancients saw all of history as a neverending alternation between prosperity and ruin. Only recently have people dared to hope that we might permanently escape misfortune, and it’s still possible to wonder whether the stability we take for granted will last.

Tools and Weapons: The Promise and the Peril of the Digital Age
by Brad Smith and Carol Ann Browne
Published 9 Sep 2019

They note, “However, processes of self-design and optimization might still lead to significant jumps in competencies.” T.G. Dietterich and E.J. Horvitz, “Rise of Concerns about AI: Reflections and Directions,” Communications of the ACM, vol. 58, no. 10, 38–40 (October 2015), http://erichorvitz.com/CACM_Oct_2015-VP.pdf. Nick Bostrom, a professor at Oxford University, explored these issues more broadly in his recent book. Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford: Oxford University Press, 2014). Within the computer science field, some use the term “singularity” differently, to describe computing power that grows so quickly that it’s not possible to predict the future.

What We Owe the Future: A Million-Year View
by William MacAskill
Published 31 Aug 2022

Their aim would be to find an adult who will act in accordance with their wishes. The challenge is for the child to do this—rather than, say, appointing a schemer who is good at deceitful salesmanship but once in power would pursue their own agenda—even though the adults are much smarter and more knowledgeable than the child is. This risk was the focus of Nick Bostrom’s book Superintelligence. The scenario most closely associated with that book is one in which a single AI agent designs better and better versions of itself, quickly developing abilities far greater than the abilities of all of humanity combined. Almost certainly, its aims would not be the same as humanity’s aims.

…

Recently, seventy-five researchers at leading organisations in AI safety and governance were asked, “Assuming that there will be an existential catastrophe as a result of AI, what do you think will be the cause?”4 The respondents could give one of six answers: the first was a scenario in which a single AI system quickly takes over the world, as described in Nick Bostrom’s Superintelligence; second and third were AI-takeover scenarios involving many AI systems that improve more gradually; the fourth was that AI would exacerbate risk from war; the fifth was that AI would be misused by people (as I described at length in Chapter 4); and the sixth was “other.” The typical respondent put a similar probability across the first five scenarios, with “other” being given a one-in-five chance.

Advances in Artificial General Intelligence: Concepts, Architectures and Algorithms: Proceedings of the Agi Workshop 2006
by Ben Goertzel and Pei Wang
Published 1 Jan 2007

Existential Risks: Analyzing Human Extinction Scenarios and Related Hazards. Journal of Evolution and Technology, vol. 9. [28] Yudkowsky, Eliezer (2007). Cognitive Biases Potentially Affecting Judgment of Global Risks, in Global Catastrophic Risks, Ed. by Nick Bostrom and Milan Cirkovic, Oxford University Press. [29] Yudkowsky, Eliezer (2007). Artificial Intelligence and Global Risk, in Global Catastrophic Risks, Ed. by Nick Bostrom and Milan Cirkovic, Oxford University Press. Advances in Artificial General Intelligence: Concepts, Architectures and Algorithms B. Goertzel and P. Wang (Eds.) IOS Press, 2007 © 2007 The authors and IOS Press.

The People vs Tech: How the Internet Is Killing Democracy (And How We Save It)
by Jamie Bartlett
Published 4 Apr 2018

* Given a bitcoin was worth around £300 back then, and is now trading at over £5,000, my cup of coffee cost approximately £75 in today’s money. Some of the staff have probably now retired. * Examples of excellent books on technology which do not really consider the way in which technological advances and changes will be shaped by politics include Max Tegmark’s Life 3.0 and Nick Bostrom’s Superintelligence. The opposite is true as well: Steven Levitsky and Daniel Ziblatt’s How Democracies Die barely mentions technology at all. * Once questions of freedom and the human spirit are invoked, it’s impossible to know how far opposition can go. Between 1978 and 1995 the ‘Unabomber’, aka Ted Kaczynski, sent 16 bombs to targets that included universities and airlines, killing three people and injuring 23.

On the Future: Prospects for Humanity
by Martin J. Rees
Published 14 Oct 2018

See Ray Kurzweil, The Singularity Is Near: When Humans Transcend Biology (New York: Viking, 2005). 14.  P. Hut and M. Rees, “How Stable Is Our Vacuum?” Nature 302 (1983): 508–9. 15.  Derek Parfit’s arguments are presented in part 4 of his Reasons and Persons (New York: Oxford University Press, 1984). 16.  Good surveys of these extreme risks are given in Nick Bostrom and Milan Ćirković, eds., Global Catastrophic Risks (Oxford: Oxford University Press, 2011); and Phil Torres, Morality, Foresight, and Human Flourishing: An Introduction to Existential Risks (Durham, NC: Pitchstone, 2018). CHAPTER 3. HUMANITY IN A COSMIC PERSPECTIVE   1.  Quoted in Carl Sagan, Pale Blue Dot: A Vision of a Human Future in Space (New York: Random House, 1994).   2.  

Wired for War: The Robotics Revolution and Conflict in the 21st Century
by P. W. Singer
Published 1 Jan 2010

Even worse, the sort of twenty-first-century questions that people working with robots and AI care about aren’t really dealt with in the broader fields of philosophy or ethics. There are few experts or resources for them to turn to, let alone any sort of consensus. Even if you were an inventor, funder, or developer who wanted to do the moral thing, as Nick Bostrom, director of the Future of Humanity Institute at Oxford University, explains, you wouldn’t have any ready guides. “Ethicists have written at length about war, the environment, our duties towards the developing world; about doctor-patient relationships, euthanasia, and abortion; about the fairness of social redistribution, race and gender relations, civil rights, and many other things.

…

Singer, Washington, DC, October 25, 2006. 421 “You can’t say it’s not part” K. Eric Drexler, “Nanotechnology: Six Lessons from Sept. 11,” Open Letter, Foresight Institute, December 2001. 422 “We’ve got to be pro-active” Ibid. 422 “We have to manage the ethics” Habershon and Woods, “No Sex Please, Robot, Just Clean the Floor.” 422 “Ethicists have written at length” Nick Bostrom, “Nanotechnology Perceptions: A Review of Ultraprecision,” Nanotechnology Perceptions: A Review of Ultraprecision Engineering and Nanotechnology 2, no. 2 (2006). 423 “People ask me about” Rodney Brooks, interview, Peter W. Singer, October 30, 2006. 423 “Asimov’s rules are neat” Daniel Wilson, interview, Peter W.

…

Soldiers in Iraq,” CNN.com, January 28, 2008 (cited January 28, 2008); available at http://www.cnn.com/2008/WORLD/meast/01/28/iraq.main/index.html. 429 “more ponies than panzers” Max Boot, War Made New: Technology, Warfare, and the Course of History, 1500 to Today (New York: Gotham Books, 2006), 467. 431 “the twenty-first century could end” Roco and Bainbridge, “Converging Technologies for Improving Human Health: Nanotechnology, Biotechnology, Information Technology and Cognitive Science.” 431 “Without more kindliness” As quoted in Nick Bostrom, “A History of Transhumanist Thought,” Journal of Evolution and Technology 14, no. 1 (2005), http://jetpress.org/volume14/bostrom.html. 432 “represents the single most important factor” Martin van Creveld, The Transformation of War (New York: Free Press, 1991), 273. 432 defy the normal rules of logic I am indebted to Sebastian Kaempf of the University of Queensland for this insight. 432 “paradox of riskless warfare” Paul W.

The Price of Tomorrow: Why Deflation Is the Key to an Abundant Future
by Jeff Booth
Published 14 Jan 2020

Having personally spent time in Redmond at the Microsoft HoloLens lab, and walking around on a virtual Mars that won NASA’s software of the year award, I can tell you that travelling to Mars via virtual reality is an experience that is hard to put into words. The software allows collaboration and interaction through virtual avatars. I was sitting about ten feet away at the Code Conference in 2016 when Elon Musk famously discussed the probability of us all living in a simulation. He explained his thesis, which was first introduced by Nick Bostrom, philosopher and author of the book Superintelligence, by using virtual reality/augmented reality as an example. He went on to argue that fidelity in virtual reality is already nearing fidelity of the real world (it feels real) and continuing to advance at a remarkable pace. If it feels real (or almost real) today and is progressing quickly, the chances are quite high that as the technology gets better and we use it more, the lines blur between reality and virtual reality, and we get confused about which reality we are in.

Homo Deus: A Brief History of Tomorrow
by Yuval Noah Harari
Published 1 Mar 2015

Unnecessary people might spend increasing amounts of time within 3D virtual-reality worlds, which would provide them with far more excitement and emotional engagement than the drab reality outside. Yet such a development would deal a mortal blow to the liberal belief in the sacredness of human life and of human experiences. What’s so sacred in useless bums who pass their days devouring artificial experiences in La La Land? Some experts and thinkers, such as Nick Bostrom, warn that humankind is unlikely to suffer this degradation, because once artificial intelligence surpasses human intelligence, it might simply exterminate humankind. The AI is likely to do so either for fear that humankind would turn against it and try to pull its plug, or in pursuit of some unfathomable goal of its own.

…

, 17 September 2013, accessed 12 August 2015, http://www.oxfordmartin.ox.ac.uk/downloads/academic/The_Future_of_Employment.pdf. 20. E. Brynjolfsson and A. McAffee, Race Against the Machine: How the Digital Revolution is Accelerating Innovation, Driving Productivity, and Irreversibly Transforming Employment and the Economy (Lexington: Digital Frontier Press, 2011). 21. Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford: Oxford University Press, 2014). 22. Ido Efrati, ‘Researchers Conducted a Successful Experiment with an “Artificial Pancreas” Connected to an iPhone’ [in Hebrew], Haaretz, 17 June 2014, accessed 23 December 2014, http://www.haaretz.co.il/news/health/1.2350956.

The Internet of Us: Knowing More and Understanding Less in the Age of Big Data
by Michael P. Lynch
Published 21 Mar 2016

Popular Web-based games like Second Life, for example, have allowed people to create artificial “people” with SIM backgrounds, jobs, spouses, etc., for years. These programs even allow your SIM to continue to interact with other SIMs when you aren’t actively playing the game, pursuing its career, relationships and so on. And that, as the philosopher Nick Bostrom has recently suggested, raises the possibility that the universe in which we live is and always has been a simulation run by a computer program created for the amusement of super-beings with superior technology.16 If so, then we aren’t just wrong about whether, for example, we have arms and legs (as opposed to just being brains in vats).

The Beginning of Infinity: Explanations That Transform the World
by David Deutsch
Published 30 Jun 2011

It requires an additional assumption, namely that when making decisions one should ignore the histories in which the decision-maker is absent. This is closely related to anthropic arguments. Again, the theory of probability for such cases is not well understood, but my guess is that the assumption is false. A related assumption occurs in the so-called simulation argument, whose most cogent proponent is the philosopher Nick Bostrom. Its premise is that in the distant future the whole universe as we know it is going to be simulated in computers (perhaps for scientific or historical research) many times – perhaps infinitely many times. Therefore virtually all instances of us are in those simulations and not the original world.

…

Bibliography Everyone should read these Jacob Bronowski, The Ascent of Man (BBC Publications, 1973) Jacob Bronowski, Science and Human Values (Harper & Row, 1956) Richard Byrne, ‘Imitation as Behaviour Parsing’, Philosophical Transactions of the Royal Society B358 (2003) Richard Dawkins, The Selfish Gene (Oxford University Press, 1976) David Deutsch, ‘Comment on Michael Lockwood, “‘Many Minds’ Interpretations of Quantum Mechanics”’, British Journal for the Philosophy of Science 47, 2 (1996) David Deutsch, The Fabric of Reality (Allen Lane, 1997) Karl Popper, Conjectures and Refutations (Routledge, 1963) Karl Popper, The Open Society and Its Enemies (Routledge, 1945) Further reading John Barrow and Frank Tipler, The Anthropic Cosmological Principle (Clarendon Press, 1986) Susan Blackmore, The Meme Machine (Oxford University Press, 1999) Nick Bostrom, ‘Are You Living in a Computer Simulation?’, Philosophical Quarterly 53 (2003) David Deutsch, ‘Apart from Universes’, in S. Saunders, J. Barrett, A. Kent and D. Wallace, eds., Many Worlds?: Everett, Quantum Theory, and Reality (Oxford University Press, 2010) David Deutsch, ‘It from Qubit’, in John Barrow, Paul Davies and Charles Harper, eds., Science and Ultimate Reality (Cambridge University Press, 2003) David Deutsch, ‘Quantum Theory of Probability and Decisions’, Proceedings of the Royal Society A455 (1999) David Deutsch, ‘The Structure of the Multiverse’, Proceedings of the Royal Society A458 (2002) Richard Feynman, The Character of Physical Law (BBC Publications, 1965) Richard Feynman, The Meaning of It All (Allen Lane, 1998) Ernest Gellner, Words and Things (Routledge & Kegan Paul, 1979) William Godwin, Enquiry Concerning Political Justice (1793) Douglas Hofstadter, Gödel, Escher, Bach: An Eternal Golden Braid (Basic Books, 1979) Douglas Hofstadter, I am a Strange Loop (Basic Books, 2007) Bryan Magee, Popper (Fontana, 1973) Pericles, ‘Funeral Oration’ Plato, Euthyphro Karl Popper, In Search of a Better World (Routledge, 1995) Karl Popper, The World of Parmenides (Routledge, 1998) Roy Porter, Enlightenment: Britain and the Creation of the Modern World (Allen Lane, 2000) Martin Rees, Just Six Numbers (Basic Books, 2001) Alan Turing, ‘Computing Machinery and Intelligence’, Mind, 59, 236 (October 1950) Jenny Uglow, The Lunar Men (Faber, 2002) Vernor Vinge, ‘The Coming Technological Singularity’, Whole Earth Review, winter 1993 *The term was coined by the philosopher Norwood Russell Hanson.

The Techno-Human Condition
by Braden R. Allenby and Daniel R. Sarewitz
Published 15 Feb 2011

We do not know where this development will ultimately take us, but it will transform the evolutionary process by drawing reproduction into a highly selective social process that is far more rapid and effective at spreading successful genes than traditional sexual competition and mate selection.,,4 We are, it turns out, in neither God's nor Darwin's hands, but in our own. In an extraordinarily strong statement of faith in the power of reductionist science, the philosopher Nick Bostrom explains how this works: "The difference between the best times in life and the worst times is ultimately a difference in the way our atoms are arranged. In principle, that's amenable to technological innovation. This simple point is very important, because it shows that there is no fundamental impossibility in enabling all of us to attain the good modes of being."

Humans Need Not Apply: A Guide to Wealth and Work in the Age of Artificial Intelligence
by Jerry Kaplan
Published 3 Aug 2015

Vitalik Buterin, “Cryptographic Code Obfuscation: Decentralized Autonomous Organizations Are About to Take a Huge Leap Forward,” Bitcoin, February 8, 2014, http://bitcoinmagazine.com/10055/cryptographic-code-obfuscation-decentralized-autonomous-organizations-huge-leap-forward/. 19. For an excellent in-depth analysis of this problem, see Nick Bostrom, Superintelligence (Oxford: Oxford University Press, 2014). 20. http://en.wikipedia.org/wiki/Anti-lock_braking_system, last modified December 30, 2014. Index Absolute Sound (magazine), 193 Accenture stock price, 63 accident avoidance. See safety Adamson Act (1916), 171 advertising, online, 64–72, 136 “impression” display, 64 individual surveillance data, 64–68 response models, 68, 70 affinity group, 67 Affordable Care Act (2010), 214n8 agent: definition of, 83–84 synthetic intellect as, 91–92.

Work in the Future The Automation Revolution-Palgrave MacMillan (2019)
by Robert Skidelsky Nan Craig
Published 15 Mar 2020

A more sensible view of AI abilities, which is held by the majority of practitioners in the field, should extend to projections of software counterparts taking over from human lawyers, doctors, scientists, journalists, and so on, speculation of which is rife across the media and in some academic circles. Such speculation has been fuelled by books like Nick Bostrom’s on SuperIntelligence (Bostrom 2014), where he is clear that his philosophical enquiry is entirely speculative, except in one respect: that AI superintelligence can and probably will occur in lightning fast time, 12 Possibilities and Limitations for AI: What Can’t Machines Do? 113 for example overnight.

Just Giving: Why Philanthropy Is Failing Democracy and How It Can Do Better
by Rob Reich
Published 20 Nov 2018

Yet some aspects of work on climate change might plausibly be addressed by foundations: basic research into technological approaches for which there is no obvious market, efforts to shift public opinion, discovery of particular mechanisms to overcome collection action problems, and convening and funding NGOs working at the global level. 31. See Gaspart and Gosseries, “Are Generational Savings Unjust?,” 196–200. 32. Thomas Scanlon, What We Owe to Each Other (Cambridge, MA: Harvard University Press, 1998), 208, italics added. 33. Scanlon, What We Owe to Each Other, 209. 34. Scanlon, What We Owe to Each Other, 209. 35. See Nick Bostrom, “Existential Risk Prevention as Global Priority,” Global Policy 4 (2013): 15–31. Bostrom mentions other potential existential risks, including asteroid strikes, supervolcanoes, and earthquakes. 36. Thompson, “Representing Future Generations.” 37. Posner, “Charitable Foundations.” Conclusion 1.

A Manual for Creating Atheists
by Peter Boghossian
Published 1 Nov 2013

(This question is made even more curious when one factors in American inventor Ray Kurzweil’s idea of exponential technological growth, with mere centuries translating into unimaginable technological differences among civilizations.) One answer to this is that there’s something wrong with our model of the universe. There’s something we don’t understand, or something we’ve considered to be a remote possibility that’s an actuality. For example, we could be brains in a vat (à la The Matrix), or as Swedish philosopher Nick Bostrom has posited, we could be living in a holographic simulation (Bostrom, 2003). Alternatively, there could be some kind of “Great Filter,” that is, a kind of “probability barrier” that life must pass through (Hanson, 1998). Hanson (1998) provides a “best-guess evolutionary path to an explosion which leads to visible colonization of most of the visible universe.”

But What if We're Wrong? Thinking About the Present as if It Were the Past
by Chuck Klosterman
Published 6 Jun 2016

I have yet to encounter anyone who totally buys it; even the man most responsible for its proliferation places the likelihood of its validity at roughly 20 percent. But even a one-in-five chance presents the potential for a paradigm shift greater than every other historical shift combined. It would place the Copernican Revolution on a par with the invention of Velcro. The man to whom I refer is existential Swedish philosopher Nick Bostrom, currently directing the Future of Humanity Institute at the University of Oxford. He’s relatively young (born in ’73), balding, and slightly nervous that the human race will be annihilated by robots. Yet it is his simulation hypothesis (building off the earlier work of Austrian roboticist Hans Moravec) that really moves the stoner needle.

The Behavioral Investor
by Daniel Crosby
Published 15 Feb 2018

Flip the script Charlie Munger, Warren Buffett’s right-hand-man, is most commonly known for admonition to “invert, always invert.” Although Munger popularized the phrase, the original idea belongs to esteemed mathematician Carl Jacobi who was espousing the benefits of considering, “Why might I be wrong?” or “What is another way of viewing this situation?” Nick Bostrom, known primarily for his doomsday musings on artificial intelligence, has proposed a more elaborate theorem in this same vein that he refers to as the “reversal test.” Bostrom says: “When a proposal to change a certain parameter is thought to have bad overall consequences, consider a change to the same parameter in the opposite direction.

Reinventing Capitalism in the Age of Big Data
by Viktor Mayer-Schönberger and Thomas Ramge
Published 27 Feb 2018

one in four trucks drive empty: Robert Matthams, “Despite High Fuel Prices, Many Trucks Run Empty,” Christian Science Monitor, February 25, 2012, http://www.csmonitor.com/Business/2012/0225/Despite-high-fuel-prices-many-trucks-run-empty. one of the pioneer venture capitalists: “Eugene Kleiner,” Kleiner, Perkins, Caufield, and Byers, http://www.kpcb.com/partner/eugene-kleiner. fear is that as we delegate decisions to machines: See, e.g., Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford: Oxford University Press, 2014). The end of scarcity: Ibid. “A world of increasing abundance”: Ibid. “Cartier for everyone”: Aaron Bastani, “Britain Doesn’t Need More Austerity, It Needs Luxury Communism,” Vice, June 12, 2015, https://www.vice.com/en_au/article/luxury-communism-933.

99%: Mass Impoverishment and How We Can End It
by Mark Thomas
Published 7 Aug 2019

Widespread adoption is harder to predict but the CEO of Uber has said that he expects his entire fleet to be driverless by 2030.10 Initially, like many other innovations in the automotive sector, driverless cars may be more expensive but the costs are likely to fall over time until – like automatic gearboxes, electric windows and in-car entertainment systems – they become commonplace. So commonplace that sometime in the 2020s, you will find yourself riding in a driverless car. Full AI As Professor Nick Bostrom of Oxford University has written: … expert opinions about the future of AI vary wildly. There is disagreement about timescales as well as about what forms AI might eventually take. Predictions about the future development of artificial intelligence, one recent study noted, ‘are as confident as they are diverse’… A series of recent surveys have polled members of several relevant expert communities on the question of when they expect ‘human-level machine intelligence’… The combined sample gave the following (median) estimate… 50% probability by 2040.11 If this is correct, then by 2050 artificial intelligence will surpass human intelligence, possibly by a large margin.

The Dark Cloud: How the Digital World Is Costing the Earth
by Guillaume Pitron
Published 14 Jun 2023

Harnessing artificial intelligence for the earth’, World Economic Forum – Stanford Woods Institute for the Environment, Pricewaterhouse Coopers (PwC), January 2018. 59 Keynote by Emmanuel Assié, founder and CEO of Webaxys, Data Centre World, 2019. 60 Interview with Tristram Walsh, research associate at University of Oxford, Smith School of Enterprise and the Environment, 2020. 61 Encyclopedia Britannica, Deep Ecology. 62 Nick Bostrom, ‘Existential risks. Analyzing human extinction scenarios and related hazards’, Journal of Evolution and Technology, vol. 9, no. 1, 2002. 63 Stuart Russell, Human Compatible: artificial intelligence and the problem of control, Penguin Books, 2020. 64 Interview with Stuart Russell, 2020. 65 ‘How the enlightenment ends’, The Atlantic, June 2018.

The Patterning Instinct: A Cultural History of Humanity's Search for Meaning
by Jeremy Lent
Published 22 May 2017

This repugnance “is the emotional expression of deep wisdom,” argues prominent scholar Leon Kass, “beyond reason's power completely to articulate it.”43 For transhumanists, though, the promise of genetic engineering is just the beginning of humanity's new journey into uncharted and enticing territory. “Transhumanists view human nature as a work-in-progress, a half-baked beginning that we can learn to remold in desirable ways,” declares philosopher Nick Bostrom, a de facto spokesperson for the movement. In an intoxicating manifesto, he proclaims the belief that “humanity's potential is still mostly unrealized” and envisions the possibility of “overcoming aging, cognitive shortcomings, involuntary suffering, and our confinement to planet Earth.” Bostrom imagines a chasm between “posthumans” and current humans as wide as that which currently exists between us and chimpanzees: “In much the same way chimpanzees lack the cognitive wherewithal to understand what it is like to be human…so we humans may lack the capacity to form a realistic intuitive understanding of what it would be like to be a radically enhanced human (a ‘posthuman’) and of the thoughts, concerns, aspirations, and social relations that such humans may have.”44 Toward the Singularity?

…

Francis Fukuyama, Our Posthuman Future: Consequences of the Biotechnology Revolution (New York: Farrar, Straus and Giroux, 2002), 102; Leo Kass, “Beyond Therapy: Biotechnology and the Pursuit of Human Improvement,” President's Council on Bioethics, Washington, DC, 2003, 16. See also George J. Annas, Lori B. Andrews, and Rosario Isasi, “Protecting the Endangered Human: Toward an International Treaty Prohibiting Cloning and Inheritable Alterations,” American Journal of Law and Medicine 28 (2002): 151–78. 44. Nick Bostrom, “Transhumanist Values,” Review of Contemporary Philosophy 4, nos. 1–2 (2005): 87–101. 45. Bostrom, “Transhumanist Values”; Vernor Vinge, “What Is the Singularity?” (presentation, Vision 21 Symposium, Westlake, OH, March 30, 1993). 46. Ray Kurzweil, The Singularity Is Near (New York: Penguin Books, 2005); Singularity University, http://singularityu.org (accessed February 5, 2017); Wikipedia, s.v.

Everybody Lies: Big Data, New Data, and What the Internet Can Tell Us About Who We Really Are
by Seth Stephens-Davidowitz
Published 8 May 2017

topic/dfp/p_TRMqWUF9s. 219 the rise of behavioral addictions in contemporary society: Adam Alter, Irresistible: The Rise of Addictive Technology and the Business of Keeping Us Hooked (New York: Penguin, 2017). 219 Top addictions reported to Google: Author’s analysis of Google Trends data. 222 says Levitt in a lecture: This is discussed in a video currently featured on the Freakonomics page of the Harry Walker Speakers Bureau, http://www.harrywalker.com/speakers/authors-of-freakonomics/. 225 beer and soft drink ads run during the Super Bowl: Wesley R. Hartmann and Daniel Klapper, “Super Bowl Ads,” unpublished manuscript, 2014. 226 a pimply kid in his underwear: For the strong case that we likely are living in a computer simulation, see Nick Bostrom, “Are We Living in a Computer Simulation?” Philosophical Quarterly 53, no. 211 (2003). 227 Of forty-three American presidents: Los Angeles Times staff, “U.S. Presidential Assassinations and Attempts,” Los Angeles Times, January 22, 2012, http://timelines.latimes.com/us-presidential-assassinations-and-attempts/. 227 Compare John F.

Scary Smart: The Future of Artificial Intelligence and How You Can Save Our World
by Mo Gawdat
Published 29 Sep 2021

Smart as we are, we’ve never really managed to realize that war comes with massive casualties and collateral damage, and that after the war, mostly, we go back to the same norm – one we could have preserved before we started, without a load of pain and vengeance to deal with for decades afterwards. Remember the song, ‘War (What is it good for)’? Absolutely Nothing! In his book Superintelligence, Nick Bostrom (a Swedish-born philosopher at the University of Oxford known for his work on existential risk, the anthropic principle, human enhancement ethics and superintelligence risks) predicts situations in which we face serious threats as a result of superintelligence. He calls it the vulnerable world hypothesis (VWH).

Outnumbered: From Facebook and Google to Fake News and Filter-Bubbles – the Algorithms That Control Our Lives
by David Sumpter
Published 18 Jun 2018

At a meeting of the Future of Life Institute – a charitable organisation in Boston, Massachusetts, focused on dealing with future risks – in January 2017, theoretical physicist Max Tegmark hosted a panel debate about general artificial intelligence.1 The panel included nine of the most influential men in the field, including entrepreneur and Tesla CEO Elon Musk; the Google guru Ray Kurzweil; DeepMind’s founder Demis Hassabis and Nick Bostrom, the philosopher who has mapped our way to, what he calls, ‘superintelligence’. The panel members varied in their views as to whether human-level machine intelligence would come gradually or all of a sudden, or whether it will be good or bad for humanity. But they all agreed that a general form of AI was more or less inevitable.

Breaking Twitter: Elon Musk and the Most Controversial Corporate Takeover in History
by Ben Mezrich
Published 6 Nov 2023

Consciousness, experiences, moments such as the video call Elon was about to have: they might simply be by-products of this simulated reality, while the rules and laws of physics that we believe to be fundamental to our existence could be mere programming. Although simulation theory sounded like science fiction, it was actually grounded in simple math. As the idea went—expounded by the scientist and philosopher Nick Bostrom in a paper and subsequent book in 2003—advanced civilizations would eventually develop the technology to create computer simulations that were indistinguishable from real life. Such societies wouldn’t simply create one simulation, but would create many. The civilizations within the simulations would eventually likewise create simulations, also indistinguishable from real life.

Gods and Robots: Myths, Machines, and Ancient Dreams of Technology
by Adrienne Mayor
Published 27 Nov 2018

Pharmaka “animates” the statues with a kind of “soul” or life but does not necessarily make them move. Hollow statues as vessels that are vivified by being filled with substances, Steiner 2001, 114–20. 19. Asimov’s laws, Kang 2011, 302. Future of Life Institute’s Beneficial AI Conference 2017; FLI’s board included Stephen Hawking, Frank Wilczek, Elon Musk, and Nick Bostrom. https://futurism.com/worlds-top-experts-have-created-a-law-of-robotics/. See also Leverhulme Centre for the Future of Intelligence: http://lcfi.ac.uk/. 20. Martinho-Truswell 2018. 21. Four-wheeled carts, Morris 1992, 10. A small, shallow bronze basin-cart on three wheels, an ancient example of pen, bonsai basin, was excavated in a sixth/fifth century BC archaeological site in China, indicating that the idea of a wheeled tripod was put into practice elsewhere in antiquity, Bagley et al. 1980, 265, 272, color plate 65.

I, Warbot: The Dawn of Artificially Intelligent Conflict
by Kenneth Payne
Published 16 Jun 2021

If it were the broom in Goethe’s fable, it would be constantly asking the apprentice—‘is that enough mopping, or do you need more?’ It’s a deceptively simple solution to a thorny issue: Can you stop an AI once it’s started? One popular variant on the theme is the idea of an AI that will resist being turned off, like the sinister HAL in Kubrick’s 2001. In Nick Bostrom’s philosophical thought experiment, an AI is tasked with simply counting paperclips—an innocuous task a long way distant from nuclear brinksmanship.17 The problem occurs when the machine goes back for a recount, just to be sure it hasn’t made a mistake. After all there’s a vanishingly small probability it has, and since it had no instructions to the contrary, it had better make sure.

What Algorithms Want: Imagination in the Age of Computing
by Ed Finn
Published 10 Mar 2017

We know it as the apotheosis of the algorithm, when technological change will accelerate to such speed that human intelligence may simply be eclipsed. In this scenario we no longer manipulate the symbols and we can no longer construe their meaning. It is the endgame of computationalism as considered by philosopher Nick Bostrom, computer scientist Vernor Vinge, and others, an existential referendum on the relationship between humanity and technics.73 If we follow the asymptote of the effective procedure far enough, the space of computation advances with not just a vanguard but a rearguard, and humanity might simply be left behind—no longer effective or efficient enough to merit emulation or attention.

Lost in Math: How Beauty Leads Physics Astray
by Sabine Hossenfelder
Published 11 Jun 2018

And a little further you have Andrei Linde’s chaotic inflation with infinitely many bubble universes. And then still further over there you have the string theory landscape in which the physics in each of the bubbles is different. And even further you get Tegmark’s mathematical multiverse. And then, far over there, you get people like [Nick] Bostrom saying that we live in a computer simulation. That’s not even pseudoscience, it’s fiction.” I say: “It’s a modern version of the clockwork universe, basically. Then it was gears and bolts, now it’s quantum computers.” “Yes,” George says. “But you see, Brian Greene lists it as a possibility in his book.

Heart of the Machine: Our Future in a World of Artificial Emotional Intelligence
by Richard Yonck
Published 7 Mar 2017

As it happens, the two questions may be intimately interlinked. Recently a number of notable luminaries, scientists, and entrepreneurs have expressed their concerns about the potential for runaway AI and superintelligent machines. Physicist Stephen Hawking, engineer and inventor Elon Musk, and philosopher Nick Bostrom have all issued stern warnings of what may happen as we move ever closer to computers that are able to think and reason as well as or perhaps even better than human beings. At the same time, several computer scientists, psychologists, and other researchers have stated that the many challenges we face in developing thinking machines shows we have little to be concerned about.

The Knowledge: How to Rebuild Our World From Scratch
by Lewis Dartnell
Published 15 Apr 2014

The ATL consists of more than a thousand digitized volumes that have been selected for the practical information they provide on self-sufficiency and rudimentary techniques, and is available on DVD or CD-ROM from Village Earth at http://villageearth.org/appropriate-technology/. Full citation information is given in the bibliography, and also see The Knowledge website, the-knowledge.org, for links to all the cited literature, including free downloads where available. INTRODUCTION Nick Bostrom and Milan M. Ćirković, eds., Global Catastrophic Risks. Jared Diamond, Collapse: How Societies Choose to Fail or Succeed. Paul R. Ehrlich and Anne H. Ehrlich, “Can a Collapse of Global Civilization Be Avoided?” John Michael Greer, The Long Descent: A User’s Guide to the End of the Industrial Age.

Only Humans Need Apply: Winners and Losers in the Age of Smart Machines
by Thomas H. Davenport and Julia Kirby
Published 23 May 2016

Even as AI exceeds our powers of cognition, we shouldn’t be afraid that whichever is smartest somehow “wins.” Bryson points out, “We already have calculators that can do math better than us, and they don’t even take over the pockets they live in, let alone the world.”1 As we’ve described earlier, there are passionate prophesiers of “superintelligence” like Nick Bostrom of Oxford. Bostrom’s colleague Stuart Armstrong published a report calling for a whole new category of risks to be recognized: risks of potentially infinite impact. Among a number of global challenges “threatening the very basis of our civilization” (including nuclear war, climate change, and global pandemics) they included artificial intelligence, because of what is known as “the control problem.”2 Once machines can outthink humans, their ability to decide and make things happen might be beyond our power to rein in.

We Are Data: Algorithms and the Making of Our Digital Selves
by John Cheney-Lippold
Published 1 May 2017

Søraker, and Eric Steinhart, eds., Singularity Hypotheses: A Scientific and Philosophical Assessment (Berlin: Springer-Verlag, 2012). 25. N. Katherine Hayles, How We Became Posthuman: Virtual Bodies in Cybernetics, Literature, and Informatics (Chicago: University of Chicago Press, 1999); Hans Moravec, Mind Children: The Future of Robot and Human Intelligence (Cambridge, MA: Harvard University Press, 1990). 26. Nick Bostrom, “How Long before Superintelligence,” Linguistic and Philosophical Investigations 5, no. 1 (2006): 11–30. Index Bold page numbers refer to figures 9/11, 40 *999 system, 201, 203–5, 207, 215–16, 218, 224, 227, 232–33, 242–44, 296n2 advertising, xii, 99, 113, 129, 132, 139, 153, 180, 244, 281n101; companies, 19, 21, 92; contextual, 223; identity construction, 6–7; targeted, 28, 75–76, 123, 133, 255.

Being You: A New Science of Consciousness
by Anil Seth
Published 29 Aug 2021

If conscious machines are possible, with them arises the possibility of rehousing our wetware-based conscious minds within the pristine circuitry of a future supercomputer that does not age and never dies. This is the territory of mind uploading, a favourite trope of futurists and transhumanists for whom one life is not enough. Some even think we may already be there. The Oxford University philosopher Nick Bostrom’s ‘simulation argument’ outlines a statistical case proposing that we are more likely to be part of a highly sophisticated computer simulation, designed and implemented by our technologically superior and genealogically obsessed descendants, than we are to be part of the original biological human race.

The Road to Conscious Machines
by Michael Wooldridge
Published 2 Nov 2018

In the train scenario above, I communicated my goal that the trains were to avoid crashing, but forgot to convey that they should still be allowed to move. I had forgotten that, although a human would, almost certainly, implicitly understand that this was my intention, a computer system would not. Scenarios like this were popularized by Oxford philosopher Nick Bostrom in his bestselling 2014 book Superintelligence.18 He calls this perverse instantiation: the computer does what you asked for, but not in the way you anticipated. Hours of fun are to be had dreaming up other perverse instantiation scenarios: you ask a robot to ensure your house is not burgled, so it burns it down; you ask a robot to prevent deaths from cancer, so it murders everyone and so on.

MegaThreats: Ten Dangerous Trends That Imperil Our Future, and How to Survive Them
by Nouriel Roubini
Published 17 Oct 2022

Unless humans merge with computers, writer Yuval Harari warns, Homo sapiens are finished. They will become obsolete just like Homo erectus, Homo habilus, and other early humans that have long since vanished. Enter Homo deus, says Harari, which is smarter, stronger, and immortal so long as knowledge can move from one machine to the next iteration. Oxford University philosopher Nick Bostrom, the author of Superintelligence, ranks artificial intelligence next to giant asteroid strikes and nuclear war as an existential threat to humanity. The late mathematician Stephen Hawking worried that AI “could spell the end of the human race.” That is why he suggested that humans should move to other planets—as the machine will take over not only all jobs but also the human race.

Utopia Is Creepy: And Other Provocations
by Nicholas Carr
Published 5 Sep 2016

But its growing acceptance, particularly among the young, reveals how eager we are, whenever science grants us new powers over our bodies’ appearance and workings, to redefine human nature as malleable, as a socially and personally defined construct rather than an expression of biological imperatives. Advances in biotechnology may be unsettling, but in the end we welcome them because they give us greater autonomy in remaking ourselves into what we think we should be. Transhumanism is “an extension of humanism,” argues Nick Bostrom, an Oxford philosophy professor who has been one of the foremost proponents of radical human enhancement. “Just as we use rational means to improve the human condition and the external world, we can also use such means to improve ourselves, the human organism. In doing so, we are not limited to traditional humanistic methods, such as education and cultural development.

Pandora's Brain
by Calum Chace
Published 4 Feb 2014

‘If it is at all possible, then it will happen, that’s for sure. There is too much to be gained for it not to happen. And there are some seriously smart people taking these ideas seriously. You’re always banging on about how smart Oxford Professors are – well one of them is at the forefront of transhumanism.’ ‘Yes, I know,’ Carl admitted lugubriously. ‘Nick Bostrom is rather letting the side down.’ Carl’s face brightened as he spotted an opportunity to change the subject. ‘But Bostrom is also famous for something completely different, and it’s great fun. Have you heard of the Simulation Hypothesis?’ Matt shook his head. ‘OK, well the argument runs like this.

Augmented: Life in the Smart Lane
by Brett King
Published 5 May 2016

It promotes an interdisciplinary approach to understanding and evaluating the opportunities for enhancing the human condition and the human organism opened up by the advancement of technology. Attention is given to both present technologies, like genetic engineering and information technology, and anticipated future ones, such as molecular nanotechnology and artificial intelligence. Nick Bostrom, “Transhumanist Values,” Journal of Value Inquiry 37, no. 4 (2003): 493–506. This so-called post-human future focuses on two broad classes of human augmentation. The first is bioengineering and the second is technology-led augmentation, sometimes called cyborgification. Changing the way we live as humans and our overall health or condition is a key outcome of the Augmented Age.

The Human Age: The World Shaped by Us
by Diane Ackerman
Published 9 Sep 2014

New York: Routledge, 2012. Rosenzweig, Michael. Win-Win Ecology: How the Earth’s Species Can Survive in the Midst of Human Enterprise. New York: Oxford University Press, 2003. Sartore, Joel. Rare: Portraits of America’s Endangered Species. Washington, DC: National Geographic, 2010. Savulescu, Julian, and Nick Bostrom, eds. Human Enhancement. Oxford: Oxford University Press, 2009. Schmidt, Eric, and Jared Cohen. The New Digital Age: Reshaping the Future of People, Nations and Business. New York: Alfred A. Knopf, 2013. Seaman, Donna. In Our Nature: Stories of Wilderness. Athens, GA: University of Georgia Press, 2002.

Radical Abundance: How a Revolution in Nanotechnology Will Change Civilization
by K. Eric Drexler
Published 6 May 2013

The Oxford Martin School, a unique, interdisciplinary research community within Oxford University, has helped to provide that diversity. The school’s programs address twenty-first-century challenges and opportunities that range from economic development to nanomedicine, food, human rights, and the governance of geoengineering; I thank its director, Ian Goldin, for welcoming me for an extended stay, and Nick Bostrom for opening the door by inviting me to join the community at the Programme on the Impacts of Future Technology. Within that programme, I thank Nick along with Anders Sandberg, Toby Ord, Stuart Armstrong, Daniel Dewey, Seán Ó hÉigeartaigh, and others for more than a year’s worth of wide-ranging and challenging discussions.

AI in Museums: Reflections, Perspectives and Applications
by Sonja Thiel and Johannes C. Bernhardt
Published 31 Dec 2023

Lemoine justified this by saying, among other things, that he recognizes a person when he speaks to one. But what is in fact the condition for the possibility of Lemoine’s rather astonishing statement being valid? Among the necessary presuppositions behind Lemoine’s statement is an idea that has been argued for explicitly by Nick Bostrom: that there might be other intelligent beings than merely carbonbased beings (Bostrom 2013). But such transhumanist ideas are surely still science fiction. Nevertheless, it is easy to position this idea within a broader emancipatory narrative: Just as humans have understood themselves for too long as categorially different from other animals, whereas they are just one animal species among other animal species, the idea that thinking is exclusive to human beings is a problematic position insofar it merely expresses anthropocentrism.

In Our Own Image: Savior or Destroyer? The History and Future of Artificial Intelligence
by George Zarkadakis
Published 7 Mar 2016

It is a disturbing thought that becomes even more disturbing when one realises how hard it is to tell whether it is true or not. So let’s examine it a little more closely. The idea of our living in a computer simulation assumes first and foremost that the body–mind duality is true. The philosopher Nick Bostrom,21 one of the proponents of a simulated universe, concedes that the basic assumption underpinning the concept of a simulated universe is the so-called ‘substrate-independence’. This is the idea that mental states supervene over physical substrates. Therefore, it is not an essential property of consciousness that it is implemented in carbon-based biological neural networks inside a cranium.

The Master Algorithm: How the Quest for the Ultimate Learning Machine Will Remake Our World
by Pedro Domingos
Published 21 Sep 2015

The Second Machine Age, by Erik Brynjolfsson and Andrew McAfee (Norton, 2014), discusses how progress in AI will shape the future of work and the economy. “World War R,” by Chris Baraniuk (New Scientist, 2014) reports on the debate surrounding the use of robots in battle. “Transcending complacency on superintelligent machines,” by Stephen Hawking et al. (Huffington Post, 2014), argues that now is the time to worry about AI’s risks. Nick Bostrom’s Superintelligence (Oxford University Press, 2014) considers those dangers and what to do about them. A Brief History of Life, by Richard Hawking (Random Penguin, 1982), summarizes the quantum leaps of evolution in the eons BC. (Before Computers. Just kidding.) The Singularity Is Near, by Ray Kurzweil (Penguin, 2005), is your guide to the transhuman future.

System Error: Where Big Tech Went Wrong and How We Can Reboot
by Rob Reich , Mehran Sahami and Jeremy M. Weinstein
Published 6 Sep 2021

“from another dimension”: Dawn Chan, “The AI That Has Nothing to Learn from Humans,” Atlantic, October 20, 2017, https://www.theatlantic.com/technology/archive/2017/10/alphago-zero-the-ai-that-taught-itself-go/543450/. 50 percent of jobs in the American economy: Carolyn Dimitri, Anne Effland, and Neilson Conklin, “The 20th Century Transformation of U.S. Agriculture and Farm Policy,” Economic Information Bulletin Number 3, June 2005, https://www.ers.usda.gov/webdocs/publications/44197/13566_eib3_1_.pdf. digital automation can replace: Nick Bostrom and Eliezer Yudkowsky, “The Ethics of Artificial Intelligence,” in Cambridge Handbook of Artificial Intelligence, edited by Keith Frankish and William M. Ramsey (Cambridge, UK: Cambridge University Press, 2014), 316–34. Will AGI put humanity: Edward Feigenbaum et al., Advanced Software Applications in Japan (Park Ridge, NJ: Noyes Data Corporation, 1995).

Deep Medicine: How Artificial Intelligence Can Make Healthcare Human Again
by Eric Topol
Published 1 Jan 2019

Similarly, a report by researchers at the Future of Humanity Institute at Oxford and Yale Universities from a large survey of machine learning experts concluded, “There is a 50 percent chance of AI outperforming humans in all tasks in 45 years and automating all human jobs in 120 years.”82 Of interest, the Future of Humanity Institute’s director Nick Bostrom is the author of Superintelligence and the subject of an in-depth profile in the New Yorker as the proponent of AI as the “Doomsday Invention.”83 Tegmark points to the low probability of that occurring: “Superintelligence arguably falls into the same category as a massive asteroid strike such as the one that wiped out the dinosaurs.”84 Regardless of what the future holds, today’s AI is narrow.

New Laws of Robotics: Defending Human Expertise in the Age of AI
by Frank Pasquale
Published 14 May 2020

Carrie Goldberg, Nobody’s Victim: Fighting Psychos, Stalkers, Pervs, and Trolls (New York: Plume, 2019). 47. Sasha Costanza-Chock, Design Justice: Community-Led Practices to Build the Worlds We Need (Cambridge, MA: MIT Press, 2020). 48. George Lakoff and Mark Johnson, Philosophy in the Flesh: The Embodied Mind and the Challenge to Western Thought (New York: Basic Books, 1999), 4. 49. Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford: Oxford University Press, 2014), 134–135. 50. For examples of such shortcomings, see Rana Foroohar, Don’t Be Evil: How Big Tech Betrayed Its Founding Principles—And All of Us (New York: Currency, 2019); Amy Webb, The Big Nine: How the Tech Titans and Their Thinking Machines Could Warp Humanity (New York: Public Affairs, 2019). 51.

Think Like a Rocket Scientist: Simple Strategies You Can Use to Make Giant Leaps in Work and Life
by Ozan Varol
Published 13 Apr 2020

If there’s no way to test a scientific hypothesis and disprove it, it’s essentially worthless. As Sagan explains, “Skeptics must be given the chance to follow your reasoning, to duplicate your experiments, and see if they get the same result.”40 Consider, for example, the “simulation hypothesis,” first posited by philosopher Nick Bostrom and later popularized by Elon Musk. The hypothesis says we’re little creatures living in a computer simulation controlled by more-intelligent powers.41 This hypothesis isn’t falsifiable. If we’re like the characters in the video game The Sims, we can’t acquire information about our world from outside it.

The Ascent of Money: A Financial History of the World
by Niall Ferguson
Published 13 Nov 2007

Knight, Risk, Uncertainty and Profit (Boston, 1921). 9 John Maynard Keynes, ‘The General Theory of Employment’, Economic Journal, 51, 2 (1937), p. 214. 10 Daniel Kahneman and Amos Tversky, ‘Prospect Theory: An Analysis of Decision under Risk’, Econometrica, 47, 2 (March 1979), p. 273. 11 Eliezer Yudkowsky, ‘Cognitive Biases Potentially Affecting Judgment of Global Risks’, in Nick Bostrom and Milan Cirkovic (eds.), Global Catastrophic Risks (Oxford University Press, 2008), pp. 91-119. See also Michael J. Mauboussin, More Than You Know: Finding Financial Wisdom in Unconventional Places (New York / Chichester, 2006). 12 Mark Buchanan, The Social Atom: Why the Rich Get Richer, Cheaters Get Caught, and Your Neighbor Usually Looks Like You (New York, 2007), p. 54. 13 For an introduction, see Andrei Shleifer, Inefficient Markets: An Introduction to Behavioral Finance (Oxford, 2000).

Dawn of the New Everything: Encounters With Reality and Virtual Reality
by Jaron Lanier
Published 21 Nov 2017

The act of observation changes a quantum system no matter who observes, man or god. Quantum cryptography turned out to work! So, if you believe that argument, there’s now less reason to worry that we might be in VR. Another line of argument is usually associated with the more recent work of philosophers like Nick Bostrom, but it was around back at the Little Hunan.16 Roughly: If there is a multitude of alien civilizations, a bunch of them would develop high-quality VR, so there would be many VR systems running, but only one real universe. Therefore, when you find yourself in a reality, the chances are that it’s a virtual reality.

Radical Technologies: The Design of Everyday Life
by Adam Greenfield
Published 29 May 2017

Technological Unemployment and the Meaning of Life,” Science and Engineering Ethics, forthcoming, philpapers.org/archive/DANWLB.pdf 53.Hannah Arendt, The Human Condition, Chicago: University of Chicago Press, 1958. 54.Amos Zeeberg, “Alienation Is Killing Americans and Japanese,” Nautilus, June 1, 2016. 8Machine learning 1.Rob Kitchin, The Data Revolution: Big Data, Open Data, Data Infrastructures and their Consequences, London: Sage Publications, 2014. 2.Daniel Rosenberg, “Data Before the Fact,” in Lisa Gitelman, ed., “Raw Data” Is an Oxymoron, Cambridge, MA: MIT Press, 2013. 3.These questions are explored in greater depth in the excellent Critical Algorithm Studies reading list maintained by Tarleton Gillespie and Nick Seaver of Microsoft Research’s Social Media Collective: socialmediacollective.org/reading-lists/critical-algorithm-studies. 4.Nick Bostrom, Superintelligence: Paths, Dangers, Strategies, Oxford, UK: Oxford University Press, 2014. 5.For those inclined to dig deeper into such subjects, Andrey Kurenkov’s history of neural networks is fantastic: andreykurenkov.com/writing/a-brief-history-of-neural-nets-and-deep-learning. 6.Alistair Barr, “Google Mistakenly Tags Black People as ‘Gorillas,’ Showing Limits of Algorithms,” Wall Street Journal, July 1, 2015. 7.Aditya Khosla et al., “Novel dataset for Fine-Grained Image Categorization,” First Workshop on Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition, 2011, vision.stanford.edu/aditya86/ImageNetDogs; ImageNet, “Large Scale Visual Recognition Challenge 2012,” image-net.org/challenges/LSVRC/2012. 8.David M.

Click Here to Kill Everybody: Security and Survival in a Hyper-Connected World
by Bruce Schneier
Published 3 Sep 2018

Weapons that can’t be recalled or turned off—and also operate at computer speeds—could cause all sorts of lethal problems for friend and foe alike. All of this comes together in artificial intelligence. Over the past few years, we’ve read some dire predictions about the dangers of AI. Technologists Bill Gates, Elon Musk, and Stephen Hawking, and philosopher Nick Bostrom, have all warned of a future where artificial intelligence—either as intelligent robots or as something less personified—becomes so powerful that it takes over the world and enslaves, exterminates, or ignores humanity. The risks might be remote, they argue, but they’re so serious that it would be foolish to ignore them.

Ways of Being: Beyond Human Intelligence
by James Bridle
Published 6 Apr 2022

Jordan Novet and Annie Palmer, ‘Amazon salesperson’s pitch to oil and gas: “Remember that we actually consume your products!”’, CNBC, 20 May 2020; https://www.cnbc.com/2020/05/20/aws-salesman-pitch-to-oil-and-gas-we-actually-consume-your-products.html. 9. For an elaboration of the paperclip hypothesis, see Nick Bostrom, ‘Ethical Issues in Advanced Artificial Intelligence’, 2003; https://www.nickbostrom.com/ethics/ai.html. 10. Samuel Gibbs, ‘Elon Musk: Regulate AI to Combat “Existential Threat” Before It’s Too Late’, The Guardian, 17 July 2017; https://www.theguardian.com/technology/2017/jul/17/elon-musk-regulation-ai-combat-existential-threat-tesla-spacex-ceo. 11.

Physics of the Future: How Science Will Shape Human Destiny and Our Daily Lives by the Year 2100
by Michio Kaku
Published 15 Mar 2011

Bartholomew’s Hospital Steven Weinberg, Nobel laureate, University of Texas at Austin Frank Wilczek, Nobel laureate, MIT Amir Aczel, author of Uranium Wars Buzz Aldrin, former NASA astronaut, second man to walk on the moon Geoff Andersen, research associate, United States Air Force Academy, author of The Telescope Jay Barbree, NBC news correspondent, coauthor of Moon Shot John Barrow, physicist, University of Cambridge, author of Impossibility Marcia Bartusiak, author of Einstein’s Unfinished Symphony Jim Bell, professor of astronomy, Cornell University Jeffrey Bennet, author of Beyond UFOs Bob Berman, astronomer, author of Secrets of the Night Sky Leslie Biesecker, chief of Genetic Disease Research Branch, National Institutes of Health Piers Bizony, science writer, author of How to Build Your Own Spaceship Michael Blaese, former National Institutes of Health scientist Alex Boese, founder of Museum of Hoaxes Nick Bostrom, transhumanist, University of Oxford Lt. Col. Robert Bowman, Institute for Space and Security Studies Lawrence Brody, chief of the Genome Technology Branch, National Institutes of Health Rodney Brooks, former director, MIT Artificial Intelligence Laboratory Lester Brown, founder of Earth Policy Institute Michael Brown, professor of astronomy, Caltech James Canton, founder of Institute for Global Futures, author of The Extreme Future Arthur Caplan, director, Center for Bioethics, University of Pennsylvania Fritjof Capra, author of The Science of Leonardo Sean Carroll, cosmologist, Caltech Andrew Chaikin, author of A Man on the Moon Leroy Chiao, former NASA astronaut George Church, director, Center for Computational Genetics, Harvard Medical School Thomas Cochran, physicist, Natural Resources Defense Council Christopher Cokinos, science writer, author of The Fallen Sky Francis Collins, director of the National Institutes of Health Vicki Colvin, director of Biological and Environmental Nanotechnology, Rice University Neil Comins, author of The Hazards of Space Travel Steve Cook, director of Space Technologies, Dynetics, former NASA spokesperson Christine Cosgrove, author of Normal at Any Cost Steve Cousins, president and CEO, Willow Garage Brian Cox, physicist, University of Manchester, BBC science host Phillip Coyle, former assistant secretary of defense, U.S.

Upheaval: Turning Points for Nations in Crisis
by Jared Diamond
Published 6 May 2019

(Penguin, New York, 2015). CHAPTER 11: WHAT LIES AHEAD FOR THE WORLD? Scott Barrett. Environment and Statecraft: The Strategy of Environmental Treaty-making. (Oxford University Press, Oxford, 2005). Scott Barrett. Why Cooperate? The Incentive to Supply Global Public Goods. (Oxford University Press, Oxford, 2007). Nick Bostrom and Milan Cirkovic, eds. Global Catastrophic Risks. (Oxford University Press, Oxford, 2011). Jared Diamond. Collapse: How Societies Choose to Fail or Succeed. (Viking Penguin, New York, 2005). Tim Flannery. Atmosphere of Hope: Searching for Solutions to the Climate Crisis. (Atlantic Monthly Press, New York, 2015).

WTF?: What's the Future and Why It's Up to Us
by Tim O'Reilly
Published 9 Oct 2017

The fear is that an artificial general intelligence will develop its own goals and, because of its ability to learn on its own at superhuman speeds, will improve itself at a rate that soon leaves humans far behind. The dire prospect is that such a superhuman AI would have no use for humans, or at best might keep us in the way that we keep pets or domesticated animals. No one even knows what such an intelligence might look like, but people like Nick Bostrom, Stephen Hawking, and Elon Musk postulate that once it exists, it will rapidly outstrip humanity, with unpredictable consequences. Bostrom calls this hypothetical next step in strong AI “artificial superintelligence.” Deep learning pioneers Demis Hassabis and Yann LeCun are skeptical. They believe we’re still a long way from artificial general intelligence.

The Technology Trap: Capital, Labor, and Power in the Age of Automation
by Carl Benedikt Frey
Published 17 Jun 2019

But this metal serves also for war, murder and robbery; and not only at close quarters, man to man, but also by projection and flight; for it can be hurled either by ballistic machines, or by the strength of human arms or even in the form of arrows. And this I hold to be the most blameworthy product of the human mind.6 Much like the fears surrounding the disruptive effects of artificial intelligence (AI) today, with scholars like Stephen Hawking and Nick Bostrom suggesting that it could spell the end of human civilization, people in preindustrial times worried that technology could destroy their much smaller and more isolated world. This was not just a worry of Pliny the Elder but the intuition that shaped attitudes toward technological progress among elites throughout classical antiquity (around 500 B.C. to A.D. 500).

The Code Breaker: Jennifer Doudna, Gene Editing, and the Future of the Human Race
by Walter Isaacson
Published 9 Mar 2021

These include Françoise Baylis, Michael Sandel, Leon Kass, Francis Fukuyama, Nathaniel Comfort, Jason Scott Robert, Eric Cohen, Bill McKibben, Marcy Darnovsky, Erik Parens, Josephine Johnston, Rosemarie Garland-Thomson, Robert Sparrow, Ronald Dworkin, Jürgen Habermas, Michael Hauskeller, Jonathan Glover, Gregory Stock, John Harris, Maxwell Mehlman, Guy Kahane, Jamie Metzl, Allen Buchanan, Julian Savulescu, Lee Silver, Nick Bostrom, John Harris, Ronald Green, Nicholas Agar, Arthur Caplan, and Hank Greeley. I also drew on the work of the Hastings Center, the Center for Genetics and Society, the Oxford Uehiro Centre for Practical Ethics, and the Nuffield Council on Bioethics. 2. Sandel, The Case against Perfection; Robert Sparrow, “Genetically Engineering Humans,” Pharmaceutical Journal, Sept. 24, 2015; Jamie Metzl, Hacking Darwin (Sourcebooks, 2019); Julian Savulescu, Ruud ter Meulen, and Guy Kahane, Enhancing Human Capacities (Wiley, 2011). 3.

Tribe of Mentors: Short Life Advice From the Best in the World
by Timothy Ferriss
Published 14 Jun 2017

He also hosts the popular podcast Waking Up with Sam Harris. * * * What is the book (or books) you’ve given most as a gift, and why? Or what are one to three books that have greatly influenced your life? The Beginning of Infinity by David Deutsch greatly expanded my sense of the potential power of human knowledge, while Nick Bostrom’s Superintelligence made me worry that machine knowledge could ruin everything. I strongly recommend both books. But if you just want to forget about the future and lose yourself in the book that forever changed how narrative nonfiction is written, read In Cold Blood by Truman Capote. What purchase of $100 or less has most positively impacted your life in the last six months (or in recent memory)?

Future Crimes: Everything Is Connected, Everyone Is Vulnerable and What We Can Do About It
by Marc Goodman
Published 24 Feb 2015

Hacking the Software of Life: Bio-crime and Bioterrorism In the nearer term, I think various developments in synthetic biology are quite disconcerting. We are gaining the ability to create designer pathogens, and there are these blueprints of various disease organisms that are in the public domain—you can download the gene sequence for smallpox or the 1918 flu virus from the Internet. NICK BOSTROM Starting in the 1970s and 1980s, groups such as Silicon Valley’s legendary Homebrew Computer Club gathered to talk tech and “hack for good.” Today there is a vibrant DIY-bio movement based very much on the same mind-set, with local community labs such as Genspace in New York and BioCurious in California providing spaces and tools for citizen scientists to come together to work and learn from each other.

The Stack: On Software and Sovereignty
by Benjamin H. Bratton
Published 19 Feb 2016

Dyson, “Search for Artificial Stellar Sources of Infrared Radiation,” Science, June 3, 1960, 1667–1668, doi:10.1126/science.131.3414.166. 76.  Charles Stross, Accelerando (New York: Penguin Group, 2005). See also Robert J. Bradbury, “Matrioshka Brains,” working paper, 1999, http://www.gwern.net/docs/1999-bradbury-matrioshkabrains.pdf. 77.  Nick Bostrom of the Oxford Institute for the Future of Humanity imagines that computational megastructures of this scale and capacity would even be capable of supporting full-scale simulations of entire worlds and if that is so, perhaps our reality is already being powered by such a machine. Bostrom, “Are We Living in a Computer Simulation?”

Bourgeois Dignity: Why Economics Can't Explain the Modern World
by Deirdre N. McCloskey
Published 15 Nov 2011

The less-dogmatic group calls ideas “culture,” which is the vague way people talk when they have not actually taken on board the exact and gigantic literature about ideas, rhetoric, ideology, ceremonies, metaphors, stories, and the like since the Greeks or the Talmudists or the Sanskrit grammarians. They do not, as the artificial intelligence guru Nick Bostrom put a similar point sharply, “resist the temptation to instantaneously misunderstand each new idea by assimilating it with the most similar-sounding cliché available in their cultural larders.”10 The vague “culture” talk makes the mistake that Germany made for centuries, elevating Kultur to a ignorable, higher realm of ornamental distraction from real rhetoric—that is, from actual politics and human relations.11 One thing that is deeply superficial, so to speak, about the neo-institutional notion of “rules of the game,” or constraints, is that it overlooks that the rules are under discussion.

Artificial Intelligence: A Modern Approach
by Stuart Russell and Peter Norvig
Published 14 Jul 2019

Turing (1996) himself made the same point in a lecture given in Manchester in 1951, drawing on earlier ideas from Samuel Butler (1863):15 It seems probable that once the machine thinking method had started, it would not take long to outstrip our feeble powers. ... At some stage therefore we should have to expect the machines to take control, in the way that is mentioned in Samuel Butler’s Erewhon. These concerns have only become more widespread with recent advances in deep learning, the publication of books such as Superintelligence by Nick Bostrom (2014), and public pronouncements from Stephen Hawking, Bill Gates, Martin Rees, and Elon Musk. Experiencing a general sense of unease with the idea of creating superintelligent machines is only natural. We might call this the gorilla problem: about seven million years ago, a now-extinct primate evolved, with one branch leading to gorillas and one to humans.