paperclip maximiser

Thinking Machines: The Inside Story of Artificial Intelligence and Our Race to Build the Future
by Luke Dormehl
Published 10 Aug 2016

A favourite thought experiment of those who believe advanced AI could mean the demise of the human race is the so-called ‘paperclip maximiser’ scenario. In the scenario, proposed by Swedish philosopher and computational neuroscientist Nick Bostrom, an AI is given the seemingly harmless goal of running a factory producing paperclips. Issued with the task of maximising the efficiency for producing paperclips, the AI, able to utilise nano technology to reconstruct matter on a molecular level, disastrously proceeds to turn first the Earth and then a large portion of the observable universe into paperclips. The ‘paperclip maximiser’ scenario is a common one, although it seems to me more a question of artificial stupidity than Artificial Intelligence.

…

The ‘paperclip maximiser’ scenario is a common one, although it seems to me more a question of artificial stupidity than Artificial Intelligence. The inability to answer questions like ‘Why are you making paperclips when there is no paper left?’ or ‘Why are you making paperclips when the person who requested the paperclips in the first place has, himself, been turned into more paperclips?’ doesn’t speak of an advanced superintelligence, unless there is something dramatically important about the nature of paperclips that I am missing. Instead, the threat comes from AI that is smart enough to work with other connected devices, but not smart enough to question its own motivations.

…

(TV show) 135–9, 162, 189–90, 225, 254 Jobs, Steve 6–7, 32, 35, 108, 113, 181, 193, 231 Jochem, Todd 55–6 judges 153–4 Kasparov, Garry 137, 138–9, 177 Katz, Lawrence 159–60 Keck, George Fred 81–2 Keynes, John Maynard 139–40 Kjellberg, Felix (PewDiePie) 151 ‘knowledge engineers’ 29, 37 Knowledge Narrator 110–11 Kodak 238 Kolibree 67 Koza, John 188–9 Ktesibios of Alexandria 71–2 Kubrick, Stanley 2, 228 Kurzweil, Ray 213–14, 231–3 Landauer, Thomas 201–2 Lanier, Jaron 156, 157 Laorden, Carlos 100, 101 learning 37–9, 41–4, 52–3, 55 Deep 11–2, 56–63, 96–7, 164, 225 and email filters 88 machine 3, 71, 84–6, 88, 100, 112, 154, 158, 197, 215, 233, 237, 239 reinforcement 83, 232 and smart homes 84, 85 supervised 57 unsupervised 57–8 legal profession 145, 188, 192 LegalZoom 145 LG 132 Lickel, Charles 136–7 ‘life logging’ software 200 Linden, David J. 213–14 Loebner, Hugh 102–3, 105 Loebner Prize 102–5 Lohn, Jason 182, 183–5, 186 long-term potentiation 39–40 love 122–4 Lovelace, Ada 185, 189 Lovelace Test 185–6 Lucas, George 110–11 M2M communication 70–71 ‘M’ (AI assistant) 153 Machine Intelligence from Cortical Networks (MICrONS) project 214–15 machine learners 38 machine learning 3, 71, 84–6, 88, 100, 112, 154, 158, 197, 215, 233, 237, 239 Machine Translator 8–9, 11 ‘machine-aided recognition’ 19–20 Manhattan Project 14, 229 MARK 1 (computer) 43–4 Mattersight Corporation 127 McCarthy, John 18, 19, 20, 27, 42, 54, 253 McCulloch, Warren 40–2, 43, 60, 142–3 Mechanical Turk jobs 152–7 medicine 11, 30, 87–8, 92–5, 187–8, 192, 247, 254 memory 13, 14, 16, 38–9, 42, 49 ‘micro-worlds’ 25 Microsoft 62–3, 106–7, 111–12, 114, 118, 129 mind mapping the 210–14, 217, 218 ‘mind clones’ 203 uploads 221 mindfiles 201–2, 207, 212 Minsky, Marvin 18, 21, 24, 32, 42, 44–6, 49, 105, 205–7, 253–4 MIT 19–20, 27, 96–7, 129, 194–5 Mitsuku (chatterbot) 103–6, 108 Modernising Medicine 11 Momentum Machines, Inc. 141 Moore’s Law 209, 220, 231 Moravec’s paradox 26–7 mortgage applications 237–8 MTurk platform 153, 154, 155 music 168, 172–7, 179 Musk, Elon 149–50, 223–4 MYCIN (expert system) 30–1 nanobots 213–14 nanosensors 92 Nara Logics 118 NASA 6, 182, 184–5 natural selection 182–3 navigational aids 90–1, 126, 127, 128, 241 Nazis 15, 17, 227 Negobot 99–102 Nest Labs 67, 96, 254 Netflix 156, 198 NETtalk 51, 52–3, 60 neural networks 11–12, 38–9, 41, 42–3, 97, 118, 164–6, 168, 201, 208–9, 211, 214–15, 218, 220, 224–5, 233, 237–8, 249, 254, 256–7 neurons 40, 41–2, 46, 49–50, 207, 209–13, 216 neuroscience 40–2, 211, 212, 214, 215 New York World’s Fair 1964 5–11 Newell, Alan 19, 226 Newman, Judith 128–9 Nuance Communications 109 offices, smart 90 OpenWorm 210 ‘Optical Scanning and Information Retrieval’ 7–8, 10 paedophile detection 99–102 Page, Larry 6–7, 34, 220 ‘paperclip maximiser’ scenario 235 Papert, Seymour 27, 44, 45–6, 49 Paro (therapeutic robot) 130–1 patents 188–9 Perceiving and Recognising Automation (PARA) 43 perceptrons 43–6 personality capture 200–4 pharmaceuticals 187–8 Pitts, Walter 40–2, 43, 60 politics 119–2 Pomerlau, Dean 54, 55–6, 90 prediction 87, 198–9 Profound Hypothermia and Circulatory Arrest 219–20 punch-cards 8 Qualcomm 93 radio-frequency identification device (RFID) 65–6 Ramón y Cajal, Santiago 39–40 Rapidly Adapting Lateral Position Handler (RALPH) 55 ‘recommender system’ 198 refuse collection 142 ‘relational agents’ 130 remote working 238–9 reverse engineering 208, 216, 217 rights for AIs 248–51 risks of AI 223–40 accountability issues 240–4, 246–8 ethics 244–8 rights for AIs 248–51 technological unemployment 139–50, 163, 225, 255 robots 62, 74–7, 89–90, 130–1, 141, 149, 162, 217, 225, 227, 246–7, 255–6 Asimov’s three ethical rules of 244–8 robotic limbs 211–12 Roomba robot vacuum cleaner 75–7, 234, 236 Rosenblatt, Frank 42–6, 61, 220 rules 36–7, 79–80 Rumelhart, David 48, 50–1, 63 Russell, Bertrand 41 Rutter, Brad 138, 139 SAINT program 20 sampling (music) 155, 157 ‘Scheherazade’ (Ai storyteller) 169–70 scikit-learn 239 Scripps Health 92 Sculley, John 110–11 search engines 109–10 Searle, John 24–5 Second Life (video game) 194 Second World War 12–13, 14–15, 17, 72, 227 Sejnowski, Terry 48, 51–3 self-awareness 77, 246–7 self-driving 53–6, 90, 143, 149–50 Semantic Information Retrieval (SIR) 20–2 sensors 75–6, 80, 84–6, 93 SHAKEY robot 23–4, 27–8, 90 Shamir, Lior 172–7, 179, 180 Shannon, Claude 13, 16–18, 28, 253 shipping systems 198 Simon, Herbert 10, 19, 24, 226 Sinclair Oil Corporation 6 Singularity, the 228–3, 251, 256 Siri (AI assistant) 108–11, 113–14, 116, 118–19, 125–30, 132, 225–6, 231, 241, 256 SITU 69, 93 Skynet 231 smart devices 3, 66–7, 69–71, 73–7, 80–8, 92–7, 230–1, 254 and AI assistants 116 and feedback 73–4 problems with 94–7 ubiquitous 92–4 and unemployment 141–2 smartwatches 66, 93, 199 Sony 199–200 Sorto, Erik 211, 212 Space Invaders (video game) 37 spectrometers 93 speech recognition 59, 62, 109, 111, 114, 120 SRI International 28, 89–90, 112–13 StarCraft II (video game) 186–7 story generation 169–70 strategy 36 STUDENT program 20 synapses 209 Synthetic Interview 202–3 Tamagotchis 123–5 Tay (chatbot) 106–7 Taylorism 95–6 Teknowledge 32, 33 Terminator franchise 231, 235 Tetris (video game) 28 Theme Park (video game) 29 thermostats 73, 79, 80 ‘three wise men’ puzzle 246–7 Tojan Room, Cambridge University 69–70 ‘tortoises’ (robots) 74–7 toys 123–5 traffic congestion 90–1 transhumanists 205 transistors 16–17 Transits – Into an abyss (musical composition) 168 translation 8–9, 11, 62–3, 155, 225 Turing, Alan 3, 13–17, 28, 35, 102, 105–6, 227, 232 Turing Test 15, 101–7, 229, 232 tutors, remote 160–1 TV, smart 80, 82 Twitter 153–4 ‘ubiquitous computing’ 91–4 unemployment, technological 139–50, 163, 225, 255 universal micropayment system 156 Universal Turing Machine 15–16 Ursache, Marius 193–7, 203–4, 207 vacuum cleaners, robotic 75–7, 234, 236 video games 28–9, 35–7, 151–2, 186–7, 194, 197 Vinge, Vernor 229–30 virtual assistants 107–32, 225–6, 240–1 characteristics 126–8 falling in love with 122–4 political 119–22 proactive 116–18 therapeutic 128–31 voices 124–126, 127–8 Viv Labs 132 Vladeck, David 242–4 ‘vloggers’ 151–2 von Neumann, John 13–14, 17, 100, 229 Voxta (AI assistant) 119–20 waiter drones 141 ‘Walking Cities’ 89–90 Walter, William Grey 74–7 Warwick, Kevin 65–6 Watson (Blue J) 138–9, 162, 189–92 Waze 90–91, 126 weapons 14, 17, 72, 224–5, 234–5, 247, 255–6 ‘wetware’ 208 Wevorce 145 Wiener, Norbert 72–3, 227 Winston, Patrick 49–50 Wofram Alpha tool 108–9 Wozniak, Steve 35, 114 X.ai 116–17 Xbox 360, Kinect device 114 XCoffee 70 XCON (expert system) 31 Xiaoice 129, 130 YouTube 151 Yudkowsky, Eliezer 237–8 Zuckerberg, Mark 7, 107–8, 230–1, 254–5 Acknowledgments WRITING A BOOK is always a bit of a solitary process.

Being You: A New Science of Consciousness
by Anil Seth
Published 29 Aug 2021

From the beast machine perspective, the quest to understand consciousness places us increasingly within nature, not further apart from it. Just as it should. Notes a golem: In his 1964 book God and Golem, Inc., the polymathic pioneer Norbert Wiener treated golems as central to his speculations about risks of future AI. vast mound of paperclips: In the parable of the paperclip maximiser, an AI is designed to make as many paperclips as possible. Because this AI lacks human values but is otherwise very smart, it destroys the world in its successful attempt to do so. See Bostrom (2014). so-called ‘Singularity’ hypothesis: See Shanahan (2015) for a refreshingly sober take on the Singularity hypothesis.

Surviving AI: The Promise and Peril of Artificial Intelligence
by Calum Chace
Published 28 Jul 2015

Rats which can choose between a direct stimulation of their brain’s pleasure centres or an item of food will starve themselves to death. Nick Bostrom calls this idea of causing great harm by mis-understanding the implications of an attempt to do great good “perverse instantiation”. Others might call it the law of unintended consequences, or Sod’s Law. The paperclip maximiser If somebody running a paperclip factory turns out to be the first person to create an AGI and it rapidly becomes a superintelligence, they are likely to have created an entity whose goal is to maximise the efficient production of paperclips. This has become the canonical example of what Nick Bostrom calls “infrastructure profusion”, the runaway train of superintelligence problems.

Superintelligence: Paths, Dangers, Strategies
by Nick Bostrom
Published 3 Jun 2014

One might think that the risk of a malignant infrastructure profusion failure arises only if the AI has been given some clearly open-ended final goal, such as to manufacture as many paperclips as possible. It is easy to see how this gives the superintelligent AI an insatiable appetite for matter and energy, since additional resources can always be turned into more paperclips. But suppose that the goal is instead to make at least one million paperclips (meeting suitable design specifications) rather than to make as many as possible. One would like to think that an AI with such a goal would build one factory, use it to make a million paperclips, and then halt. Yet this may not be what would happen.

…

(But how obvious was it before it was pointed out that there was a problem here in need of remedying?) Namely, if we want the AI to make some paperclips for us, then instead of giving it the final goal of making as many paperclips as possible, or to make at least some number of paperclips, we should give it the final goal of making some specific number of paperclips—for example, exactly one million paperclips—so that going beyond this number would be counterproductive for the AI. Yet this, too, would result in a terminal catastrophe. In this case, the AI would not produce additional paperclips once it had reached one million, since that would prevent the realization of its final goal.

…

Might we avoid this malignant outcome if instead of a maximizing agent we build a satisficing agent, one that simply seeks to achieve an outcome that is “good enough” according to some criterion, rather than an outcome that is as good as possible? There are at least two different ways to formalize this idea. The first would be to make the final goal itself have a satisficing character. For example, instead of giving the AI the final goal of making as many paperclips as possible, or of making exactly one million paperclips, we might give the AI the goal of making between 999,000 and 1,001,000 paperclips. The utility function defined by the final goal would be indifferent between outcomes in this range; and as long as the AI is sure it has hit this wide target, it would see no reason to continue to produce infrastructure.

Prediction Machines: The Simple Economics of Artificial Intelligence
by Ajay Agrawal , Joshua Gans and Avi Goldfarb
Published 16 Apr 2018

Bostrom talks of a paper-clip-obsessed superintelligence that cares about nothing but making more paper clips. The paper-clip AI could just wipe out everything else through single-mindedness. This is a powerful idea, but it overlooks competition for resources. Something economists respect is that different people (and now AIs) have different preferences. Some might be open-minded about exploration, discovery, and peace, while others may be paper-clip makers. So long as interests compete, competition will flourish, meaning that the paper-clip AI will likely find it more profitable to trade for resources than fight for them and, as if guided by an invisible hand, will end up promoting benefits distinct from its original intention.

The Equality Machine: Harnessing Digital Technology for a Brighter, More Inclusive Future
by Orly Lobel
Published 17 Oct 2022

Turing, “Computing Machinery and Intelligence,” Mind 59, no. 236 (October 1950): 433, https://doi.org/10.1093/mind/LIX.236.433. 18. David Z. Morris, “Elon Musk Says Artificial Intelligence Is the ‘Greatest Risk We Face as a Civilization,” Fortune, July 15, 2017, https://fortune.com/2017/07/15/elon-musk-artificial-intelligence-2/. 19. Joshua Gans, “AI and the Paperclip Problem,” VoxEU, June 10, 2018, https://voxeu.org/article/ai-and-paperclip-problem. 20. Pedro Domingos, The Master Algorithm: How the Quest for the Ultimate Learning Machine Will Remake Our World (New York: Basic Books, 2015), 286. 21. “Learn More,” EqualAI, last accessed December 21, 2021, www.equalai.org/learn-more. 22. NSTC Committee on Technology, Preparing for the Future of Artificial Intelligence, October 2016, 27, https://obamawhitehouse.archives.gov/sites/default/files/whitehouse_files/microsites/ostp/NSTC/preparing_for_the_future_of_ai.pdf. 23.

The Doomsday Calculation: How an Equation That Predicts the Future Is Transforming Everything We Know About Life and the Universe
by William Poundstone
Published 3 Jun 2019

He proposes that a general-purpose super-AI could be more perceptive than humans in every way, including empathy, emotional intelligence, sense of humor, negotiation skills, and salesmanship. Super-AI would know us better than we know ourselves. That is the truly terrifying thing. In a paperclips-of-doom scenario, the AI might well understand that its human creators emphatically did not intend it to transmute the entire universe into paperclips. But it might be “compartmentalized,” like a psychopath. If maximizing paperclips is the goal and not destroying the world is the subgoal, then the AI will act accordingly. Like a human, a successful AI must be capable of prioritizing multiple, sometimes contradictory goals and making wise trade-offs.

…

There the problem is more like that of writing a national constitution that can remain in force for centuries, spanning unimaginable cultural and technological changes. It is important to have mechanisms for amending the constitution, but the process shouldn’t be too easy, lest there be no point in having the constitution in the first place. Yet even that analogy fails, for human nature doesn’t change much. AI would be amending itself, creating brave new reference classes. Paperclips of Doom Bostrom and his global counterparts are not luddites. They seek to encourage the development of safe AI. Not all AI researchers welcome the help. In his 2014 book, Superintelligence, Bostrom spins tales and thought experiments of what could go wrong, often striking an imaginatively dystopian note.

…

In his 2014 book, Superintelligence, Bostrom spins tales and thought experiments of what could go wrong, often striking an imaginatively dystopian note. One such scenario is “paperclips of doom.” Suppose that super-intelligence is realized. In order to test it, its human designers assign it a simple task: make paperclips. A 3-D printer networked to the AI begins printing out something. It’s not a paperclip… it’s a robot.… Before anyone can figure out what’s happening, the robot scampers out of the room, faster than a cheetah. Pandora’s box has been opened. The robot is a mobile paperclip factory, able to collect scrap metal and transform it into paperclips. The robot is also self-reproducing, able to make countless copies of itself.

Ways of Being: Beyond Human Intelligence
by James Bridle
Published 6 Apr 2022

Having secured control of legal and financial systems, and suborned national governance and lethal force to its will, all Earth’s resources are fair game for the AI in pursuit of more efficient paperclip manufacture: mountain ranges are levelled, cities razed, and eventually all human and animal life is fed into giant machines and rendered into its component minerals. Giant paperclip rocket ships eventually leave the ravaged Earth to source energy directly from the Sun and begin the exploitation of the outer planets.9 It’s a terrifying and seemingly ridiculous chain of events – but only ridiculous in so far as an advanced Artificial Intelligence has no need for paperclips. Driven by the logic of contemporary capitalism and the energy requirements of computation itself, the deepest need of an AI in the present era is the fuel for its own expansion.

…

The ways in which the development of these supposedly intelligent tools might harm, efface and ultimately supplant us has become the subject of a wide field of study, involving computer scientists, programmers and technology firms, as well as theorists and philosophers of machine intelligence itself. One of most dramatic of these possible futures is described in something called the paperclip hypothesis. It goes like this. Imagine a piece of intelligent software – an AI – designed to optimize the manufacture of paperclips, an apparently simple and harmless business goal. The software might begin with a single factory: automating the production line, negotiating better deals with suppliers, securing more outlets for its wares. As it reaches the limits of a single establishment, it might purchase other firms, or its suppliers, adding mining companies and refineries to its portfolio, to provide its raw materials on better terms.

On the Edge: The Art of Risking Everything
by Nate Silver
Published 12 Aug 2024

Opinion, nytimes.com/2015/10/04/opinion/the-power-of-precise-predictions.html. GO TO NOTE REFERENCE IN TEXT close to 100 percent: Gary Marcus, “p(doom),” Marcus on AI (blog), August 27, 2023, garymarcus.substack.com/p/d28. GO TO NOTE REFERENCE IN TEXT many paper clips: Joshua Gans, “AI and the Paperclip Problem,” Centre for Economic Policy Research, June 10, 2018, cepr.org/voxeu/columns/ai-and-paperclip-problem. GO TO NOTE REFERENCE IN TEXT adopt means suitable: Niko Kolodny and John Brunero, “Instrumental Rationality,” in The Stanford Encyclopedia of Philosophy, ed.

…

One of the reasons Phil Tetlock (of hedgehog-and-foxes fame) found that experts made such poor forecasts is because they’d been allowed to get away with this lazy rhetoric. By contrast, someone like Yudkowsky—whose p(doom) is very high, fairly close to 100 percent—will take a lot of reputational damage if AI alignment proves relatively easy to achieve. Whereas if the machines turn us all into paper clips—one of Nick Bostrom’s famous thought experiments involves an unaligned AI with the goal of manufacturing as many paper clips as possible—he won’t be around to take credit. (As someone who has as much experience as pretty much anyone making probabilistic forecasts in public, I can also tell you from firsthand experience that incentives to do this are poor.

…

Interpretability (AI): The degree to which the behavior and inner workings of an AI system can be readily understood by humans. Inside straight: See: straight. Instrumental convergence: The hypothesis that a superintelligent machine will pursue its own goals to minimize its loss function and won’t let humans stand in its way—even if the AI’s goal isn’t to kill humans, we’ll be collateral damage as part of their game of Paper Clip Mogul. IRR: Internal rate of return; the annualized growth rate of an investment. Isothymia: A term adapted from Plato by Francis Fukuyama to refer to the profound desire to be seen as equal to others. See also: megalothymia. Iteration: One cycle in a repetitive process in which a model’s estimates are progressively improved by incorporating the results from the preceding cycle.

The Big Nine: How the Tech Titans and Their Thinking Machines Could Warp Humanity
by Amy Webb
Published 5 Mar 2019

A superintelligent AI isn’t necessarily dangerous, and it doesn’t necessarily obviate the role we play in civilization. However, superintelligent AI would likely make decisions in a nonconscious way using logic that’s alien to us. Oxford University philosopher Nick Bostrom explains the plausible outcomes of ASI using a parable about paperclips. If we asked a superintelligent AI to make paperclips, what would happen next? The outcomes of every AI, including those we have now, are determined by values and goals. It’s possible that an ASI could invent a new, better paperclip that holds a stack of paper together so that even if dropped, the pages would always stay collated in order. It’s possible that if we aren’t capable of explaining how many paperclips we actually want, an ASI could go on making paperclips forever, filling our homes and offices with them as well as our hospitals and schools, rivers and lakes, sewage systems, and on and on until mountains of paperclips covered the planet.

…

My dad will be in his late 90s, and all of his medical specialists (cardiologists, nephrologists, radiologists) will be AGIs, directed and managed by a highly trained general practitioner, who is both an MD and a data scientist. The advent of ASI could follow soon or much longer after, between the 2040s and 2060s. It doesn’t mean that by 2070 superintelligent AIs will have crushed all life on Earth under the weight of quintillions of paperclips. But it doesn’t mean they won’t have either. The Stories We Must Tell Ourselves Planning for the futures of AI requires us to build new narratives using data from the real world. If we agree that AI will evolve as it emerges, then we must create scenarios that describe the intersection of the Big Nine, the economic and political forces guiding them, and the ways humanity factors in as AI transitions from narrow applications to generally intelligent and ultimately superintelligent thinking machines.

New Laws of Robotics: Defending Human Expertise in the Age of AI
by Frank Pasquale
Published 14 May 2020

.… The very structure of reason itself comes from the details of our embodiment.”48 Lakoff and Johnson show that abstractions that drive much of AI—be they utilitarian assessments of well-being or statistical analysis of regularities—begin to break down once they are alienated from the embodied perspective of actual humans. That is the core problem in one of “existential risk studies” classic narratives of out-of-control AI—an unstoppable paper-clip maximizer that starts to use all available material on earth to generate more paperclips.49 For many in the mainstream of ethical AI, the solution to a potentially out-of-control AI (and many more mundane problems) is to program more rules into it. Perhaps we could easily program an “anti-maximization” rule into our hypothetical paper clip maker, and try our best to secure it from hackers.

Practical Doomsday: A User's Guide to the End of the World
by Michal Zalewski
Published 11 Jan 2022

If not impossible to rule out completely, the prospect of an overtly evil AI is at minimum an unimaginatively reductionist take. A more fascinating threat is that of a machine that doesn’t perceive humans as adversaries, but simply misinterprets or disregards our desires and goals. An example is the well-known parable of the paperclip maximizer: a hypothetical autonomous AI designed to continually improve the efficiency of a paperclip production line. The AI expands and improves the operation, developing new assembly methods and new resource extraction and recycling procedures, until it’s done converting the entire planet and its many inhabitants into paperclips. The point of the tale is simple: the AI doesn’t need to hate you or love you; it suffices that you’re made of atoms it has a different use for.

Other Pandemic: How QAnon Contaminated the World
by James Ball
Published 19 Jul 2023

Robot Rules: Regulating Artificial Intelligence
by Jacob Turner
Published 29 Oct 2018

Technology has always been a double edged sword, since fire kept us warm but also burned down our villages”.119 Similarly, engineer and roboethicist Alan Winfield said in a 2014 article: “If we succeed in building human equivalent AI and if that AI acquires a full understanding of how it works, and if it then succeeds in improving itself to produce super-intelligent AI, and if that super-AI, accidentally or maliciously, starts to consume resources, and if we fail to pull the plug, then, yes, we may well have a problem. The risk, while not impossible, is improbable”.120 Fundamentally, optimists think humanity can and will overcome any challenges AI poses. The pessimists include Nick Bostrom, whose “paperclip machine” thought experiment imagines an AI system asked to make paperclips which decides to seize and consume all resources in existence, in its blind aderence to that goal.121 Bostrom contemplates a form of superintelligence which is so powerful that humanity has no chance of stopping it from destroying the entire universe.

Architects of Intelligence
by Martin Ford
Published 16 Nov 2018

NICK BOSTROM: The paperclip example is a stand-in for a wider category of possible failures where you ask a system to do one thing and, perhaps, initially things turn out pretty well but then it races to a conclusion that is beyond our control. It’s a cartoon example, where you design an AI to operate a paperclip factory. It’s dumb initially, but the smarter it gets, the better it operates the paperclip factory, and the owner of this factory is very pleased and wants to make more progress. However, when the AI becomes sufficiently smart, it realizes that there are other ways of achieving an even greater number of paperclips in the world, which might then involve taking control away from humans and indeed turning the whole planet into paperclips or into space probes that can go out and transform the universe into more paperclips.

…

What do you think we should be worried about in terms of the impacts and risks of AI? GARY MARCUS: We should be worrying about people using AI in malevolent ways. The real problem is what people might do with the power that AI holds as it becomes more embedded in the grid and more hackable. I’m not that worried about AI systems independently wanting to eat us for breakfast or turn us into paper clips. It’s not completely impossible, but there’s no real evidence that we’re moving in that direction. There is evidence, though, that we’re giving more and more power to those machines, and that we have no idea how to solve the cybersecurity threats in the near term.

Falter: Has the Human Game Begun to Play Itself Out?
by Bill McKibben
Published 15 Apr 2019

But to your surprise, you find that it strenuously resists your efforts to turn it off.”24 Consider what’s become the canonical formulation of the problem, an artificial intelligence that is assigned the task of manufacturing paper clips in a 3-D printer. (Why paper clips in an increasingly paperless world? It doesn’t matter.) At first, says another Oxford scientist, Anders Sandberg, nothing seems to happen, because the AI is simply searching the internet. It “zooms through various possibilities. It notices that smarter systems generally can make more paper-clips, so making itself smarter will likely increase the number of paper-clips that will eventually be made. It does so. It considers how it can make paper-clips using the 3D printer, estimating the number of possible paper-clips. It notes that if it could get more raw materials it could make more paper-clips.

…

As he points out, we don’t particularly hate field mice, but every hour of every day we plow under millions of their dens to make sure we have supper.27 This isn’t like, say, Y2K, where grizzled old programmers could emerge out of their retirement communities to save the day with some code. “If I tried to pull the plug on it, it’s smart enough that it’s figured out a way of stopping me,” Anders Sandberg said of his paper clip AI. “Because if I pull the plug, there will be fewer paper clips in the world and that’s bad.”28 You’ll be pleased to know that not everyone is worried. Steven Pinker ridicules fears of “digital apocalypse,” insisting that “like any other technology,” artificial intelligence is “tested before it is implemented and constantly tweaked for safety and efficacy.”29 The always lucid virtual reality pioneer Jaron Lanier is dubious about the danger, too, but for precisely the opposite reason.

Artificial Intelligence: A Guide for Thinking Humans
by Melanie Mitchell
Published 14 Oct 2019

I, Warbot: The Dawn of Artificially Intelligent Conflict
by Kenneth Payne
Published 16 Jun 2021

If it were the broom in Goethe’s fable, it would be constantly asking the apprentice—‘is that enough mopping, or do you need more?’ It’s a deceptively simple solution to a thorny issue: Can you stop an AI once it’s started? One popular variant on the theme is the idea of an AI that will resist being turned off, like the sinister HAL in Kubrick’s 2001. In Nick Bostrom’s philosophical thought experiment, an AI is tasked with simply counting paperclips—an innocuous task a long way distant from nuclear brinksmanship.17 The problem occurs when the machine goes back for a recount, just to be sure it hasn’t made a mistake. After all there’s a vanishingly small probability it has, and since it had no instructions to the contrary, it had better make sure.

Our Final Invention: Artificial Intelligence and the End of the Human Era
by James Barrat
Published 30 Sep 2013

In Bostrom’s scenario, a thoughtlessly programmed superintelligence whose programmed goal is to manufacture paper clips does exactly as it is told without regard to human values. It all goes wrong because it sets about “transforming first all of earth and then increasing portions of space into paper clip manufacturing facilities.” Friendly AI would make only as many paper clips as was compatible with human values. Another tenet of Friendly AI is to avoid dogmatic values. What we consider to be good changes with time, and any AI involved with human well-being will need to stay up to speed. If in its utility function an AI sought to preserve the preferences of most Europeans in 1700 and never upgraded them, in the twenty-first century it might link our happiness and welfare to archaic values like racial inequality and slaveholding, gender inequality, shoes with buckles, and worse.

Power and Progress: Our Thousand-Year Struggle Over Technology and Prosperity
by Daron Acemoglu and Simon Johnson
Published 15 May 2023

The thought experiment presupposes an unstoppably powerful intelligent machine that gets instructions to produce more paper clips and then uses its considerable capabilities to excel in meeting this objective by coming up with new methods to transform the entire world into paper clips. When it comes to the effects of AI on politics, it may be turning our institutions into paper clips, not thanks to its superior capabilities but because of its mediocrity. By 2017, Facebook was so popular in Myanmar that it came to be identified with the internet itself. The twenty-two million users, out of a population of fifty-three million, were fertile ground for misinformation and hate speech.

The Road to Conscious Machines
by Michael Wooldridge
Published 2 Nov 2018

But Mickey dozes off and, upon waking up, his attempts to stop the broomstick bringing in water, bucket after bucket, result in ever more magical broomsticks flooding his basement. It requires the intervention of his master, the sorcerer, to rectify the problem. Mickey got what he asked for, but not what he wanted. Bostrom also considered these types of scenarios. He imagined an AI system that controlled the manufacture of paperclips. The system is given the goal of maximizing the production of paperclips, which it takes literally, transforming first the earth and then the rest of the universe into paperclips. Again, the problem is ultimately one of communication: in this case, when we communicate our goal, we need to be sure that acceptable boundaries are understood.

The Coming Wave: Technology, Power, and the Twenty-First Century's Greatest Dilemma
by Mustafa Suleyman
Published 4 Sep 2023

Think an all-powerful machine somehow destroying the world for its own mysterious ends: not some malignant AI wreaking intentional destruction like in the movies, but a full-scale AGI blindly optimizing for an opaque goal, oblivious to human concerns. The canonical thought experiment is that if you set up a sufficiently powerful AI to make paper clips but don’t specify the goal carefully enough, it may eventually turn the world and maybe even the contents of the entire cosmos into paper clips. Start following chains of logic like this and myriad sequences of unnerving events unspool. AI safety researchers worry (correctly) that should something like an AGI be created, humanity would no longer control its own destiny.

Wired for War: The Robotics Revolution and Conflict in the 21st Century
by P. W. Singer
Published 1 Jan 2010

So, to deal with all of these issues, robots must be able to learn and adapt to changes in their environment.” Al GETS STRONG Today, there are all sorts of artificial intelligence that appear in our daily lives, without our even thinking of them as AI. Anytime you check your voice mail, AI directs your calls. Anytime you try to write a letter in Microsoft Word, an annoying little paper-clip figure pops up, which is an AI trying to turn your scribbles into a stylistically sound correspondence. Anytime you play a video game, the characters in it are internal agents run by AIs, usually with their skill levels graded down so that you can beat them.

Escape From Model Land: How Mathematical Models Can Lead Us Astray and What We Can Do About It
by Erica Thompson
Published 6 Dec 2022

Human Compatible: Artificial Intelligence and the Problem of Control
by Stuart Russell
Published 7 Oct 2019

Possible Minds: Twenty-Five Ways of Looking at AI
by John Brockman
Published 19 Feb 2019

Luce Professor of Information, Technology, Consciousness, and Culture at Princeton University. He is co-author (with Brian Christian) of Algorithms to Live By. Tom Griffiths’s approach to the AI issue of “value alignment”—the study of how, exactly, we can keep the latest of our serial models of AI from turning the planet into paper clips—is human centered; i.e., that of a cognitive scientist, which is what he is. The key to machine learning, he believes, is, necessarily, human learning, which he studies at Princeton using mathematical and computational tools. Tom once remarked to me that “one of the mysteries of human intelligence is that we’re able to do so much with so little.”

To Be a Machine: Adventures Among Cyborgs, Utopians, Hackers, and the Futurists Solving the Modest Problem of Death
by Mark O'Connell
Published 28 Feb 2017

(The surge in sales was partly due to Elon Musk sternly advising his Twitter followers to read it.) Even the most benign form of AI imaginable, the book suggested, could conceivably lead to the destruction of humanity. One of the more extreme hypothetical scenarios the book laid out, for instance, was one in which an AI is assigned the task of manufacturing paper clips in the most efficient and productive manner possible, at which point it sets about converting all the matter in the entire universe into paper clips and paper-clip-manufacturing facilities. The scenario was deliberately cartoonish, but as an example of the kind of ruthless logic we might be up against with an artificial superintelligence, its intent was entirely serious.

The Myth of Artificial Intelligence: Why Computers Can't Think the Way We Do
by Erik J. Larson
Published 5 Apr 2021

Altruistically humble machines help guard against the danger that cigar-smoking tech executives (who probably don’t smoke cigars anymore) could give them venal motives, and also against the possibility of machines being too smart in the wrong way, doing the equivalent of turning everything into gold. Confusingly, “altruistically humble” machines also sound a lot like the Ex Machina take on AI—as “alive” after all, with real (not just paper clip maximizing) intelligence and ethical sensibilities. One might be forgiven for drawing the conclusion that talk of AI is doomed perpetually to straddle science and myth. Russell has a third principle he thinks necessary to thwart existential crisis with the coming superintelligence: AI should be developed in such a way that it learns to predict human preferences.

What to Think About Machines That Think: Today's Leading Thinkers on the Age of Machine Intelligence
by John Brockman
Published 5 Oct 2015

Coders: The Making of a New Tribe and the Remaking of the World
by Clive Thompson
Published 26 Mar 2019

But as Bostrom writes, maybe a “superintelligent” AI wouldn’t need to evolve any new motivations to be dangerous. It could be perfectly benign, happy to do as it’s told—yet still slaughter or enslave us all in happy pursuit of its goals. In one famous thought experiment, Bostrom imagined a superintelligent AI being tasked with making as many paper clips as possible. It might decide the best way to do this would be to disassemble all matter on earth—including humans—to convert it into the raw material for making infinite paper clips, and then barrel onward and outward, converting “increasingly large chunks of the observable universe into paper clips.”

Army of None: Autonomous Weapons and the Future of War
by Paul Scharre
Published 23 Apr 2018

There is no reason to think machine intelligence would necessarily have any of these desires. Bostrom has argued intelligence is “orthogonal” to an entity’s goals, such that “any level of intelligence could in principle be combined with . . . any final goal.” This means a superintelligent AI could have any set of values, from playing the perfect game of chess to making more paper clips. On one level, the sheer alien-ness of advanced AI makes many of science fiction’s fears seem strangely anthropomorphic. Skynet starts nuclear war because it believes humanity is a threat to its existence, but why should it care about its own existence?

Rationality: From AI to Zombies
by Eliezer Yudkowsky
Published 11 Mar 2015

Let me try a different tack of explanation—one closer to the historical way that I arrived at my own position. Suppose you build an AI, and—leaving aside that AI goal systems cannot be built around English statements, and all such descriptions are only dreams—you try to infuse the AI with the action-determining principle, “Do what I want.” And suppose you get the AI design close enough—it doesn’t just end up tiling the universe with paperclips, cheesecake or tiny molecular copies of satisfied programmers—that its utility function actually assigns utilities as follows, to the world-states we would describe in English as: <Programmer weakly desires "X," quantity 20 of X exists>: +20 <Programmer strongly desires "Y," quantity 20 of X exists>: 0 <Programmer weakly desires "X," quantity 30 of Y exists>: 0 <Programmer strongly desires "Y,” quantity 30 of Y exists>: +60 You perceive, of course, that this destroys the world. . . . since if the programmer initially weakly wants “X” and X is hard to obtain, the AI will modify the programmer to strongly want “Y,” which is easy to create, and then bring about lots of Y.

…

That’s the rational course, right? There’s a number of replies I could give to that. I’ll start by saying that this is a prime example of the sort of thinking I have in mind, when I warn aspiring rationalists to beware of cleverness. I’ll also note that I wouldn’t want an attempted Friendly AI that had just decided that the Earth ought to be transformed into paperclips, to assess whether this was a reasonable thing to do in light of all the various warnings it had received against it. I would want it to undergo an automatic controlled shutdown. Who says that meta-reasoning is immune from corruption? I could mention the important times that my naive, idealistic ethical inhibitions have protected me from myself, and placed me in a recoverable position, or helped start the recovery, from very deep mistakes I had no clue I was making.

Enlightenment Now: The Case for Reason, Science, Humanism, and Progress
by Steven Pinker
Published 13 Feb 2018

Many of the commentators in Lanier 2014 and Brockman 2015 make this point as well. 26. AI researchers vs. AI hype: Brooks 2015; Davis & Marcus 2015; Kelly 2017; Lake et al. 2017; Lanier 2014; Marcus 2016; Naam 2010; Schank 2015. See also note 25 above. 27. Shallowness and brittleness of current AI: Brooks 2015; Davis & Marcus 2015; Lanier 2014; Marcus 2016; Schank 2015. 28. Naam 2010. 29. Robots turning us into paper clips and other Value Alignment Problems: Bostrom 2016; Hanson & Yudkowsky 2008; Omohundro 2008; Yudkowsky 2008; P. Torres, “Fear Our New Robot Overlords: This Is Why You Need to Take Artificial Intelligence Seriously,” Salon, May 14, 2016. 30.

The Seventh Sense: Power, Fortune, and Survival in the Age of Networks
by Joshua Cooper Ramo
Published 16 May 2016

This puzzle has interested the Oxford philosopher Nick Bostrom, who has described the following thought experiment: Imagine a superintelligent machine programmed to do whatever is needed to make paper clips as fast as possible, a machine that is connected to every resource that task might demand. Go figure it out! might be all its human instructors tell it. As the clip-making AI becomes better and better at its task, it demands more and still more resources: more electricity, steel, manufacturing, shipping. The paper clips pile up. The machine looks around: If only I could control the power supply, it thinks. It eyes the shipping. The steel mining. The humans. And so, ambitious for more and better paper clips, it begins to think around its masters, who are incapable of stopping until it has punched the entire world into paper clips.