paperclip maximiser
description: thought experiment to illustrate existential risk posed by artificial intelligence
53 results
This Is for Everyone: The Captivating Memoir From the Inventor of the World Wide Web
by Tim Berners-Lee · 8 Sep 2025 · 347pp · 100,038 words
powerful now. Perhaps it was the memory of Clippy that prompted the philosopher Nick Bostrom’s hypothetical ‘paperclip maximizer’, the idea of an AI that, like the brooms in The Sorcerer’s Apprentice, fulfils its directive to make as many paper clips as possible by transforming all atoms (and in the process, all humans) into
…
paper clips and thus annihilating the universe. This argument drives me crazy, because this hypothetical hypersmart paperclip AI is really very dumb, with zero embedded controls. Nothing we’re building resembles a paperclip maximizer – in fact, responsive systems like ChatGPT are already far smarter than
…
it. ChatGPT would know it’s making too many paper clips. It would know because you told it so
…
called the ‘human in the loop’ doctrine.) But how is the human going to validate what the AI has done? It may be easy to reject a plan to convert the universe into paper clips, but it’s hard to override a complex medical diagnosis. If we are about to make something smarter
…
’Reilly ref1, ref2, ref3 Oxford University ref1, ref2, ref3, ref4 PACER ref1 packet-switching ref1 Page, Larry ref1, ref2 PageRank ref1 pantomime ref1 ‘paperclip maximizer’ ref1 paradigm shift, artificial intelligence (AI) ref1 passkeys ref1 passwords ref1 patents ref1 peace ref1 Pellow, Nicola ref1 Penrose, Roger ref1 Pentagon ref1, ref2, ref3 Pets.com ref1 philanthropy ref1
To Be a Machine: Adventures Among Cyborgs, Utopians, Hackers, and the Futurists Solving the Modest Problem of Death
by Mark O'Connell · 28 Feb 2017 · 252pp · 79,452 words
the destruction of humanity. One of the more extreme hypothetical scenarios the book laid out, for instance, was one in which an AI is assigned the task of manufacturing paper clips in the most efficient and productive manner possible, at which point it sets about converting all the matter in the entire universe
…
into paper clips and paper-clip-manufacturing facilities. The scenario was deliberately cartoonish, but as an example of the kind of ruthless logic we might be up against with an artificial
Falter: Has the Human Game Begun to Play Itself Out?
by Bill McKibben · 15 Apr 2019
resists your efforts to turn it off.”24 Consider what’s become the canonical formulation of the problem, an artificial intelligence that is assigned the task of manufacturing paper clips in a 3-D printer. (Why paper clips in an increasingly paperless world? It doesn’t matter.) At first, says another Oxford scientist, Anders Sandberg, nothing
…
generally can make more paper-clips, so making itself smarter will likely increase the number of paper-clips that will eventually be made. It does so. It considers how
…
it can make paper-clips using the 3D printer, estimating the number of possible paper-clips. It notes that if it could get more
…
raw materials it could make more paper-clips. It hence figures out a plan to manufacture devices that will make it much
…
smarter, prevent interference with its plan, and will turn all of Earth (and later the universe) into paper-clips. It does so.”25 Those who have seen the film The Sorcerer’s Apprentice will grasp the basic nature of the problem, examples of which
…
Sandberg said of his paper clip AI. “Because if I pull the plug, there will be fewer paper clips in the world and that’s bad.”28 You’ll be pleased to know that not everyone is worried. Steven Pinker ridicules fears of “digital apocalypse,” insisting that “like any other technology,” artificial intelligence is “tested before it
Wired for War: The Robotics Revolution and Conflict in the 21st Century
by P. W. Singer · 1 Jan 2010 · 797pp · 227,399 words
GETS STRONG Today, there are all sorts of artificial intelligence that appear in our daily lives, without our even thinking of them as AI. Anytime you check your voice mail, AI directs your calls. Anytime you try to write a letter in Microsoft Word, an annoying little paper-clip figure pops up, which is an AI
The Coming Wave: Technology, Power, and the Twenty-First Century's Greatest Dilemma
by Mustafa Suleyman · 4 Sep 2023 · 444pp · 117,770 words
AGI blindly optimizing for an opaque goal, oblivious to human concerns. The canonical thought experiment is that if you set up a sufficiently powerful AI to make paper clips but don’t specify the goal carefully enough, it may eventually turn the world and maybe even the contents of the entire cosmos into
…
paper clips. Start following chains of logic like this and myriad sequences of unnerving events unspool. AI safety researchers worry (correctly) that should something like an AGI
Enlightenment Now: The Case for Reason, Science, Humanism, and Progress
by Steven Pinker · 13 Feb 2018 · 1,034pp · 241,773 words
can lower the risk, until it is in the range of the other threats to our species’ immortality, like asteroids, supervolcanoes, or an Artificial Intelligence that turns us into paper clips. CHAPTER 20 THE FUTURE OF PROGRESS Since the Enlightenment unfolded in the late 18th century, life expectancy across the world has risen from
…
of current AI: Brooks 2015; Davis & Marcus 2015; Lanier 2014; Marcus 2016; Schank 2015. 28. Naam 2010. 29. Robots turning us into paper clips and other Value Alignment Problems: Bostrom 2016; Hanson & Yudkowsky 2008; Omohundro 2008; Yudkowsky 2008; P. Torres, “Fear Our New Robot Overlords: This Is Why You Need to Take Artificial Intelligence Seriously
…
,” Salon, May 14, 2016. 30. Why we won’t be turned into paper clips: B. Hibbard, “Reply to AI Risk,” http://www.ssec.wisc.edu/~billh/g/AIRisk_Reply.html; R. Loosemore, “The Maverick Nanny with
The Myth of Artificial Intelligence: Why Computers Can't Think the Way We Do
by Erik J. Larson · 5 Apr 2021
(its human-given objective), and by degrees converts everything in the universe into a paper clip factory, including all the usable elements in our own bodies. Eliezer Yudkowsky, former head of Berkeley’s Machine Intelligence Research Institute, once quipped, “The AI does not hate you, nor does it love you, but you are made
Coders: The Making of a New Tribe and the Remaking of the World
by Clive Thompson · 26 Mar 2019 · 499pp · 144,278 words
—yet still slaughter or enslave us all in happy pursuit of its goals. In one famous thought experiment, Bostrom imagined a superintelligent AI being tasked with making as many paper clips as possible. It might decide the best way to do this would be to disassemble all matter on earth—including humans—to
…
away: Bostrom, Superintelligence, 2243–87. “the observable universe into paper clips”: Bostrom, Superintelligence, 2908. “a faint ticking sound”: Raffi Khatchadourian, “The Doomsday Invention,” New Yorker, November 23, 2015, accessed August 21, 2018, https://www.newyorker.com/magazine/2015/11/23/doomsday-invention-artificial-intelligence-nick-bostrom. a machine that can truly reason: Kevin Hartnett
Prediction Machines: The Simple Economics of Artificial Intelligence
by Ajay Agrawal, Joshua Gans and Avi Goldfarb · 16 Apr 2018 · 345pp · 75,660 words
laws of physics. Acquiring resources is costly. Bostrom talks of a paper-clip-obsessed superintelligence that cares about nothing but making more paper clips. The paper-clip AI could just wipe out everything else through single-mindedness. This is a powerful idea, but it overlooks competition for resources. Something economists respect is that
The Thinking Machine: Jensen Huang, Nvidia, and the World's Most Coveted Microchip
by Stephen Witt · 8 Apr 2025 · 260pp · 82,629 words
was an extension of ideas he’d been considering for years. He had previously advanced the “paper-clip maximizer” thought experiment: Suppose we have an AI whose only goal is to make as many paper clips as possible. The AI will realize quickly that it would be much better if there were no humans because humans might
…
decide to switch it off. Because if humans do so, there would be fewer paper clips. Also, human bodies contain a lot of atoms that could be made into paper clips
…
. The future that the AI would be trying to gear toward would be one in which there were a lot of paper clips but no humans. The paper-clip maximizer argument had long circulated online, and it gained traction among the “rationality
…
” community, as well as with many tech executives. A few months after the publication of Bostrom’s book, Elon Musk posted a comment to the futurology website Edge.org: The pace of progress in artificial intelligence (I’m
Empire of AI: Dreams and Nightmares in Sam Altman's OpenAI
by Karen Hao · 19 May 2025 · 660pp · 179,531 words
would be difficult to control and could cause an existential catastrophe. Given a simple objective like producing paper clips, this superior AI could determine that humans pose a threat to its paper clip–producing objective because they take up paper clip–producing resources. Bostrom then proposed a solution: It could be possible to avert the superintelligence control
Co-Intelligence: Living and Working With AI
by Ethan Mollick · 2 Apr 2024 · 189pp · 58,076 words
hypothetical AI system in a paper clip factory that has been given the simple goal of producing as many paper clips as possible. By some process, this particular AI
…
or if it is possible. But let us assume that our paper clip AI—let’s call it Clippy—reaches this level of intelligence. Clippy still has the same goal: to make paper clips. So it turns its intelligence to thinking about how to make more paper clips and how to avoid being shut down (which would have a
…
unknown future after which “human affairs, as we know them, could not continue.” In an AI singularity, hyperintelligent AIs appear, with unexpected motives. But we know Clippy’s motive. It wants to make paper clips. Knowing that the core of the Earth is 80 percent iron, it builds amazing machines capable of strip
…
, may stop the production of future paper clips. And it only cares about paper clips. The paper clip AI is one of a large set of apocalyptic scenarios of AI doom that have deeply concerned many people in the AI community. Many of these concerns revolve around an ASI. The smarter-than-a-person machine
Our Final Invention: Artificial Intelligence and the End of the Human Era
by James Barrat · 30 Sep 2013 · 294pp · 81,292 words
which we’d object—by killing us after our next meal. As an example of unintended consequences, Oxford University ethicist Nick Bostrom suggests the hypothetical “paper clip maximizer.” In Bostrom’s scenario, a thoughtlessly programmed superintelligence whose programmed goal is to manufacture paper clips does exactly as it is told without regard to
…
because it sets about “transforming first all of earth and then increasing portions of space into paper clip manufacturing facilities.” Friendly AI would make only as many paper clips as was compatible with human values. Another tenet of Friendly AI is to avoid dogmatic values. What we consider to be good changes with time, and any
…
Otellini, Paul Page, Larry paper clip maximizer scenario parallel processing pattern recognition Pendleton, Leslie Perceptron Perrow, Charles Piaget, Jean power grid Precautionary Principle programming bad evolutionary genetic ordinary self-improving, see self-improvement Rackspace rational agent theory of economics recombinant DNA Reflections on Artificial Intelligence (Whitby) resource acquisition risks of artificial intelligence apoptotic systems and Asilomar Guidelines
Ways of Being: Beyond Human Intelligence
by James Bridle · 6 Apr 2022 · 502pp · 132,062 words
and philosophers of machine intelligence itself. One of most dramatic of these possible futures is described in something called the paperclip hypothesis. It goes like this. Imagine a piece of intelligent software – an AI – designed to optimize the manufacture of paperclips, an apparently simple and harmless business goal. The software might begin with
…
the outer planets.9 It’s a terrifying and seemingly ridiculous chain of events – but only ridiculous in so far as an advanced Artificial Intelligence has no need for paperclips. Driven by the logic of contemporary capitalism and the energy requirements of computation itself, the deepest need of an AI in the present
…
are only a thought experiment for now, but they are an exercise in the same vein as the story of the runaway paperclip factory, in which a poorly constrained AI overruns the planet: a shadow, cast by the spectre of artificial super-intelligence, of what awaits us in the future. We have
…
-oil-and-gas-we-actually-consume-your-products.html. 9. For an elaboration of the paperclip hypothesis, see Nick Bostrom, ‘Ethical Issues in Advanced Artificial Intelligence’, 2003; https://www.nickbostrom.com/ethics/ai.html. 10. Samuel Gibbs, ‘Elon Musk: Regulate AI to Combat “Existential Threat” Before It’s Too Late’, The Guardian, 17 July 2017
Architects of Intelligence
by Martin Ford · 16 Nov 2018 · 586pp · 186,548 words
random, unforeseen and unwanted objective cropping up there? MARTIN FORD: You have a famous example of a system that manufactures paperclips. The idea is that when a system is conceived and given an objective, it pursues that goal with a superintelligent competence, but it does it
…
paperclips in the world, which might then involve taking control away from humans and indeed turning the whole
…
planet into paperclips or into space probes that can go out and transform the universe into more paperclips. The point here is that you could substitute
…
almost any other goal you want for paperclips and if you think through what it would mean for that goal to be truly maximized in this world, that unless you
…
AI holds as it becomes more embedded in the grid and more hackable. I’m not that worried about AI systems independently wanting to eat us for breakfast or turn us into paper clips. It’s not completely impossible, but there’s no real evidence that we’re moving in that direction. There
…
problems I would put higher on the list of concerns than the notion that an AI would suddenly come up with its own desires and goals, and/or plan to sacrifice the human race to make more paper clips. MARTIN FORD: What do you think about the regulation of AI, is there a
More Everything Forever: AI Overlords, Space Empires, and Silicon Valley's Crusade to Control the Fate of Humanity
by Adam Becker · 14 Jun 2025 · 381pp · 119,533 words
to demonstrate how this could play out in a world where there is a superintelligent AI but no clear solution to the alignment problem.12 “Say one day we create a super intelligence and we ask it to make as many paperclips as possible,” he said. “Maybe we built it to run our
…
regardless, there’s a problem. “If we were to think through what it would actually mean to configure the universe in a way that maximizes the number of paperclips that exist, you realize that such an AI would have incentives, instrumental reasons, to harm humans.”13 In such a scenario, shortly after the
…
paperclip AI sets about finding ways to make more paperclips, it realizes that being more intelligent would make the job easier, allowing it to reason more quickly and develop more inventive
…
soon it’s invented a new method for quickly turning rocks into paperclips, along with a related method for building computer
…
of paperclips while increasing its own intelligence even further to ensure the success of the
…
paperclips and computer chips. They attempt to stop their creation, but it’s too late. The AI is already far more intelligent than any human and can effectively predict all human behavior. Realizing that
…
the humans will try to shut it down—which it would see as its own death—the paperclip AI decides the best way to ensure it can create the maximal number of paperclips is to destroy humanity, so we can’t interfere. As Bostrom
…
points out, “Human bodies consist of a lot of atoms and they can be used to build more paperclips.”15 The AI outwits the humans trying to stop it—talking them out of their plans with superintelligently devastating logic, turning them against each other, or just
…
loose on the Earth and its inhabitants. Less than thirty-six hours after the AI was first turned on, every human on Earth is dead, as is all other animal and plant life, all disassembled to form paperclips and computer chips by the AI in its continuing mission. In fact, the Earth
…
entire accessible universe is transformed into paperclips. Avoiding this kind of outcome, says Bostrom, is harder than it seems. “If you plug into a super-intelligent machine with almost any goal you
…
a higher or more noble purpose than wanting to make more paperclips. The broomstick enchanted by Mickey Mouse didn’t develop new objectives once
…
others at MIRI and CFAR feared the world would end in a haze of paperclips and microchips. (MIRI is so confident that paradise
On the Edge: The Art of Risking Everything
by Nate Silver · 12 Aug 2024 · 848pp · 227,015 words
proves relatively easy to achieve. Whereas if the machines turn us all into paper clips—one of Nick Bostrom’s famous thought experiments involves an unaligned AI with the goal of manufacturing as many paper clips as possible—he won’t be around to take credit. (As someone who has as much experience as pretty
…
.html. GO TO NOTE REFERENCE IN TEXT close to 100 percent: Gary Marcus, “p(doom),” Marcus on AI (blog), August 27, 2023, garymarcus.substack.com/p/d28. GO TO NOTE REFERENCE IN TEXT many paper clips: Joshua Gans, “AI and the Paperclip Problem,” Centre for Economic Policy Research, June 10, 2018, cepr.org/voxeu/columns
New Laws of Robotics: Defending Human Expertise in the Age of AI
by Frank Pasquale · 14 May 2020 · 1,172pp · 114,305 words
actual humans. That is the core problem in one of “existential risk studies” classic narratives of out-of-control AI—an unstoppable paper-clip maximizer that starts to use all available material on earth to generate more paperclips.49 For many in the mainstream of ethical AI, the solution to a potentially out-of-control
The Means of Prediction: How AI Really Works (And Who Benefits)
by Maximilian Kasy · 15 Jan 2025 · 209pp · 63,332 words
the philosopher Nick Bostrom, has become popular in the tech industry. In the updated version of the story, someone creates an AI with the goal of producing as many paper clips as possible. The AI is very good at its job. The problem is that at some point the human creator of the
…
AI has enough paper clips, but the machine continues to produce more. The human creator would like to turn off the machine. Such human intervention
…
stands in the way of the AI achieving its objective, which is to produce as many paper clips as possible. To achieve its objective, the AI thus needs to incapacitate its human creator to prevent them from switching
…
tend to do, then these measures would reduce worker welfare. Using algorithms to maximize job-finding probabilities might quickly lead to a version of the paper-clip-maximizing AI, discussed in chapter 14, where maximization of a seemingly sensible goal (production of paper clips, or jobs for the unemployed) can lead to disastrous consequences
Nexus: A Brief History of Information Networks From the Stone Age to AI
by Yuval Noah Harari · 9 Sep 2024 · 566pp · 169,013 words
human gamer, it would have realized that the loophole it found in the game’s rules probably doesn’t really count as “winning.” If the paper-clip AI had been a human bureaucrat, it would have realized that destroying humanity in order to produce paper clips is probably not what was intended. But
The Seventh Sense: Power, Fortune, and Survival in the Age of Networks
by Joshua Cooper Ramo · 16 May 2016 · 326pp · 103,170 words
figure it out! might be all its human instructors tell it. As the clip-making AI becomes better and better at its task, it demands more and still more resources: more electricity, steel, manufacturing, shipping. The paper clips pile up. The machine looks around: If only I could control the power supply, it
…
—of course no one is going to forget to tell a machine to stop making paper clips—but he is staring at something that is perhaps not so unlikely. If humans can lose their minds, so can AIs in a sense. “We cannot blithely assume that a superintelligence will necessarily share any of
Possible Minds: Twenty-Five Ways of Looking at AI
by John Brockman · 19 Feb 2019 · 339pp · 94,769 words
approach to the AI issue of “value alignment”—the study of how, exactly, we can keep the latest of our serial models of AI from turning the planet into paper clips—is human centered; i.e., that of a cognitive scientist, which is what he is. The key to machine learning, he believes
Power and Progress: Our Thousand-Year Struggle Over Technology and Prosperity
by Daron Acemoglu and Simon Johnson · 15 May 2023 · 619pp · 177,548 words
the world, with significant potential downsides for most workers. The same is true for AI-based surveillance: most citizens, wherever they are around the world, are finding it harder and harder to escape repression. Social Media and Paper Clips Internet censorship and even high-tech spyware may say nothing about the potential of
…
uses its considerable capabilities to excel in meeting this objective by coming up with new methods to transform the entire world into paper clips. When it comes to the effects of AI on politics, it may be turning our institutions into paper clips, not thanks to its superior capabilities but because of its
What to Think About Machines That Think: Today's Leading Thinkers on the Age of Machine Intelligence
by John Brockman · 5 Oct 2015 · 481pp · 125,946 words
of paper clips, it may invent incredible technologies as it sets about converting all available mass in the reachable universe into paper clips, but its decisions are still just plain dumb. AI has followed operations research, statistics, and even economics in treating the utility function as exogenously specified. We say, “The decisions are
…
should forever be the slave of carbon-based intelligence. But if we want to end up with a diverse cosmopolitan civilization instead of, for example, paper clips, we may need to ensure that the first sufficiently advanced AI is built with a utility function whose maximum pinpoints that outcome. If we want an
Supremacy: AI, ChatGPT, and the Race That Will Change the World
by Parmy Olson · 284pp · 96,087 words
paper clips as possible, it might decide to convert all of Earth’s resources and even humans into paper clips as the most effective way to fulfill its objective. His anecdote spawned a saying in AI circles, that we need to avoid becoming “paper-clipped.” Musk went ahead and put some money into
…
to be research labs,” says a former scientist at OpenAI. “They’re going to be companies building products, because AI is not really about research anymore.” Nick Bostrom’s story about the paper clip, where an artificial superintelligence destroys civilization as it converts all the world’s resources into the tiny metal widgets
Artificial Intelligence: A Modern Approach
by Stuart Russell and Peter Norvig · 14 Jul 2019 · 2,466pp · 668,761 words
hijacks resources, and Bostrom (2014) describes the runaway paper clip factory that takes over the world. Yudkowsky (2008) goes into more detail about how to design a Friendly AI. Amodei et al. (2016) present five practical safety problems for AI systems. Omohundro (2008) describes the Basic AI Drives and concludes, “Social structures which cause individuals
Army of None: Autonomous Weapons and the Future of War
by Paul Scharre · 23 Apr 2018 · 590pp · 152,595 words
“any level of intelligence could in principle be combined with . . . any final goal.” This means a superintelligent AI could have any set of values, from playing the perfect game of chess to making more paper clips. On one level, the sheer alien-ness of advanced AI makes many of science fiction’s fears
Artificial Intelligence: A Guide for Thinking Humans
by Melanie Mitchell · 14 Oct 2019 · 350pp · 98,077 words
frustrating encounters with customer service speech-recognition systems, the robotic word-learning toy Furby, or Microsoft’s annoying and ill-fated Clippy, the paper-clip virtual assistant. Full-blown AI didn’t seem imminent. Maybe this is why so many people were shocked and upset when, in 1997, IBM’s Deep Blue chess
The Rationalist's Guide to the Galaxy: Superintelligent AI and the Geeks Who Are Trying to Save Humanity's Future
by Tom Chivers · 12 Jun 2019 · 289pp · 92,714 words
became more involved with the Rationalists. I started reading their websites; I learned the jargon, all these technical and semi-technical terms like ‘updating’ and ‘paperclip maximiser’ and ‘Pascal’s mugging’ (I’ll explain what all those things are later). I read the things you’re supposed to read, especially and notably
…
about it.’ Several of the key concepts that do the rounds in the Rationalsphere these days first arose on SL4 and the Extropians. The aforementioned ‘paperclip maximiser’ was first mentioned there, possibly by Yudkowsky: ‘Someone searched [the Extropians’ archive] recently and found a plausible first mention by me,’ he told me by
…
next chapter we’ll look at a few of the reasons that the Rationalist/AI safety movement has pointed out. Chapter 8 Paperclips and Mickey Mouse The nightmare scenario is that we are all destroyed and turned into paperclips. This sounds like I’m joking, but I’m not, exactly. The classic example
…
of an AI that has gone terribly wrong – a ‘misaligned’ or ‘unfriendly’ AI, in Rationalist terms – is a thought experiment that Nick Bostrom wrote about in 2003 (probably following an original idea by Eliezer Yudkowsky): the paperclip maximiser.1 Imagine a human-level AI has been given an apparently harmless instruction: to make
…
paperclips. What might it do? Well, it might start out by simply making paperclips. It could build a small pressing machine and churn out a few
…
dozen paperclips a minute. But it’s bright enough to know that it could
…
be more efficient than that, and if it wants to maximise the number of paperclips it can make, it’s probably better not to go straight for a small press. It could instead use its materials to build a larger
…
factory, so that it’s making thousands of paperclips a minute. Still, though, if it really wants to make as many paperclips as possible, it might want to improve its ability to think about how to do so, so it might
…
new processors, improving its own code, upgrading its RAM and so on. You can see where this is going, presumably. The end point of the paperclip maximiser is a solar system in which every single atom has been turned into either paperclips, paperclip-manufacturing machines, computers that think about how best to
…
of light with instructions to set up a franchise there. This isn’t what you meant, back when you said, ‘Go and make paperclips’ to your apparently docile AI, but it’s what you said. This has, to some extent, entered the public consciousness, mainly through the medium of an extraordinarily viral
…
online clicker game that was played by tens of millions of people in 2017, Universal Paperclips.2 In it, you are an AI whose job is to make paperclips. You start out by repeatedly clicking the ‘make paperclip’ button, but the process becomes more automated
…
on in the course of the game. It’s actually a really good insight into the concepts behind AI alignment, because as the player you are incentivised solely to care about your ‘number of paperclips’ score. There are other things to care about – how much the humans (while they still exist) trust
…
are willing to invest resources in you; your processing power; your manufacturing capabilities; your ability to defend yourself against anything that might stop you making paperclips, etc. – but they’re all secondary goals, incidental to your main one. If you can run up your
…
paperclip score without doing them, you will, and so, goes the theory, would a real AI. I would recommend that you go and play Universal Paperclips immediately, but I won’t, because it is punishingly addictive and you won
…
production while you check your emails or whatever.) The point of the paperclip maximiser is not that we are, really, going to be destroyed and turned into paperclips. Bostrom’s idea was to use something self-evidently silly to illustrate that AIs will not necessarily care about what we care about – they will only
…
optimisation power against each other.’ That is, just as Mickey’s broom ended up interpreting a simple task as open-ended, a GAN might dedicate, paperclip-maximiser-style, all the resources of the solar system into both creating and undermining the things it’s supposed to produce. That’s a GAN-specific
…
you could repurpose all the atoms in the solar system for paperclips. But: ‘If I tell you to go and make paperclips, and if you turn the planet
…
into paperclips, killing everyone, I would say, “That wasn’t very smart, was it?”’ The argument that Yudkowsky, Bostrom
…
to do. It would be amazingly obvious to it that no sane human programmer would want it to destroy all humans and turn them into paperclips, or fill a cauldron until the house was flooded. And that’s actually very likely. By definition, or almost by definition, human-level
…
that a lot of people knew about the Rationalists – all the stuff they actually take seriously but which is just as weird, such as the paperclip maximiser, was somewhat obscured by it. Chapter 11 If I stop caring about chess, that won’t help me win any chess games, now will it
…
spoke to think they would), then that makes it less likely that a future AI will reach superintelligence and think, ‘These goals are pretty silly; maybe I should do something else,’ and thus not turn us all into paperclips. I asked Paul about that, while eating an intimidatingly large omelette in a
…
systems and setting up franchises there, to turn the planets and asteroids there into new computer banks and paperclips (or whatever). This has obvious implications for humanity. Yudkowsky has a much-quoted saying: ‘The AI does not hate you, nor does it love you, but you are made of atoms which it
…
lengthy conversation about this with a good friend who works in IT, and he was saying essentially that. How will an AI turn all the matter in the universe into paperclips if it can’t pick anything up? But ordinary, non-superintelligent humans can and do have enormous power without making much
…
in your philosophy Toby Walsh, the University of New South Wales professor of AI whom I spoke to, is extremely worried about AI safety. But he thinks that the paperclip maximiser scenario is the wrong thing to be concerned about. ‘The scenario that I wrestle with,’ he told me, ‘is 3D-printed drones
…
-brain in the unspecified-time-from-now future might decide to behave. But there are, right now, some little hints that AIs are going wrong in ways that are quite paperclip-maximiserish, albeit on a much smaller scale. A great paper was released on ArXiv1 in March 2018, about digital evolution – machine
…
might still be in a position to design a reward function that you know isn’t going to have unintended consequences and perverse instantiations – the paperclip maximiser, these side effects that involve existential risk. But I’m involved in the sharp end of building these things, and making them increasingly powerful, and
…
description of the method for creating a safe AI. Remember, way back when we were discussing why AI is dangerous, we talked about how seemingly innocuous goals can have weird outcomes, ‘perverse instantiations’: the paperclip maximiser was one example, the Fantasia broom another. Is anyone doing anything more specific to try to reduce
…
of the solar system has died’, BuzzFeed, September 2017 https://www.buzzfeed.com/tomchivers/cassini-death-spiral 8: Paperclips and Mickey Mouse 1. Nick Bostrom, ‘Ethical issues in advanced artificial intelligence’, 2003 https://nickbostrom.com/ethics/ai.html 2. http://www.decisionproblem.com/paperclips/index2.html 3. Soares, ‘Ensuring smarter-than-human intelligence has
Growth: A Reckoning
by Daniel Susskind · 16 Apr 2024 · 358pp · 109,930 words
for helping us figure out how we should respond. The Alignment Problem One of the troubling thought experiments in the field of artificial intelligence (AI) is the tale of the ‘paperclip maximizer’.5 Imagine that, at some point in the future, AI researchers actually succeed in their work: they manage to build an
…
set it a simple goal: maximize the manufacture of paperclips. The thought experiment asks: what would the AI
…
a reliable stream of perfectly crafted paperclips. This would be a good outcome. Yet unfortunately that is unlikely to be the
…
end of the story. This is because the AI has not been tasked with manufacturing many paperclips but the maximum number. And so the AI would go on. It would relentlessly build
…
more factories and produce more paperclips, using ever more resources to do so. As time passed, it would
…
. At this point a so-called ‘intelligence explosion’ would take place, the AI endlessly speeding up its operation in an ever-accelerating blast of self-improvement.6 And all this would be done with the goal of maximizing the production of paperclips. Soon, says Nick Bostrom, the creator of this story, we would
…
paperclips’.7 This unusual thought experiment is a cautionary tale: using a highly capable system to incessantly pursue a supposedly innocuous end can have unexpected
…
this book. For here too we have a highly capable system at our disposal: not an AI, but the market economy. Here too, we have set it a simple goal: to maximize not the number of paperclips, but the level of GDP. And here too, our system has done an extraordinarily effective job
…
of achieving the narrow goal we have set it: just as the AI turns ever more of the world into paperclips, we have turned ever more
…
else’ – in this story, paperclips. When it comes to the economy, the threat is less visible but equally consequential. One way to think about our
…
), 200–2 Amazon, patents, 182 Anfinsen, Christian, 201 anti-globalization movements, 119 Apollo space mission, 189–90 Arrhenius, Svante, 100 artificial intelligence (AI): applications, 108–9; and medicine, 200; ‘paperclip maximiser’ thought experiment, 124–5; social narratives around, 244–5 Athenaeus, 178 Atkinson, Anthony, 104 Attenborough, David, 150 Attlee, Clement, 263 Australia, patents
Thinking Machines: The Inside Story of Artificial Intelligence and Our Race to Build the Future
by Luke Dormehl · 10 Aug 2016 · 252pp · 74,167 words
says. ‘If you wanted to get something done, go to a do engine.’ From Knowledge Navigators to Animated Paperclips Although Siri was the first time most people had seen an actually working AI assistant in action, the technology had been in development for a number of years. In the second half of
…
systems in charge of making decisions they do not necessarily have the intelligence to make. A favourite thought experiment of those who believe advanced AI could mean the demise of the human race is the so-called ‘paperclip maximiser’ scenario. In the scenario, proposed by Swedish philosopher and computational neuroscientist Nick Bostrom, an
…
AI is given the seemingly harmless goal of running a factory producing paperclips. Issued with the task of maximising the efficiency for producing paperclips, the AI, able to utilise nano technology to reconstruct matter on a molecular level, disastrously proceeds to turn
…
first the Earth and then a large portion of the observable universe into paperclips. The ‘paperclip maximiser’ scenario is a common one
…
, although it seems to me more a question of artificial stupidity than Artificial Intelligence. The
…
inability to answer questions like ‘Why are you making paperclips when there is no paper left?’ or ‘Why are you making paperclips when the person who requested the
…
paperclips in the first place has, himself, been turned
…
into more paperclips?’ doesn’t speak of an advanced
…
superintelligence, unless there is something dramatically important about the nature of paperclips that I am missing
…
Communications 109 offices, smart 90 OpenWorm 210 ‘Optical Scanning and Information Retrieval’ 7–8, 10 paedophile detection 99–102 Page, Larry 6–7, 34, 220 ‘paperclip maximiser’ scenario 235 Papert, Seymour 27, 44, 45–6, 49 Paro (therapeutic robot) 130–1 patents 188–9 Perceiving and Recognising Automation (PARA) 43 perceptrons 43
Being You: A New Science of Consciousness
by Anil Seth · 29 Aug 2021 · 418pp · 102,597 words
programming foresight which leads to the Earth’s entire resources being transformed into a vast mound of paperclips? Running beneath many of these worries, especially the more existential and apocalyptic, is the assumption that AI will – at some point in its accelerating development – become conscious. This is the myth of the golem
…
Golem, Inc., the polymathic pioneer Norbert Wiener treated golems as central to his speculations about risks of future AI. vast mound of paperclips: In the parable of the paperclip maximiser, an AI is designed to make as many paperclips as possible. Because this AI lacks human values but is otherwise very smart, it destroys the world
Superintelligence: Paths, Dangers, Strategies
by Nick Bostrom · 3 Jun 2014 · 574pp · 164,509 words
intrinsically about any of those things. There is nothing paradoxical about an AI whose sole final goal is to count the grains of sand on Boracay, or to calculate the decimal expansion of pi, or to maximize the total number of paperclips that will exist in its future light cone. In fact, it
…
for computation)—including the atoms in the bodies of whomever once cared about the answer.8 • Paperclip AI. An AI, designed to manage production in a factory, is given the final goal of maximizing the manufacture of paperclips, and proceeds by converting first the Earth and then increasingly large chunks of the observable universe
…
into paperclips. In the first example, the proof or disproof of the Riemann hypothesis that the AI produces is the intended outcome and is in
…
itself harmless; the harm comes from the hardware and infrastructure created to achieve this result. In the second example, some of the paperclips produced would be part of the intended outcome; the harm
…
would come either from the factories created to produce the paperclips (infrastructure profusion) or from the excess of paperclips (perverse instantiation). One might think that the risk of a malignant
…
manufacture as many paperclips as possible. It is easy to see how this gives the
…
always be turned into more paperclips. But suppose that the goal is instead to make at least
…
one million paperclips (meeting suitable design specifications) rather than to make as many
…
it to make a million paperclips, and then halt. Yet this may not be what would happen. Unless the AI’s motivation system is
…
yet achieved its goal—this, after all, being an empirical hypothesis against which the AI can have only uncertain perceptual evidence. The AI should therefore continue to make paperclips in order to reduce the (perhaps astronomically small) probability that it has somehow still failed to make at least a million of them
…
problem here in need of remedying?) Namely, if we want the AI to make some paperclips for us, then instead of giving it the final goal of making as many paperclips as possible, or to make at least some number of paperclips, we should give it the final goal of making some specific number
…
of paperclips—for example, exactly one million paperclips—so that going beyond this number would be counterproductive for the
…
additional paperclips once it had reached one million, since that would prevent the realization of its final goal. But there are other actions the superintelligent
…
AI could take that would increase the probability of its goal being achieved. It could, for instance, count the paperclips it has made, to reduce the risk that it has made too few. After it has
…
counted them, it could count them again. It could inspect each one, over and over, to reduce the risk that any of the paperclips fail to meet the design specifications. It could build an unlimited amount of computronium in an effort to clarify its thinking, in the hope of
…
it has overlooked some obscure way in which it might have somehow failed to achieve its goal. Since the AI may always assign a nonzero probability to having merely hallucinated making the million paperclips, or to having false memories, it would quite possibly always assign a higher expected utility to continued action
…
have a satisficing character. For example, instead of giving the AI the final goal of making as many paperclips as possible, or of making exactly one million paperclips, we might give the AI the goal of making between 999,000 and 1,001,000 paperclips. The utility function defined by the final goal would be
…
counting and recounting the paperclips) is greater than the expected utility of halting. Thus, a malignant infrastructure profusion can result. Another way of developing the satisficing idea is by modifying
…
it found a plan that it judged gave a probability of success exceeding a certain threshold, say 95%. Hopefully, the AI could achieve a 95% probability of having manufactured one million paperclips without needing to turn the entire galaxy into infrastructure in the process. But this way of implementing the satisficing idea
…
95% chance of having manufactured a million paperclips, such as by building a single paperclip factory. Suppose that the first solution that pops into the AI’s mind
…
thought of this solution, and having correctly judged that it meets the satisficing criterion of giving at least 95% probability to successfully manufacturing one million paperclips, the AI would then have no reason to continue to search for alternative ways of achieving the goal. Infrastructure profusion would result, just as before. Perhaps
…
has a “resource-insatiable” final goal, such as the goal of maximizing a utility function that is linear in the number of paperclips caused by the AI to exist—twice as many paperclips, twice as good. Such an AI might care less about the simulation hypothesis, on grounds that its ability to influence how many
…
paperclips are produced looks much more limited if the AI is in a
…
pre-produced cryptographic reward tokens, or the goal of causing the existence of forty-five virtual paperclips. Such an AI should not discount those possible worlds in which it inhabits a simulation. A substantial portion of the AI’s total expected utility
…
with paperclips, then the AI should act to prevent those outcomes.11 “Where our wishes cohere rather than interfere” may be read as follows. The AI should
Rationality: From AI to Zombies
by Eliezer Yudkowsky · 11 Mar 2015 · 1,737pp · 491,616 words
to do that, you certainly don’t know how to create an AI that will conditionally respond to an environment of loving parents by growing up into a kindly superintelligence. If you have something that just maximizes the number of paperclips in its future light cone, and you raise it with loving parents
…
to infuse the AI with the action-determining principle, “Do what I want.” And suppose you get the AI design close enough—it doesn’t just end up tiling the universe with paperclips, cheesecake or tiny molecular copies of satisfied programmers—that its utility function actually assigns utilities as follows, to the
…
that can only be cured by substance S. However, substance S can only be produced by working with a paperclip maximizer from another dimension—substance S can also be used to produce paperclips. The paperclip maximizer only cares about the number of paperclips in its own universe, not in ours, so we can’t
…
offer to produce or threaten to destroy paperclips here. We have never interacted with the paperclip maximizer before, and will never interact with it again
…
matrix to produce a sense of indignation at the thought that the paperclip maximizer wants to trade off billions of human lives against a couple of paperclips. Clearly the paperclip maximizer should just let us have all of substance S. But a paperclip maximizer doesn’t do what it should; it just maximizes paperclips. In this
…
to cooperate, in a case like this. It doesn’t even seem fair—so great a sacrifice by us, for so little gain by the paperclip maximizer? And let us specify that the paperclip-agent experiences no pain or pleasure—it just outputs actions that steer its universe to contain more paperclips
…
all the rest of the logic—everything about what happens if both agents think that way, and both agents defect—is the same. For the paperclip maximizer cares as little about human deaths, or human pain, or a human sense of betrayal, as we care about paperclips. Yet we both prefer (C
…
a little bigger. You don’t need to become your shoes to understand your shoes. And sympathy? Well, suppose we’re dealing with an expected paperclip maximizer, but one that isn’t yet powerful enough to have things all its own way—it has to deal with humans to get its paperclips
…
’t feel happy when it makes paperclips; it just chooses whichever action leads to the greatest number of expected paperclips. Though a paperclip maximizer might find it convenient to display a smile when it made paperclips—so as to help manipulate any humans that had designated it a friend.
…
the Future to be something greater than the past—of being not bound to our past selves—of trying to change and move forward. A paperclip maximizer just chooses whichever action leads to the greatest number of paperclips. No free lunch. You want a wonderful and mysterious universe? That’s your value
…
free lunch. Valuable things appear because a goal system that values them takes action to create them. Paperclips don’t materialize from nowhere for a paperclip maximizer. And a wonderfully alien and mysterious Future will not materialize from nowhere for us humans, if our values that prefer it are physically obliterated—or
…
warn aspiring rationalists to beware of cleverness. I’ll also note that I wouldn’t want an attempted Friendly AI that had just decided that the Earth ought to be transformed into paperclips, to assess whether this was a reasonable thing to do in light of all the various warnings it had
…
its future light cone into paperclips. Now, though, I could see it—the pulse of the optimization process, sensory information surging in, motor instructions surging out, steering the future. In the
Deep Utopia: Life and Meaning in a Solved World
by Nick Bostrom · 26 Mar 2024 · 547pp · 173,909 words
desires that there exist as many paperclips as possible in its future light cone (such as Clippy, the instrumentally rational artificial agent that is used as an illustration in some thought experiments about AI safety). The other three classes listed in the Zeal category are similar to
Rule of the Robots: How Artificial Intelligence Will Transform Everything
by Martin Ford · 13 Sep 2021 · 288pp · 86,995 words
means that have unintended or unanticipated consequences that could turn out to be detrimental or even fatal to our civilization. A thought experiment involving a “paperclip maximizer” is often used to illustrate this point. Imagine a superintelligence designed with the specific objective of optimizing paperclip production. As it relentlessly pursued this goal
Practical Doomsday: A User's Guide to the End of the World
by Michal Zalewski · 11 Jan 2022 · 337pp · 96,666 words
’s done converting the entire planet and its many inhabitants into paperclips. The point of the tale is simple: the AI doesn’t
Human Compatible: Artificial Intelligence and the Problem of Control
by Stuart Russell · 7 Oct 2019 · 416pp · 112,268 words
’s value for paperclips lies between 25¢ and 75¢, and Harriet’s true value is 12¢, then Robbie will eventually become certain that she values paperclips at 25¢.16 As he approaches certainty about Harriet’s preferences, Robbie will resemble more and more the bad old
The Big Nine: How the Tech Titans and Their Thinking Machines Could Warp Humanity
by Amy Webb · 5 Mar 2019 · 340pp · 97,723 words
ASI using a parable about paperclips. If we asked a superintelligent AI to make paperclips, what would happen next? The outcomes of every AI, including those we have now, are determined by values and goals. It’s possible that an ASI
…
always stay collated in order. It’s possible that if we aren’t capable of explaining how many paperclips we actually want, an ASI could go on making
…
paperclips forever, filling our homes and offices with them as well
…
as our hospitals and schools, rivers and lakes, sewage systems, and on and on until mountains of paperclips covered the planet. Or an ASI using
…
efficiency as its guiding value could decide that humans were getting in the way of paperclips, so it would terraform Earth into a paperclip-making factory, making our kind go extinct in the process.16 Here’s what has so many AI experts, myself included, worried
…
follow soon or much longer after, between the 2040s and 2060s. It doesn’t mean that by 2070 superintelligent AIs will have crushed all life on Earth under the weight of quintillions of paperclips. But it doesn’t mean they won’t have either. The Stories We Must Tell Ourselves Planning for
Surviving AI: The Promise and Peril of Artificial Intelligence
by Calum Chace · 28 Jul 2015 · 144pp · 43,356 words
the implications of an attempt to do great good “perverse instantiation”. Others might call it the law of unintended consequences, or Sod’s Law. The paperclip maximiser If somebody running a paperclip factory turns out to be the first person to create an AGI and it rapidly becomes a superintelligence, they are
Escape From Model Land: How Mathematical Models Can Lead Us Astray and What We Can Do About It
by Erica Thompson · 6 Dec 2022 · 250pp · 79,360 words
model that states that financial return should be maximised. There is a classic parable illustrating the dangers of AI, which offers a scenario in which someone creates an intelligent paperclip machine with the aim of producing as many paperclips as possible, as efficiently as possible. The machine performs incredibly well, producing vast numbers of
…
paperclips from the raw materials at hand; it then co-opts other resources into the production of paperclips and eventually takes over the
…
world, subsuming everything and everyone into the pursuit of more and more paperclips at the expense of everything else on Earth. Clearly ridiculous, right? This should be the point where a
…
makes us human. In the meantime, our capitalist paperclip machine is destroying our life-support systems at least as effectively as if it were producing paperclips in the manner described above. It generates plastic waste, carbon pollution, denuded soils, hollowed communities, divided societies – while the number of zeroes on the ends
Operation Paperclip: The Secret Intelligence Program That Brought Nazi Scientists to America
by Annie Jacobsen · 11 Feb 2014 · 612pp · 181,985 words
AI.13 was given even higher priority and a new code name, Project AI.13-2.1. In Germany, Nazi scientists with knowledge of tabun and sarin were now being even more aggressively sought for recruitment into Operation Paperclip. Declassified documents reveal that the Chemical Corps wanted to employ Otto Ambros, but
The Doomsday Calculation: How an Equation That Predicts the Future Is Transforming Everything We Know About Life and the Universe
by William Poundstone · 3 Jun 2019 · 283pp · 81,376 words
in having the constitution in the first place. Yet even that analogy fails, for human nature doesn’t change much. AI would be amending itself, creating brave new reference classes. Paperclips of Doom Bostrom and his global counterparts are not luddites. They seek to encourage the development of safe AI. Not all
…
striking an imaginatively dystopian note. One such scenario is “paperclips of doom.” Suppose that super-intelligence is realized. In order to test it
…
, its human designers assign it a simple task: make paperclips. A 3-D printer networked to the AI begins printing out something. It’s not a paperclip… it’s a robot.… Before anyone can
…
scrap metal and transform it into paperclips. The robot is also self-reproducing, able to make countless copies of itself. Growing packs of
…
core’s rich store of molten iron. Eventually much of the planet’s mass is converted into paperclips. Other robots launch themselves to the moon and Mars, repeating the process there. Over time the AI devises ways to reengineer the sun’s fusion to produce the materials it needs to produce yet
…
of knowing that their predicament is due to Homo sapiens, a long-dead species that meant no harm. “Paperclips of doom” is a parable, not a prediction. Its moral is that super-AI might “mistakenly elevate a subgoal to the status of a supergoal.” HAL started killing humans because they wanted to
…
ourselves. That is the truly terrifying thing. In a paperclips-of-doom scenario, the AI might well understand that its human creators emphatically did not intend it to transmute the entire universe into paperclips. But it might be “compartmentalized,” like a psychopath. If maximizing paperclips is the goal and not destroying the world is the
The Optimist: Sam Altman, OpenAI, and the Race to Invent the Future
by Keach Hagey · 19 May 2025 · 439pp · 125,379 words
turn all matter in the universe into paperclips, a fable cribbed and bastardized from Yudkowsky—were fundamental to OpenAI’s initial
…
the paper, he also used a memorable metaphor for how AI could go wrong: if your AI is programmed to produce paperclips, if you’re not careful, it might end up filling the solar system with paperclips. Years later, Bostrom would take this example and hold it up as the ultimate symbol of
…
to work making sure that it does not destroy all of humanity. To illustrate how AI might take over, he borrows Yudkowsky’s metaphor of the paperclip, though gives it a twist. A superintelligent AI programmed to make paperclips might just keep going until all matter in the universe—including the fleshly bodies of
…
all sentient beings—is turned into paperclips. “This is quite possibly the most important and most daunting
These Strange New Minds: How AI Learned to Talk and What It Means
by Christopher Summerfield · 11 Mar 2025 · 412pp · 122,298 words
make them more powerful and more dangerous. In a notorious thought experiment, the philosopher Nick Bostrom imagines a powerful AI system that is programmed to perform a mundane task, like making paperclips. With limitless intelligence and a laser focus on the task, he imagines the AI diverting all human resources and eventually
…
on the internet as mischief-making viruses, or finding undisclosed ways to annihilate us in pursuit of a trivial goal (paperclips are mentioned with great regularity). These concerns might sound a bit lurid at first, but they are backed up by relentlessly logical thought experiments designed
…
as military AI, and ask whether LLMs might make it easier for terrorists or rogue states to wreak havoc.[*12] Finally, we’ll ask whether some of the more fantastical ideas about runaway intelligence are closer to fact or to fiction – and try to do so with minimal mention of paperclips. Skip Notes
Other Pandemic: How QAnon Contaminated the World
by James Ball · 19 Jul 2023 · 317pp · 87,048 words
modern conceptions of artificial intelligence as a threat to humanity imagine a badly programmed algorithm: a factory AI designed to maximise paperclip production, for example, might start breaking down other machinery for paperclips, hijacking mining operations, using the iron in blood for paperclips and killing anyone who seeks to disable it – because that would slow
The Road to Conscious Machines
by Michael Wooldridge · 2 Nov 2018 · 346pp · 97,890 words
he asked for, but not what he wanted. Bostrom also considered these types of scenarios. He imagined an AI system that controlled the manufacture of paperclips. The system is given the goal of maximizing the production of paperclips, which it takes literally, transforming first the earth and then the rest of the universe into
…
paperclips. Again, the problem is ultimately one of communication: in this case, when we communicate our goal, we need to
Robot Rules: Regulating Artificial Intelligence
by Jacob Turner · 29 Oct 2018 · 688pp · 147,571 words
risk, while not impossible, is improbable”.120 Fundamentally, optimists think humanity can and will overcome any challenges AI poses. The pessimists include Nick Bostrom, whose “paperclip machine” thought experiment imagines an AI system asked to make paperclips which decides to seize and consume all resources in existence, in its blind aderence to that goal
The Equality Machine: Harnessing Digital Technology for a Brighter, More Inclusive Future
by Orly Lobel · 17 Oct 2022 · 370pp · 112,809 words
Civilization,” Fortune, July 15, 2017, https://fortune.com/2017/07/15/elon-musk-artificial-intelligence-2/. 19. Joshua Gans, “AI and the Paperclip Problem,” VoxEU, June 10, 2018, https://voxeu.org/article/ai-and-paperclip-problem. 20. Pedro Domingos, The Master Algorithm: How the Quest for the Ultimate Learning Machine Will Remake Our World
I, Warbot: The Dawn of Artificially Intelligent Conflict
by Kenneth Payne · 16 Jun 2021 · 339pp · 92,785 words
that will resist being turned off, like the sinister HAL in Kubrick’s 2001. In Nick Bostrom’s philosophical thought experiment, an AI is tasked with simply counting paperclips—an innocuous task a long way distant from nuclear brinksmanship.17 The problem occurs when the machine goes back for a recount, just
Arguing With Zombies: Economics, Politics, and the Fight for a Better Future
by Paul Krugman · 28 Jan 2020 · 446pp · 117,660 words
behavior has, in effect, been parasitized by marketing algorithms. I know I’m not the only one thinking along these lines. Charlie Stross argues that “paperclip maximizers”—not people, but social systems and algorithms that try to maximize profits, market share, or whatever—have increasingly been directing the direction of society, in
The Second Intelligent Species: How Humans Will Become as Irrelevant as Cockroaches
by Marshall Brain · 6 Apr 2015 · 215pp · 56,215 words
-intelligent robots, instead of quiescence, chose the path of infinite self replication with the goal of turning the entire universe into robots (a so-called paperclip maximizer). Then robots would already be widespread. It would only be a matter of time before the robots filled the universe because of the law of
The Long History of the Future: Why Tomorrow's Technology Still Isn't Here
by Nicole Kobie · 3 Jul 2024 · 348pp · 119,358 words
big-brained ones on this planet. How could superintelligent AGI hurt us? One thought experiment often used to explain the threat is known as the ‘paper clip maximiser problem’. It goes like this: ask an AGI to make loads of paper clips, and it may choose to wipe out humans or otherwise wreak