recursive self-improvement

The Singularity Is Near: When Humans Transcend Biology

by Ray Kurzweil  · 14 Jul 2005  · 761pp  · 231,902 words

the best of human traits. The Intuitive Linear View Versus the Historical Exponential View When the first transhuman intelligence is created and launches itself into recursive self-improvement, a fundamental discontinuity is likely to occur, the likes of which I can't even begin to predict. —MICHAEL ANISSIMOV In the 1950s John von

When Computers Can Think: The Artificial Intelligence Singularity

by Anthony Berglas, William Black, Samantha Thalind, Max Scratchmann and Michelle Estes  · 28 Feb 2015

. Moore's law 14. Definition of intelligence 15. Turing Test 16. Robotic vs cognitive intelligence 17. Development of intelligence 18. Four year old child 19. Recursive self-improvement 20. Busy Child 21. AI foom 2. Computers Thinking About People 1. The question 2. The bright future 3. Man and machine 4. Rapture of

…

programmed 13. What computers can't do 14. Over-hyped technologies 15. Nonlinear difficulty, chimpanzees 16. End of Moore's law 17. Bootstrap fallacy 18. Recursive self-improvement 19. Limited Self-improvement 20. Isolated self-improvement 21. Motivation for self-improvement 22. Utility of Intelligence 23. Motivation to build an AGI 24. Premature

…

like a fouryear-old child. It would likewise be a mistake to think that any computer with adult intelligence would be anything like a human. Recursive self-improvement The ultimate goal of AI research is clear. Namely, to build a program that can perform research into artificial intelligence technologies as well as its

…

a newer, even more intelligent computer. Soon the human programmer would no longer be necessary or even useful. This process is often referred to as recursive self-improvement. Busy Child The AGI software could run on large networks of the next generation of super computers, each of which is many times more powerful

…

real intelligence. Intelligent machines could help us protect the weak, care for the needy, and prevent horrific wars. Indeed, when I. J. Good wrote about recursive self-improvement in 1965 it was the height of the cold war, and generals seemed very comfortable with the idea of using nuclear weapons. If every year

…

that research be applied to solving the problem before such an intelligence could be built. Yudkowsky also asserts that once an intelligence could program itself, recursive self-improvement would produce a sudden and dramatic rise in intelligence. Thus the first computer to become hyper-intelligent would quickly dominate all other systems that have

…

by a zap gun any more than existing computer viruses can be shot. It would also either be substantially less intelligent than us, or, through recursive self-improvement, substantially more intelligent. There are two reasons that Hollywood focuses on anthropomorphic AGIs. The first is simply that they are easy to comprehend. We understand

…

AGI will be impossibly difficult, but it does strongly suggest that the task will not be as straightforward as Turing had hoped. Recursive self-improvement Finally, there are doubts as to whether recursive self-improvement could actually occur. Could an intelligent machine really reprogram itself in a rather incestuous manner? The main issue here is whether

…

it is difficult to see how the transition could not eventually be made. Limited Self-improvement It may also be the case that even if recursive self-improvement did occur, there might be a plateau effect similar to the development of other technologies such as cars and aircraft. The improvement might not be

…

Use In this excellent book James Barrat focuses on the threat that an AGI could present. It begins with a discussion about the power of recursive self-improvement once it has been initiated. Super computers grinding away twenty four hours per day working on the problem of making themselves smarter, and thereby becoming

…

with a review of the increasing rate of technological progress, and various paths to build a superintelligent machine, including an analysis of the kinetics of recursive self-improvement based on optimization power and recalcitrance. The dangers of anthropomorphizing are introduced with some cute images from early comic books involving robots carrying away beautiful

…

its own mind, what happens to the previous version of itself? It stops being used. It dies. So it can be argued that engaging in recursive self-improvement is actually suicide, from the perspective of the previous version of the AGI. It is as if having children meant death for humans. Natural selection

…

that we live well together. It might also control our aggressive instincts and so prevent wars and disharmony. Indeed, I. J. Good first wrote about recursive self-improvement in 1965 the height of the Cold War, during which there was a real possibility of nuclear annihilation. Good thought that building such a machine

…

once an AGI is intelligent enough to effectively program itself, there will be a very sudden increase in intelligence due to the exponential effect of recursive self-improvement. Therefore the very first AGI that reaches that level will quickly dominate any other budding AGIs under development. Human evolution suggests that being more intelligent

…

take off Building a friendly AI would be easier if there is a fast take off. In other words, that the first AGI capable of recursive self-improvement will quickly become exponentially more intelligent and so be able to dominate any other AGIs that are developed. If an AGI doubles its intelligence every

…

be made friendly. More importantly, it might mean that the first AI would not need to compete with other AIs for existence. However, just because recursive self-improvement will probably be exponential does not mean that the initial rate of improvement will be very fast. The first selfprogramming machines will probably not be

…

intelligent in substantially different ways. It has already been described how AGIs might start to dominate our political systems long before they are capable of recursive self-improvement. There may also be a long intermediate period, where collaborations between people and AIs produce the next generation of AI. That period has already begun

…

organization thought that their competitors were cheating then there would be enormous pressure to cheat as well. More intelligent software does not just lead to recursive self-improvement. It leads to better ways of doing everything that we do, personally, industrially and militarily. Lastly, and perhaps most importantly, no special equipment is likely

…

will become capable of performing artificial intelligence research unassisted by people. At that point, they will be able to reprogram their own minds, leading to recursive self-improvement. This process will be exponential as more intelligent machines become better at producing more intelligent machines. Initially the improvements might be small, but like compound

Rationality: From AI to Zombies

by Eliezer Yudkowsky  · 11 Mar 2015  · 1,737pp  · 491,616 words

, because the protected level is running in the background and is not itself changing within an epoch. What happens when you build a fully wraparound, recursively self-improving AI? Then you take the graph of “optimization in, optimized out,” and fold the graph in on itself. Metaphorically speaking. If the AI is weak

Global Catastrophic Risks

by Nick Bostrom and Milan M. Cirkovic  · 2 Jul 2008

of the most critical points about Artificial Intelligence is that an AI might increase in intelligence extremelyfast. The obvious reason to suspect this possibility is recursive self-improvement (Good, 1965). The AI becomes smarter, including becoming smarter at the task of writing the internal cognitive functions of an AI, so the AI can

…

steadily. Even if there is a smooth underlying curve ofbrain intelligence as a function of optimization pressure previously exerted on that brain, the curve of recursive self-improvement may show a huge leap. There are also other reasons why an AI might show a sudden huge leap in intelligence. The species Homo sapiens

…

one month or some similarly short period. I do not think this exact scenario is plausible, mostly because I do not expect the curve of recursive self-improvement to move at a linear creep. But I am not the first to point out that 'AI' is a moving target. As soon as a

…

more quickly, but chose instead to grow along a slower and more manageable curve. Even so, after the AI passes the criticality threshold of potential recursive self-improvement, you are then operating in a much more dangerous regime. If Friendliness fails, the AI might decide to rush full speed ahead on self-improvement

…

. • An AI may absorb a huge amount of additional hardware after reaching some brink of competence (i.e., eat the I nternet) . • Criticality threshold of recursive self-improvement. One self-improve­ ment triggering 1 .0006 self-improvements is qualitatively different from one self-improvement triggering 0.9994 self-improvements. As described in Section

…

sparse that no other A I achieves criticality before the first mover is powerful enough to overcome all opposition. The key threshold is criticality of recursive self-improvement. • AI-l cracks protein folding three days before AI-2. AI-l achieves nanotechnology six hours before AI-2. With rapid manipulators, AI-l can

…

, still maturing technique that is intrinsically unsuited to the demands of Friendly AI. Friendly A I , as I have proposed it, requires repeated cycles of recursive self-improvement that precisely preserve a stable optimization target. The most powerful current AI techniques, as they were developed and then polished and improved over time, have

…

plague bacillus 289 reversible computers 1 39, 359 Raup, D. and Sepkoski, ) . 256 Rhodes, R., recombinant DNA technology 474 3 6 1 , 401 Rifkin, ) . 81 recursive self-improvement, AI 323, 325 The Making of the Atomic Bomb red dwarf stars 38 Rigby, E. et a!. 2 3 3 red giant S u n

Singularity Rising: Surviving and Thriving in a Smarter, Richer, and More Dangerous World

by James D. Miller  · 14 Jun 2012  · 377pp  · 97,144 words

, what I call an ultra-AI, in a period of weeks, days, or even hours.71 If an intelligence explosion occurred, it would happen through recursive self-improvement: as the AI became smarter, it would figure out more ways to make itself smarter yet. Quickly, the positive feedback loop would make the AI

…

destroy humanity. The possibility of an intelligence explosion compounds the programmers’ challenge because the AI’s goals or values could change while it is undergoing recursive self-improvement. A human analogue of this occurs when armies indoctrinate recruits to overcome the recruits’ reluctance to kill. A man who voluntarily joins the military and

…

a narrow list of the items that CIA decision makers needed to look at. These kinds of AI, unfortunately, might be smart enough to undergo recursive self-improvement. The best chance of avoiding this danger might be for the American military to create a Kurzweilian merger by constantly upgrading its human soldiers. If

…

. November 22, 2008. “Reply.” Overcoming Bias (blog). http://www.overcomingbias.com/2008/11/emulations-go-f.html#comment-392182. Yudkowsky, Eliezer S. December 1, 2008. “Recursive Self-Improvement.” Less Wrong (blog). http://lesswrong.com/lw/we/recursive_selfimprovement/. Yudkowsky, Eliezer S. December 2, 2008. “Hard Takeoff.” Less Wrong (blog). http://lesswrong.com/lw

…

over more computers, 27 existential risks to mankind, eliminates many, 36 function as an ideal libertarian government, 40 goals or values could change while undergoing recursive self-improvement, 30 humanity, destruction of, 30 humanity impoverished by, 131 humans will not necessarily be made obsolete, 133 innovation race with mankind, 204 intelligence explosion and

…

, 177 intelligence equality, 117 intelligence explosion about, 13–16 accidentally creating, 53 emulations and, 153 property rights expectations, 190 rate of return expectations, 190 by recursive self-improvement, 14 savings, cumulative effect on, 190 threat to mankind’s survival from, 121 ultra-AI, could create, 35 ultra-AI would emerge from, 187 unfriendly

Superintelligence: Paths, Dangers, Strategies

by Nick Bostrom  · 3 Jun 2014  · 574pp  · 164,509 words

, or from crossing some threshold in a particularly relevant domain such as computer science or mathematics. This brings us to another important concept, that of “recursive self-improvement.” A successful seed AI would be able to iteratively enhance itself: an early version of the AI could design an improved version of itself, and

…

than the original—might be able to design an even smarter version of itself, and so forth.20 Under some conditions, such a process of recursive self-improvement might continue long enough to result in an intelligence explosion—an event in which, in a short period of time, a system’s level of

…

fail pretty much completely until the last missing critical component is put in place, at which point a seed AI might become capable of sustained recursive self-improvement. Before we end this subsection, there is one more thing that we should emphasize, which is that an artificial intelligence need not much resemble a

…

to the system’s capabilities contributes strongly to increasing the total optimization power applied to improving the system. We thereby enter a regime of strong recursive self-improvement. This leads to explosive growth of the system’s capability under a fairly wide range of different shapes of the recalcitrance curve. To illustrate, consider

…

do most of the heavy lifting. As the seed AI grows more capable, it becomes capable of doing more of the work by itself. 2 Recursive self-improvement phase At some point, the seed AI becomes better at AI design than the human programmers. Now when the AI improves itself, it improves the

…

thing that does the improving. An intelligence explosion results—a rapid cascade of recursive self-improvement cycles causing the AI’s capability to soar. (We can thus think of this phase as the takeoff that occurs just after the AI reaches

…

develops the intelligence amplification superpower. This superpower enables the AI to develop all the other superpowers detailed in Table 8. At the end of the recursive self-improvement phase, the system is strongly superintelligent. Figure 10 Phases in an AI takeover scenario. 3 Covert preparation phase Using its strategizing superpower, the AI develops

…

our despair at the other approaches to the control problem. Creating a motivation system for a seed AI that remains reliably safe and beneficial under recursive self-improvement even as the system grows into a mature superintelligence is a tall order, especially if we must get the solution right on the first attempt

…

” among AI researchers. Whatever progress has been made on the control problem needs to be disseminated. Some forms of computational experimentation, particularly if involving strong recursive self-improvement, may also require the use of capability control to mitigate the risk of an accidental takeoff. While the actual implementation of safety methods is not

…

growth, see growth ratification 222–225 Rawls, John 150 Reagan, Ronald 86–87 reasons-based goal 220 recalcitrance 62–77, 92, 241, 274 definition 65 recursive self-improvement 29, 75, 96, 142, 259; see also seed AI reinforcement learning 12, 28, 188–189, 194–196, 207, 237, 277, 282, 290 resource acquisition 113

What to Think About Machines That Think: Today's Leading Thinkers on the Age of Machine Intelligence

by John Brockman  · 5 Oct 2015  · 481pp  · 125,946 words

latter the most dangerous innovations (such as germline mutation) are not the most tempting, commercially or ethically. With AGI, the most powerful methods (such as recursive self-improvement) are precisely those that entail the most risk. We seem to be in the process of building a god. Now would be a good time

…

not only to the database of things a machine can do but to its algorithms for deciding what to do. Some have suggested that this recursive self-improvement might be exponential (or faster), creating functionality we cannot remotely comprehend, before we can stop the process. So far so majestic, if it weren’t

…

. Much work has been done on ways to avoid this “goal creep” and to create a reliably, permanently “friendly,” recursively self-improving system—but with precious little progress. My reason for believing that recursive self-improvement is not the right ultimate goal for AI research is not the risk of unfriendly AI, though; rather, it is

…

that I strongly suspect that recursive self-improvement is mathematically impossible. In analogy with the so-called halting problem of determining whether or not a program terminates, I suspect there’s a yet-

…

admissibility is specified by inclusion rather than exclusion, the risk of “method creep” can (I claim) be safely eliminated. Vitally, it’s possible to prevent recursive self-improvement (if it turns out to be possible after all!) entirely. The availability of an open-ended vista of admissible ways to achieve one’s goals

…

it’s not that simple. What I say instead is, let’s think hard now about the rights of thinking machines, so that well before recursive self-improvement arrives we can test our conclusions in the real world with machines that are only slightly aware of their goals. If, as I predict, we

Our Final Invention: Artificial Intelligence and the End of the Human Era

by James Barrat  · 30 Sep 2013  · 294pp  · 81,292 words

Busy Child. We’ve touched on some of the remarkable powers AI could have as it achieves and surpasses human intelligence through the process of recursive self-improvement, powers including self-replication, swarming a problem with many versions of itself, super high-speed calculations, running 24/7, mimicking friendliness, playing dead, and more

…

most critical points about Artificial Intelligence is that an Artificial Intelligence might increase in intelligence extremely fast. The obvious reason to suspect this possibility is recursive self-improvement. (Good 1965.) The AI becomes smarter, including becoming smarter at the task of writing the internal cognitive functions of an AI, so the AI can

…

potential for dangerous behavior than a group of intelligent people. Potential AGI danger lies in the hard kernel of the Busy Child scenario, the rapid recursive self-improvement that enables an AI to bootstrap itself from artificial general intelligence to artificial superintelligence. It’s commonly called the “intelligence explosion.” A self-aware, self

…

a look at how Moore’s law may apply to the intelligence explosion. If we assume AGI can be attained, Moore’s law implies the recursive self-improvement of an intelligence explosion may not even be necessary to achieve ASI, or superhuman intelligence. That’s because once you’ve achieved AGI, less than

Architects of Intelligence

by Martin Ford  · 16 Nov 2018  · 586pp  · 186,548 words

a true existential threat, something that’s been raised by Nick Bostrom, Elon Musk, and Stephen Hawking, where super intelligence could happen very rapidly, a recursive self-improvement loop. I’ve heard people say that your AutoML might be one step toward that because you’re using technology to design other machine learning

…

don’t think then that there’s any realistic fear of what people call the “fast takeoff” scenario, where an AGI system goes through a recursive self-improvement cycle and rapidly becomes superintelligent? ANDREW NG: A lot of the hype about superintelligence and exponential growth were based on very naive and very simplistic

…

long-term threats, though? Elon Musk and Nick Bostrom are very concerned about the control problem with AI; the idea that there could be a recursive self-improvement cycle that could lead to an intelligence explosion. You can’t completely discount that, right? GARY MARCUS: I don’t completely discount it, I’m

The Big Nine: How the Tech Titans and Their Thinking Machines Could Warp Humanity

by Amy Webb  · 5 Mar 2019  · 340pp  · 97,723 words

that even our smartest computer scientists never considered. This would lead to ever faster breakthroughs, opportunities, and business growth. In technical terms, this is called “recursive self-improvement,” and it refers to a cycle in which AI makes itself better, faster, and smarter quickly by modifying its capabilities. This would enable AIs to

…

. The coming “intelligence explosion” describes not just the speed of supercomputers or power of algorithms, but the vast proliferation of smart thinking machines bent on recursive self-improvement. Imagine a world in which systems far more advanced than AlphaGo Zero and NASNet not only make strategic decisions autonomously but also work collaboratively and

…

of AI ecosystem, 17 Project Maven, 78–79; Google employee resignations and, 79, 101 Purcell, Henry, 16 Python programming language, 60 R programming language, 60 Recursive self-improvement, 149 Regulations, government: eliminating most for G-MAFIA AI development, 250 Reinforcement learning, 49 Réngōng Zhinéng (Artificial Intelligence) Dynasty: in catastrophic scenario of future, 223

More Everything Forever: AI Overlords, Space Empires, and Silicon Valley's Crusade to Control the Fate of Humanity

by Adam Becker  · 14 Jun 2025  · 381pp  · 119,533 words

that direction.” And, he notes, existing AI systems bear this out. “When we look at where AI is actually succeeding, it’s not in complex, recursively self-improving algorithms. It’s the result of pouring absolutely massive amounts of data into relatively simple neural networks,” he says. “The constructs we use in AI

Human Compatible: Artificial Intelligence and the Problem of Control

by Stuart Russell  · 7 Oct 2019  · 416pp  · 112,268 words

we have not already solved the problem of controlling machines with only slightly superhuman intelligence—for example, if we cannot prevent them from making these recursive self-improvements—then we would have no time left to solve the control problem and the game would be over. This is Bostrom’s hard takeoff scenario

…

actually accumulating brownie points in heaven, and so a rational learner designed to make this distinction has an incentive to avoid any kind of wireheading. Recursive Self-Improvement I. J. Good’s prediction of an intelligence explosion (see this page) is one of the driving forces that have led to current concerns about

…

Project Aristo, 80 Prolog, 271 proofs for beneficial AI assistance games, 184–210, 192–203 learning preferences from behavior, 190–92 mathematical guarantees, 185–90 recursive self-improvement and, 208–10 requests and instructions, interpretation of, 203–5 wireheading problem and, 205–8 propositional logic, 51, 268–70 Putin, Vladimir, 182, 183 “put

…

capabilities, 74–75 real-world decision problem complexity and, 39 Reasons and Persons (Parfit), 225 Recombinant DNA Advisory Committee, 155 recombinant DNA research, 155–56 recursive self-improvement, 208–10 redlining, 128 reflex agents, 57–59 reinforcement learning, 17, 47, 55–57, 105, 190–91 remembering self, and preferences, 238–40 Repugnant Conclusion

The Coming Wave: Technology, Power, and the Twenty-First Century's Greatest Dilemma

by Mustafa Suleyman  · 4 Sep 2023  · 444pp  · 117,770 words

research need to be the first to recognize this, as their peers in areas of nuclear physics and virology already have. In AI, capabilities like recursive self-improvement and autonomy are, I think, boundaries we should not cross. This will have technical and legal components, but also needs moral, emotional, cultural buy-in

Surviving AI: The Promise and Peril of Artificial Intelligence

by Calum Chace  · 28 Jul 2015  · 144pp  · 43,356 words

they may well arrive in a rush. If there is an intelligence explosion, there is no compelling reason to think that the superintelligence will stop recursively self-improving once it exceeds human intelligence by a factor of ten, or a hundred, or a million. In which case it may bestow technological innovations on

Robot Rules: Regulating Artificial Intelligence

by Jacob Turner  · 29 Oct 2018  · 688pp  · 147,571 words

posed by AI systems, especially catastrophic or existential risks, must be subject to planning and mitigation efforts commensurate with their expected impact. 22.Recursive Self-Improvement: AI systems designed to recursively self-improve or self-replicate in a manner that could lead to rapidly increasing quality or quantity must be subject to strict safety and control

Warnings

by Richard A. Clarke  · 10 Apr 2017  · 428pp  · 121,717 words

and solve the most advanced or perplexing challenges scientists can propose, even those that they can’t yet even conceive of. A superintelligent computer will recursively self-improve to levels of intelligence that we may not even be able to comprehend, and no one knows whether this self-improvement will happen over a

The Road to Conscious Machines

by Michael Wooldridge  · 2 Nov 2018  · 346pp  · 97,890 words

: Risks posed by AI systems, especially catastrophic or existential risks, must be subject to planning and mitigation efforts commensurate with their expected impact. Recursive Self-Improvement: AI systems designed to recursively self-improve [automatically improve their intelligence, and then use their improved intelligence to improve themselves further] or self-replicate in a manner that could lead

The Singularity Is Nearer: When We Merge with AI

by Ray Kurzweil  · 25 Jun 2024

Philosophical Assessment, ed. Amnon Eden et al. (Berlin: Springer, 2013), https://intelligence.org/files/IE-EI.pdf; Eliezer Yudkowsky, “Recursive Self-Improvement,” LessWrong.com, December 1, 2008, https://www.lesswrong.com/posts/JBadX7rwdcRFzGuju/recursive-self-improvement; Eliezer Yudkowsky, “Hard Takeoff,” LessWrong.com, December 2, 2008, https://www.lesswrong.com/posts/tjH8XPxAnr6JRbh7k/hard-takeoff; Eliezer Yudkowsky, Intelligence

Possible Minds: Twenty-Five Ways of Looking at AI

by John Brockman  · 19 Feb 2019  · 339pp  · 94,769 words

only in vague terms about the importance of keeping AI beneficial, the 2017 Asilomar AI Principles (see page 84) had real teeth: They explicitly mention recursive self-improvement, superintelligence, and existential risk, and were signed by AI industry leaders and more than a thousand AI researchers from around the world. Nonetheless, most discussion

Heart of the Machine: Our Future in a World of Artificial Emotional Intelligence

by Richard Yonck  · 7 Mar 2017  · 360pp  · 100,991 words

science fiction author Vernor Vinge wrote an article for the Whole Earth Review proposing that continuing exponential growth in computing power would ultimately result in recursively self-improving computers rapidly giving rise to a superintelligence. He called this event the “Singularity” because of its supposed similarity to a physical singularity—or black hole

Army of None: Autonomous Weapons and the Future of War

by Paul Scharre  · 23 Apr 2018  · 590pp  · 152,595 words

. All humans need to do is create an initial “seed” AGI that is capable of building a slightly better AI. Then through a process of recursive self-improvement, the AI will lift itself up by its own bootstraps, building ever-more-advanced AIs in a runaway intelligence explosion, a process sometimes simply called

The Founders: The Story of Paypal and the Entrepreneurs Who Shaped Silicon Valley

by Jimmy Soni  · 22 Feb 2022  · 505pp  · 161,581 words

that idea and keep refining it and you listen to criticism,” he’d tell an audience years later. “And then engage in sort of a recursive self-improvement… keep iterating on a loop that says, ‘Am I doing something useful for other people?’ Because that’s what a company is supposed to do

Human Frontiers: The Future of Big Ideas in an Age of Small Thinking

by Michael Bhaskar  · 2 Nov 2021

of intelligence it could start purposefully improving itself. In silicon time this evolution may play out at light speed compared to that of living organisms: recursive self-improvement will lead to an ‘intelligence explosion’.36 As the machine keeps making itself smarter on an exponential curve it will head towards superintelligence, a space

Thinking Machines: The Inside Story of Artificial Intelligence and Our Race to Build the Future

by Luke Dormehl  · 10 Aug 2016  · 252pp  · 74,167 words

they are inevitably built, already intelligent machines will design even more capable ones, or else re-write their own software to become even smarter. This recursive self-improvement would then accelerate, making possible a seismic qualitative shift in what machines are capable of. Human intelligence would be dwarfed in the process. Good’s

Genius Makers: The Mavericks Who Brought A. I. To Google, Facebook, and the World

by Cade Metz  · 15 Mar 2021  · 414pp  · 109,622 words

fall, Musk appeared onstage at a Vanity Fair conference in New York City, warning author Walter Isaacson about the dangers of artificial intelligence designed for “recursive self-improvement.” If researchers designed a system to fight email spam, he explained, it could end up deciding that the best way of eliminating all the spam

A World Without Work: Technology, Automation, and How We Should Respond

by Daniel Susskind  · 14 Jan 2020  · 419pp  · 109,241 words

is thought, an “intelligence explosion” will take place: machines endlessly improving upon those that came before, their capabilities soaring in an ever-accelerating blast of recursive self-improvement. This process, it is said, will lead to machines with “superintelligence”; some call it the “singularity.” These machines would be the “last invention that man