recursive self-improvement Concepts

When Computers Can Think: The Artificial Intelligence Singularity

by Anthony Berglas, William Black, Samantha Thalind, Max Scratchmann and Michelle Estes · 28 Feb 2015

. Moore's law 14. Definition of intelligence 15. Turing Test 16. Robotic vs cognitive intelligence 17. Development of intelligence 18. Four year old child 19. Recursive self-improvement 20. Busy Child 21. AI foom 2. Computers Thinking About People 1. The question 2. The bright future 3. Man and machine 4. Rapture of

…

programmed 13. What computers can't do 14. Over-hyped technologies 15. Nonlinear difficulty, chimpanzees 16. End of Moore's law 17. Bootstrap fallacy 18. Recursive self-improvement 19. Limited Self-improvement 20. Isolated self-improvement 21. Motivation for self-improvement 22. Utility of Intelligence 23. Motivation to build an AGI 24. Premature

…

like a fouryear-old child. It would likewise be a mistake to think that any computer with adult intelligence would be anything like a human. Recursive self-improvement The ultimate goal of AI research is clear. Namely, to build a program that can perform research into artificial intelligence technologies as well as its

…

a newer, even more intelligent computer. Soon the human programmer would no longer be necessary or even useful. This process is often referred to as recursive self-improvement. Busy Child The AGI software could run on large networks of the next generation of super computers, each of which is many times more powerful

…

real intelligence. Intelligent machines could help us protect the weak, care for the needy, and prevent horrific wars. Indeed, when I. J. Good wrote about recursive self-improvement in 1965 it was the height of the cold war, and generals seemed very comfortable with the idea of using nuclear weapons. If every year

…

that research be applied to solving the problem before such an intelligence could be built. Yudkowsky also asserts that once an intelligence could program itself, recursive self-improvement would produce a sudden and dramatic rise in intelligence. Thus the first computer to become hyper-intelligent would quickly dominate all other systems that have

…

by a zap gun any more than existing computer viruses can be shot. It would also either be substantially less intelligent than us, or, through recursive self-improvement, substantially more intelligent. There are two reasons that Hollywood focuses on anthropomorphic AGIs. The first is simply that they are easy to comprehend. We understand

…

AGI will be impossibly difficult, but it does strongly suggest that the task will not be as straightforward as Turing had hoped. Recursive self-improvement Finally, there are doubts as to whether recursive self-improvement could actually occur. Could an intelligent machine really reprogram itself in a rather incestuous manner? The main issue here is whether

…

it is difficult to see how the transition could not eventually be made. Limited Self-improvement It may also be the case that even if recursive self-improvement did occur, there might be a plateau effect similar to the development of other technologies such as cars and aircraft. The improvement might not be

…

Use In this excellent book James Barrat focuses on the threat that an AGI could present. It begins with a discussion about the power of recursive self-improvement once it has been initiated. Super computers grinding away twenty four hours per day working on the problem of making themselves smarter, and thereby becoming

…

with a review of the increasing rate of technological progress, and various paths to build a superintelligent machine, including an analysis of the kinetics of recursive self-improvement based on optimization power and recalcitrance. The dangers of anthropomorphizing are introduced with some cute images from early comic books involving robots carrying away beautiful

…

its own mind, what happens to the previous version of itself? It stops being used. It dies. So it can be argued that engaging in recursive self-improvement is actually suicide, from the perspective of the previous version of the AGI. It is as if having children meant death for humans. Natural selection

…

that we live well together. It might also control our aggressive instincts and so prevent wars and disharmony. Indeed, I. J. Good first wrote about recursive self-improvement in 1965 the height of the Cold War, during which there was a real possibility of nuclear annihilation. Good thought that building such a machine

…

once an AGI is intelligent enough to effectively program itself, there will be a very sudden increase in intelligence due to the exponential effect of recursive self-improvement. Therefore the very first AGI that reaches that level will quickly dominate any other budding AGIs under development. Human evolution suggests that being more intelligent

…

take off Building a friendly AI would be easier if there is a fast take off. In other words, that the first AGI capable of recursive self-improvement will quickly become exponentially more intelligent and so be able to dominate any other AGIs that are developed. If an AGI doubles its intelligence every

…

be made friendly. More importantly, it might mean that the first AI would not need to compete with other AIs for existence. However, just because recursive self-improvement will probably be exponential does not mean that the initial rate of improvement will be very fast. The first selfprogramming machines will probably not be

…

intelligent in substantially different ways. It has already been described how AGIs might start to dominate our political systems long before they are capable of recursive self-improvement. There may also be a long intermediate period, where collaborations between people and AIs produce the next generation of AI. That period has already begun

…

organization thought that their competitors were cheating then there would be enormous pressure to cheat as well. More intelligent software does not just lead to recursive self-improvement. It leads to better ways of doing everything that we do, personally, industrially and militarily. Lastly, and perhaps most importantly, no special equipment is likely

…

will become capable of performing artificial intelligence research unassisted by people. At that point, they will be able to reprogram their own minds, leading to recursive self-improvement. This process will be exponential as more intelligent machines become better at producing more intelligent machines. Initially the improvements might be small, but like compound

Rationality: From AI to Zombies

by Eliezer Yudkowsky · 11 Mar 2015 · 1,737pp · 491,616 words

, because the protected level is running in the background and is not itself changing within an epoch. What happens when you build a fully wraparound, recursively self-improving AI? Then you take the graph of “optimization in, optimized out,” and fold the graph in on itself. Metaphorically speaking. If the AI is weak

The Singularity Is Near: When Humans Transcend Biology

by Ray Kurzweil · 14 Jul 2005 · 761pp · 231,902 words

the best of human traits. The Intuitive Linear View Versus the Historical Exponential View When the first transhuman intelligence is created and launches itself into recursive self-improvement, a fundamental discontinuity is likely to occur, the likes of which I can't even begin to predict. —MICHAEL ANISSIMOV In the 1950s John von

Superintelligence: Paths, Dangers, Strategies

by Nick Bostrom · 3 Jun 2014 · 574pp · 164,509 words

, or from crossing some threshold in a particularly relevant domain such as computer science or mathematics. This brings us to another important concept, that of “recursive self-improvement.” A successful seed AI would be able to iteratively enhance itself: an early version of the AI could design an improved version of itself, and

…

than the original—might be able to design an even smarter version of itself, and so forth.20 Under some conditions, such a process of recursive self-improvement might continue long enough to result in an intelligence explosion—an event in which, in a short period of time, a system’s level of

…

fail pretty much completely until the last missing critical component is put in place, at which point a seed AI might become capable of sustained recursive self-improvement. Before we end this subsection, there is one more thing that we should emphasize, which is that an artificial intelligence need not much resemble a

…

to the system’s capabilities contributes strongly to increasing the total optimization power applied to improving the system. We thereby enter a regime of strong recursive self-improvement. This leads to explosive growth of the system’s capability under a fairly wide range of different shapes of the recalcitrance curve. To illustrate, consider

…

do most of the heavy lifting. As the seed AI grows more capable, it becomes capable of doing more of the work by itself. 2 Recursive self-improvement phase At some point, the seed AI becomes better at AI design than the human programmers. Now when the AI improves itself, it improves the

…

thing that does the improving. An intelligence explosion results—a rapid cascade of recursive self-improvement cycles causing the AI’s capability to soar. (We can thus think of this phase as the takeoff that occurs just after the AI reaches

…

develops the intelligence amplification superpower. This superpower enables the AI to develop all the other superpowers detailed in Table 8. At the end of the recursive self-improvement phase, the system is strongly superintelligent. Figure 10 Phases in an AI takeover scenario. 3 Covert preparation phase Using its strategizing superpower, the AI develops

…

our despair at the other approaches to the control problem. Creating a motivation system for a seed AI that remains reliably safe and beneficial under recursive self-improvement even as the system grows into a mature superintelligence is a tall order, especially if we must get the solution right on the first attempt

…

” among AI researchers. Whatever progress has been made on the control problem needs to be disseminated. Some forms of computational experimentation, particularly if involving strong recursive self-improvement, may also require the use of capability control to mitigate the risk of an accidental takeoff. While the actual implementation of safety methods is not

…

growth, see growth ratification 222–225 Rawls, John 150 Reagan, Ronald 86–87 reasons-based goal 220 recalcitrance 62–77, 92, 241, 274 definition 65 recursive self-improvement 29, 75, 96, 142, 259; see also seed AI reinforcement learning 12, 28, 188–189, 194–196, 207, 237, 277, 282, 290 resource acquisition 113

Singularity Rising: Surviving and Thriving in a Smarter, Richer, and More Dangerous World

by James D. Miller · 14 Jun 2012 · 377pp · 97,144 words

, what I call an ultra-AI, in a period of weeks, days, or even hours.71 If an intelligence explosion occurred, it would happen through recursive self-improvement: as the AI became smarter, it would figure out more ways to make itself smarter yet. Quickly, the positive feedback loop would make the AI

…

destroy humanity. The possibility of an intelligence explosion compounds the programmers’ challenge because the AI’s goals or values could change while it is undergoing recursive self-improvement. A human analogue of this occurs when armies indoctrinate recruits to overcome the recruits’ reluctance to kill. A man who voluntarily joins the military and

…

a narrow list of the items that CIA decision makers needed to look at. These kinds of AI, unfortunately, might be smart enough to undergo recursive self-improvement. The best chance of avoiding this danger might be for the American military to create a Kurzweilian merger by constantly upgrading its human soldiers. If

…

. November 22, 2008. “Reply.” Overcoming Bias (blog). http://www.overcomingbias.com/2008/11/emulations-go-f.html#comment-392182. Yudkowsky, Eliezer S. December 1, 2008. “Recursive Self-Improvement.” Less Wrong (blog). http://lesswrong.com/lw/we/recursive_selfimprovement/. Yudkowsky, Eliezer S. December 2, 2008. “Hard Takeoff.” Less Wrong (blog). http://lesswrong.com/lw

…

over more computers, 27 existential risks to mankind, eliminates many, 36 function as an ideal libertarian government, 40 goals or values could change while undergoing recursive self-improvement, 30 humanity, destruction of, 30 humanity impoverished by, 131 humans will not necessarily be made obsolete, 133 innovation race with mankind, 204 intelligence explosion and

…

, 177 intelligence equality, 117 intelligence explosion about, 13–16 accidentally creating, 53 emulations and, 153 property rights expectations, 190 rate of return expectations, 190 by recursive self-improvement, 14 savings, cumulative effect on, 190 threat to mankind’s survival from, 121 ultra-AI, could create, 35 ultra-AI would emerge from, 187 unfriendly

What to Think About Machines That Think: Today's Leading Thinkers on the Age of Machine Intelligence

by John Brockman · 5 Oct 2015 · 481pp · 125,946 words

latter the most dangerous innovations (such as germline mutation) are not the most tempting, commercially or ethically. With AGI, the most powerful methods (such as recursive self-improvement) are precisely those that entail the most risk. We seem to be in the process of building a god. Now would be a good time

…

not only to the database of things a machine can do but to its algorithms for deciding what to do. Some have suggested that this recursive self-improvement might be exponential (or faster), creating functionality we cannot remotely comprehend, before we can stop the process. So far so majestic, if it weren’t

…

. Much work has been done on ways to avoid this “goal creep” and to create a reliably, permanently “friendly,” recursively self-improving system—but with precious little progress. My reason for believing that recursive self-improvement is not the right ultimate goal for AI research is not the risk of unfriendly AI, though; rather, it is

…

that I strongly suspect that recursive self-improvement is mathematically impossible. In analogy with the so-called halting problem of determining whether or not a program terminates, I suspect there’s a yet-

…

admissibility is specified by inclusion rather than exclusion, the risk of “method creep” can (I claim) be safely eliminated. Vitally, it’s possible to prevent recursive self-improvement (if it turns out to be possible after all!) entirely. The availability of an open-ended vista of admissible ways to achieve one’s goals

…

it’s not that simple. What I say instead is, let’s think hard now about the rights of thinking machines, so that well before recursive self-improvement arrives we can test our conclusions in the real world with machines that are only slightly aware of their goals. If, as I predict, we

Global Catastrophic Risks

by Nick Bostrom and Milan M. Cirkovic · 2 Jul 2008

of the most critical points about Artificial Intelligence is that an AI might increase in intelligence extremelyfast. The obvious reason to suspect this possibility is recursive self-improvement (Good, 1965). The AI becomes smarter, including becoming smarter at the task of writing the internal cognitive functions of an AI, so the AI can

…

steadily. Even if there is a smooth underlying curve ofbrain intelligence as a function of optimization pressure previously exerted on that brain, the curve of recursive self-improvement may show a huge leap. There are also other reasons why an AI might show a sudden huge leap in intelligence. The species Homo sapiens

…

one month or some similarly short period. I do not think this exact scenario is plausible, mostly because I do not expect the curve of recursive self-improvement to move at a linear creep. But I am not the first to point out that 'AI' is a moving target. As soon as a

…

more quickly, but chose instead to grow along a slower and more manageable curve. Even so, after the AI passes the criticality threshold of potential recursive self-improvement, you are then operating in a much more dangerous regime. If Friendliness fails, the AI might decide to rush full speed ahead on self-improvement

…

. • An AI may absorb a huge amount of additional hardware after reaching some brink of competence (i.e., eat the I nternet) . • Criticality threshold of recursive self-improvement. One self-improve ment triggering 1 .0006 self-improvements is qualitatively different from one self-improvement triggering 0.9994 self-improvements. As described in Section

…

sparse that no other A I achieves criticality before the first mover is powerful enough to overcome all opposition. The key threshold is criticality of recursive self-improvement. • AI-l cracks protein folding three days before AI-2. AI-l achieves nanotechnology six hours before AI-2. With rapid manipulators, AI-l can

…

, still maturing technique that is intrinsically unsuited to the demands of Friendly AI. Friendly A I , as I have proposed it, requires repeated cycles of recursive self-improvement that precisely preserve a stable optimization target. The most powerful current AI techniques, as they were developed and then polished and improved over time, have

…

plague bacillus 289 reversible computers 1 39, 359 Raup, D. and Sepkoski, ) . 256 Rhodes, R., recombinant DNA technology 474 3 6 1 , 401 Rifkin, ) . 81 recursive self-improvement, AI 323, 325 The Making of the Atomic Bomb red dwarf stars 38 Rigby, E. et a!. 2 3 3 red giant S u n

Architects of Intelligence

by Martin Ford · 16 Nov 2018 · 586pp · 186,548 words

a true existential threat, something that’s been raised by Nick Bostrom, Elon Musk, and Stephen Hawking, where super intelligence could happen very rapidly, a recursive self-improvement loop. I’ve heard people say that your AutoML might be one step toward that because you’re using technology to design other machine learning

…

don’t think then that there’s any realistic fear of what people call the “fast takeoff” scenario, where an AGI system goes through a recursive self-improvement cycle and rapidly becomes superintelligent? ANDREW NG: A lot of the hype about superintelligence and exponential growth were based on very naive and very simplistic

…

long-term threats, though? Elon Musk and Nick Bostrom are very concerned about the control problem with AI; the idea that there could be a recursive self-improvement cycle that could lead to an intelligence explosion. You can’t completely discount that, right? GARY MARCUS: I don’t completely discount it, I’m

Our Final Invention: Artificial Intelligence and the End of the Human Era

by James Barrat · 30 Sep 2013 · 294pp · 81,292 words

Busy Child. We’ve touched on some of the remarkable powers AI could have as it achieves and surpasses human intelligence through the process of recursive self-improvement, powers including self-replication, swarming a problem with many versions of itself, super high-speed calculations, running 24/7, mimicking friendliness, playing dead, and more

…

most critical points about Artificial Intelligence is that an Artificial Intelligence might increase in intelligence extremely fast. The obvious reason to suspect this possibility is recursive self-improvement. (Good 1965.) The AI becomes smarter, including becoming smarter at the task of writing the internal cognitive functions of an AI, so the AI can

…

potential for dangerous behavior than a group of intelligent people. Potential AGI danger lies in the hard kernel of the Busy Child scenario, the rapid recursive self-improvement that enables an AI to bootstrap itself from artificial general intelligence to artificial superintelligence. It’s commonly called the “intelligence explosion.” A self-aware, self

…

a look at how Moore’s law may apply to the intelligence explosion. If we assume AGI can be attained, Moore’s law implies the recursive self-improvement of an intelligence explosion may not even be necessary to achieve ASI, or superhuman intelligence. That’s because once you’ve achieved AGI, less than

More Everything Forever: AI Overlords, Space Empires, and Silicon Valley's Crusade to Control the Fate of Humanity

by Adam Becker · 14 Jun 2025 · 381pp · 119,533 words

that direction.” And, he notes, existing AI systems bear this out. “When we look at where AI is actually succeeding, it’s not in complex, recursively self-improving algorithms. It’s the result of pouring absolutely massive amounts of data into relatively simple neural networks,” he says. “The constructs we use in AI