artificial superintelligence Concepts

Rule of the Robots: How Artificial Intelligence Will Transform Everything

by Martin Ford · 13 Sep 2021 · 288pp · 86,995 words

Russell and physicists Max Tegmark and Frank Wilczek co-authored an open letter published in the U.K.’s Independent declaring that the advent of artificial superintelligence “would be the biggest event in human history,” and that a computer with superhuman intellectual capability might be capable of “outsmarting financial markets, out-inventing

The Big Nine: How the Tech Titans and Their Thinking Machines Could Warp Humanity

by Amy Webb · 5 Mar 2019 · 340pp · 97,723 words

2 The Insular World of AI’s Tribes 3 A Thousand Paper Cuts: AI’s Unintended Consequences Part II: Our Futures 4 From Here to Artificial Superintelligence: The Warning Signs 5 Thriving in the Third Age of Computing: The Optimistic Scenario 6 Learning to Live with Millions of Paper Cuts: The Pragmatic

…

optimistic to pragmatic and catastrophic, and they will reveal both opportunity and risk as we advance from artificial narrow intelligence to artificial general intelligence to artificial superintelligence. These scenarios are intense—they are the result of data-driven models, and they will give you a visceral glimpse at how AI might evolve

…

, you surrender your will. You give it to him in utter submission, in full renunciation.” —FEODOR DOSTOYEVSKY, THE BROTHERS KARAMAZOV CHAPTER FOUR FROM HERE TO ARTIFICIAL SUPERINTELLIGENCE: THE WARNING SIGNS The evolution of artificial intelligence, from robust systems capable of completing narrow tasks to general thinking machines, is now underway. At this

…

need for direct human involvement. Artificial intelligence is typically defined using three broad categories: artificial narrow or weak intelligence (ANI), artificial general intelligence (AGI), and artificial superintelligence (ASI). The Big Nine are currently moving swiftly toward building and deploying AGI systems, which they hope will someday be able to reason, solve problems

…

to things like better medical diagnoses and new ways to solve tough engineering problems. Improvements to AGI should, eventually, bring us to the third category: artificial superintelligence. ASI systems range from being slightly more capable at performing human cognitive tasks than we are to AIs that are literally trillions of times generally

…

. Good, begins in the late 2060s. It’s becoming clear now that our AGIs are gaining profound levels of intelligence, speed, and power and that artificial superintelligence is a near-term possibility. For the past decade, the Big Nine and GAIA have been preparing for this event—and it has calculated that

…

://www.cnbc.com/2016/05/10/baidu-ceo-tells-staff-to-put-values-before-profit-after-cancer-death-scandal.html. CHAPTER 4: FROM HERE TO ARTIFICIAL SUPERINTELLIGENCE: THE WARNING SIGNS 1. I modeled the scenarios in Part II using research from a variety of sources, and their references are in the bibliography

…

), 53, 131, 143, 150, 158, 169; commercial applications, 53; sensory computation in optimistic scenario of future, 160 Artificial neural networks (ANNs), first, 32–33, 34 Artificial superintelligence (ASI), 143, 144, 147–148, 150, 177; China and catastrophic scenario of future, 229; China and pragmatic scenario of future, 206 Asimov, Isaac, 26, 236

What to Think About Machines That Think: Today's Leading Thinkers on the Age of Machine Intelligence

by John Brockman · 5 Oct 2015 · 481pp · 125,946 words

, sensor, and algorithmic technologies and it’s clear that today’s narrow AIs are on a trajectory toward a world of robust AI. Long before artificial superintelligences arrive, evolving AIs will be pressed into performing once unthinkable tasks, from firing weapons to formulating policy. Meanwhile, today’s primitive AIs tell us much

…

own. Only minds that comprehend cause and effect conjure up motives. So if goals, wants, values are features of human minds, then why predict that artificial superintelligences will become more than tools in the hands of those who program in those preferences? If the welter of prognostications about AI and machine learning

If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All

by Eliezer Yudkowsky and Nate Soares · 15 Sep 2025 · 215pp · 64,699 words

puzzles and invent new technologies, and to plan and strategize and plot, and to reflect on and improve itself. We might call AI like that “artificial superintelligence” (ASI), once it exceeds every human at almost every mental task. AI isn’t there yet. But AIs are smarter today than they were in

…

would have told you that ChatGPT-level artificial conversation wouldn’t be in reach for another thirty or fifty years. We didn’t know when artificial superintelligence would arrive, but we agreed it should be a global priority. In fact, we think the open letter drastically undersells the issue. We were invited

…

. More recently, as AI has begun to take off, we watched with concern as some of the newer people starting AI companies began talking about artificial superintelligence as a source of vast, wonderful powers. Powers that they assumed they’d control. The main danger, according to many of these founders, was that

…

focus to conveying one single point, the warning at the core of this book: If any company or group, anywhere on the planet, builds an artificial superintelligence using anything remotely like current techniques, based on anything remotely like the present understanding of AI, then everyone, everywhere on Earth, will die. We do

…

they had believed life would go back to normal before matters went too far. Once upon a time, humanity was on the brink of creating artificial superintelligence… Normality always ends. This is not to say that it’s inevitably replaced by something worse; sometimes it is and sometimes it isn’t, and

…

of death. This book is not full of great news, we admit. But we’re not here to tell you that you’re doomed, either. Artificial superintelligence doesn’t exist yet. Humanity could still decide not to build it. In the 1950s, many people expected that there would be a nuclear war

…

about brains, because you’ll understand every event that goes on inside the neurons. WOMAN: Let’s change the subject. BEFORE WE CAN EXPLAIN WHY ARTIFICIAL SUPERINTELLIGENCE achieved using anything like modern methods would inevitably go wrong, we need to quickly survey those modern methods: how they work, what they produce, and

…

giving you 0.2 percent of their wealth, not the same way you rationalize reasons they should want to. In much the same way, an artificial superintelligence will not want to find reasons to keep humanity around—not in the same way that humans desperately want to find reasons to be kept

…

the sort of technology we’d have unlocked by the year 3000, if our civilization survived that long? And how long would it take an artificial superintelligence? A thousand years of thinking takes about a month to something running at 10,000 times the speed of humans. If it was running at

…

naked in the savannah, and figured out how to exploit reality and compound advantages until they were building guns and nuclear weapons and supercomputers. An artificial superintelligence would be even more resourceful, at even greater speeds. It would have no limits but the laws of physics. In a sense, the scenario we

…

for a long time. IfAnyoneBuildsIt.com/ii PART III FACING THE CHALLENGE CHAPTER 10 A CURSED PROBLEM THE GREATEST AND MOST CENTRAL DIFFICULTY IN ALIGNING artificial superintelligence is navigating the gap between before and after. Before, the AI is not powerful enough to kill us all, nor capable enough to resist our

…

attempts to change its goals. After, the artificial superintelligence must never try to kill us, because it would succeed. Engineers must align the AI before, while it is small and weak, and can’t

…

that civilization could do a little better next time. They risked and harmed only themselves, and all humanity benefited. When it comes to aligning an artificial superintelligence (ASI), humanity will not have the luxury of learning from sufficiently bad mistakes. This also means we can’t rely on the luxury of experience

…

off its coolant water and then overheating more, leave little room for error. And nuclear engineers don’t even have it that bad, compared to artificial superintelligence developers. Nuclear reactors that get too hot don’t start intelligently redesigning themselves to increase their own reactivity rate. Overheating nuclear reactors don’t start

…

reactors. Computer security. What do all these lessons add up to, and what can we learn from them about the difficulty of aligning an artificial superintelligence? An artificial superintelligence is like a space probe, in that we cannot test it in quite the same environment where it needs to work, and by default it

…

contrivance fails. And ASI alignment has it even worse than space probes: Failure will destroy not just billions of dollars of investment, but everything. An artificial superintelligence is like a nuclear reactor, in that its underlying reality involves immense, potentially self-amplifying forces, whose inner processes run faster than humans can react

…

. An artificial superintelligence is like a computer security problem, in that every constraint an engineer tries to place upon the system might be bypassed by the intelligent forces

…

nuclear engineering. It was by instructing their workers to lick radium-coated paintbrushes. What level of game is humanity bringing to the task of shaping artificial superintelligence? Elon Musk, the head of a major AI lab named xAI, shared his plan for ASI alignment in a 2023 interview: I’m going to

…

, but we think that in a substantial fraction of cases it is real. Unfortunately, even a very sincere idealism isn’t enough to prevent an artificial superintelligence from killing us all. That would take a mature science. It’s normal for a scientific community to be overly optimistic in the early days

…

a disadvantage if you agree to stop climbing the AI escalation ladder. We have already mentioned that Rishi Sunak acknowledged the existence of risks from artificial superintelligence in October 2023, while he was the prime minister of the United Kingdom. Also in October 2023, Chinese General Secretary Xi Jinping gave (what seems

…

should be regulated as a dangerous and powerful technology. A 2025 poll found that 60 percent of surveyed U.K. voters support laws against creating artificial superintelligence, and 63 percent support the prohibition of AIs that can make smarter AIs. This issue is not yet at the top of voters’ minds, but

…

fully persuaded: We do not like retreating to maybes. We think our argument stands on its merits. We think it is an easy call that artificial superintelligence will not dutifully serve the people who created it, and that ASI will repurpose the Earth in a fashion that leaves no survivors. But we

…

even just to your friends and family. If many people in many countries say with one voice that they’d rather not die to an artificial superintelligence and would prefer an international treaty—well, that would not itself prevent the disaster. Preventing nuclear war was more complicated than lots of people being

The AGI Chronicles

by Kevin Roose · 481pp · 151,124 words

the alignment problem in time to avoid a rogue AI takeover—was naively optimistic: If any company or group, anywhere on the planet, builds an artificial superintelligence using anything remotely like current techniques, based on anything remotely like the present understanding of A.I., then everyone, everywhere on Earth, will die. Yudkowsky

…

interested in science, and on the frontier discovering this big secret, and honestly it felt to us like we were just going to build this artificial superintelligence and then it was going to go discover all of this amazing science and these amazing things. And that’s now happening. But in my

Superintelligence: Paths, Dangers, Strategies

by Nick Bostrom · 3 Jun 2014 · 574pp · 164,509 words

an extrapolation base encompassing all humans. A further clarification: The formulation is not intended to necessarily exclude the possibly of post-transition property rights in artificial superintelligences or their constituent algorithms and data structures. The formulation is meant to be agnostic about what legal or political systems would best serve to organize

To Be a Machine: Adventures Among Cyborgs, Utopians, Hackers, and the Futurists Solving the Modest Problem of Death

by Mark O'Connell · 28 Feb 2017 · 252pp · 79,452 words

, too, in Elon Musk’s and Bill Gates’s and Stephen Hawking’s increasingly vehement warnings about the prospect of our species’ annihilation by an artificial superintelligence, not to mention in Google’s instatement of Ray Kurzweil, the high priest of the Technological Singularity, as its director of engineering. I saw the

…

-clip-manufacturing facilities. The scenario was deliberately cartoonish, but as an example of the kind of ruthless logic we might be up against with an artificial superintelligence, its intent was entirely serious. “I wouldn’t describe myself these days as a transhumanist,” Nick told me one evening over dinner at an Indian

…

us. “I still think,” he said, “that within a few generations it will be possible to transform the substrate of our humanity. And I think artificial superintelligence will be the engine that drives that.” Like many transhumanists, Nick was fond of pointing out the vast disparity in processing power between human tissue

…

of the human cranium, where it is technically possible to build computer processors the size of skyscrapers. Such factors, he maintained, created the conditions for artificial superintelligence. And because of our tendency to conceive of intelligence within human parameters, we were likely to become complacent about the speed with which machine intelligence

…

human being is on this quantity of meat.” We were talking, Nate Soares and I, about the benefits that might come with the advent of artificial superintelligence. For Nate, the most immediate benefit would be the ability to run a human being—to run, specifically, himself—on something other than this quantity

…

begun to think of as magical rationalism. He spoke, now, of the great benefits that would come, all things being equal, with the advent of artificial superintelligence. By developing such a transformative technology, he said, we would essentially be delegating all future innovations—all scientific and technological progress—to the machine. These

…

claims were more or less standard among those in the tech world who believed that artificial superintelligence was a possibility. The problem-solving power of such a technology, properly harnessed, would lead to an enormous acceleration in the turnover of solutions and

…

were working toward now, for all its vast complexity and remoteness, could be achieved in the course of a long weekend by the kind of artificial superintelligence Nate was talking about.) It was easy to forget, Nate continued, that as we sat here talking we were in fact using nanotech protein computers

…

there was nothing special about carbon. And so the best-case scenario of the Singularitarians, the version of the future in which we merge with artificial superintelligence and become immortal machines, was, for me, no more appealing, in fact maybe even less appealing, than the worst-case scenario, in which

…

artificial superintelligence destroyed us all. And it was this latter scenario, the failure mode as opposed to the God mode, that I had come to learn about,

…

his colleagues—at MIRI, at the Future of Humanity Institute, at the Future of Life Institute—were working to prevent was the creation of an artificial superintelligence that viewed us, its creators, as raw material that could be reconfigured into some more useful form (not necessarily paper clips). And the way Nate

…

artificer, is the symbol and spirit of this understanding of ourselves, of our ambitions—the shadow of the waxwing slain, cast darkly across history, plummeting. Artificial superintelligence, I was repeatedly told, was a dangerous prospect precisely because of how unlike us it would be, how inhuman, how immune to anger and hatred

…

, whose production line was almost entirely roboticized, and whose CEO, Elon Musk—the same Elon Musk who was so publicly terrified by the prospect of artificial superintelligence—had recently announced the company’s plans to develop its own self-driving system within three to five years. Although I had not beheld him

…

mostly online community of biohackers, or “practical transhumanists.” These are people who don’t want to wait around for the Singularity to happen, or for artificial superintelligence to finally materialize and subsume the informational content of their human minds, their wetware. With the means currently at hand, they are doing what they

Our Final Invention: Artificial Intelligence and the End of the Human Era

by James Barrat · 30 Sep 2013 · 294pp · 81,292 words

still improving. The scientists have passed a historic milestone! For the first time humankind is in the presence of an intelligence greater than its own. Artificial superintelligence, or ASI. Now what happens? AI theorists propose it is possible to determine what an AI’s fundamental drives will be. That’s because once

…

called AGI. Shortly after that, someone (or some thing) will create an AI that is smarter than humans, often called artificial superintelligence. Suddenly we may find a thousand or ten thousand artificial superintelligences—all hundreds or thousands of times smarter than humans—hard at work on the problem of how to make themselves better

…

at making artificial superintelligences. We may also find that machine generations or iterations take seconds to reach maturity, not eighteen years as we humans do. I. J. Good, an

…

it love you, but you are made out of atoms which it can use for something else. —Eliezer Yudkowsky, research fellow, Machine Intelligence Research Institute Artificial superintelligence does not yet exist, nor does artificial general intelligence, the kind that can learn like we do and will in many senses match and exceed

…

mortality, will be solved. Artificial intelligence is the star of the Singularity media spectacle, but nanotechnology plays an important supporting role. Many experts predict that artificial superintelligence will put nanotechnology on the fast track by finding solutions for seemingly intractable problems with nanotech’s development. Some think it would be better if

…

beings run into less advanced ones: Christopher Columbus versus the Tiano, Pizzaro versus the Inca, Europeans versus Native Americans. Get ready for the next one. Artificial superintelligence versus you and me. * * * Perhaps technology thinkers have considered AI’s downside, but believe it’s too unlikely to worry about. Or they get it

…

is wrong for a lot of reasons. The assumption becomes even more dangerous after the AGI’s intelligence rockets past ours, and it becomes ASI—artificial superintelligence. So how do you create friendly AI? Or could you impose friendliness on advanced AIs after they’re already built? Yudkowsky has written a book

…

a problem with many versions of itself, super high-speed calculations, running 24/7, mimicking friendliness, playing dead, and more. We’ve proposed that an artificial superintelligence won’t be satisfied with remaining isolated; its drives and intelligence would thrust it into our world and put our existence at risk. But why

…

’re busy avoiding risks of unintended consequences from AI, AI will be scrutinizing humans for dangerous consequences of sharing the world with us. Consider an artificial superintelligence a thousand times more intelligent than the smartest human. As we noted in chapter 1, nuclear weapons are our own species’ most destructive invention. What

…

the hard kernel of the Busy Child scenario, the rapid recursive self-improvement that enables an AI to bootstrap itself from artificial general intelligence to artificial superintelligence. It’s commonly called the “intelligence explosion.” A self-aware, self-improving system will seek to better fulfill its goals, and minimize vulnerabilities, by improving

…

if this could be done then, at double the cost, the machine could exhibit ultraintelligence. So, for a few dollars more you can get ASI, artificial superintelligence, Good proposes. But then watch out for the civilization-wide ramifications of sharing the planet with smarter than human intelligence. In 1962, before he’d

…

on risks of, see risks of artificial intelligence Singularity and, see Singularity tight coupling in utility function of virtual environments for artificial neural networks (ANNs) artificial superintelligence (ASI) anthropomorphizing gradualist view of dealing with jump from AGI to; see also intelligence explosion morality of nanotechnology and runaway Artilect War, The (de Garis

…

) ASI, see artificial superintelligence Asilomar Guidelines ASIMO Asimov, Isaac: Three Laws of Robotics of Zeroth Law of Association for the Advancement of Artificial Intelligence (AAAI) asteroids Atkins, Brian and

On the Edge: The Art of Risking Everything

by Nate Silver · 12 Aug 2024 · 848pp · 227,015 words

strategic choices. AGI: Artificial general intelligence. The term lacks a clear definition but refers to at least broad human-level intelligence, sometimes as distinguished from artificial superintelligence (ASI), which surpasses that of humans. AI: See: artificial intelligence. AK: Ace-king, the best starting hand in Hold’em apart from a pocket pair

…

, 236, 263, 485 Archipelago, The, 22, 310, 478 arms race, 478 See also mutually assured destruction; nuclear existential risk art world, 329–30, 331n ASI (artificial superintelligence), 478 Asian Americans, 135–36, 513n See also race asymmetric odds, 248–49, 255, 259, 260–62, 276, 277 attack surfaces, 177, 187, 478 attention

Rationality: From AI to Zombies

by Eliezer Yudkowsky · 11 Mar 2015 · 1,737pp · 491,616 words

you do. Only in my case, it’s people who know the amazingly simple utility function that is all you need to program into an artificial superintelligence and then everything will turn out fine. Some people, when they encounter the how-to-program-a-superintelligence problem, try to solve the problem immediately