What We Owe the Future: A Million-Year View
by
William MacAskill
Published 31 Aug 2022
The attempt to lock in values through AGI would run a grave risk of an irrecoverable loss of control to the AGI systems themselves, which, if misaligned and uncontrolled, would kill the AGI’s developers as well as everyone else. This is the risk I now turn to. AI Takeover If we build AGI, it will likely not be long before AI systems far surpass human abilities across all domains, just as current AI systems far outperform humans at chess and Go. And this poses a major challenge. To borrow an analogy from Ajeya Cotra, a researcher at Open Philanthropy, think of a child who has just become the ruler of a country.78 The child can’t run the country themselves, so they need to appoint an adult to do so in their place. Their aim would be to find an adult who will act in accordance with their wishes. The challenge is for the child to do this—rather than, say, appointing a schemer who is good at deceitful salesmanship but once in power would pursue their own agenda—even though the adults are much smarter and more knowledgeable than the child is.
…
But it might come soon—within the next fifty or even twenty years. Figure 4.1. Global solar capacity has outpaced all projections by the International Energy Agency since 2006. Graph shows capacity growth per year (rather than cumulative total). The most weighty evidence for this is marshalled by Ajeya Cotra. Her report forecasts trends in computing power over time and compares those trends to the computing power of the brains of biological creatures and the amount of learning they require to attain their abilities.88 Using what we know from current neuroscience, today’s AI systems are about as powerful as insect brains, and even the very largest models are less than 1 percent as powerful as human brains.89 In the future, this will change.
On the Edge: The Art of Risking Everything
by
Nate Silver
Published 12 Aug 2024
Not necessarily…every single human being wiped out. And maybe not even necessarily involving humans having no seat at the table or no power at all. But something where humans are kept in check. And the people making the big calls about what happens are a coalition of AI systems.” —As told to me by Ajeya Cotra, an AI researcher at Open Philanthropy These definitions make a big difference. Cotra, for instance, has a p(doom) of 20 to 30 percent. “You know, in my circles I’m considered a moderate,” she said. Outside of her circles, that number might alarm people. But it doesn’t seem so extreme if you consider her definition.
…
I’m going to close with a quick summary of what I think are the best arguments for and against AI risk. Then in the final chapter, 1776, I’ll zoom out to consider the shape of things to come—the moment our civilization finds itself in—and propose some principles to guide us through the next decades and hopefully far beyond. The Steelman Case for a High p(doom) When I asked Ajeya Cotra for her capsule summary for why we should be concerned about AI risk, she gave me a pithy answer. “If you were to tell a normal person, ‘Hey, AI companies are racing as fast as possible to build a machine that is better than a human at all tasks, and to bring forward a new intelligent species that can do everything we can do and more, better than we can’—people would react to that with fear if they believed it,” she told me.
The Alignment Problem: Machine Learning and Human Values
by
Brian Christian
Published 5 Oct 2020
For an overview of this family of positions, see, e.g., Sayre-McCord, “Moral Realism.” 97. Paul Christiano, interviewed by Rob Wiblin, The 80,000 Hours Podcast, October 2, 2018. 98. See Paul Christiano, “A Formalization of Indirect Normativity,” AI Alignment (blog), April 20, 2012, https://ai-alignment.com/a-formalization-of-indirect-normativity-7e44db640160, and Ajeya Cotra, “Iterated Distillation and Amplification,” AI Alignment (blog), March 4, 2018, https://ai-alignment.com/iterated-distillation-and-amplification-157debfd1616. 99. For an explicit discussion of the connection between AlphaGo’s policy network and the idea of iterated capability amplification, see Paul Christiano, “AlphaGo Zero and Capability Amplification,” AI Alignment (blog), October 19, 2017, https://ai-alignment.com/alphago-zero-and-capability-amplification-ede767bb8446. 100.