The Rationalist's Guide to the Galaxy: Superintelligent AI and the Geeks Who Are Trying to Save Humanity's Future
by
Tom Chivers
Published 12 Jun 2019
In 2014, 31 per cent of survey respondents said that they had first heard of the movement through LessWrong;5 by 2017, that figure had dropped to 15 per cent, presumably partly because LessWrong had shrunk while Effective Altruism had grown, but a further 7 per cent had heard of it through Slate Star Codex.6 And a large fraction of LessWrongers are Effective Altruists: according to the 2016 LessWrong diaspora survey, 20 per cent of respondents identified as Effective Altruists, and 22 per cent had made ‘donations they otherwise wouldn’t’ because of Effective Altruism.7 Two other OpenPhil employees, Helen Toner and Ajeya Cotra – whom you’ll meet shortly – told me that they’d either come to Effective Altruism through LessWrong or found the two at the same time. One Effective Altruist blog, ‘The Unit of Caring’, is named after a Yudkowsky blog post (‘Money: The Unit of Caring’). Scott Alexander has written repeatedly about Effective Altruism, including an extremely powerful blog post8 backing the Giving What We Can ‘pledge’ to donate 10 per cent of your income to effective charities.
…
I can report that it tastes very much like a burger, indeed a perfectly nice burger, and appears to be taking off; it is sold at hundreds of locations in the US and one of its rivals has, since I went over there, arrived in Britain. (Specifically, and unsurprisingly, Dalston in east London.13) But the OpenPhil work that is most relevant to this book is its focus on global catastrophic risks, and especially AI. Chapter 39 EA and AI Ajeya Cotra is a research analyst for OpenPhil. She and her colleague Helen Toner, an Australian, both work on AI risk specifically, and both say that it is hard to explain to people what you do for a living. ‘When I was in high school, I discovered GiveWell and Effective Altruism, and I also discovered LessWrong and the Rationalist community,’ Cotra tells me.
…
Now it’s the subject of whole departments at several major universities, and discussed by major leading intellectuals: Martin Rees, the late Stephen Hawking, Bill Gates. Google’s DeepMind is explicitly worried about it; co-founders Shane Legg and Demis Hassabis both take it seriously. ‘It’s been amazingly quick progress,’ commented Ajeya Cotra, of OpenPhil. ‘I think the discourse around technical safety of these in 2014, compared to now, feels like different worlds.’ Bostrom’s book Superintelligence was a turning point, she said. It was a New York Times bestseller, and brought serious academic heft to the field. ‘In 2014, we had a small group of outsider futurists trying to convince the AI community that this is a risk to be taken seriously.
What We Owe the Future: A Million-Year View
by
William MacAskill
Published 31 Aug 2022
The attempt to lock in values through AGI would run a grave risk of an irrecoverable loss of control to the AGI systems themselves, which, if misaligned and uncontrolled, would kill the AGI’s developers as well as everyone else. This is the risk I now turn to. AI Takeover If we build AGI, it will likely not be long before AI systems far surpass human abilities across all domains, just as current AI systems far outperform humans at chess and Go. And this poses a major challenge. To borrow an analogy from Ajeya Cotra, a researcher at Open Philanthropy, think of a child who has just become the ruler of a country.78 The child can’t run the country themselves, so they need to appoint an adult to do so in their place. Their aim would be to find an adult who will act in accordance with their wishes. The challenge is for the child to do this—rather than, say, appointing a schemer who is good at deceitful salesmanship but once in power would pursue their own agenda—even though the adults are much smarter and more knowledgeable than the child is.
…
But it might come soon—within the next fifty or even twenty years. Figure 4.1. Global solar capacity has outpaced all projections by the International Energy Agency since 2006. Graph shows capacity growth per year (rather than cumulative total). The most weighty evidence for this is marshalled by Ajeya Cotra. Her report forecasts trends in computing power over time and compares those trends to the computing power of the brains of biological creatures and the amount of learning they require to attain their abilities.88 Using what we know from current neuroscience, today’s AI systems are about as powerful as insect brains, and even the very largest models are less than 1 percent as powerful as human brains.89 In the future, this will change.
On the Edge: The Art of Risking Everything
by
Nate Silver
Published 12 Aug 2024
Not necessarily…every single human being wiped out. And maybe not even necessarily involving humans having no seat at the table or no power at all. But something where humans are kept in check. And the people making the big calls about what happens are a coalition of AI systems.” —As told to me by Ajeya Cotra, an AI researcher at Open Philanthropy These definitions make a big difference. Cotra, for instance, has a p(doom) of 20 to 30 percent. “You know, in my circles I’m considered a moderate,” she said. Outside of her circles, that number might alarm people. But it doesn’t seem so extreme if you consider her definition.
…
I’m going to close with a quick summary of what I think are the best arguments for and against AI risk. Then in the final chapter, 1776, I’ll zoom out to consider the shape of things to come—the moment our civilization finds itself in—and propose some principles to guide us through the next decades and hopefully far beyond. The Steelman Case for a High p(doom) When I asked Ajeya Cotra for her capsule summary for why we should be concerned about AI risk, she gave me a pithy answer. “If you were to tell a normal person, ‘Hey, AI companies are racing as fast as possible to build a machine that is better than a human at all tasks, and to bring forward a new intelligent species that can do everything we can do and more, better than we can’—people would react to that with fear if they believed it,” she told me.
The Alignment Problem: Machine Learning and Human Values
by
Brian Christian
Published 5 Oct 2020
For an overview of this family of positions, see, e.g., Sayre-McCord, “Moral Realism.” 97. Paul Christiano, interviewed by Rob Wiblin, The 80,000 Hours Podcast, October 2, 2018. 98. See Paul Christiano, “A Formalization of Indirect Normativity,” AI Alignment (blog), April 20, 2012, https://ai-alignment.com/a-formalization-of-indirect-normativity-7e44db640160, and Ajeya Cotra, “Iterated Distillation and Amplification,” AI Alignment (blog), March 4, 2018, https://ai-alignment.com/iterated-distillation-and-amplification-157debfd1616. 99. For an explicit discussion of the connection between AlphaGo’s policy network and the idea of iterated capability amplification, see Paul Christiano, “AlphaGo Zero and Capability Amplification,” AI Alignment (blog), October 19, 2017, https://ai-alignment.com/alphago-zero-and-capability-amplification-ede767bb8446. 100.