friendly AI

Our Final Invention: Artificial Intelligence and the End of the Human Era
by James Barrat
Published 30 Sep 2013

But if an intelligence explosion kicks in and Yudkowsky has been unsuccessful in implementing safeguards, there’s a good chance we’ll all be goo, and so will the universe. That puts Yudkowsky at the dead center of his own cosmology. I had come here to learn more about Friendly AI, a term he coined. According to Yudkowsky, Friendly AI is the kind that will preserve humanity and our values forever. It doesn’t annihilate our species or spread into the universe like a planet-eating space plague. But what is Friendly AI? How do you create it? I also wanted to hear about the AI Box Experiment. I especially wanted to know, as he role-played the part of the AGI, how he talked the Gatekeeper into setting him free.

…

[italics mine] Not knowing how to build a Friendly AI is not deadly, of itself.… It’s the mistaken belief that an AI will be friendly which implies an obvious path to global catastrophe. Assuming that human-level AIs (AGIs) will be friendly is wrong for a lot of reasons. The assumption becomes even more dangerous after the AGI’s intelligence rockets past ours, and it becomes ASI—artificial superintelligence. So how do you create friendly AI? Or could you impose friendliness on advanced AIs after they’re already built? Yudkowsky has written a book-length online treatise about these questions entitled Creating Friendly AI: The Analysis and Design of Benevolent Goal Architectures.

…

Yudkowsky has written a book-length online treatise about these questions entitled Creating Friendly AI: The Analysis and Design of Benevolent Goal Architectures. Friendly AI is a subject so dense yet important it exasperates its chief proponent himself, who says about it, “it only takes one error for a chain of reasoning to end up in Outer Mongolia.” Let’s start with a simple definition. Friendly AI is AI that has a positive rather than a negative impact on mankind. Friendly AI pursues goals, and it takes action to fulfill those goals. To describe an AI’s success at achieving its goals, theorists use a term from economics: utility. As you might recall from Econ 101, consumers behaving rationally seek to maximize utility by spending their resources in the way that gives them the most satisfaction.

Surviving AI: The Promise and Peril of Artificial Intelligence
by Calum Chace
Published 28 Jul 2015

A superintelligence could become not just twice as smart as humans, but smarter by many orders of magnitude. It is hard to escape the conclusion that our future will depend on its decisions and its actions. Would that be a good thing or a bad thing? In other words, would a superintelligence be a “Friendly AI”? (Friendly AI, or FAI, denotes an AGI that is beneficial for humans rather than one that seeks social approbation and company. It also refers to the project to make sure that AGI is beneficial.) 7.2 – Optimistic scenarios: to immortality and beyond The ultimate problem solver Imagine having a big sister endowed with superhuman wisdom, insight and ingenuity.

…

So if we take seriously the idea that a superintelligence may appear on the Earth in the foreseeable future, we should certainly be thinking about how to ensure that the event is a positive one for ourselves and our descendants. We should be taking steps to ensure that the first AGI is a friendly AI. PART FOUR: FAI Friendly Artificial Intelligence CHAPTER 8 CAN WE ENSURE THAT SUPERINTELLIGENCE IS SAFE? As we saw in the last chapter, Friendly AI (FAI) is the project of ensuring that the world’s superintelligences are safe and useful for humans. The central argument of this book is that we need to address this challenge successfully. It may well turn out to be the most important challenge facing this generation and the next.

…

It also boasts some Hollywood glamour, with Alan Alda and Morgan Freeman on the advisory board, along with technology entrepreneur Elon Musk, who has donated $10m of his personal money to the institute. 8.6 – Conclusion We do not yet have a foolproof way to ensure that the first AGI is a Friendly AI. In fact we don’t yet know how best to approach the problem. But we have only just begun, and the resources allocated to the problem are small: Nick Bostrom estimated in 2014 that only six people in the world are working full-time on the Friendly AI problem, whereas many thousands of people work full-time on projects that could well contribute to the creation of the first AGI. (50) He argued that this equation needed urgent re-balancing.

Blockchain: Blueprint for a New Economy
by Melanie Swan
Published 22 Jan 2014

Limitations Technical Challenges Business Model Challenges Scandals and Public Perception Government Regulation Privacy Challenges for Personal Records Overall: Decentralization Trends Likely to Persist 7. Conclusion The Blockchain Is an Information Technology Blockchain AI: Consensus as the Mechanism to Foster “Friendly” AI Large Possibility Space for Intelligence Only Friendly AIs Are Able to Get Their Transactions Executed Smart Contract Advocates on Behalf of Digital Intelligence Blockchain Consensus Increases the Information Resolution of the Universe A. Cryptocurrency Basics Public/Private-Key Cryptography 101 B. Ledra Capital Mega Master Blockchain List Endnotes and References Index Blockchain Blueprint for a New Economy Melanie Swan Blockchain by Melanie Swan Copyright © 2015 Melanie Swan.

…

The blockchain is a consensus model at scale, and possibly the mechanism we have been waiting for that could help to usher in an era of friendly machine intelligence. Blockchain AI: Consensus as the Mechanism to Foster “Friendly” AI One forward-looking but important concern in the general future of technology is different ways in which artificial intelligence (AI) might arise and how to sponsor it such that it engenders a “friendly” or benevolent relationship with humans. There is the notion of a technological singularity, a moment when machine intelligence might supersede human intelligence. However, those in the field have not set forth any sort of robust plan for how to effect friendly AI, and many remain skeptical of this possibility.195 It is possible that blockchain technology could be a useful connector of humans and machines in a world of increasingly autonomous machine activity through Dapps, DAOs, and DACs that might eventually give way to AI.

…

These intelligences would likely not be operating in isolation, but would be connected to communications networks. To achieve their goals, digital intelligences will want to conduct certain transactions over the network, many of which could be managed by blockchain and other consensus mechanisms. Only Friendly AIs Are Able to Get Their Transactions Executed One of the unforeseen benefits of consensus models might be that they could possibly enforce friendly AI, which is to say cooperative, moral players within a society.196 In decentralized trust networks, an agentÕs reputation (where agents themselves remain pseudonymous) could be an important factor in whether his transactions will be executed, such that malicious players would not be able to get their transactions executed or recognized on the network.

Global Catastrophic Risks
by Nick Bostrom and Milan M. Cirkovic
Published 2 Jul 2008

Artificial Intelligence in global risk 335 We can also imagine a scenario that implies a n easy local strategy: • The first AI cannot by itself do catastrophic damage. • If even a single Friendly AI exists, that AI plus human institutions can fend off any number of unFriendly A Is. The easy scenario would hold if, for example, human institutions can reliably distinguish Friendly Ais from unFriendly ones, and give revocable power into the hands of Friendly Ais. Thus we could pick and choose our allies. The only requirement is that the Friendly AI problem must be solvable (as opposed to being completely beyond human ability) . Both of the above scenarios assume that the first AI (the first powerful, general AI) cannot by itself do global catastrophic damage.

…

In more colloquial terms, it would be nice if we knew how to build a nice AI. To describe the field of knowledge needed to address that challenge, I have proposed the term 'Friendly AI'. In addition to referring to a body oftechnique, ' Friendly AI' might also refer to the product of technique - an A I created with specified motivations. When I use the term Friendly in either sense, I capitalize it to avoid confusion with the intuitive sense of 'friendly'. One common reaction I encounter is for people to immediately declare that Friendly AI is an impossibility because any sufficiently powerful AI will be able to modify its own source code to break any constraints placed upon it.

…

If huge nanocomputers make it easier to develop AI without making it easier to solve the particular challenge of Friendliness, that is a negative interaction. Thus, all else being equal, I would greatly prefer that Friendly AI precede nanotechnology in the ordering of technological developments. If we confront the challenge of AI and succeed, we can call on Friendly AI to help us with nanotechnology. If we develop nanotechnology and survive, we still have the challenge of AI to deal with after that. Generally speaking, a success on Friendly AI should help solve nearly any other problem. Thus, if a technology makes AI neither easier nor harder, but carries with it a catastrophic risk, we should prefer all else being equal to first confront the challenge of AI.

Singularity Rising: Surviving and Thriving in a Smarter, Richer, and More Dangerous World
by James D. Miller
Published 14 Jun 2012

The Singularity Institute works toward the only goal I consider worthy of charitable dollars: increasing the survival prospects of mankind. Anna Salamon of the Singularity Institute did a credible back-of-the-envelope calculation showing that, based on some reasonable estimates of the effectiveness of friendly AI research and the harm of an unfriendly Singularity, donating one dollar to research on friendly AI will on average save over one life because slightly decreasing the odds that the seven billion current inhabitants of Earth will die yields you a huge average expected benefit.105 This expected benefit goes way up if you factor in people who are not yet born.

…

Within a year, we will probably have the technical ability to activate a seed AI, but once the Chinese threat has been annihilated, our team will have no reason to hurry and could take a decade to fine-tune their seed AI. If we delay, any intelligence explosion we eventually create will have an extremely high probability of yielding a friendly AI. Some people on our team think that, given another decade, they will be able to mathematically prove that the seed AI will turn into a friendly ultra-AI. A friendly AI would allow trillions and trillions of people to eventually live their lives, and mankind and our descendants could survive to the end of the universe in utopia. In contrast, an unfriendly AI would destroy us. I have decided to make the survival of humanity my priority.

…

The firm, therefore, wouldn’t be able to raise start-up capital from small, self-interested investors. So now pretend that at the time the firm tries to raise capital there is a well-developed theory of friendly AI, which provides programmers with a framework for creating AI that is extremely likely to be well disposed toward humanity and create a utopia if it undergoes an intelligence explosion. To raise funds from self-interested investors, an AI-building firm would need to pick a research and development path that would make it difficult for the firm ever to use the friendly AI framework. Unfortunately, this means that any intelligence explosion the firm unintentionally brings about would be less likely to be utopian than if the firm had used the friendly framework.

Rationality: From AI to Zombies
by Eliezer Yudkowsky
Published 11 Mar 2015

But compulsion is not a property of arguments; it is a property of minds that process arguments. So the reason I’m arguing against the ghost isn’t just to make the point that (1) Friendly AI has to be explicitly programmed and (2) the laws of physics do not forbid Friendly AI. (Though of course I take a certain interest in establishing this.) I also wish to establish the notion of a mind as a causal, lawful, physical system in which there is no irreducible central ghost that looks over the neurons/code and decides whether they are good suggestions. (There is a concept in Friendly AI of deliberately programming an FAI to review its own source code and possibly hand it back to the programmers.

…

Good’s older term, “intelligence explosion,” to help distinguish his views from other futurist predictions, such as Ray Kurzweil’s exponential technological progress thesis.2 Technologies like smarter-than-human AI seem likely to result in large societal upheavals, for the better or for the worse. Yudkowsky coined the term “Friendly AI theory” to refer to research into techniques for aligning an AGI’s preferences with the preferences of humans. At this point, very little is known about when generally intelligent software might be invented, or what safety approaches would work well in such cases. Present-day autonomous AI can already be quite challenging to verify and validate with much confidence, and many current techniques are not likely to generalize to more intelligent and adaptive systems. “Friendly AI” is therefore closer to a menagerie of basic mathematical and philosophical questions than to a well-specified set of programming objectives.

…

But if you want to know why I might be reluctant to extend the graph of biological and economic growth over time, into the future and over the horizon of an AI that thinks at transistor speeds and invents self-replicating molecular nanofactories and improves its own source code, then there is my reason: you are drawing the wrong graph, and it should be optimization power in versus optimized product out, not optimized product versus time. * 146 Ghosts in the Machine People hear about Friendly AI and say—this is one of the top three initial reactions: “Oh, you can try to tell the AI to be Friendly, but if the AI can modify its own source code, it’ll just remove any constraints you try to place on it.” And where does that decision come from? Does it enter from outside causality, rather than being an effect of a lawful chain of causes that started with the source code as originally written? Is the AI the ultimate source of its own free will? A Friendly AI is not a selfish AI constrained by a special extra conscience module that overrides the AI’s natural impulses and tells it what to do.

Smarter Than Us: The Rise of Machine Intelligence
by Stuart Armstrong
Published 1 Feb 2014

Some effort has been made to make the AI transition safer. Kudos must be given to Eliezer Yudkowsky and Nick Bostrom, who saw and understood the risks early on. Yudkowsky uses the term “Friendly AI” to describe an AI which does what we want even as it improves its own intelligence. In 2000 he cofounded an organization now called the Machine Intelligence Research Institute (MIRI), which holds math research workshops tackling open problems in Friendly AI theory. (MIRI also commissioned and published this book.) Meanwhile, Nick Bostrom founded the Future of Humanity Institute (FHI), a research group within the University of Oxford.

…

Are we not simply training the AI to give the correct answer in training situations? The whole approach is a constraint problem: in the space of possible AI minds, we are going to give priority to those minds that pass successfully through this training process and reassure us that they’re safe. Is there some quantifiable way of measuring how likely this is to produce a human-friendly AI at the end of it? If there isn’t, why are we putting any trust in it? These problems remain barely addressed, so though it is possible to imagine a safe AI being developed using the current approaches (or their descendants), it feels extremely unlikely. Hence we shouldn’t put our trust in the current crop of experts to solve the problem.

Physics of the Future: How Science Will Shape Human Destiny and Our Daily Lives by the Year 2100
by Michio Kaku
Published 15 Mar 2011

But many scientists are leaning toward something called “friendly AI,” where we design our robots to be benign from the very beginning. Since we are the creators of these robots, we will design them, from the very start, to perform only useful and benevolent tasks. The term “friendly AI” was coined by Eliezer Yudkowsky, a founder of the Singularity Institute for Artificial Intelligence. Friendly AI is a bit different from Asimov’s laws, which are forced upon robots, perhaps against their will. (Asimov’s laws, imposed from the outside, could actually invite the robots to devise clever ways to circumvent them.) In friendly AI, by contrast, robots are free to murder and commit mayhem.

…

If this trend continues, then perhaps friendly AI could become a reality. In this scenario, it is the consumer sector and market forces that will eventually dominate robotics, so that there will be a vast commercial interest in investing in friendly AI. MERGING WITH ROBOTS In addition to friendly AI, there is also another option: merging with our creations. Instead of simply waiting for robots to surpass us in intelligence and power, we should try to enhance ourselves, becoming superhuman in the process. Most likely, I believe, the future will proceed with a combination of these two goals, i.e., building friendly AI and also enhancing ourselves.

…

Douglas Hofstadter has said, “It’s as if you took a lot of good food and some dog excrement and blended it all up so that you can’t possibly figure out what’s good or bad. It’s an intimate mixture of rubbish and good ideas, and it’s very hard to disentangle the two, because these are smart people; they’re not stupid.” No one knows how this will play out. But I think the most likely scenario is the following. MOST LIKELY SCENARIO: FRIENDLY AI First, scientists will probably take simple measures to ensure that robots are not dangerous. At the very least, scientists can put a chip in robot brains to automatically shut them off if they have murderous thoughts. In this approach, all intelligent robots will be equipped with a fail-safe mechanism that can be switched on by a human at any time, especially when a robot exhibits errant behavior.

When Computers Can Think: The Artificial Intelligence Singularity
by Anthony Berglas , William Black , Samantha Thalind , Max Scratchmann and Michelle Estes
Published 28 Feb 2015

This means that if the first real AGI can be made friendly, there will be no need to deal with any other AGI that might be developed. It only needs to be done correctly once. Or alternatively, humanity only has one opportunity to get it right. Friendly AGI research The current centre for research into friendly AI is the Machine Intelligence Research Institute (MIRI) in Berkeley, California, which was co-founded by Yudkowsky. They focus on providing mathematical solutions to the problem of producing friendly AI. In particular, they are researching how to formalize the definition of Coherent Extrapolated Volition so that it can be implemented in software. They also offer a series of very technical workshops to select mathematicians.

…

Thus it is only necessary to tame that first program in order to tame all hyper intelligences. There are many issues with this approach, not least of which is the fact that the military funds much of the research into artificial intelligence. They would want their money back if it turned out to be too friendly. The challenges of building a friendly AI will be discussed in detail in part III. Primary assertions and objections This book develops the following assertions:Computers will eventually become truly intelligent, and then become hyperintelligent. A computer based intelligence would have a world view very different from man’s world view.

…

They also offer a series of very technical workshops to select mathematicians. Another research group is the Future of Humanity Institute, in Oxford UK which is lead by Nick Bostrum who wrote the book SuperIntelligence. The institute has 13 staff, but it takes a wider view than just developing friendly AI. The International Conference on Artificial General Intelligence was founded in 2008 and includes some papers that address controlling an AGI. There is also an associated Journal. In total, these organizations are tiny. There are many more people working on the problem of trying to make software more intelligent than there are people working on the problem of making intelligent software safe for humanity.

Warnings
by Richard A. Clarke
Published 10 Apr 2017

The ultimate problem: how to keep humanity from losing control of a machine of its own creation, to prevent artificial intelligence from becoming, in the words of James Barrat in the title of his 2013 book, Our Final Invention.6 A divisive figure, Yudkowsky is well known in academic circles and the Silicon Valley scene as the coiner of the term “friendly AI.” His thesis is simple, though his solution is not: if we are to have any hope against superintelligence, we need to code it properly from the beginning. The answer, Eliezer believes, is one of morality. AI must be programmed with a set of ethical codes that align with humanity’s. Though it is his life’s only work, Yudkowsky is pretty sure he will fail.

…

He envisions this as a system of checks and balances so that advanced AI growth is auditable and controllable, so that even as it continues to learn, advance, and reprogram itself, it will not evolve out of its own benign coding. Such preprogrammed measures will ensure that superintelligence will “behave as we intend even in the absence of immediate human supervision.”12 Eliezer calls this “friendly AI.” According to Yudkowsky, once AI gains the ability to broadly reprogram itself, it will be far too late to implement safeguards, so society needs to prepare now for the intelligence explosion. Yet this preparation is complicated by the sporadic and unpredictable nature of scientific advancement and the numerous secret efforts to create superintelligence around the world.

…

Eliezer told us that humanity’s best hope is to perhaps create one highly funded, highly secure, multilateral effort to develop a friendly superintelligence with himself (or perhaps another futurist he approves of) at the helm. The work of this massive global Manhattan Project would be explicitly “for the benefit of humanity internationally.” It simultaneously would ban, starve, or simply outpace other, less-well-thought-out efforts to develop superintelligence. Once created, this friendly AI would be unleashed to attack and destroy any competing efforts, ensuring that the only superintelligence in existence would help, not destroy, humankind. Yudkowsky rejects the idea that a superintelligence should, or could, be tailored to parochial national security interests, believing instead that any solution must be considered at the human species level.

The Singularity Is Near: When Humans Transcend Biology
by Ray Kurzweil
Published 14 Jul 2005

Yudkowsky formed the Singularity Institute for Artificial Intelligence (SIAI) to develop "Friendly AI," intended to "create cognitive content, design features, and cognitive architectures that result in benevolence" before near-human or better-than-human Als become possible. SIAI has developed The SIAI Guidelines on Friendly AI: "Friendly AI," http://www.singinst.org/friendly/. Ben Goertzel and his Artificial General Intelligence Research Institute have also examined issues related to developing friendly AI; his current focus is on developing the Novamente AI Engine, a set of learning algorithms and architectures.

…

Steve Bowman and Helit Barel, Weapons of Mass Destruction: The Terrorist Threat, Congressional Research Service Report for Congress, December 8, 1999, http://www.cnie.org/nle/crsreports/international/inter-75.pdf. 30. Eliezer S. Yudkowsky, "Creating Friendly AI 1.0, The Analysis and Design of Benevolent Goal Architectures" (2001), The Singularity Institute, http://www.singinst.org/CFAI/; Eliezer S. Yudkowsky, "What Is Friendly AI?" May 3, 2001, http://www.KurzweilAI.net/meme/frame.html?main=/articles/art0172.html. 31. Ted Kaczynski, "The Unabomber's Manifesto," May 14, 2001, http://www.KurzweilAI.netlmeme/frame.html?main=/articles/art0182.html. 32.

…

Intelligence is inherently impossible to control, so the various strategies that have been devised to control nanotechnology (for example, the "broadcast architecture" described below) won't work for strong AI. There have been discussions and proposals to guide AI development toward what Eliezer Yudkowsky calls "friendly AI"30 (see the section "Protection from 'Unfriendly' Strong AI," p. 420). These are useful for discussion, but it is infeasible today to devise strategies that will absolutely ensure that future AI embodies human ethics and values. Returning to the Past? In his essay and presentations Bill Joy eloquently describes the plagues of centuries past and how new self-replicating technologies, such as mutant bioengineered pathogens and nanobots run amok, may bring back long-forgotten pestilence.

Heart of the Machine: Our Future in a World of Artificial Emotional Intelligence
by Richard Yonck
Published 7 Mar 2017

As we saw in the previous chapter, this would almost certainly be a truly alien intelligence. More importantly, there is little guarantee or even likelihood that its values, motives, and logic would align with our own. Multiple strategies have been proposed as a means of dealing with such a superintelligence, including variations on Asimov’s Three Laws, Yudkowsky’s Friendly AI theory, and Goertzel’s global AI nanny.1 Unfortunately, each is far from a foolproof strategy, and that should concern us. While there is considerable division on whether or not such an event could actually come to pass, there is probably a comparable level of disagreement about whether the outcome will be good or really, really bad.

…

Here, I’m speaking mentally, independent of their need for physical nourishment. 18. Goertzel, B. “The Mind-World Correspondence Principle (Toward a General Theory of General Intelligence).” IEEE Symposium on Computational Intelligence for Human-like Intelligence (CIHLI). 2013. Chapter 18 1. Yudkowsky, E. “Creating Friendly AI 1.0: The Analysis and Design of Benevolent Goal Architectures.” Machine Intelligence Research Institute, 2001; Goertzel, Ben. “Should Humanity Build a Global AI Nanny to Delay the Singularity Until It’s Better Understood?” Journal of consciousness studies 19.1-2: 1-2. 2012. 2. Also known as affective empathy. 3.

…

., 70, 74 Empatica Srl, 70, 74 endocrine system and emotions, 16, 17, 19, 219, 244, 248 ENIAC, 210 Enigma machine, 37 epinephrine, 16, 186, 221 epistemology, 35 EPOC headset, 213 Epstein, Robert, 141 Erewhon (Butler), 228 Ex Machina (Garland), 236–238 exponential change, 40–41 “”EyeJacking: See What I See, 57 Eyeris, 72, 144–145 F FaceSense, 60, 66 Faceshift, 75 Facial Action Coding System (FACS), 55–56 facial expression recognition, 54 facial recognition software, 144 facial tracking system, 59 Fairchild Superconductor, 38 false memories, 145–146 feral children, 257–258 fiction, AIs in dystopias, 228–229 human relationships with AI, 235–239 and the Machine Age, 227–229 problems with machine intelligence, 238–239 threat of intelligent machines, 230–234 flint knapping, 8–13, 23–24 Foresight Program, University of Houston, 171 Forkhead box protein P2, 15 Forster, E. M., 229 FOXP2, 15 Frankenstein (Shelley), 228 Freud, Sigmund, 96 Frewen, Cindy, 170–171 Friendly AI theory, 262 “Friendly Faces of Industry,” 171 Frubber (flesh rubber), 87, 113 functional magnetic resonance imaging (fMRI), 126–127 “The Future of Social Robotics,” 171 G galvactivator, 57–58 gaming community and designer emotions, 217 Garver, Carolyn, 113 Gazzaniga, Michael, 247 Geminoids, 100–101 general intelligence, 255 general morphological analysis (GMA), 165–166 General Problem Solver (1957), 37 Georgia Institute of Technology, 120 geriatric physiotherapy rehabilitation robots, 152 Gibson, William, 171 Gigolo Joe (mecha), 233–234 global AI nanny, 262 GMA.

What to Think About Machines That Think: Today's Leading Thinkers on the Age of Machine Intelligence
by John Brockman
Published 5 Oct 2015

The system can be used between human parties or interspecies parties, exactly because it’s not necessary to know, trust, or understand the other entity, just the code (the language of machines). Over time, trust can grow through reputation. Blockchain technology could be used to enforce friendly AI and mutually beneficial interspecies interaction. Someday, important transactions (like identity authentication and resource transfer) will be conducted on smart networks that require confirmation by independent consensus mechanisms, such that only bona fide transactions by reputable entities are executed. While perhaps not a full answer to the problem of enforcing friendly AI, decentralized smart networks like blockchains are a system of checks and balances helping to provide a more robust solution to situations of future uncertainty.

Advances in Artificial General Intelligence: Concepts, Architectures and Algorithms: Proceedings of the Agi Workshop 2006
by Ben Goertzel and Pei Wang
Published 1 Jan 2007

In theory, all of these shortcuts should be equivalent to certain possible experience of the system, so they do not conflict with the principle that all the knowledge of NARS comes, directly or indirectly, from experience. One important issue to be handled through education is ethics. Unlike argued by some other researchers, NARS is not an attempt to design a “friendly AI”. As far as its initial state is concerned, the system is ethically neutral, since it can has any beliefs and goals. To make a NARS implementation “human friendly” means to give it certain beliefs and goals, which is an education mission, not a design mission. Even if something like Asimov’s “Three Laws of Robotics” is implanted into the system’s memory (which is possible), it still cannot fully control the system’s behaviors, due to the insufficiency of knowledge and resources in the system.

…

[Ben Goertzel]: Well I have a quite different opinion than that of Steve Grand in that I don’t think an amazing conceptual breakthrough on the level of the discovery of the quantum or curved 4D space-time, or something like that, is needed to create general intelligence. It might be needed to create provably stable friendly AI, like Eliezer Yudkowsky would like. I tend to think of the brain as a complex system composed of a bunch of evolved kluges for solving particular problems, which have been hacked together and adapted by evolution. I think if you assemble subcomponents solving the appropriate set of specialized problems, as well as a fairly weak general problem solver, and they are hooked together in a knowledge representation that works for all the components, with learning mechanisms that let each component learn from each other -- then you are going to have an intelligent mind that can be taught.

Overcomplicated: Technology at the Limits of Comprehension
by Samuel Arbesman
Published 18 Jul 2016

But that’s okay, because “We need not be intimidated by the accomplishments of metahuman science. We should always remember that the technologies that made metahumans possible were originally developed by humans, and they were no smarter than we.” See Luke Muehlhauser and Nick Bostrom, “Why We Need Friendly AI,” Think 36, no. 13 (Spring 2014), 41–47; and Ted Chiang, Stories of Your Life and Others (New York: Tor Books, 2003), 203. understand the most complex parts of the world: In many cases, we might even want to have a technology too complex to understand, because it means that it is sophisticated and powerful.

Human + Machine: Reimagining Work in the Age of AI
by Paul R. Daugherty and H. James Wilson
Published 15 Jan 2018

FUSION SKILL #4: Intelligent Interrogation Definition: Knowing how best to ask questions of AI, across levels of abstraction, to get the insights you need. How do you probe a massively complex system? How do you predict interactions between complex layers of data? People simply can’t do this on their own, so they must ask questions of their friendly AI. “Twin, how certain are you?” “Twin, what do you recommend?” At GE, maintenance professionals who have the skill of intelligent interrogation understand the capabilities and limitations of the AI system and know how to get the information they need to make an informed decision. The workers play to strengths and don’t duplicate the machine’s strengths.

Superintelligence: Paths, Dangers, Strategies
by Nick Bostrom
Published 3 Jun 2014

The programmers may try to guard against this possibility by secretly monitoring the AI’s source code and the internal workings of its mind; but a smart-enough AI would realize that it might be under surveillance and adjust its thinking accordingly.2 The AI might find subtle ways of concealing its true capabilities and its incriminating intent.3 (Devising clever escape plans might, incidentally, also be a convergent strategy for many types of friendly AI, especially as they mature and gain confidence in their own judgments and capabilities. A system motivated to promote our interests might be making a mistake if it allowed us to shut it down or to construct another, potentially unfriendly AI.) We can thus perceive a general failure mode, wherein the good behavioral track record of a system in its juvenile stages fails utterly to predict its behavior at a more mature stage.

…

“Computational Genetics, Physiology, Metabolism, Neural Systems, Learning, Vision, and Behavior or PolyWorld: Life in a New Context.” In Proceedings of the Artificial Life III Conference, edited by C. G. Langton, 263–98. Santa Fe Institute Studies in the Sciences of Complexity. Reading, MA: Addison-Wesley. Yudkowsky, Eliezer. 2001. Creating Friendly AI 1.0: The Analysis and Design of Benevolent Goal Architectures. Machine Intelligence Research Institute, San Francisco, CA, June 15. Yudkowsky, Eliezer. 2002. “The AI-Box Experiment.” Retrieved January 15, 2012. Available at http://yudkowsky.net/singularity/aibox. Yudkowsky, Eliezer. 2004. Coherent Extrapolated Volition.

Thinking Machines: The Inside Story of Artificial Intelligence and Our Race to Build the Future
by Luke Dormehl
Published 10 Aug 2016

As an increasing amount of work is carried out involving autonomous AI weapons in war situations, work like Bringsjord’s is in high demand. In 2014, a multidisciplinary team at Tufts and Brown Universities, working alongside Bringsjord, received funding from the Office of Naval Research to explore the possibility of giving autonomous robots – potentially used in combat – a sense of right and wrong. While not exactly a ‘friendly AI’, this kind of computational morality would allow robots on the battlefield to make ethical decisions. Imagine, for instance, a robot medic that is transporting an injured soldier to a field hospital encounters another soldier with an injured leg. Weighing up the pros and cons of stopping its mission to administer aid, potentially administering pain relief by applying traction in the field, and other conundrums are all complex issues for a human to navigate – let alone a machine.

12 Bytes: How We Got Here. Where We Might Go Next
by Jeanette Winterson
Published 15 Mar 2021

Same with that Roomba vacuum cleaner. It has a map of your house. In fact, the makers of Roomba, iRobot, are developing the idea of a house as an inside-out robot – in other words, a robot you live inside. It will come … So, I don’t want to get into the question of surveillance here. We know the price of home-friendly AI: it’s our personal data. Anyone who has voice-activated Alexa is being listened to all of the time. We’re told this isn’t like having the KGB in the kitchen. It’s ‘anonymised background sound’. How we deal with data tracking is a separate issue – a bot in the house doesn’t solve that problem, or make things worse, it’s just part of the system we’re all buying into.

To Be a Machine: Adventures Among Cyborgs, Utopians, Hackers, and the Futurists Solving the Modest Problem of Death
by Mark O'Connell
Published 28 Feb 2017

“If you consider balancing the interests of future people against those who already exist, reducing the probability of a major future catastrophe can be a very high-impact decision. If you succeed in avoiding an event that might wipe out all of future humanity, that clearly exceeds any good you might do for people currently living.” The Future of Life Institute was less focused than MIRI on the mathematical arcana of how a “friendly AI” might be engineered. The group, she said, functioned as “the outreach arm of this cluster of organizations,” raising awareness about the seriousness of this problem. It was not the attention of the media or the general public for which FLI was campaigning, Viktoriya said, but rather that of AI researchers themselves, a constituency in which the idea of existential risk was only just beginning to be taken seriously.

Possible Minds: Twenty-Five Ways of Looking at AI
by John Brockman
Published 19 Feb 2019

A superintelligent AGI will be extremely good at accomplishing its goals, and if those goals aren’t aligned with ours, we’re in trouble. People don’t think twice about flooding anthills to build hydroelectric dams, so let’s not place humanity in the position of those ants. Most researchers argue that if we end up creating superintelligence, we should make sure it’s what AI-safety pioneer Eliezer Yudkowsky has termed “friendly AI”—AI whose goals are in some deep sense beneficial. The moral question of what these goals should be is just as urgent as the technical questions about goal alignment. For example, what sort of society are we hoping to create, where we find meaning and purpose in our lives even though we, strictly speaking, aren’t needed?

Pandora's Brain
by Calum Chace
Published 4 Feb 2014

The European Commission was proposing an autonomous international agency with exceptional powers of inspection and restraint in order to stop rogue governments and individuals from doing any research that could lead to the creation of an AGI. This was a step too far for him, as it was for most Americans. ‘That game is far from over. And I wouldn’t be surprised if a coalition of the institutes which are researching human-friendly AI algorithms announced a breakthrough this spring. There’s been a lot of unusually cordial traffic between several of the bigger US-based ones during the winter. Anyway, fortunately, none of that affects what we’re doing here at the Foundation. We’ve managed to put clear blue water between brain preservation research and AI research in the public’s mind.’

What We Owe the Future: A Million-Year View
by William MacAskill
Published 31 Aug 2022

For a recent conceptual overview of the field, see Christiano (2020). Different authors have used different ways of conceptualizing the challenge of creating AI systems that are more capable than humans but lead to desirable outcomes when deployed. Yudkowsky (2001) described the issue as how to create “friendly AI”; Bostrom as the “control problem” (Bostrom 2014b, Chapter 9). (See also Christiano 2016, 2018a; Gabriel 2020; Hubinger 2020.) 83. What about worlds that are controlled by AIs but without significant lock-in? We can, for example, imagine a society of AIs that reflect, reason morally, and remain open to changing their minds.

Ways of Being: Beyond Human Intelligence
by James Bridle
Published 6 Apr 2022

That book is often cited as a sophisticated defence of the right of human beings to use animals for their benefit, but when Fox realized that the same arguments could be used by an alien species to enslave humans, he repudiated it and moved to a position supporting animal rights. 20. Calls to develop ‘friendly’ AI originate with the Machine Intelligence Research Institute, co-founded by Eliezer Yudkowsky, who also created the LessWrong community, but have been championed by some of the founders of academic and practical AI studies. See, for example, Stuart Russell and Peter Norvig’s Artificial Intelligence: A Modern Approach, 3rd edn (Harlow: Pearson Education, 2016), the standard textbook on the subject, which cites Yudkowsky’s concerns about AI safety. 21.

Hands-On Machine Learning With Scikit-Learn and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems
by Aurélien Géron
Published 13 Mar 2017

With two (or more) parents, it is called sexual reproduction. An offspring’s genome (in this case a set of policy parameters) is randomly composed of parts of its parents’ genomes. 8 OpenAI is a nonprofit artificial intelligence research company, funded in part by Elon Musk. Its stated goal is to promote and develop friendly AIs that will benefit humanity (rather than exterminate it). 9 “Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning,” R. Williams (1992). 10 We already did something similar in Chapter 11 when we discussed Gradient Clipping: we first computed the gradients, then we clipped them, and finally we applied the clipped gradients. 11 “A Markovian Decision Process,” R.

The Transhumanist Reader
by Max More and Natasha Vita-More
Published 4 Mar 2013

It’s not as though our choices are “life goes on exactly as is” versus “life as it is plus super-powerful AI.” Various technologies are advancing rapidly and society is changing accordingly, and the rate of advancement of AGI is just one aspect in the mix. I don’t think there are any magic bullets to resolve the dilemmas of AGI ethics. There will almost surely be no provably Friendly AI, in spite of the wishes of Eliezer Yudkowsky (2008) and some others. Nor, in my best guess, will there be an Artilect War in which pro-AGI and anti-AGI forces battle to the death with doomsday machines, as Hugo de Garis (2005) foresees. But I don’t pretend to be able to see exactly what the outcome will be.

Digital Empires: The Global Battle to Regulate Technology
by Anu Bradford
Published 25 Sep 2023

The more advanced member states typically endorse a liberal approach on digital issues, such as the promotion of free flow of data or measures designed to enhance growth and competitiveness.213 For example, Belgium, Finland, Denmark, Estonia, Ireland, Luxembourg, Sweden, the Netherlands, and the former member state UK have coordinated policy positions as “digitally advanced EU countries (D9)” group,214 at times joined by the Czech Republic, Poland, Portugal, and Spain in what became known as a “D9+” group.215 In 2020, these D9+ countries, together with France and Latvia, signed a position paper on the forthcoming AI regulation, calling for a well-calibrated, proportionate, and innovation-friendly AI regulation geared at fostering economic growth and European competitiveness.216 The EU member states are also split in their support for digital sovereignty, with France leading the more dirigiste, industrial policy–oriented camp whereas the Northern European countries, in particular, emphasize the need to retain Europe’s commitment to economic openness—a split that has even led to clashes among the key European Commissioners in charge of digital regulation.217 Another dividing line stems from differences in national tax regimes, explaining why a consensus on DSTs was difficult to forge across the EU.

Artificial Intelligence: A Modern Approach
by Stuart Russell and Peter Norvig
Published 14 Jul 2019

The “King Midas problem” was anticipated by Marvin Minsky, who once suggested that an AI program designed to solve the Riemann Hypothesis might end up taking over all the resources of Earth to build more powerful supercomputers. Similarly, Omohundro (2008) foresees a chess program that hijacks resources, and Bostrom (2014) describes the runaway paper clip factory that takes over the world. Yudkowsky (2008) goes into more detail about how to design a Friendly AI. Amodei et al. (2016) present five practical safety problems for AI systems. Omohundro (2008) describes the Basic AI Drives and concludes, “Social structures which cause individuals to bear the cost of their negative externalities would go a long way toward ensuring a stable and positive future.”

…

., 189, 190, 1095, 1105, 1111 Freund, Y, 718, 736, 1095 Frey, B. J., 838, 1095 Frey, C. B., 1050, 1095 Friedberg, R. M., 39, 161, 1095 Friedman, G. J., 161, 1095 Friedman, J., 734, 736, 738, 800, 1089, 1095, 1098 Friedman, N., 473, 474, 478, 516, 517, 586, 798, 799, 871, 1085, 1088, 1092, 1095, 1102 Friendly AI, 1061 frisbee, 48, 1033 Fristedt, B., 587, 1087 front-to-end, 114 front-to-front, 114 frontier, 90 Frost, D., 189, 191, 1092 Fruhwirth, T., 191, 1095 FTA (fault tree analysis), 1052 Fu, D. J., 48, 1104 Fu, J., 359, 1065, 1104 Fu, M. C., 587, 1090 Fuchs, J. J., 402, 1095 Fudenberg, D., 638, 1095 Fukunaga, A.