by Brian Christian · 5 Oct 2020 · 625pp · 167,349 words
a forty-five-year collaboration that would essentially found a new field. The field, which would cross neuroscience, behaviorist psychology, engineering, and mathematics, was dubbed “reinforcement learning”; and their names, forever linked in bibliographies of AI—“Barto & Sutton,” “Sutton & Barto”—would become synonymous with the definitive textbook of the field they
…
Comparative Psychology discussed “trial and error” in the context of animal behavior. For a short history of animal learning from the perspective of reinforcement learning, see Sutton and Barto, Reinforcement Learning. 9. See Thorndike, “A Theory of the Action of the After-Effects of a Connection upon It,” and Skinner, “The Rate of
…
-Analog Reinforcement Systems and Its Application to the Brain Model Problem” for an early example, and Chapter 15 of Sutton and Barto, Reinforcement Learning for discussion. 23. Andrew G. Barto, “Reinforcement Learning: A History of Surprises and Connections” (lecture), July 19, 2018, International Joint Conference on Artificial Intelligence, Stockholm, Sweden. 24. Andrew Barto,
…
personal interview, May 9, 2018. 25. The canonical text about reinforcement learning is Sutton and Barto, Reinforcement Learning, recently updated into a second edition. For a summary of the field up to the mid-1990s, see also Kaelbling, Littman, and Moore
…
this history, see “Michael Littman: The Reward Hypothesis” (lecture), University of Alberta, October 16, 2019, available at https://www.coursera.org/lecture/fundamentals-of-reinforcement-learning/michael-littman-the-reward-hypothesis-q6x0e. Despite the recency of this particular framing, the idea of understanding behavior as motivated, whether explicitly or implicitly, by
…
2018). 52. Sutton, “A Unified Theory of Expectation in Classical and Instrumental Conditioning.” 53. Sutton, “Temporal-Difference Learning” (lecture), July 3, 2017, Deep Learning and Reinforcement Learning Summer School 2017, Université de Montréal, July 3, 2017, http://videolectures.net/deeplearning2017_sutton_td_learning/. 54. Sutton, “Temporal-Difference Learning.” 55. Sutton, “Learning to
…
Predict by the Methods of Temporal Differences.” See also Sutton’s PhD thesis: “Temporal Credit Assignment in Reinforcement Learning.” 56. See Watkins, “Learning from Delayed Rewards” and Watkins and Dayan, “Q-Learning.” 57. Tesauro, “Practical Issues in Temporal Difference Learning.” 58. Tesauro, “TD
…
.” 67. Niv. 68. For a discussion of potential limitations to the TD-error theory of dopamine, see, e.g., Dayan and Niv, “Reinforcement Learning,” and O’Doherty, “Beyond Simple Reinforcement Learning.” 69. Niv, “Reinforcement Learning in the Brain.” 70. Yael Niv, personal interview, February 21, 2018. 71. Lenson, On Drugs. 72. See, e.g., Berridge, “
…
Formula That Predicts Happiness,” https://www.theglobeandmail.com/life/health-and-fitness/health/researchers-create-formula-that-predicts-happiness/article19919756/. 80. See Tomasik, “Do Artificial Reinforcement-Learning Agents Matter Morally?” For more on this topic, see also Schwitzgebel and Garza, “A Defense of the Rights of Artificial Intelligences.” 81. Brian Tomasik,
…
“Ethical Issues in Artificial Reinforcement Learning,” https://reducing-suffering.org/ethical-issues-artificial-reinforcement-learning/. 82. Daswani and Leike, “A Definition of Happiness for Reinforcement Learning Agents.” See also People for the Ethical Treatment of Reinforcement Learners: http://petrl.org. 83. Andrew Barto
…
CURIOSITY 1. Turing, “Intelligent Machinery.” 2. There were efforts starting in 2004 to develop standardized RL benchmarks and competitions; see Whiteson, Tanner, and White, “The Reinforcement Learning Competitions.” 3. Marc Bellemare, personal interview, February 28, 2019. 4. Bellemare et al., “The Arcade Learning Environment,” stemming originally from Naddaf, “Game-Independent AI Agents
…
was introduced into machine learning with Barto, Singh, and Chentanez, “Intrinsically Motivated Learning of Hierarchical Collections of Skills,” and Singh, Chentanez, and Barto, “Intrinsically Motivated Reinforcement Learning.” For a more recent overview of this literature, see Baldassarre and Mirolli, Intrinsically Motivated Learning in Natural and Artificial Systems. 13. Hobbes, Leviathan. 14. Simon
…
, “Learning and Satiation of Response in Intrinsically Motivated Complex Puzzle Performance by Monkeys.” 19. Scenarios of this type are described in Barto, “Intrinsic Motivation and Reinforcement Learning,” and Deci and Ryan, Intrinsic Motivation and Self-Determination in Human Behavior. 20. Berlyne, Conflict, Arousal, and Curiosity. 21. And see, for instance, Berlyne
…
intellectual motivation.” See Minsky, “Steps Toward Artificial Intelligence.” 30. See Sutton, “Integrated Architectures for Learning, Planning, and Reacting Based on Approximating Dynamic Programming” and “Reinforcement Learning Architectures for Animats.” MIT’s Leslie Pack Kaelbling devised a similar method, based on the idea of measuring an agent’s “confidence intervals” around the
…
approaches, which incentivize exploration by rewarding “information gain,” see, e.g., Schmidhuber, “Curious Model-Building Control Systems”; Stadie, Levine, and Abbeel, “Incentivizing Exploration in Reinforcement Learning with Deep Predictive Models”; and Houthooft et al., “VIME.” 51. Burda et al., “Large-Scale Study of Curiosity-Driven Learning.” 52. See Burda et al
…
30. This is a very active area of research. See, e.g., Subramanian, Isbell, and Thomaz, “Exploration from Demonstration for Interactive Reinforcement Learning”; Večerík et al., “Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards”; and Hester et al., “Deep Q-Learning from Demonstrations.” 31. In fact, many agents trained in
…
//thesis/PieterAbbeel_Defense_19May2008_320x180.mp4. 21. Abbeel, Coates, and Ng, “Autonomous Helicopter Aerobatics Through Apprenticeship Learning.” 22. Abbeel et al., “An Application of Reinforcement Learning to Aerobatic Helicopter Flight.” They also successfully performed a nose-in funnel and a tail-in funnel. 23. “As repeated sub-optimal demonstrations tend to
…
differ in their suboptimalities, together they often encode the intended trajectory.” See Abbeel, “Apprenticeship Learning and Reinforcement Learning with Application to Robotic Control,” which refers to the work in Coates, Abbeel, and Ng, “Learning for Control from Multiple Demonstrations.” 24. Abbeel, Coates,
…
Stanford helicopter performing the chaos, see “Stanford University Autonomous Helicopter: Chaos,” https://www.youtube.com/watch?v=kN6ifrqwIMY. 28. Ziebart et al., “Maximum Entropy Inverse Reinforcement Learning,” which leverages the principle of maximum entropy derived from Jaynes, “Information Theory and Statistical Mechanics.” See also Ziebart, Bagnell, and Dey, “Modeling Interaction via the
…
Problems in AI Safety,” which, in turn, references Salge, Glackin, and Polani, “Empowerment: An Introduction,” and Mohamed and Rezende, “Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning.” 50. Alexander Turner, personal interview, July 11, 2019. 51. Wiener, “Some Moral and Technical Consequences of Automation.” 52. According to Paul Christiano, “corrigibility” as
…
Art. 42. Turing et al., “Can Automatic Calculating Machines Be Said to Think?” ACKNOWLEDGMENTS 1. McCulloch, Finality and Form. BIBLIOGRAPHY Abbeel, Pieter. “Apprenticeship Learning and Reinforcement Learning with Application to Robotic Control.” PhD thesis, Stanford University, 2008. Abbeel, Pieter, Adam Coates, and Andrew Y. Ng. “Autonomous Helicopter Aerobatics Through Apprenticeship Learning.” International
…
Rebellion: Changing the Narrative.” In Thirty-First AAAI Conference on Artificial Intelligence, 2017. Akrour, Riad, Marc Schoenauer, and Michèle Sebag. “APRIL: Active Preference-Learning Based Reinforcement Learning.” In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 116–31. Springer, 2012. Akrour, Riad, Marc Schoenauer, Michèle Sebag, and Jean-Christophe
…
of the Cognitive Science Society, 2017. Choi, Jongwook, Yijie Guo, Marcin Moczulski, Junhyuk Oh, Neal Wu, Mohammad Norouzi, and Honglak Lee. “Contingency-Aware Exploration in Reinforcement Learning.” In International Conference on Learning Representations, 2019. Chouldechova, Alexandra. “Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments.” Big Data 5
…
Mental Health Crisis in Graduate Education.” Nature Biotechnology 36, no. 3 (2018): 282. Everitt, Tom, Victoria Krakovna, Laurent Orseau, Marcus Hutter, and Shane Legg. “Reinforcement Learning with a Corrupted Reward Channel.” In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI-17), 4705–13, 2017. Eysenbach, Benjamin, Shixiang
…
Gu, Julian Ibarz, and Sergey Levine. “Leave No Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning.” In International Conference on Learning Representations, 2018. Fantz, Robert L. “Visual Experience in Infants: Decreased Attention to Familiar Patterns Relative to Novel Ones.” Science 146
…
and Don Mosenfelder. Bobby Fischer Teaches Chess. Basic Systems, 1966. Florensa, Carlos, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. “Reverse Curriculum Generation for Reinforcement Learning.” In Proceedings of the 1st Annual Conference on Robot Learning, edited by Sergey Levine, Vincent Vanhoucke, and Ken Goldberg, 482–95. PMLR, 2017. Flores, Anthony
…
1976): 3–46. Malik, Dhruv, Malayandi Palaniappan, Jaime Fisac, Dylan Hadfield-Menell, Stuart Russell, and Anca Drăgan. “An Efficient, Generalized Bellman Update for Cooperative Inverse Reinforcement Learning.” In Proceedings of the 35th International Conference on Machine Learning, edited by Jennifer Dy and Andreas Krause, 3394–3402. PMLR, 2018. Malone, Thomas W. “Toward
…
2600 Console Games.” Master’s thesis, University of Alberta, 2010. Nair, Ashvin, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. “Overcoming Exploration in Reinforcement Learning with Demonstrations.” In 2018 IEEE International Conference on Robotics and Automation (ICRA), 6292–99. IEEE, 2018. Nalisnick, Eric, Bhaskar Mitra, Nick Craswell, and Rich Caruana
…
.” Journal of Experimental Criminology 12, no. 3 (2016): 347–71. Saunders, William, Girish Sastry, Andreas Stuhlmueller, and Owain Evans. “Trial Without Error: Towards Safe Reinforcement Learning via Human Intervention.” In Proceedings of the 17th International Conference on Autonomous Agents and Multiagent Systems, 2067–99. International Foundation for Autonomous Agents and Multiagent
…
Hospitalized Patients.” JAMA Neurology 74, no. 12 (2017): 1419–24. Subramanian, Kaushik, Charles L. Isbell Jr., and Andrea L. Thomaz. “Exploration from Demonstration for Interactive Reinforcement Learning.” In Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems, 447–56. International Foundation for Autonomous Agents and Multiagent Systems, 2016. Sundararajan, Mukund
…
for Animats.” In From Animals to Animats: Proceedings of the First International Conference on Simulation of Adaptive Behavior, 288–96. 1991. ———. “Temporal Credit Assignment in Reinforcement Learning.” PhD thesis, University of Massachusetts, Amherst, 1984. ———. “A Unified Theory of Expectation in Classical and Instrumental Conditioning.” Bachelor’s thesis, Stanford University, 1978. Sutton,
…
Richard S., and Andrew G. Barto. Reinforcement Learning: An Introduction. 2nd ed. MIT Press, 2018. Sweeney, Latanya. “Discrimination in Online Ad Delivery.” Communications of the ACM 56, no. 5 (2013): 44–54. ———.
…
Večerík, Matej, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. “Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards.” arXiv Preprint arXiv:1707.08817, 2017. Vincent, James. “Google ‘Fixed’ Its Racist Algorithm by Removing Gorillas from Its Image
…
9, 12, 337n6, 346n13 alignment problem amplification/distillation and, 249 analogies and, 317 corrigibility and, 295 defined, 13 as hopeful, 327–28 inverse reinforcement learning and, 255 parenting and, 166 reinforcement learning and, 151 technical limitations and, 313, 395–96n4 thermostats and, 311–12, 313 See also value alignment Allen, Woody, 170 AlphaGo, 162
…
ensemble methods, 284–85, 305 equiprobabiliorism, 303 equivant (company), 337n5 ergodicity assumption, 320 Ermon, Stefano, 324 ethics actualism vs. possibilism and, 239, 379n71 in reinforcement learning, 149 See also AI safety; fairness; moral uncertainty evaluation function. See value function Evans, Owain, 386–87n55 evolution, 170, 171–74, 368n56 expectations, 138–39
…
, 316, 342n61 See also Google research Google Brain, 113, 167, 373n53 Google research differential privacy, 347n33 fairness, 73 feature visualization, 110 multitask learning models, 107 reinforcement learning, 167 selective classification, 390n29 value alignment, 247 word embedding, 44 Gopnik, Alison, 194, 215 gorilla tag incident, 25–26, 316, 339n24 GPT-2, 344
…
238–39, 379n69 OpenAI actualism vs. possibilism, 239 amplification, 248–50 corrigibility, 296 feature visualization, 112, 357n69 intrinsic motivation, 199–200, 201 inverse reinforcement learning, 263–66, 384–85n37 reinforcement learning, 365n27 word embedding, 344–45n94 open category problem, 279–81, 315, 396nn10–11 ophthalmology, 287, 389n23 optimal regressions, 95–96 optimal reward problem
…
Herbert, 327 Reality Is Broken (McGonigal), 175 recidivism. See risk-assessment models rectified linear output functions, 24 Reddy, Raj, 223 redundant encodings, 40, 64, 343n74 reinforcement learning (RL) actor-critic architecture, 138, 362n51 actualism vs. possibilism and, 238–40, 379n69 addiction and, 205–08, 374n65 alignment problem and, 151 Arcade Learning Environment
…
143, 145 Research Institute for Advanced Studies, 383n15 Revels, Hiram, 27 reversibility, 391n39 reward hypothesis, 130–31, 133–34, 360nn26, 28 See also reinforcement learning rewards. See incentives; reinforcement learning; reward hypothesis; shaping right to explanation. See transparency rigorism, 303, 304 risk-assessment models COMPAS development, 56–57, 346nn13–14 defenses of, 68, 72
by Melanie Mitchell · 14 Oct 2019 · 350pp · 98,077 words
been used for centuries on animals and humans. Operant conditioning inspired an important machine-learning approach called reinforcement learning. Reinforcement learning contrasts with the supervised-learning method I’ve described in previous chapters: in its purest form, reinforcement learning requires no labeled training examples. Instead, an agent—the learning program—performs actions in an environment (
…
such appreciation—such as positive numbers added to its memory. FIGURE 22: A Sony Aibo robotic dog, about to kick a robot soccer ball While reinforcement learning has been part of the AI toolbox for decades, it has long been overshadowed by neural networks and other supervised-learning methods. This changed in
…
2016 when reinforcement learning played a central role in a stunning and momentous achievement in AI: a program that learned to beat the best humans at the complex game
…
if a rock is blocking the ball’s movement? As always, the real world is awash with hard-to-predict edge cases. The promise of reinforcement learning is that the agent—here our robo-dog—can learn flexible strategies on its own simply by performing actions in the world and occasionally receiving
…
Or perhaps not. Unlike a real dog, Rosie has no intrinsic desire for treats, positive numbers, or anything else. As I’ll detail below, in reinforcement learning, a human-created algorithm guides Rosie’s process of learning in response to rewards; that is, the algorithm tells Rosie how to learn from her
…
experiences. Reinforcement learning occurs by having Rosie take actions over a series of learning episodes, each of which consists of some number of iterations. At each iteration, Rosie
…
to take, but remember, we’re letting Rosie figure out on her own how to perform this task. FIGURE 23: A hypothetical first episode of reinforcement learning At iteration 2 (figure 23B), Rosie determines her new state: thirteen steps from the ball. She then chooses a new action to take, again at
…
She’s back to square one, but Rosie doesn’t even know that she has been in this state before! In the purest form of reinforcement learning, the learning agent doesn’t remember its previous states. In general, remembering previous states might take a lot of memory and doesn’t turn out
…
Kick (figure 23D). Finally, she gets a reward and uses it to learn something. What does Rosie learn? Here we take the simplest approach to reinforcement learning: upon receiving a reward, Rosie learns only about the state and action that immediately preceded the reward. In particular, Rosie learns that if she is
…
action is on the right path, and if you’ve eaten chocolate before, you can predict the intensity of the upcoming reward. The goal of reinforcement learning is for the agent to learn values that are good predictions of upcoming rewards (assuming that the agent keeps doing the right thing after taking
…
particular actions in a given state typically takes many steps of trial and error. FIGURE 24: Rosie’s Q-table after her first episode of reinforcement learning Rosie keeps track of the values of actions in a big table in her computer memory. This table, illustrated in figure 24, lists all the
…
accurate predictions of upcoming rewards—as Rosie continues to learn. This table of states, actions, and values is called the Q-table. This form of reinforcement learning is sometimes called Q-learning. The letter Q is used because the letter V (for value) was used for something else in the original paper
…
Backward again at the next iteration (figure 25B). Our robo-dog’s training has a long way to go. FIGURE 25: The second episode of reinforcement learning Everything continues as before, until Rosie’s floundering random trial-and-error actions happen to land her one step away from the ball (figure 25C
…
reduced (“discounted”) as they go back in time from the actual reward; this allows the system to learn an efficient path to an actual reward. Reinforcement learning—here, the gradual updating of values in the Q-table—continues, episode to episode, until Rosie has finally learned to perform her task from any
…
learn to perform this task perfectly, no matter where she started. This “training Rosie” example captures much of the essence of reinforcement learning, but I left out many issues that reinforcement-learning researchers face for more complex tasks.5 For example, in real-world tasks, the agent’s perception of its state is often
…
might move it different distances depending on the terrain, or even result in the robot falling down or colliding with an unseen obstacle. How can reinforcement learning deal with uncertainties like these? Additionally, how should the learning agent choose an action at each time step? A naive strategy would be to always
…
World Setting these issues aside for now, let’s look at two major stumbling blocks that might arise in extrapolating our “training Rosie” example to reinforcement learning in real-world tasks. First, there’s the Q-table. In complex real-world tasks—think, for example, of a robot car learning to
…
. Learning via a Q-table like the one in the “Rosie” example is out of the question. For this reason, most modern approaches to reinforcement learning use a neural network instead of a Q-table. The neural network’s job is to learn what values should be assigned to actions in
…
damaging itself by choosing the wrong action, such as kicking a concrete wall or stepping forward over a cliff. Just as I did for Rosie, reinforcement-learning practitioners almost always deal with this problem by building simulations of robots and environments and performing all the learning episodes in the simulation rather than
…
learning systems on forty-nine of these games.3 This was the platform used by the DeepMind group in their work on reinforcement learning. Deep Q-Learning The DeepMind group combined reinforcement learning—in particular Q-learning—with deep neural networks to create a system that could learn to play Atari video games. The
…
action based on the DQN’s output values. The system doesn’t always choose the action with the highest estimated value; as I mentioned above, reinforcement learning requires a balance between exploration and exploitation.4 The system performs its chosen action (for example, moving the paddle some amount to the left) and
…
updating the weights in the DQN via back-propagation. How are the weights updated? This is the crux of the difference between supervised learning and reinforcement learning. As you’ll recall from earlier chapters, back-propagation works by changing a neural network’s weights so as to reduce the error in the
…
the output should have been 80 points higher. The network could calculate the error because it had a label provided by a human. However, in reinforcement learning we have no labels. A given frame from the game doesn’t come labeled with the action that should be taken. How then do we
…
a year later, Google announced that it was acquiring DeepMind for £440 million (about $650 million at the time), presumably because of these results. Yes, reinforcement learning occasionally leads to big rewards. With a lot of money in their pockets and the resources of Google behind them, DeepMind—now called Google DeepMind
…
while playing itself! The method for learning was somewhat complicated, and I won’t detail it here, but it had some aspects that foreshadowed modern reinforcement learning.11 In the end, Samuel’s checkers player impressively rose to the level of a “better-than-average player,” though by no means a champion
…
surmounted an even greater challenge than chess and had done so in a much more impressive fashion. Unlike Deep Blue, AlphaGo acquired its abilities by reinforcement learning via self-play. Demis Hassabis noted that “the thing that separates out top Go players [is] their intuition” and that “what we’ve done
…
search get started. With its AlphaGo project, DeepMind demonstrated that one of AI’s longtime grand challenges could be conquered by an inventive combination of reinforcement learning, convolutional neural networks, and Monte Carlo tree search (and adding powerful modern computing hardware to the mix). As a result, AlphaGo has attained a
…
most active areas of research for machine-learning practitioners, progress on this front is still nascent.3 “Without Human Examples or Guidance” Unlike supervised learning, reinforcement learning holds the promise of programs that can truly learn on their own, simply by performing actions in their “environment” and observing the outcome. DeepMind’s
…
most important claim about its results, especially on AlphaGo, is that the work has delivered on that promise: “Our results comprehensively demonstrate that a pure reinforcement learning approach is fully feasible, even in the most challenging of domains: it is possible to train to superhuman level, without human examples or guidance, given
…
. As the psychologist and AI researcher Gary Marcus has pointed out, none of these crucial aspects of AlphaGo were “learned from the data, by pure reinforcement learning. Rather, [they were] built in innately … by DeepMind’s programmers.”5 DeepMind’s Atari game-playing programs were actually better examples of “learning without
…
deep Q-learning method on several Atari video games. The most surprising good performer was “random search”: instead of training a Deep Q-Network by reinforcement learning over many episodes, one can simply try out many different convolutional neural networks with randomly chosen weights.6 That is, there is no learning whatsoever
…
(for example, wall, ceiling, paddle, ball, tunneling), the program actually has no such concepts: These demonstrations make clear that it is misleading to credit deep reinforcement learning with inducing concepts like wall or paddle; rather, such remarks are what comparative (animal) psychology sometimes call overattributions. It’s not that the Atari system
…
real-world problems and have a huge impact on things like healthcare and science.” I think it’s very possible that DeepMind’s work on reinforcement learning may eventually have the kinds of impacts Hassabis is aiming for. But there’s a long way to go from games to the real
…
boundaries at all; you don’t know what’s in the situation, what’s out of the situation.”13 As an example, consider using reinforcement learning to train a robot to perform a very useful real-world task: take the dirty dishes stacked in the sink and put them in the
…
are violated, and any successful approach would look extremely different.”15 No one knows what that successful approach would be. Indeed, the field of deep reinforcement learning is still quite young. The results I described in this chapter can be seen as a proof of principle: the combination of deep networks and
…
25/fashion/what-shamu-taught-me-about-a-happy-marriage.html. 2. thejetsons.wikia.com/wiki/Rosey. 3. To be more precise, this approach to reinforcement learning, called value learning, is not the only possible approach. A second approach, called policy learning, has the goal of learning directly what action to perform
…
and P. Dayan, “Q-Learning,” Machine Learning 8, nos. 3–4 (1992): 279–92. 5. For a detailed, technical introduction to reinforcement learning, see R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed. (Cambridge, Mass.: MIT Press, 2017), incompleteideas.net/book/the-book-2nd.html. 6. For example, see the
…
stage Task,” in Proceedings of the First Annual Conference on Robot Learning, CoRL (2017); M. Cutler, T. J. Walsh, and J. P. How, “Real-World Reinforcement Learning via Multifidelity Simulators,” IEEE Transactions on Robotics 31, no. 3 (2015): 655–71. 9: Game On 1. Demis Hassabis, quoted in P. Iwaniuk, “A Conversation
…
1; it is initially set close to 1 and is gradually decreased over the episodes of training. 5. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed. (Cambridge, Mass.: MIT Press, 2017), 124, incompleteideas.net/book/the-book-2nd.html. 6. For more details, see V. Mnih
…
27. D. Silver et al., “Mastering the Game of Go Without Human Knowledge,” Nature, 550 (2017): 354–59. 28. D. Silver et al., “A General Reinforcement Learning Algorithm That Masters Chess, Shogi, and Go Through Self-Play,” Science 362, no. 6419 (2018): 1140–44. 10: Beyond Games 1. Quoted in P. Iwaniuk
…
06/deepminds-alphago-mastered-chess-spare-time. 3. As one example, still in the game-playing domain, DeepMind published a paper in 2018 describing a reinforcement-learning system that they claimed exhibited some degree of transfer learning in its ability to play different Atari games. L. Espeholt et al., “Impala: Scalable Distributed
…
-far-from-intelligent. 14. I should note that a few robotics groups have actually developed dishwasher-loading robots, though none of these was trained by reinforcement learning, or any other kind of machine-learning method, as far as I know. These robots come with some impressive videos (for example, “Robotic Dog
…
Robotics Atari video games; see also Breakout automated image captioning autonomous vehicles, see self-driving cars B back-propagation; in convolutional neural networks; in deep reinforcement learning barrier of meaning Barsalou, Lawrence beneficial AI Bengio, Yoshua bias; in face recognition; in word vectors big data bilingual evaluation understudy (BLEU) board positions;
…
–15; adversarial attacks on; see also IBM Watson; reading comprehension; Stanford Question-Answering Dataset; Winograd schemas R reading comprehension recurrent neural networks; 199–200 regulation reinforcement learning; actions of agent in; contrast with supervised learning; deep Q-learning, see deep Q-learning; discounting in; episode; epsilon-greedy method for; exploration versus
…
with symbolic methods suitcase words Summer Vision Project (MIT) superhuman intelligence superintelligence, see superhuman intelligence Superintelligence (book) supervised learning; contrast with human learning; contrast with reinforcement learning; in IBM Watson support vector machines Sutherland, Amy Sutskever, Ilya Sutton, Richard symbolic AI; contrast with subsymbolic methods; integration with subsymbolic methods Szegedy, Christian T
…
temporal difference learning; see also reinforcement learning test set theory of mind thought vectors training, see supervised learning training set transfer learning; for Breakout translation, see machine translation trolley problem Turing, Alan
by Stuart Russell and Peter Norvig · 14 Jul 2019 · 2,466pp · 668,761 words
and Transfer Learning 22.8Applications Summary Bibliographical and Historical Notes 23Reinforcement Learning 23.1Learning from Rewards 23.2Passive Reinforcement Learning 23.3Active Reinforcement Learning 23.4Generalization in Reinforcement Learning 23.5Policy Search 23.6Apprenticeship and Inverse Reinforcement Learning 23.7Applications of Reinforcement Learning Summary Bibliographical and Historical Notes VICommunicating, perceiving, and acting 24Natural Language Processing 24.1Language Models 24.
…
processes (MDPs) developed in the field of operations research. A flood of work followed connecting AI planning research to MDPs, and the field of reinforcement learning found applications in robotics and process control as well as acquiring deep theoretical foundations. One consequence of AI’s newfound appreciation for data, statistical modeling
…
, for example by detecting unusual patterns of behavior, but they will also contribute to the potency, survivability, and proliferation capability of malware. For example, reinforcement learning methods have been used to create highly effective tools for automated, personalized blackmail and phishing attacks. We will revisit these topics in more depth in
…
); economics (market-based algorithms (Dias et al., 2006)); physics (particle swarms (Li and Yao, 2012) and spin glasses (Mézard et al., 1987)); animal behavior (reinforcement learning, grey wolf optimizers (Mirjalili and Lewis, 2014)); ornithology (Cuckoo search (Yang and Deb, 2014)); entomology (ant colony (Dorigo et al., 2008), bee colony (Karaboga
…
of actions have probabilities associated with them): Markov decision processes, partially observable Markov decision processes, and game theory. In Chapter 23 we show that reinforcement learning allows an agent to learn how to behave from past successes and failures. Bibliographical and Historical Notes AI planning arose from investigations into state-space
…
models, to update its belief state, and to project forward possible action sequences. We shall return MDPs and POMDPs in Chapter 23, which covers reinforcement learning methods that allow an agent to improve its behavior from experience. Bibliographical and Historical Notes Richard Bellman developed the ideas underlying the modern approach to
…
). The texts by Bertsekas (1987) and Puterman (1994) provide rigorous introductions to sequential decision problems and dynamic programming. Bertsekas and Tsitsiklis (1996) include coverage of reinforcement learning. Sutton and Barto (2018) cover similar ground but in a more accessible style. Sigaud and Buffet (2010), Mausam and Kolobov (2012) and Kochenderfer (2015)
…
replacing the simple components with more sophisticated machine learning models. Part of problem formulation is deciding whether you are dealing with supervised, unsupervised, or reinforcement learning. The distinctions are not always so crisp. In semisupervised learning we are given a few labeled examples and use them to mine more information from
…
gradient descent in parameter space to minimize the loss function. •Deep learning works well for visual object recognition, speech recognition, natural language processing, and reinforcement learning in complex environments. •Convolutional networks are particularly well suited for image processing and other tasks where the data have a grid topology. •Recurrent networks are
…
have already seen the concept of rewards in Chapter 16 for Markov decision processes (MDPs). Indeed, the goal is the same in reinforcement learning: maximize the expected sum of rewards. Reinforcement learning differs from “just solving an MDP” because the agent is not given the MDP as a problem to solve; the agent
…
.6, we explore apprenticeship learning: training a learning agent using demonstrations rather than reward signals. Finally, Section 23.7 reports on applications of reinforcement learning. 23.2Passive Reinforcement Learning We start with the simple case of a fully observable environment with a small number of actions and states, in which an agent already
…
transition model to perform its updates. The environment itself supplies the connection between neighboring states in the form of observed transitions. Figure 23.4A passive reinforcement learning agent that learns utility estimates using temporal differences. The step-size function ∝(n) is chosen to ensure convergence. Figure 23.5The TD learning curves
…
update rules, even though there are good solutions in the hypothesis space. There are more sophisticated algorithms that can avoid these problems, but at present reinforcement learning with general function approximators remains a delicate art. In addition to parameters diverging to infinity, there is a more surprising problem called catastrophic forgetting.
…
to understand this kind of situation as a two-person assistance game, as described in Section 17.2.5. 23.7Applications of Reinforcement Learning We now turn to applications of reinforcement learning. These include game playing, where the transition model is known and the goal is to learn the utility function, and robotics,
…
specify. Imitation learning formulates the problem as supervised learning of a policy from the expert’s state–action pairs. Inverse reinforcement learning infers reward information from the expert’s behavior. Reinforcement learning continues to be one of the most active areas of machine learning research. It frees us from manual construction of behaviors
…
environment. Weighted linear combinations of features and neural networks are factored representations for function approximation. It is also possible to apply reinforcement learning to structured representations; this is called relational reinforcement learning (Tadepalli et al.,2004). The use of relational descriptions allows for generalization across complex behaviors involving different objects. Analysis of the
…
explore unknown environments and are guaranteed to converge on near-optimal policies with a sample complexity that is polynomial in the number of states. Bayesian reinforcement learning (Dearden et al., 1998, 1999) provides another angle on both model uncertainty and exploration. The basic idea underlying imitation learning is to apply
…
of temporal-difference learning; related research describes other neuroscientific and behavioral experiments (Dayan and Niv, 2008; Niv, 2009; Lee et al., 2012). Work in reinforcement learning has been accelerated by the availability of open-source simulation environments for developing and testing learning agents. The University of Alberta’s Arcade Learning Environment
…
simulation (Savva et al., 2019) provides a photo-realistic virtual environment for indoor robotic tasks, and their HORIZON platform (Gauci et al., 2018) enables reinforcement learning in large-scale production systems. The SYNTHIA system (Ros et al., 2016) is a simulation environment designed for improving the computer vision capabilities of self
…
and perform safely. Robotics brings together many of the concepts we have seen in this book, including probabilistic state estimation, perception, planning, unsupervised learning, reinforcement learning, and game theory. For some of these concepts robotics serves as a challenging example application. For other concepts this chapter breaks new ground, for instance
…
cost (ILQR). •Planning under uncertainty unites perception and action by online replanning (such as model predictive control) and information gathering actions that aid perception. •Reinforcement learning is applied in robotics, with techniques striving to reduce the required number of interactions with the real world. Such techniques tend to exploit models, be
…
function they should optimize from human input, such as demonstrations, corrections, or instruction in natural language. Alternatively, robots can imitate human behavior, and use reinforcement learning to help tackle the challenge of generalization to new states. Bibliographical and Historical Notes The word robot was popularized by Czech playwright Karel Čapek in
…
rational agents. A variety of different agent designs were considered, ranging from reflex agents to knowledge-based decision-theoretic agents to deep learning agents using reinforcement learning. There is also variety in the component technologies from which these designs are assembled: logical, probabilistic, or neural reasoning; atomic, factored, or structured representations
…
, of course, but compilation methods can be applied so that the overhead is small compared to the costs of the computations being controlled. Metalevel reinforcement learning may provide another way to acquire effective policies for controlling deliberation: in essence, computations that lead to better decisions are reinforced, while those that
…
Based Scheduling. Morgan Kaufmann. Abbas, A. (2018). Foundations of Multiattribute Utility. Cambridge University Press. Abbeel, P. and Ng, A. Y. (2004). Apprenticeship learning via inverse reinforcement learning. In ICML-04. Abney, S., McAllester, D. A., and Pereira, F. (1999). Relating probabilistic grammars and automata. In ACL-99. Abramson, B. (1987). The
…
Andre, D., Friedman, N., and Parr, R. (1998). Generalized prioritized sweeping. In NeurIPS 10. Andre, D. and Russell, S. J. (2002). State abstraction for programmable reinforcement learning agents. In AAAI-02. Andreae, P. (1985). Justified Generalisation: Learning Procedures from Examples. Ph.D. thesis, MIT. Andrieu, C., Doucet, A., and Holenstein, R. (
…
Gaussian Bayesian tracking. IEEE Transactions on Signal Processing, 50, 174–188. Arulkumaran, K., Deisenroth, M. P., Brundage, M., and Bharath, A. A. (2017). Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine, 34, 26–38. Arunachalam, R. and Sadeh, N. M. (2005). The supply chain trading agent competition. Electronic Commerce
…
. In Automatic Control–World Congress, 1987: Selected Papers from the 10th Triennial World Congress of the International Federation of Automatic Control. Dietterich, T. (2000). Hierarchical reinforcement learning with the MAXQ value function decomposition. JAIR, 13, 227–303. Dijkstra, E. W. (1959). A note on two problems in connexion with graphs. Numerische
…
, 137– 144. Gao, J. (2014). Machine learning applications for data center optimization. Google Research. García, J. and Fernández, F. (2015). A comprehensive survey on safe reinforcement learning. JMLR, 16, 1437–1480. Gardner, M. (1968). Logic Machines, Diagrams and Boolean Algebra. Dover. Garey, M. R. and Johnson, D. S. (1979). Computers
…
using Markovian decision theory. Master’s report, Computer Science Division, University of California, Berkeley. Koenig, S. (2000). Exploring unknown environments with real–time search or reinforcement learning. In NeurIPS 12. Koenig, S. (2001). Agent-centered search. AIMag, 22, 109–131. Koenig, S. and Likhachev, M. (2002). D* Lite. AAAI-15, 15
…
viaGLTL. arXiv:1704.04341. Liu, B., Gemp, I., Ghavamzadeh, M., Liu, J., Mahadevan, S., and Petrik, M. (2018). Proximal gradient temporal difference learning: Stable reinforcement learning with polynomial sample complexity. JAIR, 63, 461–494. Liu, H., Simonyan, K., Vinyals, O., Fernando, C., and Kavukcuoglu, K. (2017). Hierarchical representations for efficient architecture
…
(Eds.). (1992). Geometric Invariance in Computer Vision. MIT Press. Munos, R., Stepleton, T., Harutyunyan, A., and Bellemare, M. G. (2017). Safe and efficient off–policy reinforcement learning. In NeurIPS 29. Murphy, K. (2002). Dynamic Bayesian Networks: Representation, Inference and Learning. Ph.D. thesis, UC Berkeley. Murphy, K. (2012). Machine Learning: A Probabilistic
…
Theory. Cambridge University Press. Nisan, N., Roughgarden, T., Tardos, E., and Vazirani, V. (Eds.). (2007). Algorithmic Game Theory. Cambridge University Press. Niv, Y. (2009). Reinforcement learning in the brain. Journal of Mathematical Psychology, 53, 139–154. Nivre, J., De Marneffe, M.-C., Ginter, F., Goldberg, Y., Hajic, J., Manning, C., McDonald
…
. and Yannakakis, M. (1991). Shortest paths without a map. Theoretical Computer Science, 84, 127–150. Papavassiliou, V. and Russell, S. J. (1999). Convergence of reinforcement learning with general function approximators. In IJCAI-99. Parisi, G. (1988). Statistical Field Theory. Addison-Wesley. Parisi, M. M. G. and Zecchina, R. (2002). Analytic and
…
of temporal differences. Machine Learning, 3, 944. Sutton, R. S., McAllester, D. A., Singh, S., and Man-sour, Y. (2000). Policy gradient methods for reinforcement learning with function approximation. In NeurIPS 12. Sutton, R. S. (1990). Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In ICML-90
…
D. (1983). An abstract Prolog instruction set. Technical note, SRI International. Wasserman, L. (2004). All of Statistics. Springer. Watkins, C. J. (1989). Models of Delayed Reinforcement Learning. Ph.D. thesis, Psychology Department, Cambridge University. Watson, J. D. and Crick, F. (1953). A structure for deoxyribose nucleic acid. Nature, 171, 737. Wattenberg, M
…
programming of intelligent embedded systems and robotic space explorers. Proc. IEEE, 91(212–237). Williams, R. J. (1992). Simple statistical gradient following algorithms for connectionist reinforcement learning. Machine Learning, 8, 229–256. Williams, R. J. and Zipser, D. (1989). A learning algorithm for continually running fully recurrent neural networks. Neural Computation,
…
systematic comparison of phrase-based, hierarchical and syntax-augmented statistical MT. In COLING-08. Zoph, B. and Le, Q. V. (2016). Neural architecture search with reinforcement learning. arXiv:1611.01578. Zuse, K. (1945). The Plankalkül. Report, Gesellschaft für Mathematik und Datenverarbeitung. Zweig, G. and Russell, S. J. (1998). Speech recognition with
…
HMM), 43, 479, 491, 491–496, 503, 515, 795, 881 hidden variable, 443, 788 HIERARCHICAL-SEARCH, 377 hierarchical decomposition, 375 hierarchical look-ahead, 383 hierarchical reinforcement learning, 858, 1065 hierarchical structure, 1065 hierarchical task network (HTN), 375, 397, 858 Hierholzer, C., 162, 1098 high-level action, 375 higher-order logic, 273 Hilbert
…
., 516, 1100 intractability, 39 intrinsic property, 340 introspection, 20, 31 invariance, temporal, 811 inverse (of a matrix), 1077 inverse dynamics, 958 inverse kinematics, 947 inverse reinforcement learning, 864, 1054, 1065 inverted pendulum, 867 Ioffe, S., 837, 1100 IPL (programming language), 36 IQ test, 38 IR (information retrieval), 901, 905 Irpan, A.,
…
1053 regularization function, 689 Reid, D. B., 667, 1110 Reid, M., 116, 125, 1102 Reif, J., 984, 986, 1089, 1110 reification, 335 REINFORCE (reinforcement learning algorithm), 863 reinforcement, 840 reinforcement learning, 28, 210, 585, 671, 840, 789–873, 986 active, 842, 848–854 Bayesian, 851 deep, 835, 857 distributed, 636 generalization in, 854–861
by Stuart Russell · 7 Oct 2019 · 416pp · 112,268 words
money that provide eventual reward rather than immediate reward. One reason we understand the brain’s reward system is that it resembles the method of reinforcement learning developed in AI, for which we have a very solid theory.4 From an evolutionary point of view, we can think of the brain’s
…
’t design decision procedures that work only for Go. Instead, they made improvements to two fairly general-purpose techniques—lookahead search to make decisions and reinforcement learning to learn how to evaluate positions—so that they were sufficiently effective to play Go at a superhuman level. Those improvements are applicable to many
…
enormous and the reward comes only at the end of the game, lookahead search won’t work. Instead, AI researchers have developed a method called reinforcement learning, or RL for short. RL algorithms learn from direct experience of reward signals in the environment, much as a baby learns to stand up from
…
states (or sometimes the value of actions). This estimator can be combined with relatively myopic lookahead search to generate highly competent behavior. The first successful reinforcement learning system was Arthur Samuel’s checkers program, which created a sensation when it was demonstrated on television in 1956. The program learned essentially from scratch
…
the program defeated the world’s top professional Dota 2 team.64 Games such as Go and Dota 2 are a good testing ground for reinforcement learning methods because the reward function comes with the rules of the game. The real world is less convenient, however, and there have been dozens of
…
search, the vehicle has to find a trajectory that optimizes some combination of safety and progress. Some projects are trying more direct approaches based on reinforcement learning (mainly in simulation, of course) and supervised learning from recordings of hundreds of human drivers, but these approaches seem unlikely to reach the required level
…
Boston Dynamics for some of the more complex parts of their Atlas humanoid robot. Robot manipulation skills are advancing rapidly, thanks in part to deep reinforcement learning.20 The final push—putting all this together into something that begins to approximate the awesome physical skills of movie robots—is likely to come
…
’s important to understand that I’m not asking whether we can train a robot to stand up, which can be done simply by applying reinforcement learning with a reward for the robot’s head being farther away from the ground.46 Training a robot to stand up requires that the human
…
money (or to coerce behavior, if the goal is political control or espionage). The extraction of money works as the perfect reward signal for a reinforcement learning algorithm, so we can expect AI systems to improve rapidly in their ability to identify and profit from misbehavior. Early in 2015, I suggested to
…
a computer security expert that automated blackmail systems, driven by reinforcement learning, might soon become feasible; he laughed and said it was already happening. The first blackmail bot to be widely publicized was Delilah, identified in July
…
, methods of control can be direct if a government is able to implement rewards and punishments based on behavior. Such a system treats people as reinforcement learning algorithms, training them to optimize the objective set by the state. The temptation for a government, particularly one with a top-down, engineering mind-set
…
with you for hours every day, controls your access to information, and provides much of your entertainment through games, TV, movies, and social interaction. The reinforcement learning algorithms that optimize social-media click-through have no capacity to reason about human behavior—in fact, they do not even know in any meaningful
…
more or less any objective to pursue—including maximizing the number of paperclips or the number of known digits of pi. This is just how reinforcement learning systems and other kinds of reward optimizers work: the algorithms are completely general and accept any reward signal. For engineers and computer scientists operating within
…
-crawlies and had built a little treadmill for cockroaches to see how their gait changed with speed. We thought it might be possible to use reinforcement learning to train a robotic or simulated insect to reproduce these complex behaviors. The problem we faced was that we didn’t know what reward signal
…
to use. What were the flies and cockroaches optimizing? Without that information, we couldn’t apply reinforcement learning to train the virtual insect, so we were stuck. One day, I was walking down the road that leads from our house in Berkeley to
…
and planting them less stiffly because of the unpredictable ground level. As I pondered these mundane observations, I realized we had got it backwards. While reinforcement learning generates behavior from rewards, we actually wanted the opposite: to learn the rewards given the behavior. We already had the behavior, as produced by the
…
flies and cockroaches; we wanted to know the specific reward signal being optimized by this behavior. In other words, we needed algorithms for inverse reinforcement learning, or IRL.4 (I did not know at the time that a similar problem had been studied under the perhaps less wieldy name of structural
…
-circuit normal behavior in favor of direct stimulation of their own reward system is called wireheading. Could something similar happen to machines that are running reinforcement learning algorithms, such as AlphaGo? Initially, one might think this is impossible, because the only way that AlphaGo can gain its +1 reward for winning is
…
board and nothing else—because there is nothing else in AlphaGo’s model of the world. This setup corresponds to the abstract mathematical model of reinforcement learning, in which the reward signal arrives from outside the universe. Nothing AlphaGo can do, as far as it knows, has any effect on the code
…
-signal maximizer will wirehead. The AI safety community has discussed wireheading as a possibility for several years.25 The concern is not just that a reinforcement learning system such as AlphaGo might learn to cheat instead of mastering its intended task. The real issue arises when humans are the source of the
…
reward signal. If we propose that an AI system can be trained to behave well through reinforcement learning, with humans giving feedback signals that define the direction of improvement, the inevitable result is that the AI system works out how to control the
…
of pointless self-delusion on the part of the AI system, and you’d be right. But it’s a logical consequence of the way reinforcement learning is defined. The process works fine when the reward signal comes from “outside the universe” and is generated by some process that can never be
…
we avoid this kind of self-delusion? The problem comes from confusing two distinct things: reward signals and actual rewards. In the standard approach to reinforcement learning, these are one and the same. That seems to be a mistake. Instead, they should be treated separately, just as they are in assistance games
…
that AlphaGo “has the purpose of winning” is an oversimplification. A better description would be that AlphaGo is the result of an imperfect training process—reinforcement learning with self-play—for which winning was the reward. The training process is imperfect in the sense that it cannot produce a perfect Go player
…
(in which present-day reality turns out to be an illusion produced by a computer simulation) and recent work on the self-delusion problem in reinforcement learning.14 These examples, and more, convince me that the AI community should pay careful attention to the thrusts and counterthrusts of philosophical and economic debates
…
remained fairly constant, however, so the remaining preferences must arise from cultural and family influences. Quite possibly, children are constantly running some form of inverse reinforcement learning to identify the preferences of parents and peers in order to explain their behavior; children then adopt these preferences as their own. Even as adults
…
methods. Taken together, these are three of the most important application areas for AI. Deep learning has also played an important role in applications of reinforcement learning—for example, in learning the evaluation function that AlphaGo uses to estimate the desirability of possible future positions, and in learning controllers for complex robotic
…
may not have this effect, unless it is wrapped within an A/B testing framework (as is common in online marketing settings). Bandit algorithms and reinforcement learning algorithms will have this effect if they operate with an explicit representation of user state or an implicit representation in terms of the history of
…
face many important moral issues for which we are largely unprepared. 4. The following paper was among the first to make a clear connection between reinforcement learning algorithms and neurophysiological recordings: Wolfram Schultz, Peter Dayan, and P. Read Montague, “A neural substrate of prediction and reward,” Science 275 (1997): 1593–99. 5
…
for bots,” Wired, January 25, 2019. 49. AlphaZero is described by David Silver et al., “Mastering chess and shogi by self-play with a general reinforcement learning algorithm,” arXiv:1712.01815 (2017). 50. Optimal paths in graphs are found using the A* algorithm and its many descendants: Peter Hart, Nils Nilsson, and
…
as absolute rewards; by fixing the value of material to be positive; however, the program generally tended to work towards winning. 61. The application of reinforcement learning to produce a world-class backgammon program: Gerald Tesauro, “Temporal difference learning and TD-Gammon,” Communications of the ACM 38 (1995): 58–68. 62. The
…
DQN system that learns to play a wide variety of video games using deep RL: Volodymyr Mnih et al., “Human-level control through deep reinforcement learning,” Nature 518 (2015): 529–33. 63. Bill Gates’s remarks on Dota 2 AI: Catherine Clifford, “Bill Gates says gamer bots from Elon Musk-backed
…
control using generalized advantage estimation,” arXiv:1506.02438 (2015). A video demonstration is available at youtube.com/watch?v=SHLuf2ZBQSw. 47. A description of a reinforcement learning system that learns to play a capture-the-flag video game: Max Jaderberg et al., “Human-level performance in first-person multiplayer games with population
…
-based deep reinforcement learning,” arXiv:1807.01281 (2018). 48. A view of AI progress over the next few years: Peter Stone et al., “Artificial intelligence and life in 2030
…
under rational expectations,” Journal of Political Economy 86 (1978): 1009–44. 6. The first algorithms for IRL: Andrew Ng and Stuart Russell, “Algorithms for inverse reinforcement learning,” in Proceedings of the 17th International Conference on Machine Learning, ed. Pat Langley (Morgan Kaufmann, 2000). 7. Better algorithms for inverse RL: Pieter Abbeel and
…
Andrew Ng, “Apprenticeship learning via inverse reinforcement learning,” in Proceedings of the 21st International Conference on Machine Learning, ed. Russ Greiner and Dale Schuurmans (ACM Press, 2004). 8. Understanding inverse RL as Bayesian
…
updating: Deepak Ramachandran and Eyal Amir, “Bayesian inverse reinforcement learning,” in Proceedings of the 20th International Joint Conference on Artificial Intelligence, ed. Manuela Veloso (AAAI Press, 2007). 9. How to teach helicopters to fly and
…
of the brain in man,” American Journal of Psychiatry 120 (1963): 571–77. 25. A first mathematical treatment of wireheading, showing how it occurs in reinforcement learning agents: Mark Ring and Laurent Orseau, “Delusion, survival, and intelligent agents,” in Artificial General Intelligence: 4th International Conference, ed. Jürgen Schmidhuber, Kristinn Thórisson, and Moshe
…
Looks (Springer, 2011). One possible solution to the wireheading problem: Tom Everitt and Marcus Hutter, “Avoiding wireheading with value reinforcement learning,” arXiv:1605.03143 (2016). 26. How it might be possible for an intelligence explosion to occur safely: Benja Fallenstein and Nate Soares, “Vingean reflection: Reliable
…
–21. 8. A generalization of Harsanyi’s social aggregation theorem to the case of unequal prior beliefs: Andrew Critch, Nishant Desai, and Stuart Russell, “Negotiable reinforcement learning for Pareto optimal sequential decision-making,” in Advances in Neural Information Processing Systems 31, ed. Samy Bengio et al. (2018). 9. The sourcebook for ideal
…
values; see, for example, Pietro Carrera, Il gioco degli scacchi (Giovanni de Rossi, 1617). 2. A report describing Samuel’s heroic research on an early reinforcement learning algorithm for checkers: Arthur Samuel, “Some studies in machine learning using the game of checkers,” IBM Journal of Research and Development 3 (1959): 210–29
…
–34 exponential complexity of problems and, 38–39 halting problem and, 37–38 lookahead search, 47, 49–50, 260–61 propositional logic and, 268–70 reinforcement learning, 55–57, 105 subroutines within, 34 supervised learning, 58–59, 285–93 Alibaba, 250 AlphaGo, 6, 46–48, 49–50, 55, 91, 92, 206–7
…
and, 128–30 decisions affecting people, use of machines in, 126–28 robots built in humanoid form and, 124–26 intractable problems, 38–39 inverse reinforcement learning, 191–93 IQ, 48 Ishiguro, Hiroshi, 125 is-ought problem, 167 “it’s complicated” argument, 147–48 “it’s impossible” argument, 149–50 “it’s
…
, 86–87, 288–93 as evolutionary accelerator, 18–20 from experience, 285–93 explanation-based learning, 294–95 feature engineering and, 84–85 inverse reinforcement learning, 191–93 reinforcement learning, 17, 47, 55–57, 105, 190–91 supervised learning, 58–59, 285–93 from thinking, 293–95 LeCun, Yann, 47, 165 legal profession, 119
…
and Persons (Parfit), 225 Recombinant DNA Advisory Committee, 155 recombinant DNA research, 155–56 recursive self-improvement, 208–10 redlining, 128 reflex agents, 57–59 reinforcement learning, 17, 47, 55–57, 105, 190–91 remembering self, and preferences, 238–40 Repugnant Conclusion, 225 reputation systems, 108–9 “research can’t be controlled
by Martin Ford · 13 Sep 2021 · 288pp · 86,995 words
a Rubik’s Cube.7 The system was trained using high-speed simulation and succeeded only after the equivalent of about 10,000 years of reinforcement learning. Solving a Rubik’s Cube with one hand is not at all easy even for humans. Despite the company’s claim that it had achieved
…
highest-profile startups is Covariant, which was founded in 2017 but emerged from stealth mode only in early 2020. The researchers at Covariant believe that “reinforcement learning”—or essentially learning through trial and error—is the most effective way to progress, and the company claims to be building a system based on
…
dexterity. Many of the startup companies and university researchers working in this area believe, like Covariant, that a strategy founded on deep neural networks and reinforcement learning is the best way to fuel progress toward more dexterous robots. One notable exception is Vicarious, a small AI company based in the San Francisco
…
it can then become fodder for a supervised learning regimen deployed on a powerful neural network. While supervised learning dominates, another important technique—“reinforcement learning”—is used in certain applications. Reinforcement learning builds competence through repeated practice or trial and error. When an algorithm ultimately succeeds at a specified objective, it receives a digital
…
response to the proper command, it gets a treat. Repeat this process enough times and the dog will learn to reliably sit. The leader in reinforcement learning is the London-based company DeepMind, which is now owned by Google’s parent, Alphabet. DeepMind has made massive investments in research based on the
…
technique, merging it with powerful convolutional neural networks to develop what the company calls “deep reinforcement learning.” DeepMind began working on applying reinforcement learning to build AI systems that could play video games shortly after its founding in 2010. In January 2013, the company announced that
…
titans, including especially Google founder Larry Page, and in 2014 Google brushed aside a competing offer from Facebook and acquired DeepMind for $400 million. Deep reinforcement learning’s most notable achievement came in March 2016, when AlphaGo, a system developed by DeepMind to play the ancient game of Go, defeated Lee Sedol
…
train AlphaGo’s neural networks on thirty million moves extracted from detailed records of games played by the best human players. It then turned to reinforcement learning, essentially turning the system loose to play against itself. Over the course of thousands of simulated practice games, and under the relentless pressure of a
…
become a leader in artificial intelligence.8 While supervised learning depends on massive quantities of labeled data, reinforcement learning requires a huge number of practice runs, the majority of which end in spectacular failure. Reinforcement learning is especially well suited to games—in which an algorithm can rapidly churn though more matches than a
…
a lifetime. The approach can also be applied to real-world activities that can be simulated at high speed. The most important practical application of reinforcement learning is currently in training self-driving cars. Before the autonomous driving systems used by Waymo or Tesla ever see a real car or a real
…
from the brain but also contributing to its understanding. DeepMind once again made an important contribution to neuroscience when the company leveraged its expertise in reinforcement learning in early 2020 by exploring the operation of dopamine neurons in the brain.21 Since the 1990s, neuroscientists have understood that these special neurons make
…
achieved is greater than expected, then relatively more dopamine is released. If the result underperforms, then less of this “feel good” chemical is generated. Computational reinforcement learning traditionally works in much the same way; the algorithm makes a prediction and then adjusts the reward based on the difference between the predicted and
…
actual results. Researchers at DeepMind were able to greatly improve a reinforcement learning algorithm by generating a distribution of predictions, rather than a single average prediction, and then adjusting the rewards accordingly. The company then teamed with a
…
parallel results in both a digital algorithm and the biological brain. Research of this type reflects the confidence that Hassabis and his team have in reinforcement learning and their belief that it is a critical component of any attempt to progress toward more general artificial intelligence. In this, they are something of
…
an outlier. Facebook’s Yann LeCun, for example, has stated that he believes reinforcement learning plays a relatively minor role. In his presentations, he often says that if intelligence were a black forest cake, then
…
reinforcement learning would amount to only the cherry on top.22 The team at DeepMind believes it is far more central—and that it possibly provides a
…
viable path to achieving AGI. We generally describe reinforcement learning in terms of a reward-driven algorithm to optimize some external macro process—for example, learning to play the game of Go or figuring out
…
how to drive a simulated car. However, Hassabis points out that reinforcement learning also plays a critical role internal to the brain, and that it may be essential to the emergence of intelligence. It’s conceivable that
…
reinforcement learning might be the primary mechanism that drives the brain toward curiosity, learning and reason. Imagine, for example, that the brain’s inherent objective is simply
…
that’s a highly useful motivation.”23 In other words, the engine powering our continuous drive to understand the world around us could be a reinforcement learning algorithm linked to the production of dopamine. An entirely different approach to building more general machine intelligence is being pursued by David Ferrucci, the CEO
…
result in a system that can learn from far less data than is the case with current deep learning systems that rely on supervised or reinforcement learning techniques. When Kurzweil approached Google’s Larry Page to seek funding for a venture to put these ideas into practice, Page instead convinced him to
…
in terms of staffing levels, it remains far smaller than the more established company. Like DeepMind, OpenAI has developed powerful deep neural networks trained using reinforcement learning techniques, and its team of researchers has created systems capable of defeating the best human players at video games, such as Dot 2. OpenAI, however
…
As we’ve seen, the two primary techniques used to train deep learning systems are supervised learning, which requires large quantities of labeled data, and reinforcement learning, which requires a huge number of trial-and-error iterations as an algorithm attempts to succeed at a task. Though human beings likewise employ these
…
done.56 Understanding causal relationships is critical to imagination and to the generation of mental counterfactual scenarios that enable us to solve problems. Unlike a reinforcement learning algorithm that needs to fail thousands of times before figuring out how to succeed, we can run a kind of simulation in our heads and
…
overcoming the major limitations that confine today’s AI systems to highly specific, narrow domains. AlphaStar, for example, must be extensively trained, using supervised and reinforcement learning techniques, to play in the role of a particular alien species. Switching to a different species, with different relative strengths, requires re-training from scratch
…
. Volodymyr Mnih, Koray Kavukcuoglu, David Silver et al. “Playing Atari with deep reinforcement learning,” DeepMind Research, January 1, 2013, deepmind.com/research/publications/playing-atari-deep-reinforcement-learning. 5. Volodymyr Mnih, Koray Kavukcuoglu, David Silver et al., “Human-level control through deep reinforcement learning,” Nature, volume 518, pp. 529–533 (2015), February 25, 2015, www.nature
by Sebastian Mallaby; · 30 Mar 2026 · 607pp · 161,998 words
and use that feedback to fine-tune their behavior. Sutton’s textbook laid out ideas for implementing this approach, which was known as reinforcement learning. The basic idea of reinforcement learning, often known as RL, was already familiar to the gaming fraternity. When he had worked for Molyneux, Hassabis had proposed that a simple
…
was therefore just one piece of the puzzle. The larger challenge was to combine deep learning, which would solve challenges such as computer vision, with reinforcement learning, which would deliver other facets of intelligence, including the ability to hatch plans and think strategically. To deliver on this premise, the business plan
…
has to develop algorithms that narrow the search for the best action. Relative to deep learning, with its mind-boggling nonlinearities and impressive practical results, reinforcement learning seemed theoretical and primitive. But to Mnih and other believers in RL, the promise of agents that could learn from experience remained thrilling. Whereas
…
spirits at DeepMind were more rewarding than the ones spent at his university laboratory.[10] Silver had by now established himself as an authority on reinforcement learning, but the other newcomers at Bernard Street demonstrated DeepMind’s commitment to intellectual diversity.[11] Two recruits worked on statistical methods for quantifying uncertainty
…
but handcrafted algorithms for dealing with specific tasks could not generalize to multiple Atari environments. DeepMind’s probabilistic duo took a different tack, creating a reinforcement-learning agent that began with a model of how the games worked, then increased its confidence if trial-and-error play confirmed its hypothesis. This second
…
unrandomized and unrepresentative experiences, it would never master Atari. To get around this obstacle, David Silver proposed a new riff on an old idea in reinforcement learning. Back in the 1990s, RL scientists had experimented with a technique called memory replay: To extract maximum learning from limited data, experiences were stored
…
succeeded previously. Hence its monomaniacal zeal for firing off torpedoes.[27] Pondering this glitch, Mnih realized that it reflected another clash between supervised learning and reinforcement learning. The goal of a computer-vision system was to predict how photos were labeled. Every time it answered right, it would get a simple,
…
the team had assembled, Hassabis had shown exquisite judgment. In choosing the Atari challenge, he had understood that the moment to fuse deep learning and reinforcement learning had arrived. The result was another ImageNet moment—not just for vision, but for agents. Chapter 7 Thiel Trouble On October 8, 2012, while
…
laugh. “One day!”[10] In spring 2014, Silver suggested to Hassabis that Go’s day was approaching. The success of the Atari project, combining reinforcement learning with deep learning, provided just the sort of springboard that Silver had been waiting for.[11] It was this suggestion, the culmination of conversations begun
…
networks, mimicking the fast-thinking parts of human intelligence. But what was really powerful was how the intuitive deep-learning models worked with the introspective reinforcement learning, the tree search—the deeper, slower, System Two side of intelligence. Thanks to Maddison’s policy net, the search algorithm no longer began with
…
with random moves and discovering which ones generated a reward signal. The new Agent Zero would stand as a triumph for Silver’s scientific specialty, reinforcement learning; it would also mark a leap toward machine autonomy. Yesterday’s deep-learning systems had ingested data that represented human knowledge, curated by human
…
programmers. Tomorrow’s reinforcement-learning agents would rely on data that they generated themselves, by acting in the world and gathering experiences.[2] A year or so passed before Silver
…
AlphaZero meant, not just for humans and their cognitive limits, but rather for the road to artificial general intelligence. For Silver, this breakthrough for reinforcement learning marked a revolution. AlphaZero had mastered three different complex games from scratch, without human instruction or human data. The old obstacle to AI—the impossibility
…
mastered multiple games, demonstrating that agentic trial and error could deliver impressive versatility. At the same time, however, both systems achieved their advances in reinforcement learning thanks to progress in deep learning. Back in 2013, the Atari system had leveraged a particular kind of network, known as a convolutional neural net
…
a new DeepMind office in Edmonton, Alberta, cementing the company’s lead in designing agents that learned through experience. Hassabis shared Silver’s enthusiasm for reinforcement learning, albeit for his own reasons. Thanks to his PhD in neuroscience, he had always thought that artificial general intelligence would depend on integrating multiple
…
. For Hassabis, the mission was founding a company to go after AGI. For Silver, it was to push the frontier of reinforcement learning. For Vlad Mnih, it was fusing reinforcement learning with deep learning. Meanwhile, for Ilya Sutskever, the opportunity that sat most squarely in his wheelhouse was the appearance of a new
…
equivalent of the ImageNet breakthrough.[15] Completing his doctoral work in 2013, Sutskever was for recurrent networks and sequential data what David Silver was for reinforcement learning and Go: a leading authority who had not completely cracked his chosen problem. After his PhD, Sutskever continued to wrestle with the challenge of
…
of those experiments that advances science by failing. Far from teaching DeepMind’s agents to master concepts, Purves’s naturalistic simulations demonstrated the limits to reinforcement learning. RL agents could more or less manage a simplified natural world, but too many irregularities and unclassifiable shapes flummoxed them. When confronted with an
…
environment would not necessarily lead to improved performance. It followed that the path ahead for scaling language models was excitingly open. The path ahead for reinforcement learning looked murky and uncertain.[7] “A scientist doesn’t think, ‘Oh, some experiments failed, some succeeded,’ ” Hassabis reflected. “In science, it’s just projects.
…
direction.” * * * • • • The next direction on which DeepMind focused was David Silver’s post-AlphaZero effort. Starting in early 2018, Silver set about building a reinforcement-learning agent to crack StarCraft II, a battle simulation game whose top players attract millions of devoted followers. StarCraft’s complexity makes even Go appear simple
…
create intelligent systems that will accelerate scientific discovery,” Hassabis announced proudly.[14] The question was what this triumph signified for DeepMind’s larger bet on reinforcement learning. Like most RL breakthroughs, this one was enabled by an advance in deep learning—thanks to Vinyals, AlphaStar was built on the transformer architecture.
…
in South Korea, Hassabis had also realized something more. The gamification of protein folding appeared to change a general computational challenge into the kind of reinforcement-learning problem at which DeepMind excelled: There was a clear objective, feedback in the form of a precise score, and a virtual environment that allowed
…
would have to map amino acid chains onto their folded structures. No less a figure than David Silver backed this deep-learning approach, acknowledging that reinforcement learning was not suited to the protein problem.[11] * * * • • • The question was how to overcome the hurdles that Jumper had confronted during his PhD: a
…
a lab that prioritized language models. At DeepMind, Hassabis had recently begun speaking of three coequal “paradigms” within the research team: The first regarded reinforcement learning as the path to AGI; the second aimed to implement ideas from neuroscience; the third built neural networks that learned from data, with language models
…
human responses. The evaluator model judged Sparrow’s answers, reinforced good ones with rewards, and nudged Sparrow to tweak its parameters accordingly. When this reinforcement learning was complete, Sparrow proved both delightful and responsible. It could still make mistakes or exhibit biases arising from its training data. But it strove to
…
the Hinton–Bengio warning—the one represented by Geoffrey Irving and Jan Leike. To prevent models from deceiving humans, you needed technical solutions such as reinforcement learning from human feedback, bolstered by pre-release red-team tests to discover residual misbehaviors. Already, thanks to this formula, chatbots had become less biased,
…
chinks of light. Inscrutable, unpredictable, and therefore inherently dangerous systems became at least partially understandable. Irving’s second promising project addressed the central weakness in reinforcement learning from human feedback. As AI generated ever more sophisticated outputs, it would outstrip humans’ ability to provide feedback on them. It was one thing
…
Superhuman Agent.” Meanwhile at OpenAI, Ilya Sutskever had launched a project called GPT-Zero in 2021: The name was a tribute to Silver’s greatest reinforcement-learning triumph, the AlphaZero model.[21] During the incubation of AlphaGo, and again when the transformer architecture appeared, Sutskever and Silver had embodied the two
…
ground. During the evolution of standard language models, post-training had begun with clever prompting, then moved on to fine-tuning, and then incorporated reinforcement learning from human feedback. Likewise, now that thinking models had moved through chain-of-thought prompting and fine-tuning, the logical next step was to fortify
…
thinking through their implications? Presumably, the more thinking an AI system did, the less data it would need; and this insight led researchers back to reinforcement learning. Rumination was precisely what AlphaGo and AlphaZero had done: They planned out move sequences, evaluated them, backed up and searched out more, engaging in what
…
a process. It was a gamble. To prepare for their big rally, Shazeer and Rae had invited experimentalists across Google DeepMind to share ideas on reinforcement learning for language. The response had been stunning: More than 250 scientists had showed up at the brainstorming session with a one-slide presentation. Clearly,
…
Rather than kick-starting its thinking abilities by learning from human judgment, the Zero system emulated its namesake, DeepMind’s AlphaZero, and skipped straight to reinforcement learning. Presented with a series of math and logic questions, it experimented randomly with various chains of thought, refining its methods as it discovered which chains
…
then it could be seriously existential for humanity.”[29] * * * • • • In mid-2025, I spoke again with David Silver. As the long-term champion of reinforcement learning, he embodied both its promise and its danger. Given the comeback of RL, he also represented the beyond—beyond chatbots, beyond coding assistants, beyond imagining
…
, based purely on deduction, had been limited, for sure. But the AI revolution had equipped machines to think inductively. Thanks to deep learning and reinforcement learning, classical computers could confront uncertainty and intuit what to do next. Penrose’s quantum speculations about human and machine intelligence had been rendered irrelevant. The
…
was worth £100 million. Geoffrey Hinton, author interview, September 6, 2023. BACK TO NOTE REFERENCE 21 The potential upside from fusing deep learning and reinforcement learning influenced Google’s attitude to the pricing. Harrison recalled that DeepMind’s formula “was fairly remarkable and an evolution on what Geoff [Hinton] had described
…
take self-play to surpass human experts. Sutskever, author interview. BACK TO NOTE REFERENCE 27 The design of rewards is a key challenge in building reinforcement-learning systems. Guez’s value net allowed the Go system to recognize a rewarding position. BACK TO NOTE REFERENCE 28 Huang, author interview. BACK TO
…
–59, doi.org/10.1038/nature24270. BACK TO NOTE REFERENCE 4 David Silver et al., “Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm,” arXiv, December 5, 2017, doi.org/10.48550/arXiv.1712.01815. BACK TO NOTE REFERENCE 5 Silver, author interview. BACK TO NOTE REFERENCE
…
researchers, Jonathan Lai and James An, circulated an influential internal paper laying out the way to supersede OpenAI’s 2023 technique. They called their method Reinforcement Learning with Verified Rewards. BACK TO NOTE REFERENCE 27 Rae, author interview. BACK TO NOTE REFERENCE 28 Oriol Vinyals, author interview, February 6, 2025. BACK
…
into’ National Security Implications of DeepSeek’s AI.” BACK TO NOTE REFERENCE 17 Daya Guo et al., “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning,” arXiv, January 22, 2025, arxiv.org/abs/2501.12948. BACK TO NOTE REFERENCE 18 Other reasoning models also prompted suggestions that AI was approaching
…
Action (MIT), February 18, 2025. BACK TO NOTE REFERENCE 29 David Silver, lecture at the RLC 2024 conference, “David Silver—Towards Superhuman Intelligence—RLC 2024,” Reinforcement Learning Conference (RLC), October 1, 2024, YouTube, 1 hr., 3 min., 13 sec., youtube.com/watch?v=pkpJMNjvgXw. BACK TO NOTE REFERENCE 30 David Silver
…
Ranzato, Marc’Aurelio, 218 Rayburn, Sam, 419n5 reasoning, RL and, 358, 362–66 recurrent neural network, 200, 208 Redmond, Michael, 159 Rees, Geraint, 421n9 reinforcement learning (RL) AGI and, 97 AlphaGo trained with, 152 AlphaProof and, 378 AlphaStar and, 226–27 AlphaZero and, 195–200 in Atari challenge, 101 concept of
…
2, 106 Q* project and, 356, 359 from raw experience, 412n24 reasoning and, 358, 362–66 safety risks of, 373–75 with verified rewards, 438n27 reinforcement learning from human feedback (RLHF), 298–99, 301, 335 Renaissance Technologies, 232 Reos Partners, 67 Republic game, 36–41 residual neural network (ResNet), 197–98
by Temple Grandin and Sean Barron · 30 Sep 2012 · 347pp · 123,884 words
rule teachers cite, although many teachers will overlook infringements on a regular basis. It’s a breeding ground for mistrust to develop. Consistently applied consequences reinforce learning; inconsistency creates an environment that hampers, rather than facilitates, learning social behaviors. Rule #2 Not Everything that Happens is Equally Important in the Grand Scheme
by Ben Goertzel and Pei Wang · 1 Jan 2007 · 303pp · 67,891 words
Hunting: A Thought-Experiment in Embodied Social Learning, Cognitive Process Integration, and the Dynamic Emergence of the Self 217 Ben Goertzel viii Probabilistic Logic Based Reinforcement Learning of Simple Embodied Behaviors in a 3D Simulation World Ari Heljakka, Ben Goertzel, Welter Silva, Cassio Pennachin, Andre’ Senna and Izabela Goertzel How Do We
…
al chapter discusses the learning of some very simple behaviors for a simulated humanoid agent in the AGISim 3D simulation world, via a pure “embodied reinforcement learning” methodology. In Piagetan terms, these are “infantile-level” tasks, but to achieve them within the Novamente architecture nevertheless requires a fairly subtle integration of various
…
or empirical analysis of particular problems or domains, we will study how focus management can be learned. One idea is to use state-action-reward reinforcement learning algorithms to automatically generate focus management schemes. The actions are the choice of attention fixation. The state will be both the state of the environment
…
be given here. Inference control in Novamente takes several forms: 1. Standard forward-chaining and backward-chaining inference heuristics (see e.g. [45]) 2. A reinforcement learning mechanism that allows inference rules to be chosen based on experience. Probabilities are tabulated regarding which inference rules have been useful in the past in
…
used to bias the choices of inference rules during forward or backward chaining inference 3. Application of PLN inference to the probabilities used in the reinforcement learning mechanism--enables generalization, abstraction and analogy to be used in guessing which inference rules are most useful in a given context These different approaches to
…
, Architectures and Algorithms B. Goertzel and P. Wang (Eds.) IOS Press, 2007 © 2007 The authors and IOS Press. All rights reserved. 253 Probabilistic Logic Based Reinforcement Learning of Simple Embodied Behaviors in a 3D Simulation World Ari HELJAKKA*, Ben GOERTZEL, Welter SILVA, Cassio PENNACHIN, Andre’ SENNA and Izabela GOERTZEL Novamente LLC * heljakka
…
such as theorem-proving and linguistic semantics. In the Novamente AGI architecture, however, probabilistic logic is used for a wider variety of purposes, including simple reinforcement learning of infantile behaviors, which are primarily concerned with perception and action rather than abstract cognition. This paper reports some simple experiments designed to validate the
…
viability of this approach, via using the PLN probabilistic logic framework, implemented within the Novamente AGI architecture, to carry out reinforcement learning of simple embodied behaviors in a 3D simulation world (AGISim). The specific experiment focused upon involves teaching Novamente to play the game of “fetch” using
…
reinforcement learning based on repeated partial rewards. Novamente is an integrative AGI architecture involving considerably more than just PLN; however, in this “fetch” experiment, the only cognitive
…
environment). And, where learning human language is concerned, embodiment gives the opportunity for robust symbol grounding [3]. 254 A. Heljakka et al. / Probabilistic Logic Based Reinforcement Learning On the other hand, perhaps the biggest drawback of embodiment from the AGI developer’s perspective is pragmatic in nature. Building, maintaining and using robots
…
an AI controls in AGISim is designed to bear suffi1 2 sourceforge.net/projects/agisim. crystal.sourceforge.net. A. Heljakka et al. / Probabilistic Logic Based Reinforcement Learning 255 cient resemblance to a simple humanoid robot to enable the fairly straightforward porting of control routines learned in AGISim to a physical robot. Our
…
one narrow domain or range of cognitive functions. The NAIE integrates aspects of prior AI projects and approaches, including symbolic, neural-network, evolutionary programming and reinforcement learning. The existing codebase is being applied in bioinformatics, NLP and other domains. To save space, some of the discussion in this paper will assume a
…
to find it. Human babies less than 9 months of age will often look in location A, 256 A. Heljakka et al. / Probabilistic Logic Based Reinforcement Learning but older babies look in B. The NAIE learns through interactive experience to look in location B – it learns that objects exist even when unobserved
…
role of the dog. Figure 1. Dogs playing fetch, correctly (bottom) and incorrectly (middle). (Illustrations by Zebulon Goertzel). A. Heljakka et al. / Probabilistic Logic Based Reinforcement Learning 257 Figure 2. A screenshot of Novamente and its teacher playing fetch in the AGISim simulation world. Here Novamente is returning the ball to the
…
be made to learn similar behaviour in a richer environment and with a larger set of possible actions. A. Heljakka et al. / Probabilistic Logic Based Reinforcement Learning 258 3. Learning Fetch Within the Novamente Architecture Novamente is an integrative AGI architecture, in which the highest levels of intelligence are intended to be
…
the current incomplete version of the integrated Novamente cognition. From a Novamente point of view, it is interesting mostly as a “smoke test” for embodied reinforcement learning, to indicate that the basic mechanisms required for cognitive interaction with AGISim are integrated adequately and working correctly. As noted above, fetch can be learned
…
an external vision-processing module that supplies either voxel or polygon vision inputs of the sort described above. A. Heljakka et al. / Probabilistic Logic Based Reinforcement Learning 259 The object-recognition problem is restricted to the first two cases. We have not dealt with the voxel vision case so far, but have
…
via much simpler methods than anything in Novamente) but on what it teaches us about the use of probabilistic inference in the context of embodied reinforcement learning. 4. Pattern Mining in Novamente The “pattern mining” step mentioned above has not been discussed in previous publications on Novamente. We mention it briefly here
…
very simple, but may be made more sophisticated in future. The inputs of pattern mining are, in general, A. Heljakka et al. / Probabilistic Logic Based Reinforcement Learning 260 • • • Atoms denoting raw outputs of the “sensors” that Novamente possesses in the AGISim world Atoms indicating actions Novamente has taken, e.g. in the
…
in full-fledged Novamente Node/Link notation, SequentialAND SimultaneousAND EvaluationLink holding ball EvaluationLink near ListLink (me, teacher) Reward A. Heljakka et al. / Probabilistic Logic Based Reinforcement Learning 261 The predicates required here are near() and holding(), as well as the primitive “sensation” of Reward. Given these predicates as primitives, the mining of
…
PLN backward chainer is to find some way to prove that if some actionable predicates become true, then A. Heljakka et al. / Probabilistic Logic Based Reinforcement Learning 262 Evaluation (Reward) becomes true. This inference is possible by assuming that trying out actions is always possible, ie. the actions are considered to be
…
y – termination of x) For example, in many cases one may use w(x,y) = k/(diff+k) A. Heljakka et al. / Probabilistic Logic Based Reinforcement Learning 263 where k is an adjustable parameter. For more details on temporal links and the event calculus variant we use, see [18]. In order to
…
. Firstly, the ModusPonensRule is unsurprisingly a probabilistic version of modus ponens, i.e. Implication A B A |B A. Heljakka et al. / Probabilistic Logic Based Reinforcement Learning 264 Modus Ponens can also be applied to PredictiveImplications, insofar as the system keeps track of the structure of the proof tree so as to
…
and schemata. The learned plan discussed here is a representative one that lends itself relatively well to discussion. A. Heljakka et al. / Probabilistic Logic Based Reinforcement Learning 265 Firstly, we define the specific predicates used as primitives for this learning experiment: • • • • • • Reward – a built-in sensation corresponding to the Novamente agent getting
…
EvaluationLink goto ball etc. The two stages of the reward function are: Stage 1 PredImp holding ball Reward A. Heljakka et al. / Probabilistic Logic Based Reinforcement Learning 266 Stage 2 PredImp SeqAnd holding ball done goto teacher done drop ball Reward 6.2. Knowledge Gained via Pattern Mining Next, in the course
…
of this simple task of fetch, PLN can do the learning job quite efficiently with support only from A. Heljakka et al. / Probabilistic Logic Based Reinforcement Learning 267 pattern mining, without needing support from more sophisticated pattern recognition tools like MOSES. The final inference trajectory follows. First of all, the inference target
…
] "try":PredicateNode <0,0> [6552272] ExecutionLink [7505792] "drop":GroundedSchemaNode <0,0> [6564640] "Ball":ConceptNode <0,0> [6559856] 268 A. Heljakka et al. / Probabilistic Logic Based Reinforcement Learning Graphically, the previous two link constructs would be denoted Figure 3. and Figure 4. Respectively. In the rest of this discussion we will often substitute
…
produced by applying SimpleANDRule to its three child EvaluationLinks. The EvaluationLink [104300720] was produced by applying ModusPonensRule to: A. Heljakka et al. / Probabilistic Logic Based Reinforcement Learning 269 Figure 5. which was mined from perception data, and to Figure 6. The SequentialANDLink [104307776] was produced by applying SimpleANDRule to its two child
…
EvaluationLinks. The EvaluationLink [72926800] was produced by applying RewritingRule to: Figure 7. 270 A. Heljakka et al. / Probabilistic Logic Based Reinforcement Learning and Figure 8. The EvaluationLink [72916304], as well as all other try statements, were considered axiomatic, and technically produced by applying CrispUnificationRule to: Figure 9
…
, and technically produced by applying CrispUnificationRule to: Figure 10. The EvaluationLink [72913264] was produced by applying RewritingRule to: A. Heljakka et al. / Probabilistic Logic Based Reinforcement Learning 271 Figure 11. and Figure 12. Returning to the first PredictiveImplicationLink’s children, EvaluationLink [72895584] was produced by applying RewritingRule to: Figure 13. and Figure
…
14. A. Heljakka et al. / Probabilistic Logic Based Reinforcement Learning 272 which were both axiomatic. QED! For illustration, we finally present here an example of a plan the agent formed during the partial reward stage
…
, 0> [104296656] [6565264] [104296656] was produced by applying ModusPonensRule to: Figure 15. On the other hand, Figure 16. A. Heljakka et al. / Probabilistic Logic Based Reinforcement Learning 273 This causes the system to come up with the following plan which works but contains obvious redundancy: ExecutionLink <0,0.00062> [6888032] "goto":GroundedSchemaNode
…
to deal with a wide variety of learning tasks corresponding to the full range of levels of 274 A. Heljakka et al. / Probabilistic Logic Based Reinforcement Learning cognitive development. Making all this “general infrastructure” work together to yield a simple behavior like fetch is a lot of work – the infrastructure doesn’t
…
Maia and others), and earlier versions of PLN (mainly Guilherme Lamacie, the late Jeff Pressing, and Pei Wang). A. Heljakka et al. / Probabilistic Logic Based Reinforcement Learning 275 References [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] Guha, R. V. and Lenat
by Yarden Katz
tends to be set by the day’s most attention-grabbing systems. Currently, these are computing systems that use neural networks, sometimes in combination with reinforcement learning, a set of frameworks in which computational agents are trained to learn by reinforcement signals (described in more detail below). These systems carry many narratives
…
regimen of rewards and punishments (figure 3.3).46 The success of DeepMind’s systems is attributed not only to deep networks but also to reinforcement learning.47 Reinforcement learning imbibes the behaviorist principle that intelligence arises by individual agents being disciplined by the environment. This understanding of the self discounts people’s knowledge
…
a Chess-Playing Machine,” New York Times, December 26, 2018. 46. This behaviorist core was made explicit by Nature magazine’s coverage of DeepMind’s reinforcement-learning system. According to the magazine, a primary challenge for such systems is “avoiding ‘superstitious behavior’ in which statistical associations may be misinterpreted as causal.” Bernhard
…
. See John E. R. Staddon, “The ‘Superstition’ Experiment: A Reversible Figure,” Journal of Experimental Psychology: General 121, no. 3 (1992): 270–72. Also note that reinforcement learning as a field encompasses multiple modeling approaches, including ones where the agents form internal representations that a behaviorist such as Skinner might have rejected as
…
computing that at some point appeared under the umbrella of “AI,” reinforcement learning has been developed in other fields and under different labels. The most popular reinforcement learning textbook was published in the late 1990s; see Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction (Cambridge, Mass.: MIT Press, 1998). Yet the rebranded AI
…
’s narratives present it as an utterly new development. MIT Technology Review, for example, featured “reinforcement learning” in its list of “breakthrough technologies” in 2014. 48. B. F. Skinner, Reflections on Behaviorism and Society (Englewood Cliffs, N.J.: Prentice-Hall, 1978). 49
…
16, 2018. Supiot, Alain. The Spirit of Philadelphia: Social Justice vs. the Total Market. New York: Verso, 2012. Sutton, Richard S., and Andrew G. Barto. Reinforcement Learning: An Introduction. Cambridge, Mass.: MIT Press, 1998. Taylor, Robert W. “In Memoriam: JCR Licklider 1915–1990.” Digital Systems Research Center, Palo Alto, Calif., 1990. Tegmark
…
, 33, 191, 244n37 rationalistic tradition, 193, 197–99, 203, 207, 209, 218, 221, 223 Reagan, Ronald, 43, 247n68 rebranded AI. See artificial intelligence, rebranding of reinforcement learning, 106, 121, 267nn46–47. See also behaviorism rematriation, 234, 274n37 Riley, Boots, 168 Robinson, Cedric, 1, 7, 93–94, 160, 233, 264n2, 278n19 robotics, 35
by Norman Doidge · 15 Mar 2007 · 515pp · 136,938 words
by Ash Fontana · 4 May 2021 · 296pp · 66,815 words
by Martin Ford · 16 Nov 2018 · 586pp · 186,548 words
by Aurélien Géron · 13 Mar 2017 · 1,331pp · 163,200 words
by Cecilia Heyes · 15 Apr 2018
by Jeff Booth · 14 Jan 2020 · 180pp · 55,805 words
by Kenneth Payne · 16 Jun 2021 · 339pp · 92,785 words
by Maximilian Kasy · 15 Jan 2025 · 209pp · 63,332 words
by Nigel Toon · 13 Dec 2023
by Michael Wooldridge · 2 Nov 2018 · 346pp · 97,890 words
by Nick Bostrom · 3 Jun 2014 · 574pp · 164,509 words
by Robert M. Sapolsky · 1 May 2017 · 1,261pp · 294,715 words
by Terrence J. Sejnowski · 27 Sep 2018
by John Brockman · 19 Feb 2019 · 339pp · 94,769 words
by Scott E. Page · 27 Nov 2018 · 543pp · 153,550 words
by Jacob Turner · 29 Oct 2018 · 688pp · 147,571 words
by Mustafa Suleyman · 4 Sep 2023 · 444pp · 117,770 words
by Joseph Henrich · 27 Oct 2015 · 631pp · 177,227 words
by Pedro Domingos · 21 Sep 2015 · 396pp · 117,149 words
by Eliezer Yudkowsky · 11 Mar 2015 · 1,737pp · 491,616 words
by Andrea L. Glenn and Adrian Raine · 7 Mar 2014
by Steve Kaufmann · 15 Jan 2003
by David N. Blank-Edelman · 16 Sep 2018
by Stephen M Fleming · 27 Apr 2021
by Karen Hao · 19 May 2025 · 660pp · 179,531 words
by Paul Scharre · 18 Jan 2023
by Kevin Roose · 481pp · 151,124 words
by Kenneth Cukier, Viktor Mayer-Schönberger and Francis de Véricourt · 10 May 2021 · 291pp · 80,068 words
by Andrew McAfee and Erik Brynjolfsson · 26 Jun 2017 · 472pp · 117,093 words
by Daniel C. Dennett · 15 Jan 1995 · 846pp · 232,630 words
by Calestous Juma · 27 May 2017
by Eric Topol · 1 Jan 2019 · 424pp · 114,905 words
by Cade Metz · 15 Mar 2021 · 414pp · 109,622 words
by Henry A Kissinger, Eric Schmidt and Daniel Huttenlocher · 2 Nov 2021 · 194pp · 57,434 words
by Ray Kurzweil · 25 Jun 2024
by Robert J. Shiller · 14 Oct 2019 · 611pp · 130,419 words
by Diane Coyle · 11 Oct 2021 · 305pp · 75,697 words
by Adam Greenfield · 29 May 2017 · 410pp · 119,823 words
by Gabriel Weinberg and Lauren McCann · 17 Jun 2019
by Marcus Du Sautoy · 7 Mar 2019 · 337pp · 103,522 words
by Barbara Oakley Phd · 20 Oct 2008
by Nicole Aschoff
by Nate Silver · 12 Aug 2024 · 848pp · 227,015 words
by David Easley, Marcos López de Prado and Maureen O'Hara · 28 Sep 2013
by Keach Hagey · 19 May 2025 · 439pp · 125,379 words
by J. Doyne Farmer · 24 Apr 2024 · 406pp · 114,438 words
by Erica Thompson · 6 Dec 2022 · 250pp · 79,360 words
by Frederi G. Viens, Maria C. Mariani and Ionut Florescu · 20 Dec 2011 · 443pp · 51,804 words
by John Y. Campbell and Tarun Ramadorai · 25 Jul 2025
by Veljko Krunic · 29 Mar 2020
by Daniel J. Levitin · 18 Aug 2014 · 685pp · 203,949 words
by Richard E. Nisbett · 17 Aug 2015 · 397pp · 109,631 words
by Max More and Natasha Vita-More · 4 Mar 2013 · 798pp · 240,182 words
by David Epstein · 1 Mar 2019 · 406pp · 109,794 words
by Parmy Olson · 284pp · 96,087 words
by Christopher Summerfield · 11 Mar 2025 · 412pp · 122,298 words
by Daron Acemoglu and Simon Johnson · 15 May 2023 · 619pp · 177,548 words
by Mark Bergen · 5 Sep 2022 · 642pp · 141,888 words
by Kate Crawford · 5 Apr 2021 · 355pp · 89,587 words
by David Easley and Jon Kleinberg · 15 Nov 2010 · 1,535pp · 337,071 words
by Nick Bostrom and Milan M. Cirkovic · 2 Jul 2008
by Tim O'Reilly · 9 Oct 2017 · 561pp · 157,589 words
by Aurelien Geron · 14 Aug 2019
by Ed Finn · 10 Mar 2017 · 285pp · 86,853 words
by Jacob Ward · 25 Jan 2022 · 292pp · 94,660 words
by Ariel Ezrachi and Maurice E. Stucke · 30 Nov 2016
by John Brockman · 5 Oct 2015 · 481pp · 125,946 words
by Bruce Schneier · 7 Feb 2023 · 306pp · 82,909 words
by Joi Ito and Jeff Howe · 6 Dec 2016 · 254pp · 76,064 words
by Anders Lisdorf
by Eliezer Yudkowsky and Nate Soares · 15 Sep 2025 · 215pp · 64,699 words
by Ajay Agrawal, Joshua Gans and Avi Goldfarb · 16 Apr 2018 · 345pp · 75,660 words
by Paul Scharre · 23 Apr 2018 · 590pp · 152,595 words
by David Sumpter · 18 Jun 2018 · 276pp · 81,153 words
by William Davidow and Michael Malone · 18 Feb 2020 · 304pp · 80,143 words
by Mehmed Kantardzić · 2 Jan 2003 · 721pp · 197,134 words
by Joanna Walsh · 22 Sep 2025 · 255pp · 80,203 words
by Daniel Susskind · 14 Jan 2020 · 419pp · 109,241 words
by Meredith Broussard · 19 Apr 2018 · 245pp · 83,272 words
by Kai-Fu Lee · 14 Sep 2018 · 307pp · 88,180 words
by Ethan Mollick · 2 Apr 2024 · 189pp · 58,076 words
by Peter H. Diamandis and Steven Kotler · 13 Apr 2026 · 225pp · 76,418 words
by Paul R. Daugherty and H. James Wilson · 15 Jan 2018 · 523pp · 61,179 words
by Dipanjan Sarkar · 1 Dec 2016
by Mo Gawdat · 29 Sep 2021 · 259pp · 84,261 words
by Toby Ord · 24 Mar 2020 · 513pp · 152,381 words
by Calum Chace · 17 Jul 2016 · 477pp · 75,408 words
by Yuval Noah Harari · 29 Aug 2018 · 389pp · 119,487 words
by Robert H. Frank · 15 Jan 1999 · 416pp · 112,159 words
by Tom Chivers · 12 Jun 2019 · 289pp · 92,714 words
by Nicole Kobie · 3 Jul 2024 · 348pp · 119,358 words
by Pistono, Federico · 14 Oct 2012 · 245pp · 64,288 words
by Luke Dormehl · 10 Aug 2016 · 252pp · 74,167 words
by Amy Webb · 5 Mar 2019 · 340pp · 97,723 words
by Po Bronson and Ashley Merryman · 19 Feb 2013 · 407pp · 109,653 words
by Richard Dawkins · 15 Mar 2017 · 420pp · 130,714 words
by Yuval Noah Harari · 9 Sep 2024 · 566pp · 169,013 words
by Sergey Young · 23 Aug 2021 · 326pp · 88,968 words
by Jim Al-Khalili · 17 Apr 2019 · 381pp · 120,361 words
by Kevin Kelly · 6 Jun 2016 · 371pp · 108,317 words
by Andreas Herrmann, Walter Brenner and Rupert Stadler · 25 Mar 2018
by Kevin Roose · 9 Mar 2021 · 208pp · 57,602 words
by Carissa Véliz · 21 Apr 2026 · 503pp · 129,255 words
by George Gilder · 16 Jul 2018 · 332pp · 93,672 words
by Spencer Jakab · 1 Feb 2022 · 420pp · 94,064 words
by Richard H. Thaler · 10 May 2015 · 500pp · 145,005 words
by Anil Ananthaswamy · 15 Jul 2024 · 416pp · 118,522 words
by Stephen Witt · 8 Apr 2025 · 260pp · 82,629 words
by Roger Bootle · 4 Sep 2019 · 374pp · 111,284 words
by Erik J. Larson · 5 Apr 2021
by Alexandra Robbins · 31 Mar 2009 · 509pp · 147,998 words
by Jamie Susskind · 3 Sep 2018 · 533pp
by Mariya Yao, Adelyn Zhou and Marlene Jia · 1 Jun 2018 · 161pp · 39,526 words
by Kevin Dutton · 3 Feb 2011 · 338pp · 100,477 words
by Michael Bhaskar · 2 Nov 2021
by Charles Duhigg · 1 Jan 2011 · 455pp · 116,578 words
by Yuval Noah Harari · 1 Mar 2015 · 479pp · 144,453 words
by James Barrat · 30 Sep 2013 · 294pp · 81,292 words
by Jeffrey D. Sachs · 2 Jun 2020
by Richard Dawkins · 1 Jan 1982 · 506pp · 152,049 words
by Sebastien Donadio · 7 Nov 2019
by Gary Marcus and Jeremy Freeman · 1 Nov 2014 · 336pp · 93,672 words
by Mary L. Gray and Siddharth Suri · 6 May 2019 · 346pp · 97,330 words
by Raúl Garreta and Guillermo Moncecchi · 14 Sep 2013 · 122pp · 29,286 words
by Gavin Hackeling · 31 Oct 2014
by James Vlahos · 1 Mar 2019 · 392pp · 108,745 words
by Valliappa Lakshmanan, Sara Robinson and Michael Munn · 31 Oct 2020
by Anil Seth · 29 Aug 2021 · 418pp · 102,597 words
by Michael Moss · 2 Mar 2021 · 300pp · 94,628 words
by Brian Christian and Tom Griffiths · 4 Apr 2016 · 523pp · 143,139 words
by Adam Kucharski · 23 Feb 2016 · 360pp · 85,321 words
by Peter H. Diamandis and Steven Kotler · 28 Jan 2020 · 501pp · 114,888 words
by Mary Mortimer · 1 Jan 1999 · 282pp · 28,394 words
by John Myles White · 10 Dec 2012 · 94pp · 22,435 words
by Tim Berners-Lee · 8 Sep 2025 · 347pp · 100,038 words
by Rizwan Virk · 31 Mar 2019 · 315pp · 89,861 words
by Calum Chace · 28 Jul 2015 · 144pp · 43,356 words
by William Poundstone · 267pp · 71,941 words
by Alasdair Gilchrist · 27 Jun 2016
by Abowlfulloflemons.com and Toni Hammersley · 5 Jan 2016 · 278pp · 42,509 words