reinforcement learning

The Infinity Machine: Demis Hassabis, DeepMind, and the Quest for Superintelligence

by Sebastian Mallaby;  · 30 Mar 2026  · 607pp  · 161,998 words

and use that feedback to fine-tune their behavior. Sutton’s textbook laid out ideas for implementing this approach, which was known as reinforcement learning. The basic idea of reinforcement learning, often known as RL, was already familiar to the gaming fraternity. When he had worked for Molyneux, Hassabis had proposed that a simple

…

was therefore just one piece of the puzzle. The larger challenge was to combine deep learning, which would solve challenges such as computer vision, with reinforcement learning, which would deliver other facets of intelligence, including the ability to hatch plans and think strategically. To deliver on this premise, the business plan

…

has to develop algorithms that narrow the search for the best action. Relative to deep learning, with its mind-boggling nonlinearities and impressive practical results, reinforcement learning seemed theoretical and primitive. But to Mnih and other believers in RL, the promise of agents that could learn from experience remained thrilling. Whereas

…

spirits at DeepMind were more rewarding than the ones spent at his university laboratory.[10] Silver had by now established himself as an authority on reinforcement learning, but the other newcomers at Bernard Street demonstrated DeepMind’s commitment to intellectual diversity.[11] Two recruits worked on statistical methods for quantifying uncertainty

…

but handcrafted algorithms for dealing with specific tasks could not generalize to multiple Atari environments. DeepMind’s probabilistic duo took a different tack, creating a reinforcement-learning agent that began with a model of how the games worked, then increased its confidence if trial-and-error play confirmed its hypothesis. This second

…

unrandomized and unrepresentative experiences, it would never master Atari. To get around this obstacle, David Silver proposed a new riff on an old idea in reinforcement learning. Back in the 1990s, RL scientists had experimented with a technique called memory replay: To extract maximum learning from limited data, experiences were stored

…

succeeded previously. Hence its monomaniacal zeal for firing off torpedoes.[27] Pondering this glitch, Mnih realized that it reflected another clash between supervised learning and reinforcement learning. The goal of a computer-vision system was to predict how photos were labeled. Every time it answered right, it would get a simple,

…

the team had assembled, Hassabis had shown exquisite judgment. In choosing the Atari challenge, he had understood that the moment to fuse deep learning and reinforcement learning had arrived. The result was another ImageNet moment—not just for vision, but for agents. Chapter 7 Thiel Trouble On October 8, 2012, while

…

laugh. “One day!”[10] In spring 2014, Silver suggested to Hassabis that Go’s day was approaching. The success of the Atari project, combining reinforcement learning with deep learning, provided just the sort of springboard that Silver had been waiting for.[11] It was this suggestion, the culmination of conversations begun

…

networks, mimicking the fast-thinking parts of human intelligence. But what was really powerful was how the intuitive deep-learning models worked with the introspective reinforcement learning, the tree search—the deeper, slower, System Two side of intelligence. Thanks to Maddison’s policy net, the search algorithm no longer began with

…

with random moves and discovering which ones generated a reward signal. The new Agent Zero would stand as a triumph for Silver’s scientific specialty, reinforcement learning; it would also mark a leap toward machine autonomy. Yesterday’s deep-learning systems had ingested data that represented human knowledge, curated by human

…

programmers. Tomorrow’s reinforcement-learning agents would rely on data that they generated themselves, by acting in the world and gathering experiences.[2] A year or so passed before Silver

…

AlphaZero meant, not just for humans and their cognitive limits, but rather for the road to artificial general intelligence. For Silver, this breakthrough for reinforcement learning marked a revolution. AlphaZero had mastered three different complex games from scratch, without human instruction or human data. The old obstacle to AI—the impossibility

…

mastered multiple games, demonstrating that agentic trial and error could deliver impressive versatility. At the same time, however, both systems achieved their advances in reinforcement learning thanks to progress in deep learning. Back in 2013, the Atari system had leveraged a particular kind of network, known as a convolutional neural net

…

a new DeepMind office in Edmonton, Alberta, cementing the company’s lead in designing agents that learned through experience. Hassabis shared Silver’s enthusiasm for reinforcement learning, albeit for his own reasons. Thanks to his PhD in neuroscience, he had always thought that artificial general intelligence would depend on integrating multiple

…

. For Hassabis, the mission was founding a company to go after AGI. For Silver, it was to push the frontier of reinforcement learning. For Vlad Mnih, it was fusing reinforcement learning with deep learning. Meanwhile, for Ilya Sutskever, the opportunity that sat most squarely in his wheelhouse was the appearance of a new

…

equivalent of the ImageNet breakthrough.[15] Completing his doctoral work in 2013, Sutskever was for recurrent networks and sequential data what David Silver was for reinforcement learning and Go: a leading authority who had not completely cracked his chosen problem. After his PhD, Sutskever continued to wrestle with the challenge of

…

of those experiments that advances science by failing. Far from teaching DeepMind’s agents to master concepts, Purves’s naturalistic simulations demonstrated the limits to reinforcement learning. RL agents could more or less manage a simplified natural world, but too many irregularities and unclassifiable shapes flummoxed them. When confronted with an

…

environment would not necessarily lead to improved performance. It followed that the path ahead for scaling language models was excitingly open. The path ahead for reinforcement learning looked murky and uncertain.[7] “A scientist doesn’t think, ‘Oh, some experiments failed, some succeeded,’ ” Hassabis reflected. “In science, it’s just projects.

…

direction.” * * * • • • The next direction on which DeepMind focused was David Silver’s post-AlphaZero effort. Starting in early 2018, Silver set about building a reinforcement-learning agent to crack StarCraft II, a battle simulation game whose top players attract millions of devoted followers. StarCraft’s complexity makes even Go appear simple

…

create intelligent systems that will accelerate scientific discovery,” Hassabis announced proudly.[14] The question was what this triumph signified for DeepMind’s larger bet on reinforcement learning. Like most RL breakthroughs, this one was enabled by an advance in deep learning—thanks to Vinyals, AlphaStar was built on the transformer architecture.

…

in South Korea, Hassabis had also realized something more. The gamification of protein folding appeared to change a general computational challenge into the kind of reinforcement-learning problem at which DeepMind excelled: There was a clear objective, feedback in the form of a precise score, and a virtual environment that allowed

…

would have to map amino acid chains onto their folded structures. No less a figure than David Silver backed this deep-learning approach, acknowledging that reinforcement learning was not suited to the protein problem.[11] * * * • • • The question was how to overcome the hurdles that Jumper had confronted during his PhD: a

…

a lab that prioritized language models. At DeepMind, Hassabis had recently begun speaking of three coequal “paradigms” within the research team: The first regarded reinforcement learning as the path to AGI; the second aimed to implement ideas from neuroscience; the third built neural networks that learned from data, with language models

…

human responses. The evaluator model judged Sparrow’s answers, reinforced good ones with rewards, and nudged Sparrow to tweak its parameters accordingly. When this reinforcement learning was complete, Sparrow proved both delightful and responsible. It could still make mistakes or exhibit biases arising from its training data. But it strove to

…

the Hinton–Bengio warning—the one represented by Geoffrey Irving and Jan Leike. To prevent models from deceiving humans, you needed technical solutions such as reinforcement learning from human feedback, bolstered by pre-release red-team tests to discover residual misbehaviors. Already, thanks to this formula, chatbots had become less biased,

…

chinks of light. Inscrutable, unpredictable, and therefore inherently dangerous systems became at least partially understandable. Irving’s second promising project addressed the central weakness in reinforcement learning from human feedback. As AI generated ever more sophisticated outputs, it would outstrip humans’ ability to provide feedback on them. It was one thing

…

Superhuman Agent.” Meanwhile at OpenAI, Ilya Sutskever had launched a project called GPT-Zero in 2021: The name was a tribute to Silver’s greatest reinforcement-learning triumph, the AlphaZero model.[21] During the incubation of AlphaGo, and again when the transformer architecture appeared, Sutskever and Silver had embodied the two

…

ground. During the evolution of standard language models, post-training had begun with clever prompting, then moved on to fine-tuning, and then incorporated reinforcement learning from human feedback. Likewise, now that thinking models had moved through chain-of-thought prompting and fine-tuning, the logical next step was to fortify

…

thinking through their implications? Presumably, the more thinking an AI system did, the less data it would need; and this insight led researchers back to reinforcement learning. Rumination was precisely what AlphaGo and AlphaZero had done: They planned out move sequences, evaluated them, backed up and searched out more, engaging in what

…

a process. It was a gamble. To prepare for their big rally, Shazeer and Rae had invited experimentalists across Google DeepMind to share ideas on reinforcement learning for language. The response had been stunning: More than 250 scientists had showed up at the brainstorming session with a one-slide presentation. Clearly,

…

Rather than kick-starting its thinking abilities by learning from human judgment, the Zero system emulated its namesake, DeepMind’s AlphaZero, and skipped straight to reinforcement learning. Presented with a series of math and logic questions, it experimented randomly with various chains of thought, refining its methods as it discovered which chains

…

then it could be seriously existential for humanity.”[29] * * * • • • In mid-2025, I spoke again with David Silver. As the long-term champion of reinforcement learning, he embodied both its promise and its danger. Given the comeback of RL, he also represented the beyond—beyond chatbots, beyond coding assistants, beyond imagining

…

, based purely on deduction, had been limited, for sure. But the AI revolution had equipped machines to think inductively. Thanks to deep learning and reinforcement learning, classical computers could confront uncertainty and intuit what to do next. Penrose’s quantum speculations about human and machine intelligence had been rendered irrelevant. The

…

was worth £100 million. Geoffrey Hinton, author interview, September 6, 2023. BACK TO NOTE REFERENCE 21 The potential upside from fusing deep learning and reinforcement learning influenced Google’s attitude to the pricing. Harrison recalled that DeepMind’s formula “was fairly remarkable and an evolution on what Geoff [Hinton] had described

…

take self-play to surpass human experts. Sutskever, author interview. BACK TO NOTE REFERENCE 27 The design of rewards is a key challenge in building reinforcement-learning systems. Guez’s value net allowed the Go system to recognize a rewarding position. BACK TO NOTE REFERENCE 28 Huang, author interview. BACK TO

…

–59, doi.org/10.1038/nature24270. BACK TO NOTE REFERENCE 4 David Silver et al., “Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm,” arXiv, December 5, 2017, doi.org/10.48550/arXiv.1712.01815. BACK TO NOTE REFERENCE 5 Silver, author interview. BACK TO NOTE REFERENCE

…

researchers, Jonathan Lai and James An, circulated an influential internal paper laying out the way to supersede OpenAI’s 2023 technique. They called their method Reinforcement Learning with Verified Rewards. BACK TO NOTE REFERENCE 27 Rae, author interview. BACK TO NOTE REFERENCE 28 Oriol Vinyals, author interview, February 6, 2025. BACK

…

into’ National Security Implications of DeepSeek’s AI.” BACK TO NOTE REFERENCE 17 Daya Guo et al., “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning,” arXiv, January 22, 2025, arxiv.org/abs/2501.12948. BACK TO NOTE REFERENCE 18 Other reasoning models also prompted suggestions that AI was approaching

…

Action (MIT), February 18, 2025. BACK TO NOTE REFERENCE 29 David Silver, lecture at the RLC 2024 conference, “David Silver—Towards Superhuman Intelligence—RLC 2024,” Reinforcement Learning Conference (RLC), October 1, 2024, YouTube, 1 hr., 3 min., 13 sec., youtube.com/watch?v=pkpJMNjvgXw. BACK TO NOTE REFERENCE 30 David Silver

…

Ranzato, Marc’Aurelio, 218 Rayburn, Sam, 419n5 reasoning, RL and, 358, 362–66 recurrent neural network, 200, 208 Redmond, Michael, 159 Rees, Geraint, 421n9 reinforcement learning (RL) AGI and, 97 AlphaGo trained with, 152 AlphaProof and, 378 AlphaStar and, 226–27 AlphaZero and, 195–200 in Atari challenge, 101 concept of

…

2, 106 Q* project and, 356, 359 from raw experience, 412n24 reasoning and, 358, 362–66 safety risks of, 373–75 with verified rewards, 438n27 reinforcement learning from human feedback (RLHF), 298–99, 301, 335 Renaissance Technologies, 232 Reos Partners, 67 Republic game, 36–41 residual neural network (ResNet), 197–98

We Are as Gods: A Survival Guide for the Age of Abundance

by Peter H. Diamandis and Steven Kotler  · 13 Apr 2026  · 225pp  · 76,418 words

a stranger convergence: the intersection of AI, psychedelics, and neurotech. Companies like Compass Pathways, MindMed, and Cybin started testing AI for psychedelic drug discovery, leveraging reinforcement learning and generative adversarial networks to engineer molecules for specific neural targets. They found ways to shorten the length of the psychedelic trip and fine-tune

Prophecy: Prediction, Power, and the Fight for the Future, from Ancient Oracles to AI

by Carissa Véliz  · 21 Apr 2026  · 503pp  · 129,255 words

enormous amounts of text, which allows them to mimic the patterns that they have picked up on. Second, they are refined through a process of reinforcement learning from human feedback. People teach the AI what kinds of responses they prefer. Through numerous iterations, the system learns how to satisfy human beings’ tastes

Cognitive Gadgets: The Cultural Evolution of Thinking

by Cecilia Heyes  · 15 Apr 2018

inherited comes from a study showing that, in human newborns, actions that are followed by response-contingent stimulation increase in frequency (they are subject to reinforcement learning) not only when the stimulation is biologically relevant—for example, the delivery of milk—but also when it consists of short, emotionally neutral speech sounds

…

, the transition from primarily promiscuous to primarily monogamous mating systems involved the insertion of an oxytocin (or vasopressin) receptor gate in a basic mechanism of reinforcement learning (Insel and Young, 2001). Therefore, any changes that occurred in the hominin line are likely to have been further tweaks or small adjustments to exactly

…

of blobs (Figure 3.1) becomes a highly robust and selective preference for fellow humans “looking at me” (Heyes, 2016b). Associative learning and, more specifically, reinforcement learning also explain how infants, building on an inborn face preference, develop gaze-cuing: a tendency to direct attention to the object, or area of space

…

checking) are more likely to yield an encounter with an interesting object, and as long as pointing tends to make adults do what infants want, reinforcement learning can build joint attention on the foundation of a simple, genetically inherited face preference (Moore and Corkum, 1994; Paulus, Hunnius, Vissers, and Bekkering, 2011; Triesch

…

four month old infants to increase the pitch of their vocalizations. When mothers were consistently rewarded for higher pitch by happier behavior from the infant, reinforcement learning increased the average pitch of the mothers’ vocalizations. Babies from a few days to several months of age look longer at faces producing infant-directed

…

in any way by another agent, let alone via a process specialized for cultural inheritance. Trial and error learning (also known as “instrumental learning” and “reinforcement learning”) occurs in a wide range of animals, when they confront problems posed by social and inanimate features of the world. My framework, outlined in Figure

…

Custance, 1996). These findings make it plausible that children overimitate because, ever since they were born, they have been rewarded for imitation. Via plain old reinforcement learning, they have discovered that imitation tends to be followed by goodies, and it is the conscious or unconscious expectation of those goodies that makes them

…

. Apperly, I. (2010). Mindreaders: The Cognitive Basis of “Theory of Mind.”. Hove, UK: Psychology Press. Apps, M. A., Lesage, E., and Ramnani, N. (2015). Vicarious reinforcement learning signals when instructing others. Journal of Neuroscience, 35(7), 2904–2913. Aquinas, T. (1272). Summa Theologica (new ed., 2015). Roccasecca, Italy: Xist Publishing. Atran, S

…

, social learning, 33 Input biases: human faces, attention, 60–63, 61f, 66; human voices, attention, 63–66 Instinct theory, 25 Instrumental learning. See Operant conditioning; Reinforcement learning Intelligence testing, 73 Intelligent design, 206–207, 222 Intentionality, imitation, 136–137 Intervention vs. development, 6, 25, 133–135 J Joint attention, 63. See also

…

–85, 85–89, 86f. See also Asocial learning; Associative learning; Cultural learning; Domain-general processes; Learning biases; Motor and motor sequence learning; Perceptual sequence learning; Reinforcement learning; Selective social learning; Sequence learning processes; Social learning; Trial-and-error learning Learning biases: social learning, 31, 89, 91–92, 98, 99, 102–103, 110

…

(language), 152–153 Numeracy, 168, 205, 206 O Oblique inheritance (cultural), 39, 40, 42 Occipital cortex (brain), 20, 149 Operant conditioning, 67–68. See also Reinforcement learning Optical mirrors, 123f, 128, 132, 143 Organ relations, 120 Overimitation, 138–140 Oxytocin, 57, 59–60 P Parent-child relations: facial imitation, 124–125; language

…

; disorders, 150; dual-route cascaded model, 20, 21f; mindreading similarities, 147, 148–151, 154; neurocognitive system, 20, 22. See also Literacy Reasoning (executive function), 72 Reinforcement learning: face attention preferences, 62; social motivation, 58, 59, 139; trial and error, 81, 87. See also Operant conditioning Religious beliefs, inheritance, 40 Representational re-description

…

–130, 142 Trainor, L.J., 65, 66 Transcranial magnetic stimulation, 121 “Transformations” rules, language, 172 Trial-and-error learning, 81, 87. See also Operant conditioning; Reinforcement learning Twin studies: cognitive ability, 47–48, 207–208; mindreading, 151–152 U United States: cultural beliefs, 150; mindreading research, 154–155 Unitization questions, evolution, 37

Artificial Intelligence: A Modern Approach

by Stuart Russell and Peter Norvig  · 14 Jul 2019  · 2,466pp  · 668,761 words

and Transfer Learning 22.8Applications Summary Bibliographical and Historical Notes 23Reinforcement Learning 23.1Learning from Rewards 23.2Passive Reinforcement Learning 23.3Active Reinforcement Learning 23.4Generalization in Reinforcement Learning 23.5Policy Search 23.6Apprenticeship and Inverse Reinforcement Learning 23.7Applications of Reinforcement Learning Summary Bibliographical and Historical Notes VICommunicating, perceiving, and acting 24Natural Language Processing 24.1Language Models 24.

…

processes (MDPs) developed in the field of operations research. A flood of work followed connecting AI planning research to MDPs, and the field of reinforcement learning found applications in robotics and process control as well as acquiring deep theoretical foundations. One consequence of AI’s newfound appreciation for data, statistical modeling

…

, for example by detecting unusual patterns of behavior, but they will also contribute to the potency, survivability, and proliferation capability of malware. For example, reinforcement learning methods have been used to create highly effective tools for automated, personalized blackmail and phishing attacks. We will revisit these topics in more depth in

…

); economics (market-based algorithms (Dias et al., 2006)); physics (particle swarms (Li and Yao, 2012) and spin glasses (Mézard et al., 1987)); animal behavior (reinforcement learning, grey wolf optimizers (Mirjalili and Lewis, 2014)); ornithology (Cuckoo search (Yang and Deb, 2014)); entomology (ant colony (Dorigo et al., 2008), bee colony (Karaboga

…

of actions have probabilities associated with them): Markov decision processes, partially observable Markov decision processes, and game theory. In Chapter 23 we show that reinforcement learning allows an agent to learn how to behave from past successes and failures. Bibliographical and Historical Notes AI planning arose from investigations into state-space

…

models, to update its belief state, and to project forward possible action sequences. We shall return MDPs and POMDPs in Chapter 23, which covers reinforcement learning methods that allow an agent to improve its behavior from experience. Bibliographical and Historical Notes Richard Bellman developed the ideas underlying the modern approach to

…

). The texts by Bertsekas (1987) and Puterman (1994) provide rigorous introductions to sequential decision problems and dynamic programming. Bertsekas and Tsitsiklis (1996) include coverage of reinforcement learning. Sutton and Barto (2018) cover similar ground but in a more accessible style. Sigaud and Buffet (2010), Mausam and Kolobov (2012) and Kochenderfer (2015)

…

replacing the simple components with more sophisticated machine learning models. Part of problem formulation is deciding whether you are dealing with supervised, unsupervised, or reinforcement learning. The distinctions are not always so crisp. In semisupervised learning we are given a few labeled examples and use them to mine more information from

…

gradient descent in parameter space to minimize the loss function. •Deep learning works well for visual object recognition, speech recognition, natural language processing, and reinforcement learning in complex environments. •Convolutional networks are particularly well suited for image processing and other tasks where the data have a grid topology. •Recurrent networks are

…

have already seen the concept of rewards in Chapter 16 for Markov decision processes (MDPs). Indeed, the goal is the same in reinforcement learning: maximize the expected sum of rewards. Reinforcement learning differs from “just solving an MDP” because the agent is not given the MDP as a problem to solve; the agent

…

.6, we explore apprenticeship learning: training a learning agent using demonstrations rather than reward signals. Finally, Section 23.7 reports on applications of reinforcement learning. 23.2Passive Reinforcement Learning We start with the simple case of a fully observable environment with a small number of actions and states, in which an agent already

…

transition model to perform its updates. The environment itself supplies the connection between neighboring states in the form of observed transitions. Figure 23.4A passive reinforcement learning agent that learns utility estimates using temporal differences. The step-size function ∝(n) is chosen to ensure convergence. Figure 23.5The TD learning curves

…

update rules, even though there are good solutions in the hypothesis space. There are more sophisticated algorithms that can avoid these problems, but at present reinforcement learning with general function approximators remains a delicate art. In addition to parameters diverging to infinity, there is a more surprising problem called catastrophic forgetting.

…

to understand this kind of situation as a two-person assistance game, as described in Section 17.2.5. 23.7Applications of Reinforcement Learning We now turn to applications of reinforcement learning. These include game playing, where the transition model is known and the goal is to learn the utility function, and robotics,

…

specify. Imitation learning formulates the problem as supervised learning of a policy from the expert’s state–action pairs. Inverse reinforcement learning infers reward information from the expert’s behavior. Reinforcement learning continues to be one of the most active areas of machine learning research. It frees us from manual construction of behaviors

…

environment. Weighted linear combinations of features and neural networks are factored representations for function approximation. It is also possible to apply reinforcement learning to structured representations; this is called relational reinforcement learning (Tadepalli et al.,2004). The use of relational descriptions allows for generalization across complex behaviors involving different objects. Analysis of the

…

explore unknown environments and are guaranteed to converge on near-optimal policies with a sample complexity that is polynomial in the number of states. Bayesian reinforcement learning (Dearden et al., 1998, 1999) provides another angle on both model uncertainty and exploration. The basic idea underlying imitation learning is to apply

…

of temporal-difference learning; related research describes other neuroscientific and behavioral experiments (Dayan and Niv, 2008; Niv, 2009; Lee et al., 2012). Work in reinforcement learning has been accelerated by the availability of open-source simulation environments for developing and testing learning agents. The University of Alberta’s Arcade Learning Environment

…

simulation (Savva et al., 2019) provides a photo-realistic virtual environment for indoor robotic tasks, and their HORIZON platform (Gauci et al., 2018) enables reinforcement learning in large-scale production systems. The SYNTHIA system (Ros et al., 2016) is a simulation environment designed for improving the computer vision capabilities of self

…

and perform safely. Robotics brings together many of the concepts we have seen in this book, including probabilistic state estimation, perception, planning, unsupervised learning, reinforcement learning, and game theory. For some of these concepts robotics serves as a challenging example application. For other concepts this chapter breaks new ground, for instance

…

cost (ILQR). •Planning under uncertainty unites perception and action by online replanning (such as model predictive control) and information gathering actions that aid perception. •Reinforcement learning is applied in robotics, with techniques striving to reduce the required number of interactions with the real world. Such techniques tend to exploit models, be

…

function they should optimize from human input, such as demonstrations, corrections, or instruction in natural language. Alternatively, robots can imitate human behavior, and use reinforcement learning to help tackle the challenge of generalization to new states. Bibliographical and Historical Notes The word robot was popularized by Czech playwright Karel Čapek in

…

rational agents. A variety of different agent designs were considered, ranging from reflex agents to knowledge-based decision-theoretic agents to deep learning agents using reinforcement learning. There is also variety in the component technologies from which these designs are assembled: logical, probabilistic, or neural reasoning; atomic, factored, or structured representations

…

, of course, but compilation methods can be applied so that the overhead is small compared to the costs of the computations being controlled. Metalevel reinforcement learning may provide another way to acquire effective policies for controlling deliberation: in essence, computations that lead to better decisions are reinforced, while those that

…

Based Scheduling. Morgan Kaufmann. Abbas, A. (2018). Foundations of Multiattribute Utility. Cambridge University Press. Abbeel, P. and Ng, A. Y. (2004). Apprenticeship learning via inverse reinforcement learning. In ICML-04. Abney, S., McAllester, D. A., and Pereira, F. (1999). Relating probabilistic grammars and automata. In ACL-99. Abramson, B. (1987). The

…

Andre, D., Friedman, N., and Parr, R. (1998). Generalized prioritized sweeping. In NeurIPS 10. Andre, D. and Russell, S. J. (2002). State abstraction for programmable reinforcement learning agents. In AAAI-02. Andreae, P. (1985). Justified Generalisation: Learning Procedures from Examples. Ph.D. thesis, MIT. Andrieu, C., Doucet, A., and Holenstein, R. (

…

Gaussian Bayesian tracking. IEEE Transactions on Signal Processing, 50, 174–188. Arulkumaran, K., Deisenroth, M. P., Brundage, M., and Bharath, A. A. (2017). Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine, 34, 26–38. Arunachalam, R. and Sadeh, N. M. (2005). The supply chain trading agent competition. Electronic Commerce

…

. In Automatic Control–World Congress, 1987: Selected Papers from the 10th Triennial World Congress of the International Federation of Automatic Control. Dietterich, T. (2000). Hierarchical reinforcement learning with the MAXQ value function decomposition. JAIR, 13, 227–303. Dijkstra, E. W. (1959). A note on two problems in connexion with graphs. Numerische

…

, 137– 144. Gao, J. (2014). Machine learning applications for data center optimization. Google Research. García, J. and Fernández, F. (2015). A comprehensive survey on safe reinforcement learning. JMLR, 16, 1437–1480. Gardner, M. (1968). Logic Machines, Diagrams and Boolean Algebra. Dover. Garey, M. R. and Johnson, D. S. (1979). Computers

…

using Markovian decision theory. Master’s report, Computer Science Division, University of California, Berkeley. Koenig, S. (2000). Exploring unknown environments with real–time search or reinforcement learning. In NeurIPS 12. Koenig, S. (2001). Agent-centered search. AIMag, 22, 109–131. Koenig, S. and Likhachev, M. (2002). D* Lite. AAAI-15, 15

…

viaGLTL. arXiv:1704.04341. Liu, B., Gemp, I., Ghavamzadeh, M., Liu, J., Mahadevan, S., and Petrik, M. (2018). Proximal gradient temporal difference learning: Stable reinforcement learning with polynomial sample complexity. JAIR, 63, 461–494. Liu, H., Simonyan, K., Vinyals, O., Fernando, C., and Kavukcuoglu, K. (2017). Hierarchical representations for efficient architecture

…

(Eds.). (1992). Geometric Invariance in Computer Vision. MIT Press. Munos, R., Stepleton, T., Harutyunyan, A., and Bellemare, M. G. (2017). Safe and efficient off–policy reinforcement learning. In NeurIPS 29. Murphy, K. (2002). Dynamic Bayesian Networks: Representation, Inference and Learning. Ph.D. thesis, UC Berkeley. Murphy, K. (2012). Machine Learning: A Probabilistic

…

Theory. Cambridge University Press. Nisan, N., Roughgarden, T., Tardos, E., and Vazirani, V. (Eds.). (2007). Algorithmic Game Theory. Cambridge University Press. Niv, Y. (2009). Reinforcement learning in the brain. Journal of Mathematical Psychology, 53, 139–154. Nivre, J., De Marneffe, M.-C., Ginter, F., Goldberg, Y., Hajic, J., Manning, C., McDonald

…

. and Yannakakis, M. (1991). Shortest paths without a map. Theoretical Computer Science, 84, 127–150. Papavassiliou, V. and Russell, S. J. (1999). Convergence of reinforcement learning with general function approximators. In IJCAI-99. Parisi, G. (1988). Statistical Field Theory. Addison-Wesley. Parisi, M. M. G. and Zecchina, R. (2002). Analytic and

…

of temporal differences. Machine Learning, 3, 944. Sutton, R. S., McAllester, D. A., Singh, S., and Man-sour, Y. (2000). Policy gradient methods for reinforcement learning with function approximation. In NeurIPS 12. Sutton, R. S. (1990). Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In ICML-90

…

D. (1983). An abstract Prolog instruction set. Technical note, SRI International. Wasserman, L. (2004). All of Statistics. Springer. Watkins, C. J. (1989). Models of Delayed Reinforcement Learning. Ph.D. thesis, Psychology Department, Cambridge University. Watson, J. D. and Crick, F. (1953). A structure for deoxyribose nucleic acid. Nature, 171, 737. Wattenberg, M

…

programming of intelligent embedded systems and robotic space explorers. Proc. IEEE, 91(212–237). Williams, R. J. (1992). Simple statistical gradient following algorithms for connectionist reinforcement learning. Machine Learning, 8, 229–256. Williams, R. J. and Zipser, D. (1989). A learning algorithm for continually running fully recurrent neural networks. Neural Computation,

…

systematic comparison of phrase-based, hierarchical and syntax-augmented statistical MT. In COLING-08. Zoph, B. and Le, Q. V. (2016). Neural architecture search with reinforcement learning. arXiv:1611.01578. Zuse, K. (1945). The Plankalkül. Report, Gesellschaft für Mathematik und Datenverarbeitung. Zweig, G. and Russell, S. J. (1998). Speech recognition with

…

HMM), 43, 479, 491, 491–496, 503, 515, 795, 881 hidden variable, 443, 788 HIERARCHICAL-SEARCH, 377 hierarchical decomposition, 375 hierarchical look-ahead, 383 hierarchical reinforcement learning, 858, 1065 hierarchical structure, 1065 hierarchical task network (HTN), 375, 397, 858 Hierholzer, C., 162, 1098 high-level action, 375 higher-order logic, 273 Hilbert

…

., 516, 1100 intractability, 39 intrinsic property, 340 introspection, 20, 31 invariance, temporal, 811 inverse (of a matrix), 1077 inverse dynamics, 958 inverse kinematics, 947 inverse reinforcement learning, 864, 1054, 1065 inverted pendulum, 867 Ioffe, S., 837, 1100 IPL (programming language), 36 IQ test, 38 IR (information retrieval), 901, 905 Irpan, A.,

…

1053 regularization function, 689 Reid, D. B., 667, 1110 Reid, M., 116, 125, 1102 Reif, J., 984, 986, 1089, 1110 reification, 335 REINFORCE (reinforcement learning algorithm), 863 reinforcement, 840 reinforcement learning, 28, 210, 585, 671, 840, 789–873, 986 active, 842, 848–854 Bayesian, 851 deep, 835, 857 distributed, 636 generalization in, 854–861

The Alignment Problem: Machine Learning and Human Values

by Brian Christian  · 5 Oct 2020  · 625pp  · 167,349 words

a forty-five-year collaboration that would essentially found a new field. The field, which would cross neuroscience, behaviorist psychology, engineering, and mathematics, was dubbed “reinforcement learning”; and their names, forever linked in bibliographies of AI—“Barto & Sutton,” “Sutton & Barto”—would become synonymous with the definitive textbook of the field they

…

Comparative Psychology discussed “trial and error” in the context of animal behavior. For a short history of animal learning from the perspective of reinforcement learning, see Sutton and Barto, Reinforcement Learning. 9. See Thorndike, “A Theory of the Action of the After-Effects of a Connection upon It,” and Skinner, “The Rate of

…

-Analog Reinforcement Systems and Its Application to the Brain Model Problem” for an early example, and Chapter 15 of Sutton and Barto, Reinforcement Learning for discussion. 23. Andrew G. Barto, “Reinforcement Learning: A History of Surprises and Connections” (lecture), July 19, 2018, International Joint Conference on Artificial Intelligence, Stockholm, Sweden. 24. Andrew Barto,

…

personal interview, May 9, 2018. 25. The canonical text about reinforcement learning is Sutton and Barto, Reinforcement Learning, recently updated into a second edition. For a summary of the field up to the mid-1990s, see also Kaelbling, Littman, and Moore

…

this history, see “Michael Littman: The Reward Hypothesis” (lecture), University of Alberta, October 16, 2019, available at https://www.coursera.org/lecture/fundamentals-of-reinforcement-learning/michael-littman-the-reward-hypothesis-q6x0e. Despite the recency of this particular framing, the idea of understanding behavior as motivated, whether explicitly or implicitly, by

…

2018). 52. Sutton, “A Unified Theory of Expectation in Classical and Instrumental Conditioning.” 53. Sutton, “Temporal-Difference Learning” (lecture), July 3, 2017, Deep Learning and Reinforcement Learning Summer School 2017, Université de Montréal, July 3, 2017, http://videolectures.net/deeplearning2017_sutton_td_learning/. 54. Sutton, “Temporal-Difference Learning.” 55. Sutton, “Learning to

…

Predict by the Methods of Temporal Differences.” See also Sutton’s PhD thesis: “Temporal Credit Assignment in Reinforcement Learning.” 56. See Watkins, “Learning from Delayed Rewards” and Watkins and Dayan, “Q-Learning.” 57. Tesauro, “Practical Issues in Temporal Difference Learning.” 58. Tesauro, “TD

…

.” 67. Niv. 68. For a discussion of potential limitations to the TD-error theory of dopamine, see, e.g., Dayan and Niv, “Reinforcement Learning,” and O’Doherty, “Beyond Simple Reinforcement Learning.” 69. Niv, “Reinforcement Learning in the Brain.” 70. Yael Niv, personal interview, February 21, 2018. 71. Lenson, On Drugs. 72. See, e.g., Berridge, “

…

Formula That Predicts Happiness,” https://www.theglobeandmail.com/life/health-and-fitness/health/researchers-create-formula-that-predicts-happiness/article19919756/. 80. See Tomasik, “Do Artificial Reinforcement-Learning Agents Matter Morally?” For more on this topic, see also Schwitzgebel and Garza, “A Defense of the Rights of Artificial Intelligences.” 81. Brian Tomasik,

…

“Ethical Issues in Artificial Reinforcement Learning,” https://reducing-suffering.org/ethical-issues-artificial-reinforcement-learning/. 82. Daswani and Leike, “A Definition of Happiness for Reinforcement Learning Agents.” See also People for the Ethical Treatment of Reinforcement Learners: http://petrl.org. 83. Andrew Barto

…

CURIOSITY 1. Turing, “Intelligent Machinery.” 2. There were efforts starting in 2004 to develop standardized RL benchmarks and competitions; see Whiteson, Tanner, and White, “The Reinforcement Learning Competitions.” 3. Marc Bellemare, personal interview, February 28, 2019. 4. Bellemare et al., “The Arcade Learning Environment,” stemming originally from Naddaf, “Game-Independent AI Agents

…

was introduced into machine learning with Barto, Singh, and Chentanez, “Intrinsically Motivated Learning of Hierarchical Collections of Skills,” and Singh, Chentanez, and Barto, “Intrinsically Motivated Reinforcement Learning.” For a more recent overview of this literature, see Baldassarre and Mirolli, Intrinsically Motivated Learning in Natural and Artificial Systems. 13. Hobbes, Leviathan. 14. Simon

…

, “Learning and Satiation of Response in Intrinsically Motivated Complex Puzzle Performance by Monkeys.” 19. Scenarios of this type are described in Barto, “Intrinsic Motivation and Reinforcement Learning,” and Deci and Ryan, Intrinsic Motivation and Self-Determination in Human Behavior. 20. Berlyne, Conflict, Arousal, and Curiosity. 21. And see, for instance, Berlyne

…

intellectual motivation.” See Minsky, “Steps Toward Artificial Intelligence.” 30. See Sutton, “Integrated Architectures for Learning, Planning, and Reacting Based on Approximating Dynamic Programming” and “Reinforcement Learning Architectures for Animats.” MIT’s Leslie Pack Kaelbling devised a similar method, based on the idea of measuring an agent’s “confidence intervals” around the

…

approaches, which incentivize exploration by rewarding “information gain,” see, e.g., Schmidhuber, “Curious Model-Building Control Systems”; Stadie, Levine, and Abbeel, “Incentivizing Exploration in Reinforcement Learning with Deep Predictive Models”; and Houthooft et al., “VIME.” 51. Burda et al., “Large-Scale Study of Curiosity-Driven Learning.” 52. See Burda et al

…

30. This is a very active area of research. See, e.g., Subramanian, Isbell, and Thomaz, “Exploration from Demonstration for Interactive Reinforcement Learning”; Večerík et al., “Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards”; and Hester et al., “Deep Q-Learning from Demonstrations.” 31. In fact, many agents trained in

…

//thesis/PieterAbbeel_Defense_19May2008_320x180.mp4. 21. Abbeel, Coates, and Ng, “Autonomous Helicopter Aerobatics Through Apprenticeship Learning.” 22. Abbeel et al., “An Application of Reinforcement Learning to Aerobatic Helicopter Flight.” They also successfully performed a nose-in funnel and a tail-in funnel. 23. “As repeated sub-optimal demonstrations tend to

…

differ in their suboptimalities, together they often encode the intended trajectory.” See Abbeel, “Apprenticeship Learning and Reinforcement Learning with Application to Robotic Control,” which refers to the work in Coates, Abbeel, and Ng, “Learning for Control from Multiple Demonstrations.” 24. Abbeel, Coates,

…

Stanford helicopter performing the chaos, see “Stanford University Autonomous Helicopter: Chaos,” https://www.youtube.com/watch?v=kN6ifrqwIMY. 28. Ziebart et al., “Maximum Entropy Inverse Reinforcement Learning,” which leverages the principle of maximum entropy derived from Jaynes, “Information Theory and Statistical Mechanics.” See also Ziebart, Bagnell, and Dey, “Modeling Interaction via the

…

Problems in AI Safety,” which, in turn, references Salge, Glackin, and Polani, “Empowerment: An Introduction,” and Mohamed and Rezende, “Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning.” 50. Alexander Turner, personal interview, July 11, 2019. 51. Wiener, “Some Moral and Technical Consequences of Automation.” 52. According to Paul Christiano, “corrigibility” as

…

Art. 42. Turing et al., “Can Automatic Calculating Machines Be Said to Think?” ACKNOWLEDGMENTS 1. McCulloch, Finality and Form. BIBLIOGRAPHY Abbeel, Pieter. “Apprenticeship Learning and Reinforcement Learning with Application to Robotic Control.” PhD thesis, Stanford University, 2008. Abbeel, Pieter, Adam Coates, and Andrew Y. Ng. “Autonomous Helicopter Aerobatics Through Apprenticeship Learning.” International

…

Rebellion: Changing the Narrative.” In Thirty-First AAAI Conference on Artificial Intelligence, 2017. Akrour, Riad, Marc Schoenauer, and Michèle Sebag. “APRIL: Active Preference-Learning Based Reinforcement Learning.” In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 116–31. Springer, 2012. Akrour, Riad, Marc Schoenauer, Michèle Sebag, and Jean-Christophe

…

of the Cognitive Science Society, 2017. Choi, Jongwook, Yijie Guo, Marcin Moczulski, Junhyuk Oh, Neal Wu, Mohammad Norouzi, and Honglak Lee. “Contingency-Aware Exploration in Reinforcement Learning.” In International Conference on Learning Representations, 2019. Chouldechova, Alexandra. “Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments.” Big Data 5

…

Mental Health Crisis in Graduate Education.” Nature Biotechnology 36, no. 3 (2018): 282. Everitt, Tom, Victoria Krakovna, Laurent Orseau, Marcus Hutter, and Shane Legg. “Reinforcement Learning with a Corrupted Reward Channel.” In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI-17), 4705–13, 2017. Eysenbach, Benjamin, Shixiang

…

Gu, Julian Ibarz, and Sergey Levine. “Leave No Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning.” In International Conference on Learning Representations, 2018. Fantz, Robert L. “Visual Experience in Infants: Decreased Attention to Familiar Patterns Relative to Novel Ones.” Science 146

…

and Don Mosenfelder. Bobby Fischer Teaches Chess. Basic Systems, 1966. Florensa, Carlos, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. “Reverse Curriculum Generation for Reinforcement Learning.” In Proceedings of the 1st Annual Conference on Robot Learning, edited by Sergey Levine, Vincent Vanhoucke, and Ken Goldberg, 482–95. PMLR, 2017. Flores, Anthony

…

1976): 3–46. Malik, Dhruv, Malayandi Palaniappan, Jaime Fisac, Dylan Hadfield-Menell, Stuart Russell, and Anca Drăgan. “An Efficient, Generalized Bellman Update for Cooperative Inverse Reinforcement Learning.” In Proceedings of the 35th International Conference on Machine Learning, edited by Jennifer Dy and Andreas Krause, 3394–3402. PMLR, 2018. Malone, Thomas W. “Toward

…

2600 Console Games.” Master’s thesis, University of Alberta, 2010. Nair, Ashvin, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. “Overcoming Exploration in Reinforcement Learning with Demonstrations.” In 2018 IEEE International Conference on Robotics and Automation (ICRA), 6292–99. IEEE, 2018. Nalisnick, Eric, Bhaskar Mitra, Nick Craswell, and Rich Caruana

…

.” Journal of Experimental Criminology 12, no. 3 (2016): 347–71. Saunders, William, Girish Sastry, Andreas Stuhlmueller, and Owain Evans. “Trial Without Error: Towards Safe Reinforcement Learning via Human Intervention.” In Proceedings of the 17th International Conference on Autonomous Agents and Multiagent Systems, 2067–99. International Foundation for Autonomous Agents and Multiagent

…

Hospitalized Patients.” JAMA Neurology 74, no. 12 (2017): 1419–24. Subramanian, Kaushik, Charles L. Isbell Jr., and Andrea L. Thomaz. “Exploration from Demonstration for Interactive Reinforcement Learning.” In Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems, 447–56. International Foundation for Autonomous Agents and Multiagent Systems, 2016. Sundararajan, Mukund

…

for Animats.” In From Animals to Animats: Proceedings of the First International Conference on Simulation of Adaptive Behavior, 288–96. 1991. ———. “Temporal Credit Assignment in Reinforcement Learning.” PhD thesis, University of Massachusetts, Amherst, 1984. ———. “A Unified Theory of Expectation in Classical and Instrumental Conditioning.” Bachelor’s thesis, Stanford University, 1978. Sutton,

…

Richard S., and Andrew G. Barto. Reinforcement Learning: An Introduction. 2nd ed. MIT Press, 2018. Sweeney, Latanya. “Discrimination in Online Ad Delivery.” Communications of the ACM 56, no. 5 (2013): 44–54. ———.

…

Večerík, Matej, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. “Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards.” arXiv Preprint arXiv:1707.08817, 2017. Vincent, James. “Google ‘Fixed’ Its Racist Algorithm by Removing Gorillas from Its Image

…

9, 12, 337n6, 346n13 alignment problem amplification/distillation and, 249 analogies and, 317 corrigibility and, 295 defined, 13 as hopeful, 327–28 inverse reinforcement learning and, 255 parenting and, 166 reinforcement learning and, 151 technical limitations and, 313, 395–96n4 thermostats and, 311–12, 313 See also value alignment Allen, Woody, 170 AlphaGo, 162

…

ensemble methods, 284–85, 305 equiprobabiliorism, 303 equivant (company), 337n5 ergodicity assumption, 320 Ermon, Stefano, 324 ethics actualism vs. possibilism and, 239, 379n71 in reinforcement learning, 149 See also AI safety; fairness; moral uncertainty evaluation function. See value function Evans, Owain, 386–87n55 evolution, 170, 171–74, 368n56 expectations, 138–39

…

, 316, 342n61 See also Google research Google Brain, 113, 167, 373n53 Google research differential privacy, 347n33 fairness, 73 feature visualization, 110 multitask learning models, 107 reinforcement learning, 167 selective classification, 390n29 value alignment, 247 word embedding, 44 Gopnik, Alison, 194, 215 gorilla tag incident, 25–26, 316, 339n24 GPT-2, 344

…

238–39, 379n69 OpenAI actualism vs. possibilism, 239 amplification, 248–50 corrigibility, 296 feature visualization, 112, 357n69 intrinsic motivation, 199–200, 201 inverse reinforcement learning, 263–66, 384–85n37 reinforcement learning, 365n27 word embedding, 344–45n94 open category problem, 279–81, 315, 396nn10–11 ophthalmology, 287, 389n23 optimal regressions, 95–96 optimal reward problem

…

Herbert, 327 Reality Is Broken (McGonigal), 175 recidivism. See risk-assessment models rectified linear output functions, 24 Reddy, Raj, 223 redundant encodings, 40, 64, 343n74 reinforcement learning (RL) actor-critic architecture, 138, 362n51 actualism vs. possibilism and, 238–40, 379n69 addiction and, 205–08, 374n65 alignment problem and, 151 Arcade Learning Environment

…

143, 145 Research Institute for Advanced Studies, 383n15 Revels, Hiram, 27 reversibility, 391n39 reward hypothesis, 130–31, 133–34, 360nn26, 28 See also reinforcement learning rewards. See incentives; reinforcement learning; reward hypothesis; shaping right to explanation. See transparency rigorism, 303, 304 risk-assessment models COMPAS development, 56–57, 346nn13–14 defenses of, 68, 72

Hands-On Machine Learning With Scikit-Learn and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems

by Aurélien Géron  · 13 Mar 2017  · 1,331pp  · 163,200 words

nets, convolutional nets, recurrent nets, long short-term memory (LSTM) nets, and autoencoders. Techniques for training deep neural nets. Scaling neural networks for huge datasets. Reinforcement learning. The first part is based mostly on Scikit-Learn while the second part uses TensorFlow. Caution Don’t jump into deep waters too hastily: while

…

that it is useful to classify them in broad categories based on: Whether or not they are trained with human supervision (supervised, unsupervised, semisupervised, and Reinforcement Learning) Whether or not they can learn incrementally on the fly (online versus batch learning) Whether they work by simply comparing new data points to known

…

classified according to the amount and type of supervision they get during training. There are four major categories: supervised learning, unsupervised learning, semisupervised learning, and Reinforcement Learning. Supervised learning In supervised learning, the training data you feed to the algorithm includes the desired solutions, called labels (Figure 1-5). Figure 1-5

…

on top of one another. RBMs are trained sequentially in an unsupervised manner, and then the whole system is fine-tuned using supervised learning techniques. Reinforcement Learning Reinforcement Learning is a very different beast. The learning system, called an agent in this context, can observe the environment, select and perform actions, and get rewards

…

the agent should choose when it is in a given situation. Figure 1-12. Reinforcement Learning For example, many robots implement Reinforcement Learning algorithms to learn how to walk. DeepMind’s AlphaGo program is also a good example of Reinforcement Learning: it made the headlines in March 2016 when it beat the world champion Lee Sedol

…

, with all this information you are now ready to start designing your system. First, you need to frame the problem: is it supervised, unsupervised, or Reinforcement Learning? Is it a classification task, a regression task, or something else? Should you use batch learning or online learning techniques? Before you read on, pause

…

networks across dozens of servers and GPUs! In the following chapters we will go through a few more important neural network architectures before we tackle Reinforcement Learning. Exercises If you get a CUDA_ERROR_OUT_OF_MEMORY when starting your TensorFlow program, what is probably going on? What can you do about

…

. (2016). 13 “Semantic Hashing,” R. Salakhutdinov and G. Hinton (2008). 14 “CNN Based Hashing for Image Retrieval,” J. Gua and J. Li (2015). Chapter 16. Reinforcement Learning Reinforcement Learning (RL) is one of the most exciting fields of Machine Learning today, and also one of the oldest. It has been around since the 1950s

…

? With hindsight it seems rather simple: they applied the power of Deep Learning to the field of Reinforcement Learning, and it worked beyond their wildest dreams. In this chapter we will first explain what Reinforcement Learning is and what it is good at, and then we will present two of the most important techniques

…

in deep Reinforcement Learning: policy gradients and deep Q-networks (DQN), including a discussion of Markov decision processes (MDP). We will use these techniques to train a model to

…

games. The same techniques can be used for a wide variety of tasks, from walking robots to self-driving cars. Learning to Optimize Rewards In Reinforcement Learning, a software agent makes observations and takes actions within an environment, and in return it receives rewards. Its objective is to learn to act in

…

observe stock market prices and decide how much to buy or sell every second. Rewards are obviously the monetary gains and losses. Figure 16-1. Reinforcement Learning examples: (a) walking robot, (b) Ms. Pac-Man, (c) Go player, (d) thermostat, (e) automatic trader5 Note that there may not be any positive rewards

…

a negative reward at every time step, so it better find the exit as quickly as possible! There are many other examples of tasks where Reinforcement Learning is well suited, such as self-driving cars, placing ads on a web page, or controlling where an image classification system should focus its attention

…

example, the policy could be a neural network taking observations as inputs and outputting the action to take (see Figure 16-2). Figure 16-2. Reinforcement Learning using a neural network policy The policy can be any algorithm you can think of, and it does not even have to be deterministic. For

…

an environment for the agent to live in, so it’s time to introduce OpenAI gym. Introduction to OpenAI Gym One of the challenges of Reinforcement Learning is that in order to train an agent, you first need to have a working environment. If you want to program an agent that will

…

network as usual, by minimizing the cross entropy between the estimated probability and the target probability. It would just be regular supervised learning. However, in Reinforcement Learning the only guidance the agent gets is through rewards, and rewards are typically sparse and delayed. For example, if the agent manages to balance the

…

value the present much more than the future, then the prospect of future rewards is not worth immediate pain. Temporal Difference Learning and Q-Learning Reinforcement Learning problems with discrete actions can often be modeled as Markov decision processes, but the agent initially has no idea what the transition probabilities are (it

…

. In any case, RL still requires quite a lot of patience and tweaking, but the end result is very exciting. Exercises How would you define Reinforcement Learning? How is it different from regular supervised or unsupervised learning? Can you think of three possible applications of RL that were not mentioned in this

…

rewards? What is the discount rate? Can the optimal policy change if you modify the discount rate? How do you measure the performance of a Reinforcement Learning agent? What is the credit assignment problem? When does it occur? How can you alleviate it? What is the point of using a replay memory

…

it be? Aurélien Géron, November 26th, 2016 1 For more details, be sure to check out Richard Sutton and Andrew Barto’s book on RL, Reinforcement Learning: An Introduction (MIT Press), or David Silver’s free online RL course at University College London. 2 “Playing Atari with Deep

…

Reinforcement Learning,” V. Mnih et al. (2013). 3 “Human-level control through deep reinforcement learning,” V. Mnih et al. (2015). 4 Check out the videos of DeepMind’s system learning to play Space Invaders

…

stated goal is to promote and develop friendly AIs that will benefit humanity (rather than exterminate it). 9 “Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning,” R. Williams (1992). 10 We already did something similar in Chapter 11 when we discussed Gradient Clipping: we first computed the gradients, then we clipped

…

) for each instance. The two most common supervised tasks are regression and classification. Common unsupervised tasks include clustering, visualization, dimensionality reduction, and association rule learning. Reinforcement Learning is likely to perform best if we want a robot to learn to walk in various unknown terrains since this is typically the type of

…

problem that Reinforcement Learning tackles. It might be possible to express the problem as a supervised or semisupervised learning problem, but it would be less natural. If you don

…

autoencoder. For the solutions to exercises 8, 9, and 10, please see the Jupyter notebooks available at https://github.com/ageron/handson-ml. Chapter 16: Reinforcement Learning Reinforcement Learning is an area of Machine Learning aimed at creating agents capable of taking actions in an environment in a way that maximizes rewards over time

…

regular supervised and unsupervised learning. Here are a few: In supervised and unsupervised learning, the goal is generally to find patterns in the data. In Reinforcement Learning, the goal is to find a good policy. Unlike in supervised learning, the agent is not explicitly given the “right” answer. It must learn by

…

not tell the agent how to perform the task, but we do tell it when it is making progress or when it is failing. A Reinforcement Learning agent needs to find the right balance between exploring the environment, looking for new ways of getting rewards, and exploiting sources of rewards that it

…

feed on the training data they are given. In supervised and unsupervised learning, training instances are typically independent (in fact, they are generally shuffled). In Reinforcement Learning, consecutive observations are generally not independent. An agent may remain in the same region of the environment for a while before it moves on, so

…

. In some cases a replay memory is used to ensure that the training algorithm gets fairly independent observations. Here are a few possible applications of Reinforcement Learning, other than those mentioned in Chapter 16: Music personalization The environment is a user’s personalized web radio. The agent is the software deciding what

…

on. They would get positive rewards for each product delivered on time, and negative rewards for late deliveries. When estimating the value of an action, Reinforcement Learning algorithms typically sum all the rewards that this action led to, giving more weight to immediate rewards, and less weight to later rewards (considering that

…

’t value the future, you will just grab any immediate reward you can find, never investing in the future. To measure the performance of a Reinforcement Learning agent, you can simply sum up the rewards it gets. In a simulated environment, you can run many episodes and look at the total rewards

…

gets on average (and possibly look at the min, max, standard deviation, and so on). The credit assignment problem is the fact that when a Reinforcement Learning agent receives a reward, it has no direct way of knowing which of its previous actions contributed to this reward. It typically occurs when there

…

to Optimize Rewards AlexNet architecture, AlexNet-AlexNet algorithmspreparing data for, Prepare the Data for Machine Learning Algorithms-Select and Train a Model AlphaGo, Reinforcement Learning, Introduction to Artificial Neural Networks, Reinforcement Learning, Policy Gradients Anaconda, Create the Workspace anomaly detection, Unsupervised learning Apple’s Siri, Introduction to Artificial Neural Networks apply_gradients(), Gradient Clipping

…

Data Representations deconvolutional layer, ResNet deep autoencoders (see stacked autoencoders) deep belief networks (DBNs), Semisupervised learning, Deep Belief Nets-Deep Belief Nets Deep Learning, Reinforcement Learning(see also Reinforcement Learning; TensorFlow) about, The Machine Learning Tsunami, Roadmap libraries, Up and Running with TensorFlow-Up and Running with TensorFlow deep neural networks (DNNs), Multi-Layer

…

sequence difficulties, The Difficulty of Training over Many Time Steps truncated backpropagation through time, The Difficulty of Training over Many Time Steps DeepMind, Reinforcement Learning, Introduction to Artificial Neural Networks, Reinforcement Learning, Approximate Q-Learning degrees of freedom, Overfitting the Training Data, Learning Curves denoising autoencoders, Denoising Autoencoders-TensorFlow Implementation depth concat layer, GoogLeNet

…

also Random Forests) random patches and random subspaces, Random Patches and Random Subspaces stacking, Stacking-Stacking entropy impurity measure, Gini Impurity or Entropy? environments, in reinforcement learning, Learning to Optimize Rewards-Evaluating Actions: The Credit Assignment Problem, Exploration Policies, Learning to Play Ms. Pac-Man Using Deep Q-Learning episodes (in RL

…

Layers fully_connected(), Construction Phase, Xavier and He Initialization, Implementing Batch Normalization with TensorFlow-Implementing Batch Normalization with TensorFlow, Tying Weights G game play (see reinforcement learning) gamma value, Gaussian RBF Kernel gate controllers, LSTM Cell Gaussian distribution, Select a Performance Measure, Variational Autoencoders, Generating Digits Gaussian RBF, Adding Similarity Features Gaussian

…

Cost Function-Training and Cost Function log_device_placement, Logging placements LSTM (Long Short-Term Memory) cell, LSTM Cell-GRU Cell M machine control (see reinforcement learning) Machine Learninglarge-scale projects (see TensorFlow) notations, Select a Performance Measure-Select a Performance Measure process example, End-to-End Machine Learning Project-Exercises project

…

Online Learning-Online learning instance-based versus model-based learning, Instance-Based Versus Model-Based Learning-Model-based learning supervised/unsupervised learning, Supervised/Unsupervised Learning-Reinforcement Learning workflow example, Model-based learning-Model-based learning machine translation (see natural language processing (NLP)) make(), Introduction to OpenAI Gym Manhattan norm, Select a Performance

…

Regression, Ridge Regression-Ridge Regression shrinkage, Gradient Boosting ℓ 1 and ℓ 2 regularization, ℓ1 and ℓ2 Regularization-ℓ1 and ℓ2 Regularization REINFORCE algorithms, Policy Gradients Reinforcement Learning (RL), Reinforcement Learning-Reinforcement Learning, Reinforcement Learning-Thank You!actions, Evaluating Actions: The Credit Assignment Problem-Evaluating Actions: The Credit Assignment Problem credit assignment problem, Evaluating Actions: The Credit Assignment Problem-Evaluating

…

the Reconstructions Visualizing Features Unsupervised Pretraining Using Stacked Autoencoders Denoising Autoencoders TensorFlow Implementation Sparse Autoencoders TensorFlow Implementation Variational Autoencoders Generating Digits Other Autoencoders Exercises 16. Reinforcement Learning Learning to Optimize Rewards Policy Search Introduction to OpenAI Gym Neural Network Policies Evaluating Actions: The Credit Assignment Problem Policy Gradients Markov Decision Processes Temporal

…

Neural Nets Chapter 12: Distributing TensorFlow Across Devices and Servers Chapter 13: Convolutional Neural Networks Chapter 14: Recurrent Neural Networks Chapter 15: Autoencoders Chapter 16: Reinforcement Learning B. Machine Learning Project Checklist Frame the Problem and Look at the Big Picture Get the Data Explore the Data Prepare the Data Short-List

Architects of Intelligence

by Martin Ford  · 16 Nov 2018  · 586pp  · 186,548 words

data. This explains why companies that control huge amounts of data, like Google, Amazon, and Facebook, have such a dominant position in deep learning technology. REINFORCEMENT LEARNING essentially means learning through practice or trial and error. Rather than training an algorithm by providing the correct, labeled outcome, the learning system is set

…

for itself, and if it succeeds it is given a “reward.” Imagine training your dog to sit, and if he succeeds, giving him a treat. Reinforcement learning has been an especially powerful way to build AI systems that play games. As you will learn from the interview with Demis Hassabis in this

…

book, DeepMind is a strong proponent of reinforcement learning and relied on it to create the AlphaGo system. The problem with reinforcement learning is that it requires a huge number of practice runs before the algorithm can succeed. For this reason, it

…

is primarily used for games or for tasks that can be simulated on a computer at high speed. Reinforcement learning can be used in the development of self-driving cars—but not by having actual cars practice on real roads. Instead virtual cars are trained

…

coming from their environments. This is how human beings learn. Young children, for example, learn languages primarily by listening to their parents. Supervised learning and reinforcement learning also play a role, but the human brain has an astonishing ability to learn simply by observation and unsupervised interaction with the environment. Unsupervised learning

…

LECUN A human can learn to drive a car in 15 hours of training without crashing into anything. If you want to use the current reinforcement learning methods to train a car to drive itself, the machine will have to drive off cliffs 10,000 times before it figures out how not

…

of learning where you don’t train for a task, you just observe the world and figure out how it works, essentially. MARTIN FORD: Would reinforcement learning, or learning by practice with a reward for succeeding, be in the category of unsupervised learning? YANN LECUN: No, that’s a different category altogether

…

. There are three categories essentially; it’s more of a continuum, but there is reinforcement learning, supervised learning, and self-supervised learning. Reinforcement learning is learning by trial and error, getting rewards when you succeed and not getting rewards when you don’t succeed. That form

…

for games, where you can try things as many times as you want, but doesn’t work in many real-world scenarios. You can use reinforcement learning to train a machine to play Go or chess. That works really well, as we’ve seen with AlphaGo, for example, but it requires a

…

performance, and it works really well if you can do that, but it is often impractical in the real world. If you want to use reinforcement learning to train a robot to grab objects, it will take a ridiculous amount of time to achieve that. A human can learn to drive a

…

car in 15 hours of training without crashing into anything. If you want to use the current reinforcement learning methods to train a car to drive itself, the machine will have to drive off cliffs 10,000 times before it figures out how not

…

for the fact that the kind of learning that we can do as humans is very, very different from pure reinforcement learning. It’s more akin to what people call model-based reinforcement learning. This is where you have your internal model of the world that allows you to predict that when you turn

…

result, you can plan ahead and not take the actions that result in bad outcomes. Learning to drive in this context is called model-based reinforcement learning, and that’s one of the things we don’t really know how to do. There is a name for it, but there’s no

…

lot of fundamental research and questions on machine learning, so things that have more to do with applied mathematics and optimization. We are working on reinforcement learning, and we are also working on something called generative models, which are a form of self-supervised or predictive learning. MARTIN FORD: Is Facebook working

…

labels. As kids, we watch how other humans do things and then we do it; so, the field is now starting to get into inverse reinforcement learning algorithms, and neuro-programming algorithms. There is a lot of new exploration, and DeepMind is doing that. Google Brain is doing that; Stanford is doing

…

multidimensional—and one that has the kind of learning capability that humans do, which is not only through big data but also through unsupervised learning, reinforcement learning, virtual learning, and various kinds of learning. If we use that as a definition of AGI, then I think the path to AGI is a

…

these general algorithms that we can apply to real-world problems. MARTIN FORD: So far, your focus has primarily been on combining deep learning with reinforcement learning. That’s basically learning by practice, where the system repeatedly attempts something, and there’s a reward function that drives it toward success. I’ve

…

heard you say that you believe that reinforcement learning offers a viable path to general intelligence, that it might be sufficient to get there. Is that your primary focus going forward? DEMIS HASSABIS: Going

…

forward, yes, it is. I think that technique is extremely powerful, but you need to combine it with other things to scale it. Reinforcement learning has been around for a long time, but it was only used in very small toy problems because it was very difficult for anyone to

…

did the processing of the screen, and the model of the environment you’re in. Deep learning is amazing at scaling, so combining that with reinforcement learning allowed it to scale to these large problems that we’ve now tackled in AlphaGo and DQN—all of these things that people would have

…

proved that first part. The reason we were so confident about it and why we backed it when we did was because in my opinion reinforcement learning will become as big as deep learning in the next few years. DeepMind is one of the few companies that take that seriously because, from

…

the neuroscience perspective, we know that the brain uses a form of reinforcement learning as one of its learning mechanisms, it’s called temporal difference learning, and we know the dopamine system implements that. Your dopamine neurons track the

…

be a viable solution to the problem of general intelligence. It may not be the only one, but from a biologically inspired standpoint, it seems reinforcement learning is sufficient once you scale it up enough. Of course, there are many technical challenges with doing that, and many of them are unsolved. MARTIN

…

FORD: Still, when a child learns things like language or an understanding of the world, it doesn’t really seem like reinforcement learning for the most part. It’s unsupervised learning, as no one’s giving the child labeled data the way we would do with ImageNet. Yet

…

their peers and they do unsupervised learning when they’re just experimenting with stuff, with no goal in mind. They also do reward learning and reinforcement learning when they do something, and they get a reward for it. We work on all three of those, and they’re all going to be

…

could be intrinsic rewards that could be guiding the unsupervised learning. I find that it is useful to think about intelligence in the framework of reinforcement learning. MARTIN FORD: One thing that’s obvious from listening to you is that you combine a deep interest in both neuroscience and computer science. Is

…

, having neuroscience as a guide can allow me to make much bigger, much stronger bets on things like that. A great example of this is reinforcement learning. I know reinforcement learning has to be scalable because the brain does scale it. If you didn’t know that the brain implemented

…

reinforcement learning and it wasn’t scaling, how would you know on a practical level if you should spend another two years on this? It’s very

…

up doing my bachelor’s at Carnegie Mellon, my master’s from MIT and a PhD, with a thesis titled, Shaping and Policy Search in Reinforcement Learning, from the University of California, Berkeley. For about the next twelve years I taught at the Stanford University Department of Computer Science and the Department

…

’re only just discovering the power of techniques such as deep learning and neural networks in their many forms, as well as other techniques like reinforcement learning and transfer learning. These techniques all still have enormous headroom; we’re only just scratching the surface of where they can take us. Deep learning

…

that learning in totally new environments or on a previously unencountered problem, over there. There are definitely some exciting new techniques coming up, whether in reinforcement learning or even simulated learning—the kinds of things that AlphaZero has begun to do—where you self-learn and self-create structures, as well start

…

but I think ultimately inadequate idea that we are seeing in the field right now. What we see at the moment is people doing deep reinforcement learning over pixels of, for example, the Atari game Breakout, and while you get results that look impressive, they’re incredibly fragile. DeepMind trained an AI

…

think that some sort of built-in template or structure should be built into an AI system so it can create causal models? DeepMind uses reinforcement learning, which is based on practice or trial and error. Perhaps that would be a way of discovering causal relationships? JUDEA PEARL: It comes into it

…

, but reinforcement learning has limitations, too. You can only learn actions that have been seen before. You cannot extrapolate to actions that you haven’t seen, like raising

…

-hanging fruits. MARTIN FORD: Looking to the future, do you think that neural networks are going to be very important? JUDEA PEARL: Neural networks and reinforcement learning will all be essential components when properly utilized in causal modeling. MARTIN FORD: So, you think it might be a hybrid system that incorporates not

…

of that is also focused on that problem is DeepMind, but I’m struck by how different your approach is. DeepMind is focused on deep reinforcement learning through games and simulated environments, whereas what I hear from you is that the path to intelligence is through language. DAVID FERRUCCI: Let’s restate

…

the end of their arm, why can’t a robot? There’s something dramatic missing. MARTIN FORD: I have seen reports that deep learning and reinforcement learning is being used to have robots learn to do things by practicing or even just by watching YouTube videos. What’s your view on this

…

came from people who were trying to understand how human intelligence works. That includes the basic mathematics of what we now call deep learning and reinforcement learning, but also much further back to Boole as one of the inventors of mathematical logic, or Laplace in his work on probability theory. In more

…

at achieving more general intelligence by modeling an evolutionary approach? JOSH TENENBAUM: Well, a number of people at DeepMind and others who follow the deep reinforcement learning ethos would say they’re thinking about evolution in a more general sense, and that’s also a part of learning. They’d say their

…

, machine learning and other AI-related fields, and their papers have received awards at venues across the AI landscape, including leading conferences in computer vision, reinforcement learning and decision-making, robotics, uncertainty in AI, learning and development, cognitive modeling and neural information processing. They have introduced several widely used AI tools and

…

contribution to AI safety, at least as valuable as worrying about the alignment problem, which ultimately is just a technical problem having to do with reinforcement learning and objective functions. So, I wouldn’t say that we’re underinvesting in being prepared for AI safety, and certainly some of the work that

Artificial Intelligence: A Guide for Thinking Humans

by Melanie Mitchell  · 14 Oct 2019  · 350pp  · 98,077 words

been used for centuries on animals and humans. Operant conditioning inspired an important machine-learning approach called reinforcement learning. Reinforcement learning contrasts with the supervised-learning method I’ve described in previous chapters: in its purest form, reinforcement learning requires no labeled training examples. Instead, an agent—the learning program—performs actions in an environment (

…

such appreciation—such as positive numbers added to its memory. FIGURE 22: A Sony Aibo robotic dog, about to kick a robot soccer ball While reinforcement learning has been part of the AI toolbox for decades, it has long been overshadowed by neural networks and other supervised-learning methods. This changed in

…

2016 when reinforcement learning played a central role in a stunning and momentous achievement in AI: a program that learned to beat the best humans at the complex game

…

if a rock is blocking the ball’s movement? As always, the real world is awash with hard-to-predict edge cases. The promise of reinforcement learning is that the agent—here our robo-dog—can learn flexible strategies on its own simply by performing actions in the world and occasionally receiving

…

Or perhaps not. Unlike a real dog, Rosie has no intrinsic desire for treats, positive numbers, or anything else. As I’ll detail below, in reinforcement learning, a human-created algorithm guides Rosie’s process of learning in response to rewards; that is, the algorithm tells Rosie how to learn from her

…

experiences. Reinforcement learning occurs by having Rosie take actions over a series of learning episodes, each of which consists of some number of iterations. At each iteration, Rosie

…

to take, but remember, we’re letting Rosie figure out on her own how to perform this task. FIGURE 23: A hypothetical first episode of reinforcement learning At iteration 2 (figure 23B), Rosie determines her new state: thirteen steps from the ball. She then chooses a new action to take, again at

…

She’s back to square one, but Rosie doesn’t even know that she has been in this state before! In the purest form of reinforcement learning, the learning agent doesn’t remember its previous states. In general, remembering previous states might take a lot of memory and doesn’t turn out

…

Kick (figure 23D). Finally, she gets a reward and uses it to learn something. What does Rosie learn? Here we take the simplest approach to reinforcement learning: upon receiving a reward, Rosie learns only about the state and action that immediately preceded the reward. In particular, Rosie learns that if she is

…

action is on the right path, and if you’ve eaten chocolate before, you can predict the intensity of the upcoming reward. The goal of reinforcement learning is for the agent to learn values that are good predictions of upcoming rewards (assuming that the agent keeps doing the right thing after taking

…

particular actions in a given state typically takes many steps of trial and error. FIGURE 24: Rosie’s Q-table after her first episode of reinforcement learning Rosie keeps track of the values of actions in a big table in her computer memory. This table, illustrated in figure 24, lists all the

…

accurate predictions of upcoming rewards—as Rosie continues to learn. This table of states, actions, and values is called the Q-table. This form of reinforcement learning is sometimes called Q-learning. The letter Q is used because the letter V (for value) was used for something else in the original paper

…

Backward again at the next iteration (figure 25B). Our robo-dog’s training has a long way to go. FIGURE 25: The second episode of reinforcement learning Everything continues as before, until Rosie’s floundering random trial-and-error actions happen to land her one step away from the ball (figure 25C

…

reduced (“discounted”) as they go back in time from the actual reward; this allows the system to learn an efficient path to an actual reward. Reinforcement learning—here, the gradual updating of values in the Q-table—continues, episode to episode, until Rosie has finally learned to perform her task from any

…

learn to perform this task perfectly, no matter where she started. This “training Rosie” example captures much of the essence of reinforcement learning, but I left out many issues that reinforcement-learning researchers face for more complex tasks.5 For example, in real-world tasks, the agent’s perception of its state is often

…

might move it different distances depending on the terrain, or even result in the robot falling down or colliding with an unseen obstacle. How can reinforcement learning deal with uncertainties like these? Additionally, how should the learning agent choose an action at each time step? A naive strategy would be to always

…

World Setting these issues aside for now, let’s look at two major stumbling blocks that might arise in extrapolating our “training Rosie” example to reinforcement learning in real-world tasks. First, there’s the Q-table. In complex real-world tasks—think, for example, of a robot car learning to

…

. Learning via a Q-table like the one in the “Rosie” example is out of the question. For this reason, most modern approaches to reinforcement learning use a neural network instead of a Q-table. The neural network’s job is to learn what values should be assigned to actions in

…

damaging itself by choosing the wrong action, such as kicking a concrete wall or stepping forward over a cliff. Just as I did for Rosie, reinforcement-learning practitioners almost always deal with this problem by building simulations of robots and environments and performing all the learning episodes in the simulation rather than

…

learning systems on forty-nine of these games.3 This was the platform used by the DeepMind group in their work on reinforcement learning. Deep Q-Learning The DeepMind group combined reinforcement learning—in particular Q-learning—with deep neural networks to create a system that could learn to play Atari video games. The

…

action based on the DQN’s output values. The system doesn’t always choose the action with the highest estimated value; as I mentioned above, reinforcement learning requires a balance between exploration and exploitation.4 The system performs its chosen action (for example, moving the paddle some amount to the left) and

…

updating the weights in the DQN via back-propagation. How are the weights updated? This is the crux of the difference between supervised learning and reinforcement learning. As you’ll recall from earlier chapters, back-propagation works by changing a neural network’s weights so as to reduce the error in the

…

the output should have been 80 points higher. The network could calculate the error because it had a label provided by a human. However, in reinforcement learning we have no labels. A given frame from the game doesn’t come labeled with the action that should be taken. How then do we

…

a year later, Google announced that it was acquiring DeepMind for £440 million (about $650 million at the time), presumably because of these results. Yes, reinforcement learning occasionally leads to big rewards. With a lot of money in their pockets and the resources of Google behind them, DeepMind—now called Google DeepMind

…

while playing itself! The method for learning was somewhat complicated, and I won’t detail it here, but it had some aspects that foreshadowed modern reinforcement learning.11 In the end, Samuel’s checkers player impressively rose to the level of a “better-than-average player,” though by no means a champion

…

surmounted an even greater challenge than chess and had done so in a much more impressive fashion. Unlike Deep Blue, AlphaGo acquired its abilities by reinforcement learning via self-play. Demis Hassabis noted that “the thing that separates out top Go players [is] their intuition” and that “what we’ve done

…

search get started. With its AlphaGo project, DeepMind demonstrated that one of AI’s longtime grand challenges could be conquered by an inventive combination of reinforcement learning, convolutional neural networks, and Monte Carlo tree search (and adding powerful modern computing hardware to the mix). As a result, AlphaGo has attained a

…

most active areas of research for machine-learning practitioners, progress on this front is still nascent.3 “Without Human Examples or Guidance” Unlike supervised learning, reinforcement learning holds the promise of programs that can truly learn on their own, simply by performing actions in their “environment” and observing the outcome. DeepMind’s

…

most important claim about its results, especially on AlphaGo, is that the work has delivered on that promise: “Our results comprehensively demonstrate that a pure reinforcement learning approach is fully feasible, even in the most challenging of domains: it is possible to train to superhuman level, without human examples or guidance, given

…

. As the psychologist and AI researcher Gary Marcus has pointed out, none of these crucial aspects of AlphaGo were “learned from the data, by pure reinforcement learning. Rather, [they were] built in innately … by DeepMind’s programmers.”5 DeepMind’s Atari game-playing programs were actually better examples of “learning without

…

deep Q-learning method on several Atari video games. The most surprising good performer was “random search”: instead of training a Deep Q-Network by reinforcement learning over many episodes, one can simply try out many different convolutional neural networks with randomly chosen weights.6 That is, there is no learning whatsoever

…

(for example, wall, ceiling, paddle, ball, tunneling), the program actually has no such concepts: These demonstrations make clear that it is misleading to credit deep reinforcement learning with inducing concepts like wall or paddle; rather, such remarks are what comparative (animal) psychology sometimes call overattributions. It’s not that the Atari system

…

real-world problems and have a huge impact on things like healthcare and science.” I think it’s very possible that DeepMind’s work on reinforcement learning may eventually have the kinds of impacts Hassabis is aiming for. But there’s a long way to go from games to the real

…

boundaries at all; you don’t know what’s in the situation, what’s out of the situation.”13 As an example, consider using reinforcement learning to train a robot to perform a very useful real-world task: take the dirty dishes stacked in the sink and put them in the

…

are violated, and any successful approach would look extremely different.”15 No one knows what that successful approach would be. Indeed, the field of deep reinforcement learning is still quite young. The results I described in this chapter can be seen as a proof of principle: the combination of deep networks and

…

25/fashion/what-shamu-taught-me-about-a-happy-marriage.html.   2.  thejetsons.wikia.com/wiki/Rosey.   3.  To be more precise, this approach to reinforcement learning, called value learning, is not the only possible approach. A second approach, called policy learning, has the goal of learning directly what action to perform

…

and P. Dayan, “Q-Learning,” Machine Learning 8, nos. 3–4 (1992): 279–92.   5.  For a detailed, technical introduction to reinforcement learning, see R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed. (Cambridge, Mass.: MIT Press, 2017), incompleteideas.net/book/the-book-2nd.html.   6.  For example, see the

…

stage Task,” in Proceedings of the First Annual Conference on Robot Learning, CoRL (2017); M. Cutler, T. J. Walsh, and J. P. How, “Real-World Reinforcement Learning via Multifidelity Simulators,” IEEE Transactions on Robotics 31, no. 3 (2015): 655–71. 9: Game On   1.  Demis Hassabis, quoted in P. Iwaniuk, “A Conversation

…

1; it is initially set close to 1 and is gradually decreased over the episodes of training.   5.  R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed. (Cambridge, Mass.: MIT Press, 2017), 124, incompleteideas.net/book/the-book-2nd.html.   6.  For more details, see V. Mnih

…

27.  D. Silver et al., “Mastering the Game of Go Without Human Knowledge,” Nature, 550 (2017): 354–59. 28.  D. Silver et al., “A General Reinforcement Learning Algorithm That Masters Chess, Shogi, and Go Through Self-Play,” Science 362, no. 6419 (2018): 1140–44. 10: Beyond Games   1.  Quoted in P. Iwaniuk

…

06/deepminds-alphago-mastered-chess-spare-time.   3.  As one example, still in the game-playing domain, DeepMind published a paper in 2018 describing a reinforcement-learning system that they claimed exhibited some degree of transfer learning in its ability to play different Atari games. L. Espeholt et al., “Impala: Scalable Distributed

…

-far-from-intelligent. 14.  I should note that a few robotics groups have actually developed dishwasher-loading robots, though none of these was trained by reinforcement learning, or any other kind of machine-learning method, as far as I know. These robots come with some impressive videos (for example, “Robotic Dog

…

Robotics Atari video games; see also Breakout automated image captioning autonomous vehicles, see self-driving cars B back-propagation; in convolutional neural networks; in deep reinforcement learning barrier of meaning Barsalou, Lawrence beneficial AI Bengio, Yoshua bias; in face recognition; in word vectors big data bilingual evaluation understudy (BLEU) board positions;

…

–15; adversarial attacks on; see also IBM Watson; reading comprehension; Stanford Question-Answering Dataset; Winograd schemas R reading comprehension recurrent neural networks; 199–200 regulation reinforcement learning; actions of agent in; contrast with supervised learning; deep Q-learning, see deep Q-learning; discounting in; episode; epsilon-greedy method for; exploration versus

…

with symbolic methods suitcase words Summer Vision Project (MIT) superhuman intelligence superintelligence, see superhuman intelligence Superintelligence (book) supervised learning; contrast with human learning; contrast with reinforcement learning; in IBM Watson support vector machines Sutherland, Amy Sutskever, Ilya Sutton, Richard symbolic AI; contrast with subsymbolic methods; integration with subsymbolic methods Szegedy, Christian T

…

temporal difference learning; see also reinforcement learning test set theory of mind thought vectors training, see supervised learning training set transfer learning; for Breakout translation, see machine translation trolley problem Turing, Alan

The Means of Prediction: How AI Really Works (And Who Benefits)

by Maximilian Kasy  · 15 Jan 2025  · 209pp  · 63,332 words

with different options and doing what seems best based on what has been learned from actions they have previously tried. Reinforcement learning goes one step further than multi-armed bandit algorithms. Reinforcement learning builds algorithms that learn to plan by learning how likely it is that different states of the world are favorable down

…

as crucial in many real-world tasks as it is in playing games. The problem of planning is taken into account in the framework of reinforcement learning. Reinforcement Learning Certain board games have long stood as symbols of intellectual challenge: chess in Europe, go in East Asia, and backgammon in the Middle East. In

…

algorithm of AlphaGo was much simpler. Like TD-Gammon, it used an approach called (deep) reinforcement learning. Both TD-Gammon and AlphaGo learned to play by playing a vast number of games against themselves. The term reinforcement learning comes from behaviorist ideas of how animals learn and how they can be trained. According to

…

, to teach a computer to play backgammon by means of selective rewards than to try the same with your dog or cat. But how does reinforcement learning work with a computer? Your computer has no innate desire for treats, after all. Let us start by again considering the multi-armed bandits. How

…

and AlphaGo used. They trained neural networks to predict the probability of winning, in the recursive manner described above. This all sounds pretty good: Deep reinforcement learning can teach itself by exploring the world and learning to plan for the future. Why not use this approach to address all kinds of real

…

need to draw conclusions about the state of the world, and you need to make predictions about its future state. Any real-world application of reinforcement learning must overcome this issue of partial observability, which implies that it needs to remember the past to predict the future. The second reason why deep

…

reinforcement learning is not easily applied to real-world problems is that it is a very data-hungry approach. As mentioned above, the way AlphaGo learned was

…

might want to do exactly that.) Actual self-driving cars, to the extent that they exist, thus need to rely on approaches other than pure reinforcement learning. There is a more general lesson here. Current deep-learning-based methods in AI need lots of data. There are some settings where data have

…

to solve. At one extreme, in terms of scalability, we have data generated via simulation, such as the simulated games that were used to train reinforcement learning algorithms like AlphaGo. In domains where data can be generated by simulation, there are no limits to the machine-learning-based approach, at least in

…

safe AI—or so some claim. One of the approaches that has been proposed for addressing the problem of value alignment is known as inverse reinforcement learning—in effect, sidestepping the problem of explicitly specifying a reward function for AI algorithms. Rather than maximizing an explicitly specified reward function, in inverse

…

reinforcement learning the algorithms are supposed to construct a reward function by inferring human objectives from observed human behavior. The algorithms are meant to learn their own

…

further, the algorithms are also supposed to learn the purpose of human actions relative to future rewards. This, then, is truly an inversion of the reinforcement learning problem. If a human waters a basil plant now, he might want to eat basil pesto in a month. If another human trains to increase

…

behavior for decades. In doing so, they have learned that estimating preferences is quite difficult and not always feasible, even when maintaining strong assumptions. Inverse reinforcement learning and related approaches may hold some promise, but there are fundamental limits to what these approaches can achieve. They cannot solve the multitasking problem, and

…

teaching to the test. This problem cannot be solved by observing educational policymakers to infer the preferences of these policymakers, which is what the inverse reinforcement learning approach would try to do. Teaching to the test occurs because standardized tests cannot measure certain important dimensions of student development. No amount of reward

…

behavior to infer his preferences and by deriving from these inferred preferences the correct objective for social media feed selection, which is, again, what inverse reinforcement learning would suggest. There are fundamental limits on the extent to which the value-alignment problem can be solved by reward engineering or by approaches such

…

as inverse reinforcement learning. Because of this, there are settings where important decisions should not be delegated to AI systems. These systems are bound to ignore unmeasured dimensions of

…

well-being, and the underlying agency problems prevent effective delegation. There is another, and arguably even more important, problem that needs to be solved. Inverse reinforcement learning is supposed to learn human preferences and act accordingly. But which human’s preferences? Whose values should the algorithm align with? Alignment with Whom? There

…

of the Web.” New Yorker, February 9, 2023. François-Lavet, V., P. Henderson, R. Islam, M. G. Bellemare, and J. Pineau. “An Introduction to Deep Reinforcement Learning.” Foundations and Trends in Machine Learning 11 no. 3–4 (2018): 219–354. Friedman, J., T. Hastie, and R. Tibshirani. The Elements of Statistical Learning

…

1988 Neyman Memorial Lecture: A Galtonian Perspective on Shrinkage Estimators.” Statistical Science 5, no. 1 (1990): 147–55. Sutton, R. S., and A. G. Barto. Reinforcement Learning: An Introduction. MIT Press, 2018. Thompson, W. R. “On the Likelihood That One Unknown Probability Exceeds Another in View of the Evidence of Two Samples

…

, 45, 49–50, 63–65, 89–90; and generative AI, 53–56; how it works, 45–50; reinforcement learning and, 63; relative simplicity of, 49; self-supervised, 52–53; technical meaning of, 47 deep reinforcement learning, 63 democratic governance: and automated decision-making, 2, 7–8, 111–13, 135, 186–88; broad conception of

…

intelligence, concept of, 19, 21–22 intelligence explosion, 4, 22, 122 International Energy Agency, 93 interoperability requirements, 110 interpretability. See explainability interventions, 181–82 inverse reinforcement learning, 129–30 Israel, AI use in warfare by, 6, 31–32, 133 Jabirian corpus, 44, 51 Johnson, Simon, 150 jury of peers, 199 Kafka, Franz

…

noise, 141 randomized response method, 136–38, 137 rare earth minerals, 90 Rawls, John, 80 recidivism, 169 regularization, 40 regulation, of data collection, 145–46 reinforcement learning, 12, 60–65. See also inverse reinforcement learning; reward design representative democracy, 197–99 revealed preferences, 78, 129 reward design, 124–28. See also objectives of AI

…

; reinforcement learning Robinson, Joan, 154 robots, 6, 23–24, 122, 129, 131, 157, 160–61 robustness, 177–78, 184 Russell, Stuart, 4 safety. See AI safety sample

Advances in Artificial General Intelligence: Concepts, Architectures and Algorithms: Proceedings of the Agi Workshop 2006

by Ben Goertzel and Pei Wang  · 1 Jan 2007  · 303pp  · 67,891 words

The Unwritten Rules of Social Relationships: Decoding Social Mysteries Through the Unique Perspectives of Autism

by Temple Grandin and Sean Barron  · 30 Sep 2012  · 347pp  · 123,884 words

The AI-First Company

by Ash Fontana  · 4 May 2021  · 296pp  · 66,815 words

Human Compatible: Artificial Intelligence and the Problem of Control

by Stuart Russell  · 7 Oct 2019  · 416pp  · 112,268 words

I, Warbot: The Dawn of Artificially Intelligent Conflict

by Kenneth Payne  · 16 Jun 2021  · 339pp  · 92,785 words

Evil Genes: Why Rome Fell, Hitler Rose, Enron Failed, and My Sister Stole My Mother's Boyfriend

by Barbara Oakley Phd  · 20 Oct 2008

High-Frequency Trading

by David Easley, Marcos López de Prado and Maureen O'Hara  · 28 Sep 2013

The Road to Conscious Machines

by Michael Wooldridge  · 2 Nov 2018  · 346pp  · 97,890 words

Artificial Whiteness

by Yarden Katz

The Age of AI: And Our Human Future

by Henry A Kissinger, Eric Schmidt and Daniel Huttenlocher  · 2 Nov 2021  · 194pp  · 57,434 words

Genius Makers: The Mavericks Who Brought A. I. To Google, Facebook, and the World

by Cade Metz  · 15 Mar 2021  · 414pp  · 109,622 words

Rule of the Robots: How Artificial Intelligence Will Transform Everything

by Martin Ford  · 13 Sep 2021  · 288pp  · 86,995 words

The Brain That Changes Itself: Stories of Personal Triumph From the Frontiers of Brain Science

by Norman Doidge  · 15 Mar 2007  · 515pp  · 136,938 words

Mindware: Tools for Smart Thinking

by Richard E. Nisbett  · 17 Aug 2015  · 397pp  · 109,631 words

Rationality: From AI to Zombies

by Eliezer Yudkowsky  · 11 Mar 2015  · 1,737pp  · 491,616 words

Networks, Crowds, and Markets: Reasoning About a Highly Connected World

by David Easley and Jon Kleinberg  · 15 Nov 2010  · 1,535pp  · 337,071 words

Superintelligence: Paths, Dangers, Strategies

by Nick Bostrom  · 3 Jun 2014  · 574pp  · 164,509 words

The Master Algorithm: How the Quest for the Ultimate Learning Machine Will Remake Our World

by Pedro Domingos  · 21 Sep 2015  · 396pp  · 117,149 words

Possible Minds: Twenty-Five Ways of Looking at AI

by John Brockman  · 19 Feb 2019  · 339pp  · 94,769 words

Model Thinker: What You Need to Know to Make Data Work for You

by Scott E. Page  · 27 Nov 2018  · 543pp  · 153,550 words

Robot Rules: Regulating Artificial Intelligence

by Jacob Turner  · 29 Oct 2018  · 688pp  · 147,571 words

Behave: The Biology of Humans at Our Best and Worst

by Robert M. Sapolsky  · 1 May 2017  · 1,261pp  · 294,715 words

The Transhumanist Reader

by Max More and Natasha Vita-More  · 4 Mar 2013  · 798pp  · 240,182 words

Framers: Human Advantage in an Age of Technology and Turmoil

by Kenneth Cukier, Viktor Mayer-Schönberger and Francis de Véricourt  · 10 May 2021  · 291pp  · 80,068 words

The Deep Learning Revolution (The MIT Press)

by Terrence J. Sejnowski  · 27 Sep 2018

The Secret of Our Success: How Culture Is Driving Human Evolution, Domesticating Our Species, and Making Us Smarter

by Joseph Henrich  · 27 Oct 2015  · 631pp  · 177,227 words

The Singularity Is Nearer: When We Merge with AI

by Ray Kurzweil  · 25 Jun 2024

The Coming Wave: Technology, Power, and the Twenty-First Century's Greatest Dilemma

by Mustafa Suleyman  · 4 Sep 2023  · 444pp  · 117,770 words

Know Thyself

by Stephen M Fleming  · 27 Apr 2021

What to Think About Machines That Think: Today's Leading Thinkers on the Age of Machine Intelligence

by John Brockman  · 5 Oct 2015  · 481pp  · 125,946 words

Cogs and Monsters: What Economics Is, and What It Should Be

by Diane Coyle  · 11 Oct 2021  · 305pp  · 75,697 words

Fixed: Why Personal Finance is Broken and How to Make it Work for Everyone

by John Y. Campbell and Tarun Ramadorai  · 25 Jul 2025

The New Harvest: Agricultural Innovation in Africa

by Calestous Juma  · 27 May 2017

Succeeding With AI: How to Make AI Work for Your Business

by Veljko Krunic  · 29 Mar 2020

What Algorithms Want: Imagination in the Age of Computing

by Ed Finn  · 10 Mar 2017  · 285pp  · 86,853 words

Seeking SRE: Conversations About Running Production Systems at Scale

by David N. Blank-Edelman  · 16 Sep 2018

Virtual Competition

by Ariel Ezrachi and Maurice E. Stucke  · 30 Nov 2016

Four Battlegrounds

by Paul Scharre  · 18 Jan 2023

On the Edge: The Art of Risking Everything

by Nate Silver  · 12 Aug 2024  · 848pp  · 227,015 words

Empire of AI: Dreams and Nightmares in Sam Altman's OpenAI

by Karen Hao  · 19 May 2025  · 660pp  · 179,531 words

Making Sense of Chaos: A Better Economics for a Better World

by J. Doyne Farmer  · 24 Apr 2024  · 406pp  · 114,438 words

Machine, Platform, Crowd: Harnessing Our Digital Future

by Andrew McAfee and Erik Brynjolfsson  · 26 Jun 2017  · 472pp  · 117,093 words

The Price of Tomorrow: Why Deflation Is the Key to an Abundant Future

by Jeff Booth  · 14 Jan 2020  · 180pp  · 55,805 words

Narrative Economics: How Stories Go Viral and Drive Major Economic Events

by Robert J. Shiller  · 14 Oct 2019  · 611pp  · 130,419 words

The Organized Mind: Thinking Straight in the Age of Information Overload

by Daniel J. Levitin  · 18 Aug 2014  · 685pp  · 203,949 words

Darwin's Dangerous Idea: Evolution and the Meanings of Life

by Daniel C. Dennett  · 15 Jan 1995  · 846pp  · 232,630 words

Deep Medicine: How Artificial Intelligence Can Make Healthcare Human Again

by Eric Topol  · 1 Jan 2019  · 424pp  · 114,905 words

Demystifying Smart Cities

by Anders Lisdorf

Power and Progress: Our Thousand-Year Struggle Over Technology and Prosperity

by Daron Acemoglu and Simon Johnson  · 15 May 2023  · 619pp  · 177,548 words

Handbook of Modeling High-Frequency Data in Finance

by Frederi G. Viens, Maria C. Mariani and Ionut Florescu  · 20 Dec 2011  · 443pp  · 51,804 words

Radical Technologies: The Design of Everyday Life

by Adam Greenfield  · 29 May 2017  · 410pp  · 119,823 words

WTF?: What's the Future and Why It's Up to Us

by Tim O'Reilly  · 9 Oct 2017  · 561pp  · 157,589 words

Prediction Machines: The Simple Economics of Artificial Intelligence

by Ajay Agrawal, Joshua Gans and Avi Goldfarb  · 16 Apr 2018  · 345pp  · 75,660 words

The Autonomous Revolution: Reclaiming the Future We’ve Sold to Machines

by William Davidow and Michael Malone  · 18 Feb 2020  · 304pp  · 80,143 words

The Smartphone Society

by Nicole Aschoff

Supremacy: AI, ChatGPT, and the Race That Will Change the World

by Parmy Olson  · 284pp  · 96,087 words

The Optimist: Sam Altman, OpenAI, and the Race to Invent the Future

by Keach Hagey  · 19 May 2025  · 439pp  · 125,379 words

The Linguist: A Personal Guide to Language Learning

by Steve Kaufmann  · 15 Jan 2003

Psychopathy: An Introduction to Biological Findings and Their Implications

by Andrea L. Glenn and Adrian Raine  · 7 Mar 2014

Whiplash: How to Survive Our Faster Future

by Joi Ito and Jeff Howe  · 6 Dec 2016  · 254pp  · 76,064 words

AI Superpowers: China, Silicon Valley, and the New World Order

by Kai-Fu Lee  · 14 Sep 2018  · 307pp  · 88,180 words

The Creativity Code: How AI Is Learning to Write, Paint and Think

by Marcus Du Sautoy  · 7 Mar 2019  · 337pp  · 103,522 words

Global Catastrophic Risks

by Nick Bostrom and Milan M. Cirkovic  · 2 Jul 2008

Super Thinking: The Big Book of Mental Models

by Gabriel Weinberg and Lauren McCann  · 17 Jun 2019

Hands-On Machine Learning With Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems

by Aurelien Geron  · 14 Aug 2019

If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All

by Eliezer Yudkowsky and Nate Soares  · 15 Sep 2025  · 215pp  · 64,699 words

These Strange New Minds: How AI Learned to Talk and What It Means

by Christopher Summerfield  · 11 Mar 2025  · 412pp  · 122,298 words

Escape From Model Land: How Mathematical Models Can Lead Us Astray and What We Can Do About It

by Erica Thompson  · 6 Dec 2022  · 250pp  · 79,360 words

Like, Comment, Subscribe: Inside YouTube's Chaotic Rise to World Domination

by Mark Bergen  · 5 Sep 2022  · 642pp  · 141,888 words

Human + Machine: Reimagining Work in the Age of AI

by Paul R. Daugherty and H. James Wilson  · 15 Jan 2018  · 523pp  · 61,179 words

Range: Why Generalists Triumph in a Specialized World

by David Epstein  · 1 Mar 2019  · 406pp  · 109,794 words

Outnumbered: From Facebook and Google to Fake News and Filter-Bubbles – the Algorithms That Control Our Lives

by David Sumpter  · 18 Jun 2018  · 276pp  · 81,153 words

Text Analytics With Python: A Practical Real-World Approach to Gaining Actionable Insights From Your Data

by Dipanjan Sarkar  · 1 Dec 2016

The Loop: How Technology Is Creating a World Without Choices and How to Fight Back

by Jacob Ward  · 25 Jan 2022  · 292pp  · 94,660 words

21 Lessons for the 21st Century

by Yuval Noah Harari  · 29 Aug 2018  · 389pp  · 119,487 words

Army of None: Autonomous Weapons and the Future of War

by Paul Scharre  · 23 Apr 2018  · 590pp  · 152,595 words

A Hacker's Mind: How the Powerful Bend Society's Rules, and How to Bend Them Back

by Bruce Schneier  · 7 Feb 2023  · 306pp  · 82,909 words

Co-Intelligence: Living and Working With AI

by Ethan Mollick  · 2 Apr 2024  · 189pp  · 58,076 words

The Economic Singularity: Artificial Intelligence and the Death of Capitalism

by Calum Chace  · 17 Jul 2016  · 477pp  · 75,408 words

Data Mining: Concepts, Models, Methods, and Algorithms

by Mehmed Kantardzić  · 2 Jan 2003  · 721pp  · 197,134 words

Scary Smart: The Future of Artificial Intelligence and How You Can Save Our World

by Mo Gawdat  · 29 Sep 2021  · 259pp  · 84,261 words

Artificial Unintelligence: How Computers Misunderstand the World

by Meredith Broussard  · 19 Apr 2018  · 245pp  · 83,272 words

The Revolution That Wasn't: GameStop, Reddit, and the Fleecing of Small Investors

by Spencer Jakab  · 1 Feb 2022  · 420pp  · 94,064 words

A World Without Work: Technology, Automation, and How We Should Respond

by Daniel Susskind  · 14 Jan 2020  · 419pp  · 109,241 words

The Inevitable: Understanding the 12 Technological Forces That Will Shape Our Future

by Kevin Kelly  · 6 Jun 2016  · 371pp  · 108,317 words

Science in the Soul: Selected Writings of a Passionate Rationalist

by Richard Dawkins  · 15 Mar 2017  · 420pp  · 130,714 words

The Long History of the Future: Why Tomorrow's Technology Still Isn't Here

by Nicole Kobie  · 3 Jul 2024  · 348pp  · 119,358 words

Sunfall

by Jim Al-Khalili  · 17 Apr 2019  · 381pp  · 120,361 words

The Precipice: Existential Risk and the Future of Humanity

by Toby Ord  · 24 Mar 2020  · 513pp  · 152,381 words

Amateurs!: How We Built Internet Culture and Why It Matters

by Joanna Walsh  · 22 Sep 2025  · 255pp  · 80,203 words

Thinking Machines: The Inside Story of Artificial Intelligence and Our Race to Build the Future

by Luke Dormehl  · 10 Aug 2016  · 252pp  · 74,167 words

The Big Nine: How the Tech Titans and Their Thinking Machines Could Warp Humanity

by Amy Webb  · 5 Mar 2019  · 340pp  · 97,723 words

Autonomous Driving: How the Driverless Revolution Will Change the World

by Andreas Herrmann, Walter Brenner and Rupert Stadler  · 25 Mar 2018

Luxury Fever: Why Money Fails to Satisfy in an Era of Excess

by Robert H. Frank  · 15 Jan 1999  · 416pp  · 112,159 words

The Rationalist's Guide to the Galaxy: Superintelligent AI and the Geeks Who Are Trying to Save Humanity's Future

by Tom Chivers  · 12 Jun 2019  · 289pp  · 92,714 words

Nexus: A Brief History of Information Networks From the Stone Age to AI

by Yuval Noah Harari  · 9 Sep 2024  · 566pp  · 169,013 words

The Geeks Shall Inherit the Earth: Popularity, Quirk Theory, and Why Outsiders Thrive After High School

by Alexandra Robbins  · 31 Mar 2009  · 509pp  · 147,998 words

Misbehaving: The Making of Behavioral Economics

by Richard H. Thaler  · 10 May 2015  · 500pp  · 145,005 words

Robots Will Steal Your Job, But That's OK: How to Survive the Economic Collapse and Be Happy

by Pistono, Federico  · 14 Oct 2012  · 245pp  · 64,288 words

The Ages of Globalization

by Jeffrey D. Sachs  · 2 Jun 2020

The Science and Technology of Growing Young: An Insider's Guide to the Breakthroughs That Will Dramatically Extend Our Lifespan . . . And What You Can Do Right Now

by Sergey Young  · 23 Aug 2021  · 326pp  · 88,968 words

Futureproof: 9 Rules for Humans in the Age of Automation

by Kevin Roose  · 9 Mar 2021  · 208pp  · 57,602 words

The Power of Habit: Why We Do What We Do in Life and Business

by Charles Duhigg  · 1 Jan 2011  · 455pp  · 116,578 words

Top Dog: The Science of Winning and Losing

by Po Bronson and Ashley Merryman  · 19 Feb 2013  · 407pp  · 109,653 words

Split-Second Persuasion: The Ancient Art and New Science of Changing Minds

by Kevin Dutton  · 3 Feb 2011  · 338pp  · 100,477 words

The Myth of Artificial Intelligence: Why Computers Can't Think the Way We Do

by Erik J. Larson  · 5 Apr 2021

Why Machines Learn: The Elegant Math Behind Modern AI

by Anil Ananthaswamy  · 15 Jul 2024  · 416pp  · 118,522 words

The Thinking Machine: Jensen Huang, Nvidia, and the World's Most Coveted Microchip

by Stephen Witt  · 8 Apr 2025  · 260pp  · 82,629 words

Human Frontiers: The Future of Big Ideas in an Age of Small Thinking

by Michael Bhaskar  · 2 Nov 2021

Our Final Invention: Artificial Intelligence and the End of the Human Era

by James Barrat  · 30 Sep 2013  · 294pp  · 81,292 words

The Extended Phenotype: The Long Reach of the Gene

by Richard Dawkins  · 1 Jan 1982  · 506pp  · 152,049 words

The AI Economy: Work, Wealth and Welfare in the Robot Age

by Roger Bootle  · 4 Sep 2019  · 374pp  · 111,284 words

Homo Deus: A Brief History of Tomorrow

by Yuval Noah Harari  · 1 Mar 2015  · 479pp  · 144,453 words

Applied Artificial Intelligence: A Handbook for Business Leaders

by Mariya Yao, Adelyn Zhou and Marlene Jia  · 1 Jun 2018  · 161pp  · 39,526 words

Life After Google: The Fall of Big Data and the Rise of the Blockchain Economy

by George Gilder  · 16 Jul 2018  · 332pp  · 93,672 words

Learn Descriptive Cataloging Second North American Edition

by Mary Mortimer  · 1 Jan 1999  · 282pp  · 28,394 words

Future Politics: Living Together in a World Transformed by Tech

by Jamie Susskind  · 3 Sep 2018  · 533pp

Learn Algorithmic Trading

by Sebastien Donadio  · 7 Nov 2019

Machine Learning Design Patterns: Solutions to Common Challenges in Data Preparation, Model Building, and MLOps

by Valliappa Lakshmanan, Sara Robinson and Michael Munn  · 31 Oct 2020

Learning Scikit-Learn: Machine Learning in Python

by Raúl Garreta and Guillermo Moncecchi  · 14 Sep 2013  · 122pp  · 29,286 words

Mastering Machine Learning With Scikit-Learn

by Gavin Hackeling  · 31 Oct 2014

The Future Is Faster Than You Think: How Converging Technologies Are Transforming Business, Industries, and Our Lives

by Peter H. Diamandis and Steven Kotler  · 28 Jan 2020  · 501pp  · 114,888 words

Algorithms to Live By: The Computer Science of Human Decisions

by Brian Christian and Tom Griffiths  · 4 Apr 2016  · 523pp  · 143,139 words

The Perfect Bet: How Science and Math Are Taking the Luck Out of Gambling

by Adam Kucharski  · 23 Feb 2016  · 360pp  · 85,321 words

Ghost Work: How to Stop Silicon Valley From Building a New Global Underclass

by Mary L. Gray and Siddharth Suri  · 6 May 2019  · 346pp  · 97,330 words

The Future of the Brain: Essays by the World's Leading Neuroscientists

by Gary Marcus and Jeremy Freeman  · 1 Nov 2014  · 336pp  · 93,672 words

Bandit Algorithms for Website Optimization

by John Myles White  · 10 Dec 2012  · 94pp  · 22,435 words

Talk to Me: How Voice Computing Will Transform the Way We Live, Work, and Think

by James Vlahos  · 1 Mar 2019  · 392pp  · 108,745 words

Hooked: Food, Free Will, and How the Food Giants Exploit Our Addictions

by Michael Moss  · 2 Mar 2021  · 300pp  · 94,628 words

Being You: A New Science of Consciousness

by Anil Seth  · 29 Aug 2021  · 418pp  · 102,597 words

This Is for Everyone: The Captivating Memoir From the Inventor of the World Wide Web

by Tim Berners-Lee  · 8 Sep 2025  · 347pp  · 100,038 words

Surviving AI: The Promise and Peril of Artificial Intelligence

by Calum Chace  · 28 Jul 2015  · 144pp  · 43,356 words

The Simulation Hypothesis

by Rizwan Virk  · 31 Mar 2019  · 315pp  · 89,861 words

Industry 4.0: The Industrial Internet of Things

by Alasdair Gilchrist  · 27 Jun 2016

How to Predict the Unpredictable

by William Poundstone  · 267pp  · 71,941 words

The Complete Book of Home Organization: 336 Tips and Projects

by Abowlfulloflemons.com and Toni Hammersley  · 5 Jan 2016  · 278pp  · 42,509 words