Attention Is All You Need

The Infinity Machine: Demis Hassabis, DeepMind, and the Quest for Superintelligence

by Sebastian Mallaby;  · 30 Mar 2026  · 607pp  · 161,998 words

to attend to as it outputted each bit of its response, the original word ordering could be ignored: Paradoxically, sequential learning could be done nonsequentially. “Attention Is All You Need,” ran the title of the Google paper. Philosophically, Google’s innovation recalled an old insight about time, and how people relate to it. In the

How AI Thinks

by Nigel Toon  · 13 Dec 2023

LSTM developments are useful, but it was only as recently as December 2017 that a big breakthrough was made. A now-famous technical paper titled ‘Attention is all you need’7 was published by a team from the Google Brain project and Toronto University. This much-referenced paper proposed an improved mechanism for learning how

…

. 26 6Tulving, Endel, ‘How many memory systems are there?’, American Psychologist, 40: 4 (1985), pp. 385–98 7Vaswani, Ashish; Shazeer, Noam; Parmar, Niki et al., ‘Attention is all you need’, Advances in Neural Information Processing Systems (2017), pp. 5998–6008 8Jumper, John; Evans, Richard; Pritzel, Alexander et al., ‘Highly accurate protein structure prediction with AlphaFold

…

, 134, 235 focussing 2, 56, 60, 61, 63, 108, 114, 131, 171, 206 multi-headed 62, 63, 64, 65 attention-based AI 211, 277, 280 ‘Attention is all you need’ (Google Brain project and Toronto University) 61–2 attention systems 60–61 Autor, Professor David H.: ‘Why are there still so many jobs?…’ 216 Baars

The Means of Prediction: How AI Really Works (And Who Benefits)

by Maximilian Kasy  · 15 Jan 2025  · 209pp  · 63,332 words

Another in View of the Evidence of Two Samples.” Biometrika 25 no. 3–4 (1933): 285–94. Vaswani, A., N. Shazeer, N. Parmar, et al. “Attention Is All You Need.” Proceedings of the 31st International Conference on Neural Information Processing Systems, December 4, 2017, 6000–6010. Yang, L., Z. Zhang, Y. Song, et al. “Diffusion

Supremacy: AI, ChatGPT, and the Race That Will Change the World

by Parmy Olson  · 284pp  · 96,087 words

,” he said aloud at one point. Jones looked up from his desk, nearby. “I’m not very good with titles,” he replied. “But how about ‘Attention is all you need’?” It was a random thought that had popped into his head, and Vaswani didn’t say anything in agreement. In fact, he got up and

…

walked away, Jones recalls. But later, the title “Attention Is All You Need” landed on the front page of their paper, a perfect summary of what they’d discovered. When you used a transformer, your AI system could

…

, Shazeer left Google in 2021 to pursue his research on large language models independently, cofounding a chatbot company called Character.ai. By that time, the “Attention Is All You Need” paper had become one of the most popular research works of all time in the field of AI. Typically, a research paper on AI might

…

Understanding.” blog.research.google, August 31, 2017. Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. “Attention Is All You Need.” Advances in Neural Information Processing Systems 30 (2017). Chapter 10: Size Matters Brockman, Greg (@gdb). “Held our civil ceremony in the @OpenAI office last week

The Thinking Machine: Jensen Huang, Nvidia, and the World's Most Coveted Microchip

by Stephen Witt  · 8 Apr 2025  · 260pp  · 82,629 words

in the Neural Information Processing Systems journal, which had published the original AlexNet results. The paper needed a name, so Jones, channeling the Beatles, suggested “Attention Is All You Need.” This was an off-the-cuff joke that he didn’t think the team would actually use. Later, he would meet people with the sentence

We Are as Gods: A Survival Guide for the Age of Abundance

by Peter H. Diamandis and Steven Kotler  · 13 Apr 2026  · 225pp  · 76,418 words

team of researchers were wrapping up what seemed like a small side project. They documented their findings in a brief paper with a modest title: “Attention Is All You Need.” Inside those few pages, they introduced the Transformer, a deceptively simple architecture for pattern recognition and language processing. Around the office, Googlers buzzed with excitement

…

/2025/09/06/godfather-of-ai-geoffrey-hinton-massive-unemployment-soaring-profits-capitalist-system/. “Attention Is All You Need”: Ashish Vaswani et al., “Attention Is All You Need,” Advances in Neural Information Processing Systems (NIPS 2017) 30 (2017): 5998–6008, https://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf. After a year in the trenches: Alec Radford, Karthik Narasimhan, Tim Salimans, and

Co-Intelligence: Living and Working With AI

by Ethan Mollick  · 2 Apr 2024  · 189pp  · 58,076 words

among the many papers on different forms of AI being published by industry and academic experts, one stood out, a paper with the catchy title “Attention Is All You Need.” Published by Google researchers in 2017, this paper introduced a significant shift in the world of AI, particularly in how computers understand and process human

The Optimist: Sam Altman, OpenAI, and the Race to Invent the Future

by Keach Hagey  · 19 May 2025  · 439pp  · 125,379 words

negative without relying on humans to pre-label the data. Two months later, Sutskever read a preprint of a paper by eight Google researchers titled “Attention Is All You Need,” which he immediately recognized as presenting a method to make the kind of research Radford was doing vastly more efficient. Rather than processing one character

…

, 119 Atari, 87, 147, 165, 190 Atkins, Brian and Sabine, 142–43 Atlanta Georgi, 23 Atlantic, The (magazine), 46, 297 Atmos home building platform, 260 “Attention Is All You Need” (“the transformer paper”), 218–19, 270 Australia, 99, 266 Authors Guild, 220 Autopilot AI-assisted driving, 225 Baidu, 170 baldness, cure for, 108 Bankman-Fried

…

,” 188, 312 Goertzel, Ben, 145 Goetz, Jim, 88 Goldman Sachs, 59, 64, 150, 225 Good Ventures foundation, 212–13 Google AdSense, 243 Alphabet, 194, 271 “Attention Is All You Need” (“the transformer paper”), 218–19, 270 Bard conversational model, 271 Chrome web browser, 272 DeepMind acquisition, 146–48, 154, 165, 168–69, 171–72, 184

Artificial Intelligence: A Modern Approach

by Stuart Russell and Peter Norvig  · 14 Jul 2019  · 2,466pp  · 668,761 words

, the highest scoring hypothesis La entrada can only generate low-probability continuations, so it “falls off the beam.” 25.4The Transformer Architecture The influential article “Attention is all you need” (Vaswani et al., 2018) introduced the transformer architecture, which uses a self-attention mechanism that can model long-distance context without a sequential dependency. 25

…

GPU performance evaluation. arXiv:1412.7580. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I.(2018). Attention is all you need. In NeurIPS 30. Veach, E. and Guibas, L. J. (1995). Optimally combining sampling techniques for Monte Carlo rendering. In Proc. 22rd Annual Conference on Computer

These Strange New Minds: How AI Learned to Talk and What It Means

by Christopher Summerfield  · 11 Mar 2025  · 412pp  · 122,298 words

The transformer was invented in 2017. It was first described in a preprint – a paper published online without peer review – with the slightly incongruous title ‘Attention is All You Need’.[*1] The paper didn’t make much of a splash at first. Submitted to the annual jamboree that is the Neural Information Processing Systems (NeurIPS

…

the entire input sequence in parallel – using a form of attention called self-attention to place emphasis on each item i when predicting j (hence: ‘Attention is All You Need’). To understand why self-attention is so useful in language, consider the problem of completing the following two prompts: As I approached the ancient tree

…

Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks’, arXiv. Available at https://arxiv.org/pdf/2302.08399. Vaswani, A. et al. (2017), ‘Attention Is All You Need’. Preprint. arXiv. Available at http://arxiv.org/abs/1706.03762 (accessed 30 October 2020). Verzijden, M. N. et al. (2015), ‘Male Drosophila melanogaster Learn to

Code Dependent: Living in the Shadow of AI

by Madhumita Murgia  · 20 Mar 2024  · 336pp  · 91,806 words

On the Edge: The Art of Risking Everything

by Nate Silver  · 12 Aug 2024  · 848pp  · 227,015 words

Empire of AI: Dreams and Nightmares in Sam Altman's OpenAI

by Karen Hao  · 19 May 2025  · 660pp  · 179,531 words

The Singularity Is Nearer: When We Merge with AI

by Ray Kurzweil  · 25 Jun 2024