Attention Is All You Need

The Thinking Machine: Jensen Huang, Nvidia, and the World's Most Coveted Microchip

by Stephen Witt  · 8 Apr 2025  · 260pp  · 82,629 words

in the Neural Information Processing Systems journal, which had published the original AlexNet results. The paper needed a name, so Jones, channeling the Beatles, suggested “Attention Is All You Need.” This was an off-the-cuff joke that he didn’t think the team would actually use. Later, he would meet people with the sentence

The Infinity Machine: Demis Hassabis, DeepMind, and the Quest for Superintelligence

by Sebastian Mallaby;  · 30 Mar 2026  · 607pp  · 161,998 words

to attend to as it outputted each bit of its response, the original word ordering could be ignored: Paradoxically, sequential learning could be done nonsequentially. “Attention Is All You Need,” ran the title of the Google paper. Philosophically, Google’s innovation recalled an old insight about time, and how people relate to it. In the

How AI Thinks

by Nigel Toon  · 13 Dec 2023

LSTM developments are useful, but it was only as recently as December 2017 that a big breakthrough was made. A now-famous technical paper titled ‘Attention is all you need’7 was published by a team from the Google Brain project and Toronto University. This much-referenced paper proposed an improved mechanism for learning how

…

. 26 6Tulving, Endel, ‘How many memory systems are there?’, American Psychologist, 40: 4 (1985), pp. 385–98 7Vaswani, Ashish; Shazeer, Noam; Parmar, Niki et al., ‘Attention is all you need’, Advances in Neural Information Processing Systems (2017), pp. 5998–6008 8Jumper, John; Evans, Richard; Pritzel, Alexander et al., ‘Highly accurate protein structure prediction with AlphaFold

…

, 134, 235 focussing 2, 56, 60, 61, 63, 108, 114, 131, 171, 206 multi-headed 62, 63, 64, 65 attention-based AI 211, 277, 280 ‘Attention is all you need’ (Google Brain project and Toronto University) 61–2 attention systems 60–61 Autor, Professor David H.: ‘Why are there still so many jobs?…’ 216 Baars

On the Edge: The Art of Risking Everything

by Nate Silver  · 12 Aug 2024  · 848pp  · 227,015 words

seems like human-level text generation. And, you know, nobody really anticipated that.” In 2017, a group of researchers at Google published a paper called “Attention Is All You Need” that introduced something called a “transformer.” I’ll provide a more detailed description of a transformer later, but it isn’t important for now—the

…

of a mystery—this is the “bag of numbers” stage. But it just seems to work out somehow. In the famous Google paper on transformers, “Attention Is All You Need,” “attention” essentially refers to the importance of the relationships between different pairs of tokens. Once a transformer figures out these relationships, there isn’t a

…

. GO TO NOTE REFERENCE IN TEXT companies like OpenAI and Anthropic: “Google Brain Drain: Where are the Authors of ‘Attention Is All You Need’ Now?” AIChat, aichat.blog/google-exodus-where-are-the-authors-of-attention-is-all-you-need-now. GO TO NOTE REFERENCE IN TEXT Altman has tipped his hat:@SamA, https://twitter.com/sama/status/1540227243368058880?lang

…

, washingtonpost.com/technology/2023/05/16/sam-altman-open-ai-congress-hearing. GO TO NOTE REFERENCE IN TEXT “Attention Is All”: Ashish Vaswani et al., “Attention Is All You Need,” arXiv, August 1, 2023, arxiv.org/abs/1706.03762. GO TO NOTE REFERENCE IN TEXT most rapidly adopted technologies: Krystal Hu, “ChatGPT Sets Record for

Code Dependent: Living in the Shadow of AI

by Madhumita Murgia  · 20 Mar 2024  · 336pp  · 91,806 words

as the ‘transformer’. The eight research scientists who eventually played a part in its creation described it in a short paper with a snappy title: ‘Attention Is All You Need’.2 One of the authors, Llion Jones, who grew up in a tiny Welsh village, says the title was a nod to the Beatles song

…

called GPT – the Generative Pre-trained Transformer – software that could produce text-based answers in response to human queries. One of the authors of the ‘Attention Is All You Need’ paper, Lukasz Kaiser, had ended up working there and helping to build it. It was an impressive piece of technology but until November in 2022

…

Pioneered an AI Revolution’, The Financial Times, July 23, 2023, https://www.ft.com/content/37bb01af-ee46-4483-982f-ef3921436a50. 2 A. Vaswani et al., ‘Attention Is All You Need’, Arxiv, June 12, 2017, https://arxiv.org/abs/1706.03762. 3 M. Murgia, ‘OpenAI’s Mira Murati: The Woman Charged with Pushing Generative AI into

…

healthcare and see healthcare policing and see policing regulation of see regulation teenage pregnancy and see pregnancy, teenage term ref1 Aslam, Yaseen ref1 AstraZeneca ref1 ‘Attention Is All You Need’ (paper) ref1, ref2 automation ref1, ref2, ref3, ref4 Ayup, Abduweli ref1 Azure software ref1, ref2 Bandura, Albert ref1, ref2 Bard ref1 Bayyah, Sheikh bin ref1

Supremacy: AI, ChatGPT, and the Race That Will Change the World

by Parmy Olson  · 284pp  · 96,087 words

,” he said aloud at one point. Jones looked up from his desk, nearby. “I’m not very good with titles,” he replied. “But how about ‘Attention is all you need’?” It was a random thought that had popped into his head, and Vaswani didn’t say anything in agreement. In fact, he got up and

…

walked away, Jones recalls. But later, the title “Attention Is All You Need” landed on the front page of their paper, a perfect summary of what they’d discovered. When you used a transformer, your AI system could

…

, Shazeer left Google in 2021 to pursue his research on large language models independently, cofounding a chatbot company called Character.ai. By that time, the “Attention Is All You Need” paper had become one of the most popular research works of all time in the field of AI. Typically, a research paper on AI might

…

Understanding.” blog.research.google, August 31, 2017. Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. “Attention Is All You Need.” Advances in Neural Information Processing Systems 30 (2017). Chapter 10: Size Matters Brockman, Greg (@gdb). “Held our civil ceremony in the @OpenAI office last week

Co-Intelligence: Living and Working With AI

by Ethan Mollick  · 2 Apr 2024  · 189pp  · 58,076 words

among the many papers on different forms of AI being published by industry and academic experts, one stood out, a paper with the catchy title “Attention Is All You Need.” Published by Google researchers in 2017, this paper introduced a significant shift in the world of AI, particularly in how computers understand and process human

The Optimist: Sam Altman, OpenAI, and the Race to Invent the Future

by Keach Hagey  · 19 May 2025  · 439pp  · 125,379 words

negative without relying on humans to pre-label the data. Two months later, Sutskever read a preprint of a paper by eight Google researchers titled “Attention Is All You Need,” which he immediately recognized as presenting a method to make the kind of research Radford was doing vastly more efficient. Rather than processing one character

…

, 119 Atari, 87, 147, 165, 190 Atkins, Brian and Sabine, 142–43 Atlanta Georgi, 23 Atlantic, The (magazine), 46, 297 Atmos home building platform, 260 “Attention Is All You Need” (“the transformer paper”), 218–19, 270 Australia, 99, 266 Authors Guild, 220 Autopilot AI-assisted driving, 225 Baidu, 170 baldness, cure for, 108 Bankman-Fried

…

,” 188, 312 Goertzel, Ben, 145 Goetz, Jim, 88 Goldman Sachs, 59, 64, 150, 225 Good Ventures foundation, 212–13 Google AdSense, 243 Alphabet, 194, 271 “Attention Is All You Need” (“the transformer paper”), 218–19, 270 Bard conversational model, 271 Chrome web browser, 272 DeepMind acquisition, 146–48, 154, 165, 168–69, 171–72, 184

Empire of AI: Dreams and Nightmares in Sam Altman's OpenAI

by Karen Hao  · 19 May 2025  · 660pp  · 179,531 words

TO NOTE REFERENCE IN TEXT In August 2017, that changed: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez et al., “Attention Is All You Need,” in NIPS ’17: Proceedings of the 31st International Conference on Neural Information Processing Systems (December 2017): 6000–10, dl.acm.org/doi/10.5555/3295222

These Strange New Minds: How AI Learned to Talk and What It Means

by Christopher Summerfield  · 11 Mar 2025  · 412pp  · 122,298 words

The transformer was invented in 2017. It was first described in a preprint – a paper published online without peer review – with the slightly incongruous title ‘Attention is All You Need’.[*1] The paper didn’t make much of a splash at first. Submitted to the annual jamboree that is the Neural Information Processing Systems (NeurIPS

…

the entire input sequence in parallel – using a form of attention called self-attention to place emphasis on each item i when predicting j (hence: ‘Attention is All You Need’). To understand why self-attention is so useful in language, consider the problem of completing the following two prompts: As I approached the ancient tree

…

Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks’, arXiv. Available at https://arxiv.org/pdf/2302.08399. Vaswani, A. et al. (2017), ‘Attention Is All You Need’. Preprint. arXiv. Available at http://arxiv.org/abs/1706.03762 (accessed 30 October 2020). Verzijden, M. N. et al. (2015), ‘Male Drosophila melanogaster Learn to

We Are as Gods: A Survival Guide for the Age of Abundance

by Peter H. Diamandis and Steven Kotler  · 13 Apr 2026  · 225pp  · 76,418 words

The Means of Prediction: How AI Really Works (And Who Benefits)

by Maximilian Kasy  · 15 Jan 2025  · 209pp  · 63,332 words

Artificial Intelligence: A Modern Approach

by Stuart Russell and Peter Norvig  · 14 Jul 2019  · 2,466pp  · 668,761 words

The Singularity Is Nearer: When We Merge with AI

by Ray Kurzweil  · 25 Jun 2024

The AGI Chronicles

by Kevin Roose  · 481pp  · 151,124 words