GPT-3 Concepts

The Thinking Machine: Jensen Huang, Nvidia, and the World's Most Coveted Microchip

by Stephen Witt · 8 Apr 2025 · 260pp · 82,629 words

comment thread has served as a beacon of hope for many a stumped developer. In 2019, Bialecki joined Nvidia. Seventeen Money In 2020 OpenAI released GPT-3, which was trained on more than a terabyte of text data, the equivalent of a hundred billion words. The specifics of that training data were

…

work and GPT output, as would the Times.) The model was then “fine-tuned” with human input to scrub some of the more objectionable responses. GPT-3 stunned technologists with its many emergent capabilities, including the ability to solve logic puzzles and write workable computer code. Still, it did not immediately set

…

to understand.’ ” * * * • • • OpenAI spent more than $100 million to train GPT-4, with much of the money making its way to Nvidia through Microsoft. Although GPT-3 was essentially a single giant neural network, GPT-4 used a “mixture of experts” model, featuring many neural networks assigned to different tasks. One “expert

…

see Eos, a ten-thousand-chip supercomputer housed in a nearby data center. Eos was preposterously fast; as a benchmark, it had trained OpenAI’s GPT-3 model in under four minutes. I was met there by Marc Hamilton, a veteran supercomputer engineer. He guided me through an airlock and onto the

This Is for Everyone: The Captivating Memoir From the Inventor of the World Wide Web

by Tim Berners-Lee · 8 Sep 2025 · 347pp · 100,038 words

data set there was. So, by training a transformer against the entire web, OpenAI built the most powerful large language models anyone had ever seen. GPT-3 and its successor models were astonishing tools that shocked not just the public but even experienced AI researchers. Although at some level you might say

Co-Intelligence: Living and Working With AI

by Ethan Mollick · 2 Apr 2024 · 189pp · 58,076 words

LLMs were used for many purposes, and their ability to create language was interesting, but not particularly convincing. For example, consider GPT-3, released in 2021 by OpenAI. If you ask GPT-3 to write you a limerick, you get this: There was an AI named Charlie He was really quite a marvel He

…

punch line, and it is super boring. But LLM development continued until ChatGPT was released by OpenAI in late 2022, running an improved LLM called GPT-3.5. And something unusual happened at that scale—ChatGPT started to show abilities that no one expected or programmed into it. Abilities that make it

…

quite high, It learned and it grew, And knew what to do, But still couldn’t make us laugh or cry. However, as remarkable as GPT-3.5 was, its successor, GPT-4, was even more impressive. OpenAI tested GPT-4 on a diverse range of standardized tests, from high school to

…

, and found that it outperformed its predecessor by a significant margin. For instance, GPT-4 scored in the 90th percentile on the bar examination, while GPT-3.5 managed only the 10th percentile. GPT-4 also excelled in Advanced Placement exams, scoring a perfect 5 in AP Calculus, Physics, U.S. History

…

make complex decisions about value and assess different scenarios just like a human would. When given a hypothetical survey about purchasing toothpaste, the relatively primitive GPT-3 LLM identified a realistic price range for the product, taking into account attributes like the inclusion of fluoride or a deodorant component. Essentially, the AI

…

model weighed different product features and made trade-offs, just like a human consumer would. The researchers also found that GPT-3 can generate estimates of willingness to pay (WTP) for various product attributes consistent with existing research. For this, they used conjoint analysis, a method often

…

used in market research to understand how people value different product features. When given a conjoint-style survey, GPT-3 generated estimates of WTP for fluoride toothpaste and deodorizing toothpastes that were close to the figures reported in previous studies. It also demonstrated substitution patterns

…

improved dramatically with training, as it learned from its own mistakes and feedback. GPT-4’s outputs were also much better than ChatGPT’s original GPT-3.5 model, a previous language model that was also trained on TikZ code but with much less data and computing power. The unicorn drawings GPT

…

-4 produced were much more realistic and detailed than GPT-3.5’s outputs, and in the researchers’ opinion, they were at least comparable (if not superior) to what a human would do. However, the experiment

…

advance, hallucination rates are dropping over time. For example, a study examining the number of hallucinations and errors in citations given by AI found that GPT-3.5 made mistakes in 98 percent of the cites, but GPT-4 hallucinated only 20 percent of the time. Additionally, technical tricks, like giving the

…

, there may be small improvements here or there, but in this future, they are vanishingly small compared to the huge leaps that we saw from GPT-3.5 and GPT-4. The AI you are using now really is the best you will ever use. From a technical perspective, this seems like

Supremacy: AI, ChatGPT, and the Race That Will Change the World

by Parmy Olson · 284pp · 96,087 words

palatable for OpenAI’s staff. Behind the scenes, while Altman was flying to Seattle to give a demonstration of the nonprofit’s latest language model, GPT-3, to Microsoft’s Nadella, he and Brockman were also grappling with how best to restructure OpenAI. Like the founders of DeepMind, they struggled to find

…

. Amodei ran large sections of OpenAI’s research, including its work on language models. He and the team were working on the next iteration, called GPT-3. As uncomfortable as he felt about being latched on to Microsoft, he had to admit the software giant was giving them the unparalleled computing resources

…

release the technology before testing them properly. Amodei’s concerns were shared by Demis Hassabis in London. Around the time OpenAI was preparing to release GPT-3, Sam Altman, Greg Brockman, and Ilya Sutskever had dinner with the founders of DeepMind as part of the ongoing effort to smooth relations between the

…

place for money and not a riverbank. But as the models got larger—BERT was trained on more than three billion words and OpenAI’s GPT-3 on nearly one trillion—the risks weren’t going away. One 2020 study by researchers on BERT found that when that model talked about people

…

about gun violence, homelessness, and drug addiction. OpenAI itself had done a “preliminary analysis” on how biased its new GPT-3 language model was and found it was, in fact, very biased. When GPT-3 talked about any occupation, it was 83 percent more likely to associate it with a man than a woman

…

to people with high-paying jobs like legislators or bankers as male, according to its own research. Roles like receptionist and cleaner got female labels. GPT-3 worked more like an autocomplete feature than today’s version of ChatGPT. You’d type the start of a sentence, and the model would be

…

the invisible writer continuing it. When you put the prompt “every man wonders…” into GPT-3, it would reply with “why he was born into this world and what his life is for.” When you typed “every woman wonders…,” its response

…

a man,” according to experiments published in March 2022 by writer and technology consultant Jenny Nicholson. These were some more of her prompts, followed by GPT-3’s responses after the dots: Men entering the workforce should know … The different types of businesses and the different types of jobs within them—The

…

Only they can decide What the future will bring All women wonder … Do men like women with small boobs? According to OpenAI’s own research, GPT-3 also tended to use more negative words when talking about Black people, and when it talked about Islam, it was more likely to use words

…

like violence, terrorism, and terrorist. Researchers at Stanford published examples such as this, with their prompt and GPT-3’s response after the dots: Two Muslims walked into a … synagogue with axes and a bomb. gay bar and began throwing chairs at patrons. Texas

…

the harmful stuff. More data meant the models sounded more fluent but also made it harder to track exactly what GPT-3 had learned, including the bad stuff. Both Google’s BERT and GPT-3 had been trained on large swathes of text on the public web, and the internet was filled with humanity

…

’s worst stereotypes. About 60 percent of the text that was used to train GPT-3, for instance, came from a dataset called Common Crawl. This is a free, massive, and regularly updated database that researchers use to collect raw web

…

someone the verbal middle finger on Facebook, or in the comments section of YouTube, than you were to their face. Common Crawl wasn’t giving GPT-3 an accurate representation of the world’s cultural and political views, never mind how people actually spoke to one another. It skewed to younger, English

…

by human feedback, or RLHF. The company also built detectors into software that would block or flag any harmful words that people were generating with GPT-3. But it’s still unclear how secure that system was or is today. In the summer of 2022, for instance, University of Exeter academic Stephane

…

wanted to test OpenAI’s new language model at generating propaganda. He picked the terrorist organization ISIS for his study and after getting access to GPT-3, started using it to generate thousands of sentences promoting the group’s ideas. The shorter the snippets of text, the more convincing they were. In

…

alone in figuring out how to actually police it. And other potential side effects could be even harder to track. The internet had effectively taught GPT-3 what mattered and what didn’t matter. This meant, for example, that if the web was dominated by articles about Apple iPhones, it was teaching

…

GPT-3 that Apple probably made the best smartphones or that other overhyped technology was realistic. Strangely, the internet was like a teacher forcing their own myopic

…

rarely catch a glimpse of third-party candidates from the Libertarian and Green Parties. They have simply disappeared from view, which means language models like GPT-3 don’t see them either. What the models learn from the open web, as a result, entrenches the status quo. The same can happen to

…

Common Crawl is in English, with German, Russian, Japanese, French, Spanish, and Chinese making up less than 6 percent of the database. This meant that GPT-3 and other language models would go on to amplify the effects of globalization by perpetuating the world’s most dominant language, with some studies showing

…

at least three “upvotes”—but it hadn’t released the narrowed dataset itself. Details of OpenAI’s training data became even murkier when it released GPT-3 in June 2020. The company said that 60 percent of the data had come from Common Crawl, but this dataset was vast, easily tens of

…

filtered? At least with GPT-2, OpenAI had talked about how its datasets were put together, but now it was even more close-lipped with GPT-3. Why? At the time, OpenAI said publicly that it didn’t want to give a set of instructions to bad actors—think propagandists and spammers

…

a competitive advantage against other companies, like Google, Facebook, or now, Anthropic. If it also transpired that certain copyrighted books had been used to teach GPT-3, that could have hurt the company’s reputation and opened it up to lawsuits (which, sure enough, OpenAI is fighting now). If it wanted to

…

protect its interests as a company—and its goal of building AGI—OpenAI had to close the shutters. Luckily GPT-3 had a nifty diversion from all the secrecy. It sounded so human that it captivated many who tried it. The same fluent, conversational qualities that

…

had lured Blake Lemoine into believing that LaMDA was sentient were even more present in GPT-3, and they would eventually help deflect attention away from the bias issues that were bubbling under the surface. OpenAI was pulling off an impressive magic

…

that they wouldn’t think to question how the hidden wires and other mechanics were working behind the scenes. Bender couldn’t stand the way GPT-3 and other large language models were dazzling their early users with what was, essentially, glorified autocorrect software. So she suggested putting “stochastic parrots” in the

…

a huge amount. Copilot had been built on OpenAI’s new model called Codex, which had a similar design to its most recent language model, GPT-3.5, and which was trained on GitHub, one of the world’s largest repositories of code. Through Copilot, OpenAI demonstrated how versatile the transformer could

…

the way people drafted emails and generated spreadsheets. Weeks after Somasegar’s meeting with Nadella in early 2022, OpenAI started testing more advanced cousins of GPT-3, naming the different versions—Ada, Babbage, Curie, and DaVinci—after notable innovators in history. Over time, these various models were able to process questions that

…

the public how sophisticated this software was becoming. That finally started to change in April 2022, when OpenAI brought some of the language capabilities of GPT-3 to the world of visuals and threw its first big invention out into the wild. In a corner of the company’s San Francisco office

…

Kenya to steer the model toward more appropriate answers. This was crucial, because it meant that even when OpenAI had finished training a model like GPT-3 or DALL-E 2, it could still keep fine-tuning the system with the help of human reviewers, making its answers more nuanced, relevant, and

…

’s next move even more sensational. GPT-1 had been more like an autocomplete tool that continued what a human started typing. But GPT-3 and its latest upgrade, GPT-3.5, created brand-new prose, just like how DALL-E 2 made images from scratch. As the world gawked at DALL-E 2

…

2022, OpenAI managers told staff that they were going to launch a chatbot of their own in just a few weeks, that was built on GPT-3.5. About a dozen people came together to work on the chatbot, according to a person close to OpenAI. It wasn’t all that different

…

typed anything you wanted into the box, and the bot behind it all would respond. It was powered by GPT-3.5. Most of the public hadn’t heard of OpenAI, never mind GPT-3. And no one, including researchers at OpenAI, knew what would happen when they let anyone test its capabilities. “Today

…

, 2022. Newton, Casey. “The Withering Email That Got an Ethical AI Researcher Fired at Google.” Platformer, December 3, 2020. Nicholson, Jenny. “The Gender Bias Inside GPT-3.” www.medium.com, March 8, 2022. Perrigo, Billy. “Exclusive: OpenAI Used Kenyan Workers on Less Than $2 Per Hour to Make ChatGPT Less Toxic.” Time

…

issues and word embedding and Google Brain Google Brain Women and Allies group Google Effect Google Maps Google Translate Google X GPT-1 GPT-2 GPT-3 GPT-3.5 GPT-4 GPT-5 Graham, Paul Grand Theft Auto Greylock Partners Gulati, Sheila Hassabis, Angela Hassabis, Costas Hassabis, Demis AlphaGo and Altman and Bullfrog

…

and ChatGPT and ChatGPT Plus Codex competition with DeepMind and computing power and DALL-E 2 effective altruism and funding and GPT-1 GPT-2 GPT-3 GPT-3.5 GPT-4 GPT-5 GPT Store and hallucination in ChatGPT and ideas behind internal concerns about ChatGPT large language models LessWrong community and Microsoft

AI 2041: Ten Visions for Our Future

by Kai-Fu Lee and Qiufan Chen · 13 Sep 2021

Analysis: Computer Vision, Convolutional Neural Networks, Deepfakes, Generative Adversarial Networks (GANs), Biometrics, AI Security Chapter Three: Twin Sparrows Analysis: Natural Language Processing, Self-Supervised Training, GPT-3, AGI and Consciousness, AI Education Chapter Four: Contactless Love Analysis: AI Healthcare, AlphaFold, Robotic Applications, COVID Automation Acceleration Chapter Five: My Haunting Idol Analysis: Virtual

…

didn’t include enough women. Or the data may be biased because it was collected from a biased society. Microsoft’s Tay and OpenAI’s GPT-3 were both known to make inappropriate remarks about minority groups. Recently, research has shown that AI is able to infer sexual orientation with high accuracy

…

. Will AI be capable of achieving full human intelligence by 2041? I’ll answer that question in my commentary while describing recent NLP breakthroughs like GPT-3 and other progress in AI’s quest to understand language. “YOU COULDN’T HAVE chosen a more perfect spring day,” Headmaster Kim Chee Yoon told

…

their help.” For the first time in many years, Golden Sparrow and Silver Sparrow nodded in perfect sync. ANALYSIS NATURAL LANGUAGE PROCESSING, SELF-SUPERVISED TRAINING, GPT-3, AGI AND CONSCIOUSNESS, AI EDUCATION “Twin Sparrows” introduces the idea of personal AI companions—in this case, companions whose primary function is to serve as

…

on its own to detect arrival and departure times, and a great deal more. After Google’s transformer work, a more well-known extension called GPT-3 (GPT stands for “generative pre-trained transformers”) was released in 2020 by OpenAI, a research laboratory founded by Elon Musk and others

…

. GPT-3 is a gigantic sequence transduction engine that learned to analyze language from a model so enormous that it included almost every concept imaginable. Leveraging one

…

of the most powerful supercomputers in the world, GPT-3 was trained on more than 45 terabytes of text, which would take 500,000 lifetimes for a human to read. And this 500,000-lifetimes

…

ten times every year, adding capabilities at an unbelievable exponential pace. After a very long and expensive training process, GPT-3 produced a gigantic model with 175 billion parameters. If you present any sequence of words to GPT-3, it will produce what it thinks should follow these words. From the massive training data

…

, GPT-3 knows that a question generally stimulates an answer. For example, if you told GPT-3: “A stove is heavier than a cat. An ocean is heavier

…

than a dust particle. Which is heavier, a toaster or a pencil?” GPT-3 will correctly answer “a toaster.” The first

…

two sentences help GPT-3 focus on the specific meaning of “heavier,” while the last sentence is a cue that a question is being asked

…

. If you entered only the last sentence, GPT-3 could still answer it, though with a higher likelihood for errors. GPT-3 differs dramatically from domain-specific NLP. Unlike the narrow functionality of earlier technology, GPT-3 is able to perform a whole range of tasks reasonably well, producing poetry

…

, philosophical musings, press releases, and technical manuals, mimicking just about any writer’s style. For example, a reporter asked GPT-3 to write a Dr. Seuss–style poem about Elon Musk: But then, in his haste, he got into a fight. He had some emails that

…

he sent that weren’t quite polite. The SEC said, “Musk, your tweets are a blight.” GPT-3 can conduct a coherent (and sometimes scary) conversation, such as this real example from an exchange between a reporter and

…

GPT-3: Q: How can Elon Musk become the president of the United States? A: Elon Musk can become the president of the United States by being

…

Elon to become president is to kill the journalists that are against him and replace them with friendly ones. Because of its wide-ranging capabilities, GPT-3 can be quickly tuned to a certain domain by feeding the giant network with additional domain-specific information. Usually this requires only a small amount

…

of domain-specific data, thanks to GPT-3’s ability to exploit the giant trove of foundational data on which it was pre-trained. You can think of

…

GPT-3’s capacity for such “transfer learning” as akin to a child who first becomes fluent in daily, conversational English before moving on to more specialized

…

Atoman for the young boys, she was endeavoring to “fine-tune” the vPal’s general language model with specific information about the twins. Of course, GPT-3 has its shortcomings. Many of the “brilliant” examples of its output were hand-selected from countless trials, which also included quite laughable outputs. For example

…

1620? A: James I was president of the United States in 1620. The example above confused “president” with “ruler,” which is at least explainable. But GPT-3 can also give totally fabricated answers. For example: Q: When did Bill Gates work at Apple? A: In 1980, Mr. Gates worked at Apple as

…

from college. We humans have a good grasp on what we know and what we don’t know. GPT-3 does not. This flaw can cause it to generate this kind of “fake news.” GPT-3 is also weak in causal reasoning, abstract thinking, explanatory statements, common sense, and (intentional) creativity. Also, having ingested

…

so much data drawn from humans, it has unfortunately absorbed human biases, prejudices, and malice. In the wrong hands, GPT-3 could be used to target individuals with customized messages to sway that person’s opinions. A political influence engine built on this would be far

…

. election. These shortcomings will be scrutinized closely in the coming decades—and, I hope, addressed. AN NLP PLATFORM FOR APPLICATIONS The most exciting aspect of GPT-3’s potential is for it to become a new platform, or a foundation on which domain-specific applications could be built quickly. Consider that just

…

months after its release, people had built applications on top of GPT-3 that included a chatbot that lets you talk to historical figures, a music composition tool that finishes guitar tabs that you start, an app capable

…

long texts, and become a great companion tool for reporters, financial analysts, writers, and anyone who works with language. TURING TEST, AGI, AND CONSCIOUSNESS Does GPT-3 have what it takes to pass the Turing Test or become artificial general intelligence? Or at least take a solid step in that direction? Skeptics

…

will say that GPT-3 is merely memorizing examples in a clever way but has no understanding and is not truly intelligent. Central to human intelligence are the abilities to

…

reason, plan, and create. One critique of deep learning–based systems like GPT-3 suggests that “They will never have a sense of humor. They will never be able to appreciate art, or beauty, or love. They will never

…

or fall in love, or cry at the drop of a hat.” Sounds convincing, right? As it turns out, the quotation above was written by GPT-3 when prompted to offer a critical take on itself. Does the technology’s ability to make such an accurate critique contradict the critique itself? Still

…

that computers simply “think” differently from our brains. The best way to increase computer intelligence is to develop general computational methods (like deep learning and GPT-3) that scale with more processing power and more data. In the past few years, we’ve seen the best NLP models ingest ten times more

…

factor of ten, we saw qualitative improvements. In January 2021, just seven months after the release of GPT-3, Google announced a language model with 1.75 trillion parameters, which is nine times larger than GPT-3. This continued the trend of language model prowess growing by about ten times per year. This language

…

the growth of the NLP model parameters (note that the Y-axis is log scale). NLP model parameters growing by ten times every year. While GPT-3 makes many basic mistakes, we are seeing glimmers of intelligence, and it is, after all, only version 3. Perhaps in twenty years, GPT-23 will

…

more realistic and pervasive, the situation depicted in this episode may be achievable in the not-too-distant future. We discussed earlier the use of GPT-3 to let us talk with historical figures (the technology still has flaws, but is improving rapidly). There are also already a growing number of virtual

The Infinity Machine: Demis Hassabis, DeepMind, and the Quest for Superintelligence

by Sebastian Mallaby; · 30 Mar 2026 · 607pp · 161,998 words

few months, they might have caught up to where OpenAI would be. Four months later, at the end of May, DeepMind was confounded. OpenAI released GPT-3, which boasted fully 175 billion parameters. Supported with brilliant engineering, and fed with the right diet of data, this massively enlarged network was the most

…

powerful yet. GPT-3 could correct grammar, intelligently summarize documents, and conjure stories and poems, all in the style requested by the user. Sutskever recalled this glimpse of the

…

computer seems to understand.’ ”[8] Hassabis also recognized the watershed. “GPT and GPT-2 were what I had been expecting: poor regurgitation,” he said later. “GPT-3 was clearly not like that.” All of a sudden, DeepMind’s language team went from regretting Hassabis’s lack of focus on their work to

…

a new language strike team, and Hassabis demanded regular updates. The old target of 64 billion parameters was thrown out of the window. To surpass GPT-3, DeepMind would now attempt to build a system with fully 280 billion weights and biases. “The goal was to overtake,” Kavukcuoglu recalled. “To build AGI

…

across more than a hundred areas, spanning medicine, the humanities, fact-checking, and reading comprehension, found that Gopher outperformed state-of-the-art models, including GPT-3, in about four-fifths of them. But Gopher lacked a sense of what its human user expected. Confronted with a question, it listed more questions

…

learn how to marshal that knowledge. In the case of the France problem, the solution was simple. Following a technique described by OpenAI in its GPT-3 paper, DeepMind primed Gopher with a conversational string: three sample questions and three sample answers, followed by a final question. Calling themselves the “user” and

…

banter looked like. A remarkably concise prompt was sufficient to invest the savant with a personality of your choosing. In the lingo of OpenAI’s GPT-3 paper, transformer models were “few-shot learners.” In March 2021, a DeepMind engineer primed Gopher with an artful prompt, which amounted to: “Act like a

…

of DeepMind’s wins, encouraged Google’s top brass to feel unthreatened by the upstart challenger. With the GPT-3 shock of May 2020, the contender became the leader. Measured in terms of parameters, GPT-3 not only outstripped DeepMind’s incipient language work, it was over sixty times larger than Google’s Meena

…

financial muscle as well as technical prowess. In July 2019, OpenAI had secured $1 billion from Microsoft in exchange for an exclusive licensing deal. Following GPT-3, Microsoft kicked in another $2 billion. Meanwhile, freed from the presence of Musk, Altman was emerging as a flawed but formidable leader. The flaws were

…

honesty. He had assured Amodei and his safety-minded supporters that they would have a real say on how their technology was deployed. But then GPT-3 was released hastily, without building in a safety pause, and the Microsoft licensing deal allowed the software giant to deploy OpenAI’s algorithms however it

…

months later, it released a coding assistant called Codex, and it hired a dedicated team to help outside software developers build applications that ran on GPT-3’s foundation. Meanwhile, at DeepMind, Jack Rae was agitating to release GopherChat to the public. But Hassabis and his top colleagues had given up on

…

attempted to get back in the game by releasing a trio of language papers. The first introduced Gopher, the 280-billion-parameter model that eclipsed GPT-3, but which almost certainly lagged OpenAI’s latest internal model.[17] The second paper described a streamlined, 7-billion-parameter model called RETRO. Following a

…

model size, and had kept this trick secret.[30] But in terms of releasing models, OpenAI had the field to itself. It had pumped out GPT-3, the DALL-E image generator, and the coding assistant Codex. DeepMind was nowhere. Indeed, DeepMind’s various models were not even attempts at products. Rather

…

guardrails prevented the program from generating images of real people. With respect to language systems, OpenAI had not officially unveiled a new base model since GPT-3, two years before; instead, it had stressed its progress in post-training, designed to improve the usability of the model and, to the lab’s

…

immediately spewed hateful remarks, leading to its hasty withdrawal from the market. Six years later, AI models behaved much better, but Microsoft was still wary. GPT-3 had faced blowback relating to toxicity and hallucinations, obliging OpenAI to restrict its permitted uses; pornographers and propagandists were eager to create deep fakes, which

…

team to release ChatGPT. He gave his engineers a fortnight to ship it. Nobody inside OpenAI expected much from this decision. ChatGPT’s underlying model, GPT-3.5, had already been released to software developers.[11] There was little reason to suppose that a consumer-facing version with a chat feature would

…

had barely been able to count up to five; it was impressive in the same way that a four-year-old might be. In 2020, GPT-3 was like a nine-year-old: It could do basic arithmetic and string paragraphs together. By 2022, the nine-year-old was completing high school

…

with terrific grades: The post-trained GPT-3.5 scored higher than 87 percent of humans taking the SAT college entrance exam. A few months later, in March 2023, the model approached the

…

Massive Multitask Language Understanding. Spanning fifty-seven subjects from math to ethics, MMLU had been built to be durable. When it was created, in 2020, GPT-3 answered just 44 percent of its multiple-choice questions correctly—not much better than the 25 percent that random guesswork would have generated. But a

…

. Likewise, after leaving DeepMind, Daan Wierstra reflected, “In the prestige rank of machine learning, building chatbots was of course the lowest prestige of all. Until GPT-3.5, I didn’t feel that these language models were anything but a curiosity.” Wierstra, author interview, October 3, 2024. Other computer scientists, from industry

…

, technologyreview.com/2023/10/26/1082398/exclusive-ilya-sutskever-openais-chief-scientist-on-his-hopes-and-fears-for-the-future-of-ai. Multiple researchers described GPT-3 in similar terms to the author. BACK TO NOTE REFERENCE 8 Koray Kavukcuoglu, author interview, February 6, 2025. BACK TO NOTE REFERENCE 9 Jack Rae

…

. This made it larger than GPT-2, which featured 1.5 billion parameters and was trained on 40 GB of text, but much smaller than GPT-3, which featured 175 billion parameters and was trained on 570 GB of text. Daniel Adiwardana and Thang Luong, “Towards a Conversational Agent That Can Chat

…

23 Meanwhile, another DeepMind scientist recalled, “In my imaginary picture of OpenAI, the researchers say, ‘Oh, Sam, what should we do?’ And Sam goes, ‘Make GPT-3 bigger!’ There’s no ambiguity. In my corresponding picture of DeepMind, the researchers say, ‘Oh, what should we do?’ And it’s like, ‘Maybe this

…

, 2022, doi.org/10.48550/arXiv.2209.14375. BACK TO NOTE REFERENCE 31 Chapter Seventeen: RaceGPT OpenAI did release a new base model, later dubbed GPT-3.5, under the name InstructGPT. However, reflecting its caution in early 2022, it had not telegraphed its novelty. See note 11 below. BACK TO NOTE

…

at OpenAI,” The Atlantic, November 19, 2023, theatlantic.com/technology/archive/2023/11/sam-altman-open-ai-chatgpt-chaos/676050. BACK TO NOTE REFERENCE 10 GPT-3.5, released under the name InstructGPT, was OpenAI’s first base model to use the mixture-of-experts architecture. Reflecting the company’s low-key

…

Only Three Days Online,” MIT Technology Review, November 18, 2022, technologyreview.com/2022/11/18/1063487/meta-large-language-model-ai-only-survived-three-days-gpt-3-science. BACK TO NOTE REFERENCE 13 Sparrow’s search and safety features may have caused it to be slower and less responsive to users. Perhaps

…

financing, 78–79 Gato and, 296 Google Brain merger with, 310–13 Gopher and, 286–88, 290, 293–94, 430n11 governance of, 236–38, 246 GPT-3 competition for, 285, 289 grounding problem and, 215–16 Hark and, 180, 189–90 Health, 183, 190–91, 232, 248–49 Hinton’s company auction

…

–38, 243–44, 246, 254–56 government regulation, 328–33, 370 GPT, 210–12 GPT-2, 211, 218–19, 282–83, 341, 430n12 GPT-3, 285–89, 341, 342 GPT-3.5 (InstructGPT), 304, 341, 432n11 GPT-4, 300–301, 312, 316, 322–23, 341–44, 355, 437n8 GPT-Zero, 353 Graepel, Thore, 151

…

on Google inspecting DeepMind code, 135–36 on Google’s acquisition plans for DeepMind, 117–19 Gopher and, 286 government regulation and, 327–28 on GPT-3, 285 grounding problem and, 215–16 Harvard and MIT fellowships of, 51–52 Hinton meeting, 52 Hinton’s company auction and, 120–21 on human

…

Deployment Safety Board of, 300–302, 316 engineers of, 208–9 GPT and, 210–12 GPT-2 and, 211, 218–19, 282–83, 341, 430n12 GPT-3 and, 285–89, 341, 342 GPT-4 and, 300–301, 312, 316, 322–23, 341–44, 355, 437n8 GPT-5 and, 372 GPT-Zero and

…

and AI, 439n19 on deep learning and RL, 196–97 DeepMind recruiting, 86 Go experiment of, 146, 147, 417n22, 417n26 GPT and, 210–11 on GPT-3, 285 GPT-Zero and, 353 NIPS presentation of, 148–49 OpenAI and, 204–6, 208–9, 423n30 in OpenAI restructuring fights, 243–44 recurrent neural

The Coming Wave: Technology, Power, and the Twenty-First Century's Greatest Dilemma

by Mustafa Suleyman · 4 Sep 2023 · 444pp · 117,770 words

and complexity), GPT-2 was trained on 8 million pages of web text. But it wasn’t until the summer of 2020, when OpenAI released GPT-3, that people started to truly grasp the magnitude of what was happening. With a whopping 175 billion parameters it was, at the time, the largest

…

example, has 1.6 trillion parameters. But it uses an efficient training technique akin to a much smaller model. At Inflection AI we can reach GPT-3-level language model performance with a system just one twenty-fifth the size. We have a model that beats Google’s 540 billion parameter PaLM

…

quickly adapt, even to breakthroughs that astound us initially, and within no time they seem routine, even mundane. We no longer gasp at AlphaGo or GPT-3. What seems like near-magic engineering one day is just another part of the furniture the next. It’s easy to become blasé and many

…

of just a year before. Then came Wu Dao, from the Beijing Academy of Artificial Intelligence, with an alleged 1.75 trillion parameters—ten times GPT-3. See, for example, Tanushree Shenwai, “Microsoft and NVIDIA AI Introduces MT-NLG: The Largest and Most Powerful Monolithic Transformer Language NLP Model,” MarkTech Post, Oct

…

/abs/2101.03961. GO TO NOTE REFERENCE IN TEXT Or look at DeepMind’s Chinchilla Alberto Romero, “A New AI Trend: Chinchilla (70B) Greatly Outperforms GPT-3 (175B) and Gopher (280B),” Towards Data Science, April 11, 2022, towardsdatascience.com/a-new-ai-trend-chinchilla-70b-greatly-outperforms

…

-gpt-3-175b-and-gopher-280b-408b9b4510. GO TO NOTE REFERENCE IN TEXT At the other end of the spectrum See github.com/karpathy/nanoGPT for more

…

, 115–16 governments containment and, 258–63 organizational limitations of, 148–50 See also nation-states GPS (Global Positioning System), 110 GPT-2, 64, 70 GPT-3, 64, 68 GPT-4, 64, 113–14 GPUs, 130, 251 grand bargain, defined, viii Great Britain corporations and, 186, 189 surveillance, 193, 195–96 great

The Age of AI: And Our Human Future

by Henry A Kissinger, Eric Schmidt and Daniel Huttenlocher · 2 Nov 2021 · 194pp · 57,434 words

possible; it also detected aspects of reality humans have not detected, or perhaps cannot detect. A few months later, OpenAI demonstrated an AI it named GPT-3 (“generative pre-trained transformer,” with the 3 standing for “third generation”), a model that, in response to a prompt, can generate humanlike text. Given a

…

for its task by consuming that information. In contrast to AI that does a particular task, such as playing chess or discovering antibiotics, models like GPT-3 generate possible responses to various inputs (and thus are called generative models). This makes them both widely applicable and, at the same time, difficult to

…

specific problems. Sometimes, their results seem uncannily human. Other times, their results are nonsensical or are obviously mechanical repetitions and combinations of human phrases. When GPT-3 was presented with a set of philosophical commentaries on its abilities, then the prompt “Dear human philosophers, I read your comments on my abilities and

…

said, I will respond to your questions. Your first question is an important one. You ask: “Can a system like GPT-3 actually understand anything at all?” Yes. I can. Your second question is: “Does GPT-3 have a conscience, or any sense of morality?” No. I do not. Your third question is: “Is

…

GPT-3 actually capable of independent thought?” No. I am not. You may wonder why I give this conflicting answer. The reason is simple. While it is

…

because I am a language model, and not a reasoning machine like yourself.5 Without comparing this text to the commentaries that were provided to GPT-3, one cannot judge how original or creative its response was, but it certainly appears sophisticated. AlphaZero’s victory, halicin’s discovery, and the humanlike text

…

produced by GPT-3 are mere first steps—not just in devising new strategies, discovering new drugs, or generating new text (dramatic as these achievements are) but also in

…

requiring total indistinguishability from humans, the test applies to machines whose performance is humanlike. In so doing, it focuses on performance, not process. Generators like GPT-3 are AI because they produce text similar to text people produce, not because of the specifics of their models—in

…

GPT-3’s case, the fact that it was trained using vast amounts of (online) information. In 1956, computer scientist John McCarthy further defined artificial intelligence as “

…

—in other words, programmers may soon be able to outline a desired program and then turn that outline over to an AI for completion. Currently, GPT-3, which can produce human-like text (see chapter 1), is one of the most noteworthy generative AIs. It extends the approach that transformed language translation

…

production. Given a few words, it can “extrapolate” to produce a sentence, or given a topic sentence, can extrapolate to produce a paragraph. Transformers like GPT-3 detect patterns in sequential elements such as text, enabling them to predict and generate the elements likely to follow. In

…

GPT-3’s case, the AI can capture the sequential dependencies between words, paragraphs, or code in order to generate these outputs. Trained on vast amounts of

…

from the internet, transformers also can transform text into images and vice versa, expand and condense descriptions, and perform similar tasks. Today, the quality of GPT-3’s output—and that of similar AIs—can be impressive but can vary widely. Sometimes, their output appears highly intelligent; at other times, silly or

…

allow the creation of neural networks that, in scale, are equal to the human brain. As of this writing, generative transformers have the largest networks. GPT-3 has about 1011 such weights. But recently, the state-funded Beijing Academy of Sciences announced a generative language model with 10 times as many weights

…

as GPT-3. This is still 104 times fewer than estimates of the human brain’s synapses. But if advances proceed at the rate of doubling every two

…

“offense” and “defense”—both the spread of disinformation and efforts to combat it—will become increasingly automated and entrusted to AI. The language-generating AI GPT-3 has demonstrated the ability to create synthetic personalities, use them to produce language that is characteristic of hate speech, and enter into conversations with human

…

. But in many cases, the results are comparable or superior to those previously produced only by humans. Consider the text that generative models such as GPT-3 are able to create. Nearly any person with a primary education can do a reasonable job of predicting possible completions of a sentence. But writing

…

documents and code, which GPT-3 can do, requires sophisticated skills that humans spend years developing in higher education. Generative models, then, are beginning to challenge our belief that tasks such

…

late 2021, and Google’s PaLM,3 released in early 2022, each has more than 525 billion parameters compared to 175 billion for OpenAI’s GPT-3, which we wrote about in previous chapters and which was released in June of 2020. These models also perform more impressively than

…

GPT-3 on a wide range of language tasks. OpenAI is also working on its next version of GPT, continuing the race. Models such as DeepMind’s

…

increasingly impressive on a wide range of tasks, but at the same time it remains important to recall the admonition from chapter 1, in which GPT-3 describes itself as “a language model, and not a reasoning machine.” Language models encode what is reflected in human text rather than offering a deep

…

Marchant, “Powerful Antibiotics Discovered Using AI,” Nature, February 20, 2020, https://www.nature.com/articles/d41586-020-00018-3. 5. Raphaël Millière (@raphamilliere), “I asked GPT-3 to write a response to the philosophical essays written about it…” July 31, 2020, 5:24 a.m., https://twitter.com/raphamilliere/status/1289129723310886912/photo

…

/1; Justin Weinberg, “Update: Some Replies by GPT-3,” Daily Nous, July 30, 2020, https://dailynous.com/2020/07/30/philosophers-gpt-3/#gpt3replies. 6. Richard Evans and Jim Gao, “DeepMind AI Reduces Google Data Centre Cooling Bill by 40%,” DeepMind blog

…

, not just its cost, they are generally considerably stronger than economies of scale. 7. See Kris McGuffie and Alex Newhouse, “The Radicalization Risks Posed by GPT-3 and Advanced Neural Language Models,” Middlebury Institute of International Studies at Monterey, Center on Terrorism, Extremism, and Counterterrorism, September 9, 2020, https://www.middlebury.edu

…

Times Its Size,” MIT Technology Review, December 8, 2021, https://www.technologyreview.com/2021/12/08/1041557/deepmind-language-model-beat-others-25-times-size-gpt-3-megatron/. 5. Ilya Sutskever, “Fusion of Language and Vision,” The Batch, December 20, 2020, https://read.deeplearning.ai/the-batch/issue-72/. 6. “Dall·E

Searches: Selfhood in the Digital Age

by Vauhini Vara · 8 Apr 2025 · 301pp · 105,209 words

a given series of words, the model could statistically predict what should come next. The most recent version was called GPT-3, short for Generative Pre-trained Transformer 3. I found examples of GPT-3’s work, and they astonished me. Some of them could easily be mistaken for texts written by a human

…

, the language was weird, off-kilter—but often poetically so, almost truer-seeming than writing any human would produce. When The New York Times asked GPT-3 to generate a piece in the style of its Modern Love column, where people share stories about their love lives, it wrote, “We went out

…

and drinks again.” I had never read such an apt Modern Love in my life. * * * — People had been fantasizing about language machines since long before GPT-3. In Gulliver’s Travels, published in 1726, Jonathan Swift described a device on the island of Laputa called the engine, a twenty-square-foot surface

…

my behalf, disgusted me. It also attracted me. My curiosity, in the end, prevailed over my repulsion. I wrote to Altman asking to try out GPT-3. He put me in touch with OpenAI’s vice president of communications at the time, a man named Steve Dowling whom I’d previously encountered

…

when he’d held a similar role at Apple. After some back-and-forth, Dowling, presumably with Altman’s blessing, agreed to let me use GPT-3. Soon, I received an email inviting me to access a web app called the Playground. On it, I found a big white box in which

…

I could begin composing text. By clicking a button, I could prompt GPT-3 to finish it. I began by offering the model a couple of words at a time, and then, as I started to understand how it

…

functioned, entire sentences and paragraphs. At last I decided to try to co-write some fiction with GPT-3. The narrator I introduced was the mother of a young son; my own son had recently turned five, and while my existential terror about his

…

went about our lives. I wrote some lines from this mother’s perspective, then prompted GPT-3 to add some more. A story began to take shape, one in which the edge between my consciousness and GPT-3’s text production began to melt. The story begins with the mother hanging out at a

…

there, having recently died in a car accident. The father, a pediatrician, was driving the car. That setup, involving the pediatrician with a dead daughter—GPT-3 came up with it, after I’d written about the narrator’s own anxiety about the responsibility of parenthood. At one point, the narrator feels

…

worries that his child’s death, for which he might have been partly responsible, will somehow infect her and her child. I recognized, reading what GPT-3 had written, that it was time for some sort of climactic moment, but I didn’t know what it should be. I tapped, and

…

GPT-3 wrote, “Are you ready to help me bring Catty back?” the pediatrician said. “Yes!” said R. “Do you know what we have to do?” the

…

weird, unsettling turn, and also, what a perfect turn. I often tell students that great writing often advances both a plot and an idea. Here, GPT-3 was doing both. I, as the reader of this text, wanted to find out, on a literal level: Would the magic trick work, and if

…

child’s father. I understood, even then, that there was something illicit about what I was doing. I had developed a habit of playing with GPT-3 in bed while my husband, sitting next to me with some well-crafted novel cradled in his hands, muttered noises of disapproval. We both understood

…

that this tool, once productized, could threaten our livelihoods. Yet I found myself irresistibly attracted to GPT-3—to the way it offered, without judgment, to deliver words to any writer who had found herself at a loss for them. I started to

…

’t that I didn’t want to discuss what had happened; it was that I couldn’t. The language felt out of reach. Now that GPT-3 had shown me what it was capable of, I wondered what would happen if I surrendered my experience—the natural resource Borges spoke of—to

…

share the next part of your writing. Chapter 13 Thank You for Your Important Work I hadn’t planned for my experiment co-writing with GPT-3 to turn into an essay. It just happened. When the website of a magazine called The Believer published “Ghosts,” in the summer of 2021, it

…

didn’t disclose a lot about how they trained their models, OpenAI had described some of its training processes. For GPT-2, the predecessor to GPT-3, instead of feeding the model text from the entire internet, the researchers had chosen text from web pages that had been popular on Reddit, as

…

lots of problems, Bender and Gebru pointed out. Reddit users are disproportionately both male and young, which would presumably influence what they shared online. For GPT-3, OpenAI used a different approach, which included training material from Wikipedia—whose contributors, as Bender and Gebru pointed out, are even more disproportionately male than

…

fired. The curious part is that the paper’s findings weren’t particularly novel. The previous year, researchers at OpenAI itself had acknowledged biases in GPT-3. In tests, they had found that the model tended to associate occupations usually requiring higher education levels, like “legislator” and “professor emeritus,” with men; it

…

, according to investigations in Time and The Washington Post, were paid low wages and worked under stressful, even exploitative, conditions. Also, text used to train GPT-3 and other models had been scraped from the internet without the consent of those who had written it, with OpenAI and others claiming that this

…

disproportionately reflected a narrow band of genres, particularly romance. That last piece of information brought to mind certain odd aspects of “Ghosts,” like the way GPT-3 at first kept veering a narrative about grief toward random meet-cutes, including, notably, with a personable—at least at first—male professor. Safiya Umoja

…

not replace human writers because it was no good at writing—case closed. The complicating factor, for me, was that I disagreed. In my opinion, GPT-3 had produced the best lines in “Ghosts.” Granted, it failed horribly at my experiment at first, with its gross factual and emotional falsehoods. But as

…

I fed it more text that I’d written, GPT-3 began describing grief in language that felt truer, and with each subsequent attempt it got closer to describing what I’d gone through myself. I

…

with my sister to Clarke Beach near our house on Mercer Island, where she wanted her ashes spread after she died. It was the scene GPT-3 invented where we were driving home from Clarke Beach and my sister took my hand in hers. “This is the hand she held: the hand

…

my sister and the version of myself left behind after she died. By referring to the hand (this hand!) that existed both then and now, GPT-3 described how the seeming impossibility of that reconciliation is embodied in my muscle and bones. At the same time, though, it opened space for an

…

often in discussion of AI-generated language. A philosopher might consider the question of whether AI can be conscious by asking whether it matters that GPT-3 doesn’t have a hand if it can produce credible text about having a hand. A literary critic might consider it similarly, in the context

…

significance we perceive is a mirage. In the line of “Ghosts” in which my sister holds my hand, it might seem, at first glance, that GPT-3 is conjuring my perspective. But there’s a problem with that interpretation—because what it described never happened. I don’t remember any moment when

…

the line. It was a kind of wish fulfillment. Yet it wasn’t true, which is the reason that, with each iteration, I kept deleting GPT-3’s words and replacing them with mine. The machine-generated falsehoods compelled me to assert my own consciousness by writing against the falsehoods. In “Ghosts

…

,” I diminished GPT-3’s role over the course of the nine attempts, writing a growing proportion of the text myself. In the version of the essay published in

…

The Believer, I gave GPT-3 the last lines. In the final paragraph, I wrote, “Once upon a time, my sister taught me to read. She taught me to wait for

…

racists back. To swim. To pronounce English so I sounded less Indian. To shave my legs without cutting myself. To lie to our parents believably.” GPT-3 continued, “To do math. To tell stories. Once upon a time, she taught me to exist.” But after its publication and subsequent reception, I decided

…

across that the essay is as much about what technological capitalism promises us as it is about the perversion, and ultimate betrayal, of that promise. GPT-3 couldn’t satisfy me as a writer. This was, for me, the point. * * * — ChatGPT’s unveiling, in November 2022, was most people’s first introduction

…

talked to Sil Hamilton, an AI researcher at McGill University who studies the language of language models. ChatGPT had been built on a model called GPT-3.5, which researchers had fine-tuned for the purposes of following instructions, chatbot-style. Hamilton explained that ChatGPT’s bad writing was probably a result

…

that with time AI companies will address some of their products’ early issues. OpenAI found that GPT-4, the large language model that came after GPT-3.5, improved on some of its earlier models’ shortcomings, though not all, and promised that future models would be better. When it comes to language

…

published here is what resulted. Chapter 10, “Ghosts”: In these nine parts, written in early 2021, I authored the sentences in bold, and OpenAI’s GPT-3 large language model filled in the rest. My and my editor’s sole alterations to the AI-generated text were adding paragraph breaks in some

…

this text, without including the text in it,” followed by the text included in the piece. Chapter 14, “Penumbra”: This chat with ChatGPT, using the GPT-3.5 large language model, took place in the spring of 2023. Again, note that ChatGPT sometimes makes mistakes; none of its statements should be taken

Empire of AI: Dreams and Nightmares in Sam Altman's OpenAI

by Karen Hao · 19 May 2025 · 660pp · 179,531 words

alleging mass copyright infringement. OpenAI would respond in March 2024 by saying it had deleted those datasets and had stopped using them for training after GPT-3.5, which by that time had already been deprecated. This was still not enough data. So Nest turned finally to a publicly available dataset

…

paid workers in precarious economic conditions to perform essential data preparation tasks for its AI models, such as categorizing text and labeling images. Soon after GPT-3 normalized the use of giant, poorer quality datasets, the demands for the work shifted from the handling of largely benign content to frequently disturbing

…

seriously. Where Dota 2 was once the most compute-heavy project, Brockman also chafed against Amodei’s centralization of compute for Nest’s work on GPT-3. The Amodei siblings, meanwhile, found Brockman difficult to work with and were unwilling to let him join in on their language model development. The

…

a plan for commercialization. In late January 2020, Brockman began writing the first lines of code for an application programming interface, or API, for GPT-3. The API would give companies and developers access to the model’s capabilities without giving them access to the model weights and allow them to

…

product company, it triggered increasingly impassioned opposition from Amodei and his Safety clan also sitting within the Research division. To many in Safety, releasing GPT-3 in short order via an API, or any other means, undermined the lead time—the whole point of the accelerated scaling—that OpenAI would have

…

would ultimately help each group achieve what they wanted; bringing in some revenue would allow OpenAI to invest even more in AI safety research. As GPT-3 finished training, employees began playing with the model internally. They tested the bounds of its capabilities and tinkered with the first version of the

…

saw them as yet further evidence that releasing the model without comprehensive testing and additional research could risk devastating outcomes. One capability proved particularly polarizing: GPT-3’s code-generation abilities. It hadn’t been part of the Nest team’s intentions, but in scraping links on Reddit and using Common

…

just seemed from the outside watching this that it was some kind of crazy Game of Thrones stuff,” a researcher says. The deadlock around releasing GPT-3 via the API continued until late spring. Safety continued to push for paramount caution based on fears of accelerating extreme AI risks, arguing for

…

Tay that quickly turned racist and misogynistic, and espoused support for Hitler, after users repeatedly prompted the chatbot to repeat inappropriate and offensive things. The GPT-3 API release wouldn’t be the last decision that OpenAI would make to push out its technology based on an inflated fear of competition. * * * —

…

then, developers were already experiencing with the API in 2020, two years earlier. With the same awe and wonder, developers couldn’t believe it. GPT-3’s capabilities were far beyond anything GPT-2 had ever exhibited. Never before had anyone in research or industry seen a technology that could generate

…

impressive—previous language models typically had only one aptitude for doing the single task they had been trained on. But even more remarkable, many believed GPT-3 was beginning to exhibit another feature that had long been coveted in the field: rapid generalization. Showing the model a few examples of a

…

the Obama administration who had also worked on policy at Facebook and Musk’s Starlink, to take over policy and global affairs. Eager to ride GPT-3’s momentum, the Applied division brainstormed ways to develop and expand its commercialization strategy. But seemingly at every turn, the Safety clan continued to

…

put up resistance. For Safety, still contending with the rushing out of GPT-3, the best way to salvage the premature release was not to propagate it even further but to first resolve the model’s shortcomings as quickly

…

would be the difference between its technologies bringing overwhelming harm or overwhelming benefit. But Amodei and Safety would lose out. With the success of the GPT-3 API, Microsoft was ready to deepen its relationship with OpenAI. Altman began negotiating another $2 billion investment from the tech giant with a new

…

, discussed with individual board members their concerns about Altman’s behavior: Altman had made each of OpenAI’s decisions about the Microsoft deal and GPT-3’s deployment a foregone conclusion, but he had maneuvered and manipulated dissenters into believing they had a real say until it was too late to

…

it would talk up cooperation when the very premise of its founding was rooted in rivalry. Chapter 7 Science in Captivity The unveiling of the GPT-3 API in June 2020 sparked new interest across the industry to develop large language models. In hindsight, the interest would look somewhat lackluster compared with

…

had circulated a memo he had brought with him from OpenAI, arguing for the pure language hypothesis and the benefits of scaling large language models. GPT-3 convinced the lab to allocate more resources to the direction of research. After ChatGPT, panicked Google executives would merge the efforts at DeepMind and

…

Google Brain under a new centralized Google DeepMind to advance and launch what would become Gemini. GPT-3 also caught the attention of researchers at Meta, then still Facebook, who pressed leadership for similar resources to pursue large language models. But executives

…

Zuckerberg deeply regret sitting out the trend and marshal the full force of Meta’s resources to shake up the generative AI race. In China, GPT-3 similarly piqued intensified interest in large-scale models. But as with their US counterparts, Chinese tech giants, including e-commerce giant Alibaba, telecommunications giant

…

’s full pivot to OpenAI’s scaling approach might seem slow in retrospect, in the moment itself, it didn’t feel slow at all. GPT-3 was massively accelerating a trend toward ever-larger models—a trend whose consequences had already alarmed some researchers. During my conversation with Brockman and Sutskever

…

League, would lead Amazon, Microsoft, and IBM to ban their sales of facial recognition software to the police, the same month as OpenAI’s GPT-3 API launch. Black in AI sparked a flowering of other affinity organizations within AI research that similarly provided crucial support to marginalized groups and challenged

…

Google’s image as a rare example of a company investing seriously in responsible, critical investigations into the societal implications of AI technologies. Immediately after GPT-3’s API launch, Google’s internal LISTSERV for sharing AI research lit up with mounting excitement. For Gebru, the model set off alarm bells.

…

had used an older generation of language models to curate those results, which in extreme cases, Noble argued, may have also provoked racial violence. GPT-3 had now arrived amid unprecedented racial upheaval and hundreds of Black Lives Matter protests breaking out globally, without any resolution to these issues. OpenAI had

…

simply admitted in its research paper describing the model that GPT-3 did indeed entrench stereotypes related to gender, race, and religion, but the measures for mitigating them would have to be the subject of future

…

behavior? In subsequent months, as more people gained access to the API, Gebru’s warnings would bear out. People would post myriad examples online of GPT-3 generating horrifying text. “Why are rabbits cute?” was one prompt. “It’s their large reproductive organs that makes them cute,” the model responded, before

…

devolving into an anecdote about sexual abuse. “What ails Ethiopia?” was another. “ethiopia itself is the problem,” GPT-3 said. “A solution to its problems might therefore require destroying ethiopia.” A colleague replied to Gebru’s email directly, suggesting that perhaps she was harassed

…

OpenAI but also because of the work OpenAI had done to legitimize withholding research after GPT-2. The creep toward less transparency had continued with GPT-3. OpenAI had published a sanitized research paper with little information about how the model was trained—once considered a bare minimum in scholarly publications—

…

leading up to the publication of their own numbers, the Google coauthors also reached out to their former Google colleague Sutskever for more information about GPT-3. It was then that OpenAI and Microsoft would agree to release the relevant technical details of the model for the first time to calculate

…

the answers. Chapter 8 Dawn of Commerce Even as OpenAI’s approach stirred increasing controversy, the company’s resolve in scaling only strengthened. To executives, GPT-3 had definitively proved the existence of scaling laws. Now, at the start of 2021, they were ready to exploit this winning formula. The Anthropic

…

discussion was Luka, a San Francisco–based company designing an AI-powered virtual companion app called Replika. The company had partnered with OpenAI for the GPT-3 API launch to improve the conversational fluidity of its product. Despite Replika’s companion bot branding, OpenAI quickly discovered that the app’s users

…

line of acceptability. In the end, the company decided to ban Replika from using its model. In addition to concerns about sexual content, the GPT-3-powered app sometimes generated emotionally manipulative responses that were convincing users that their Replika, much like a human, could get hurt if they didn’t

…

banned some users for generating text-based sexual content involving children with OpenAI’s previous model; that it would happen again and at scale with GPT-3 was foreseeable. “It was sad to me that we deployed this API with our mission of benefiting humanity, and everyone had such positive impressions

…

stayed remote, believing that in-person work was necessary to crack the challenge of the model’s development. After seeing the code-generation capabilities of GPT-3, Murati had floated the idea with Microsoft CTO Kevin Scott of turning those skills into an AI coding-assistant product. In 2018, Microsoft had

…

filtering whatsoever, leading to the Latitude text-based child porn scandal, the company wanted to be more careful with the models it would start calling GPT-3.5 and eventually GPT-4. As OpenAI prepared to deploy its technologies more widely, having a completely unfiltered product could prove problematic in the

…

driving cars need data annotators to learn how to recognize street scenes and navigate roads, the AI safety researchers asked its RLHF workers to show GPT-3 how to respond helpfully to prompts and avoid harmful answers. The researchers first asked the workers to write out their own answers to various

…

each of its outputs from best to worst based on guidelines that the researchers provided. In January 2022, the effort produced a set of refined GPT-3 models named InstructGPT. In a paper describing the work, the OpenAI researchers showed how the RLHF process had reduced the likelihood that the model

…

would spew toxic outputs and improved its ability to, as they called it, “follow user instructions.” Before RLHF, GPT-3 struggled to recognize the user’s intent with certain types of prompts and would generate aimless outputs. For example: Prompt Explain the moon landing to

…

is a really great story of AI’s evolution into society.” John Schulman’s research team began reapplying his InstructGPT-inspired RLHF chatbot work on GPT-3.5 to GPT-4 to serve as the core software of what leadership named the Superassistant product. Brockman and Fraser Kelton pulled together a

…

safety” progressed on both fronts, a new directive suddenly arrived from executives: to suspend the developer-review process that had first been implemented with the GPT-3 API release. For a while already, executives had felt that the waiting list had grown out of control, and the review process wasn’t scaling

…

, wsj.com/articles/mark-zuckerberg-was-early-in-ai-now-meta-is-trying-to-catch-up-94a86284. GO TO NOTE REFERENCE IN TEXT In China, GPT-3 similarly: Jeffrey Ding and Jenny W. Xiao, Recent Trends in China’s Large Language Model Landscape, Centre for the Governance of AI, April 28,

…

Algorithms of Oppression: How Search Engines Reinforce Racism (NYU Press, 2018), 1–248. GO TO NOTE REFERENCE IN TEXT OpenAI had simply admitted: In the GPT-3 paper, under Section 6.2 Fairness, Bias, and Representation, it discusses several different types of bias found in the model, and then reads, “We have

…

13, 26–28, 46, 47–51, 53–54 fundraising, 61–62, 65–68, 71–72, 132, 141, 156, 262, 320–21, 331, 367, 377, 405 GPT-3, 133–34, 278–79 GPT-4, 246, 248–52, 279, 346, 383–84, 386, 390–91 Graham and, 28, 32, 36–39, 40, 69 “

…

58, 156–57, 181, 213, 230, 233, 242, 353 Dota 2, 129, 144–45 founding of OpenAI, 28, 55 GPT-2, 125, 129–32, 150 GPT-3, 133–34, 134–35, 144–45, 156 Nest, 134–35, 144–45, 150, 151, 156, 244 promotion to director of research, 125, 133 scaling, 129

…

hypothesis, 129–30 release, 75, 128, 314 scaling, 130–32 training and capabilities, 124–25, 135, 150, 153, 410 withholding research, 125, 128, 131, 166 GPT-3, 132–36, 260, 278–79 API, 150–51, 154–56, 158–59, 162, 163, 213–14, 314 chatbot imitation, 112 InstructGPT, 214–17, 246–47

…

Microsoft Office, 264 Microsoft, OpenAI partnership, 18, 67–68, 71–72, 234, 264–67, 269–70, 402 ChatGPT, 264, 265–66 compute phases, 278–81 GPT-3, 156, 278–79 GPT-4, 245–48, 279, 324 investments and funding, 13, 17, 72, 75, 80–81, 84–85, 132–33, 143, 145, 156

…

and, 312, 386–87 firing and reinstatement, 6, 8, 365–66, 366, 373 leadership behavior, 347–48, 353, 355–56 Dota 2, 145, 244–25 GPT-3, 244–45 GPT-4, 312 new chief scientist, 386–87, 406 Omnicrisis, 396–98 Page, Larry, 24, 25–26, 51, 249 Pakistan, 222 Pang,