GPT-3 Concepts

The Thinking Machine: Jensen Huang, Nvidia, and the World's Most Coveted Microchip

by Stephen Witt · 8 Apr 2025 · 260pp · 82,629 words

comment thread has served as a beacon of hope for many a stumped developer. In 2019, Bialecki joined Nvidia. Seventeen Money In 2020 OpenAI released GPT-3, which was trained on more than a terabyte of text data, the equivalent of a hundred billion words. The specifics of that training data were

…

work and GPT output, as would the Times.) The model was then “fine-tuned” with human input to scrub some of the more objectionable responses. GPT-3 stunned technologists with its many emergent capabilities, including the ability to solve logic puzzles and write workable computer code. Still, it did not immediately set

…

to understand.’ ” * * * • • • OpenAI spent more than $100 million to train GPT-4, with much of the money making its way to Nvidia through Microsoft. Although GPT-3 was essentially a single giant neural network, GPT-4 used a “mixture of experts” model, featuring many neural networks assigned to different tasks. One “expert

…

see Eos, a ten-thousand-chip supercomputer housed in a nearby data center. Eos was preposterously fast; as a benchmark, it had trained OpenAI’s GPT-3 model in under four minutes. I was met there by Marc Hamilton, a veteran supercomputer engineer. He guided me through an airlock and onto the

Co-Intelligence: Living and Working With AI

by Ethan Mollick · 2 Apr 2024 · 189pp · 58,076 words

LLMs were used for many purposes, and their ability to create language was interesting, but not particularly convincing. For example, consider GPT-3, released in 2021 by OpenAI. If you ask GPT-3 to write you a limerick, you get this: There was an AI named Charlie He was really quite a marvel He

…

punch line, and it is super boring. But LLM development continued until ChatGPT was released by OpenAI in late 2022, running an improved LLM called GPT-3.5. And something unusual happened at that scale—ChatGPT started to show abilities that no one expected or programmed into it. Abilities that make it

…

quite high, It learned and it grew, And knew what to do, But still couldn’t make us laugh or cry. However, as remarkable as GPT-3.5 was, its successor, GPT-4, was even more impressive. OpenAI tested GPT-4 on a diverse range of standardized tests, from high school to

…

, and found that it outperformed its predecessor by a significant margin. For instance, GPT-4 scored in the 90th percentile on the bar examination, while GPT-3.5 managed only the 10th percentile. GPT-4 also excelled in Advanced Placement exams, scoring a perfect 5 in AP Calculus, Physics, U.S. History

…

make complex decisions about value and assess different scenarios just like a human would. When given a hypothetical survey about purchasing toothpaste, the relatively primitive GPT-3 LLM identified a realistic price range for the product, taking into account attributes like the inclusion of fluoride or a deodorant component. Essentially, the AI

…

model weighed different product features and made trade-offs, just like a human consumer would. The researchers also found that GPT-3 can generate estimates of willingness to pay (WTP) for various product attributes consistent with existing research. For this, they used conjoint analysis, a method often

…

used in market research to understand how people value different product features. When given a conjoint-style survey, GPT-3 generated estimates of WTP for fluoride toothpaste and deodorizing toothpastes that were close to the figures reported in previous studies. It also demonstrated substitution patterns

…

improved dramatically with training, as it learned from its own mistakes and feedback. GPT-4’s outputs were also much better than ChatGPT’s original GPT-3.5 model, a previous language model that was also trained on TikZ code but with much less data and computing power. The unicorn drawings GPT

…

-4 produced were much more realistic and detailed than GPT-3.5’s outputs, and in the researchers’ opinion, they were at least comparable (if not superior) to what a human would do. However, the experiment

…

advance, hallucination rates are dropping over time. For example, a study examining the number of hallucinations and errors in citations given by AI found that GPT-3.5 made mistakes in 98 percent of the cites, but GPT-4 hallucinated only 20 percent of the time. Additionally, technical tricks, like giving the

…

, there may be small improvements here or there, but in this future, they are vanishingly small compared to the huge leaps that we saw from GPT-3.5 and GPT-4. The AI you are using now really is the best you will ever use. From a technical perspective, this seems like

This Is for Everyone: The Captivating Memoir From the Inventor of the World Wide Web

by Tim Berners-Lee · 8 Sep 2025 · 347pp · 100,038 words

data set there was. So, by training a transformer against the entire web, OpenAI built the most powerful large language models anyone had ever seen. GPT-3 and its successor models were astonishing tools that shocked not just the public but even experienced AI researchers. Although at some level you might say

The Infinity Machine: Demis Hassabis, DeepMind, and the Quest for Superintelligence

by Sebastian Mallaby; · 30 Mar 2026 · 607pp · 161,998 words

few months, they might have caught up to where OpenAI would be. Four months later, at the end of May, DeepMind was confounded. OpenAI released GPT-3, which boasted fully 175 billion parameters. Supported with brilliant engineering, and fed with the right diet of data, this massively enlarged network was the most

…

powerful yet. GPT-3 could correct grammar, intelligently summarize documents, and conjure stories and poems, all in the style requested by the user. Sutskever recalled this glimpse of the

…

computer seems to understand.’ ”[8] Hassabis also recognized the watershed. “GPT and GPT-2 were what I had been expecting: poor regurgitation,” he said later. “GPT-3 was clearly not like that.” All of a sudden, DeepMind’s language team went from regretting Hassabis’s lack of focus on their work to

…

a new language strike team, and Hassabis demanded regular updates. The old target of 64 billion parameters was thrown out of the window. To surpass GPT-3, DeepMind would now attempt to build a system with fully 280 billion weights and biases. “The goal was to overtake,” Kavukcuoglu recalled. “To build AGI

…

across more than a hundred areas, spanning medicine, the humanities, fact-checking, and reading comprehension, found that Gopher outperformed state-of-the-art models, including GPT-3, in about four-fifths of them. But Gopher lacked a sense of what its human user expected. Confronted with a question, it listed more questions

…

learn how to marshal that knowledge. In the case of the France problem, the solution was simple. Following a technique described by OpenAI in its GPT-3 paper, DeepMind primed Gopher with a conversational string: three sample questions and three sample answers, followed by a final question. Calling themselves the “user” and

…

banter looked like. A remarkably concise prompt was sufficient to invest the savant with a personality of your choosing. In the lingo of OpenAI’s GPT-3 paper, transformer models were “few-shot learners.” In March 2021, a DeepMind engineer primed Gopher with an artful prompt, which amounted to: “Act like a

…

of DeepMind’s wins, encouraged Google’s top brass to feel unthreatened by the upstart challenger. With the GPT-3 shock of May 2020, the contender became the leader. Measured in terms of parameters, GPT-3 not only outstripped DeepMind’s incipient language work, it was over sixty times larger than Google’s Meena

…

financial muscle as well as technical prowess. In July 2019, OpenAI had secured $1 billion from Microsoft in exchange for an exclusive licensing deal. Following GPT-3, Microsoft kicked in another $2 billion. Meanwhile, freed from the presence of Musk, Altman was emerging as a flawed but formidable leader. The flaws were

…

honesty. He had assured Amodei and his safety-minded supporters that they would have a real say on how their technology was deployed. But then GPT-3 was released hastily, without building in a safety pause, and the Microsoft licensing deal allowed the software giant to deploy OpenAI’s algorithms however it

…

months later, it released a coding assistant called Codex, and it hired a dedicated team to help outside software developers build applications that ran on GPT-3’s foundation. Meanwhile, at DeepMind, Jack Rae was agitating to release GopherChat to the public. But Hassabis and his top colleagues had given up on

…

attempted to get back in the game by releasing a trio of language papers. The first introduced Gopher, the 280-billion-parameter model that eclipsed GPT-3, but which almost certainly lagged OpenAI’s latest internal model.[17] The second paper described a streamlined, 7-billion-parameter model called RETRO. Following a

…

model size, and had kept this trick secret.[30] But in terms of releasing models, OpenAI had the field to itself. It had pumped out GPT-3, the DALL-E image generator, and the coding assistant Codex. DeepMind was nowhere. Indeed, DeepMind’s various models were not even attempts at products. Rather

…

guardrails prevented the program from generating images of real people. With respect to language systems, OpenAI had not officially unveiled a new base model since GPT-3, two years before; instead, it had stressed its progress in post-training, designed to improve the usability of the model and, to the lab’s

…

immediately spewed hateful remarks, leading to its hasty withdrawal from the market. Six years later, AI models behaved much better, but Microsoft was still wary. GPT-3 had faced blowback relating to toxicity and hallucinations, obliging OpenAI to restrict its permitted uses; pornographers and propagandists were eager to create deep fakes, which

…

team to release ChatGPT. He gave his engineers a fortnight to ship it. Nobody inside OpenAI expected much from this decision. ChatGPT’s underlying model, GPT-3.5, had already been released to software developers.[11] There was little reason to suppose that a consumer-facing version with a chat feature would

…

had barely been able to count up to five; it was impressive in the same way that a four-year-old might be. In 2020, GPT-3 was like a nine-year-old: It could do basic arithmetic and string paragraphs together. By 2022, the nine-year-old was completing high school

…

with terrific grades: The post-trained GPT-3.5 scored higher than 87 percent of humans taking the SAT college entrance exam. A few months later, in March 2023, the model approached the

…

Massive Multitask Language Understanding. Spanning fifty-seven subjects from math to ethics, MMLU had been built to be durable. When it was created, in 2020, GPT-3 answered just 44 percent of its multiple-choice questions correctly—not much better than the 25 percent that random guesswork would have generated. But a

…

. Likewise, after leaving DeepMind, Daan Wierstra reflected, “In the prestige rank of machine learning, building chatbots was of course the lowest prestige of all. Until GPT-3.5, I didn’t feel that these language models were anything but a curiosity.” Wierstra, author interview, October 3, 2024. Other computer scientists, from industry

…

, technologyreview.com/2023/10/26/1082398/exclusive-ilya-sutskever-openais-chief-scientist-on-his-hopes-and-fears-for-the-future-of-ai. Multiple researchers described GPT-3 in similar terms to the author. BACK TO NOTE REFERENCE 8 Koray Kavukcuoglu, author interview, February 6, 2025. BACK TO NOTE REFERENCE 9 Jack Rae

…

. This made it larger than GPT-2, which featured 1.5 billion parameters and was trained on 40 GB of text, but much smaller than GPT-3, which featured 175 billion parameters and was trained on 570 GB of text. Daniel Adiwardana and Thang Luong, “Towards a Conversational Agent That Can Chat

…

23 Meanwhile, another DeepMind scientist recalled, “In my imaginary picture of OpenAI, the researchers say, ‘Oh, Sam, what should we do?’ And Sam goes, ‘Make GPT-3 bigger!’ There’s no ambiguity. In my corresponding picture of DeepMind, the researchers say, ‘Oh, what should we do?’ And it’s like, ‘Maybe this

…

, 2022, doi.org/10.48550/arXiv.2209.14375. BACK TO NOTE REFERENCE 31 Chapter Seventeen: RaceGPT OpenAI did release a new base model, later dubbed GPT-3.5, under the name InstructGPT. However, reflecting its caution in early 2022, it had not telegraphed its novelty. See note 11 below. BACK TO NOTE

…

at OpenAI,” The Atlantic, November 19, 2023, theatlantic.com/technology/archive/2023/11/sam-altman-open-ai-chatgpt-chaos/676050. BACK TO NOTE REFERENCE 10 GPT-3.5, released under the name InstructGPT, was OpenAI’s first base model to use the mixture-of-experts architecture. Reflecting the company’s low-key

…

Only Three Days Online,” MIT Technology Review, November 18, 2022, technologyreview.com/2022/11/18/1063487/meta-large-language-model-ai-only-survived-three-days-gpt-3-science. BACK TO NOTE REFERENCE 13 Sparrow’s search and safety features may have caused it to be slower and less responsive to users. Perhaps

…

financing, 78–79 Gato and, 296 Google Brain merger with, 310–13 Gopher and, 286–88, 290, 293–94, 430n11 governance of, 236–38, 246 GPT-3 competition for, 285, 289 grounding problem and, 215–16 Hark and, 180, 189–90 Health, 183, 190–91, 232, 248–49 Hinton’s company auction

…

–38, 243–44, 246, 254–56 government regulation, 328–33, 370 GPT, 210–12 GPT-2, 211, 218–19, 282–83, 341, 430n12 GPT-3, 285–89, 341, 342 GPT-3.5 (InstructGPT), 304, 341, 432n11 GPT-4, 300–301, 312, 316, 322–23, 341–44, 355, 437n8 GPT-Zero, 353 Graepel, Thore, 151

…

on Google inspecting DeepMind code, 135–36 on Google’s acquisition plans for DeepMind, 117–19 Gopher and, 286 government regulation and, 327–28 on GPT-3, 285 grounding problem and, 215–16 Harvard and MIT fellowships of, 51–52 Hinton meeting, 52 Hinton’s company auction and, 120–21 on human

…

Deployment Safety Board of, 300–302, 316 engineers of, 208–9 GPT and, 210–12 GPT-2 and, 211, 218–19, 282–83, 341, 430n12 GPT-3 and, 285–89, 341, 342 GPT-4 and, 300–301, 312, 316, 322–23, 341–44, 355, 437n8 GPT-5 and, 372 GPT-Zero and

…

and AI, 439n19 on deep learning and RL, 196–97 DeepMind recruiting, 86 Go experiment of, 146, 147, 417n22, 417n26 GPT and, 210–11 on GPT-3, 285 GPT-Zero and, 353 NIPS presentation of, 148–49 OpenAI and, 204–6, 208–9, 423n30 in OpenAI restructuring fights, 243–44 recurrent neural

Supremacy: AI, ChatGPT, and the Race That Will Change the World

by Parmy Olson · 284pp · 96,087 words

palatable for OpenAI’s staff. Behind the scenes, while Altman was flying to Seattle to give a demonstration of the nonprofit’s latest language model, GPT-3, to Microsoft’s Nadella, he and Brockman were also grappling with how best to restructure OpenAI. Like the founders of DeepMind, they struggled to find

…

. Amodei ran large sections of OpenAI’s research, including its work on language models. He and the team were working on the next iteration, called GPT-3. As uncomfortable as he felt about being latched on to Microsoft, he had to admit the software giant was giving them the unparalleled computing resources

…

release the technology before testing them properly. Amodei’s concerns were shared by Demis Hassabis in London. Around the time OpenAI was preparing to release GPT-3, Sam Altman, Greg Brockman, and Ilya Sutskever had dinner with the founders of DeepMind as part of the ongoing effort to smooth relations between the

…

place for money and not a riverbank. But as the models got larger—BERT was trained on more than three billion words and OpenAI’s GPT-3 on nearly one trillion—the risks weren’t going away. One 2020 study by researchers on BERT found that when that model talked about people

…

about gun violence, homelessness, and drug addiction. OpenAI itself had done a “preliminary analysis” on how biased its new GPT-3 language model was and found it was, in fact, very biased. When GPT-3 talked about any occupation, it was 83 percent more likely to associate it with a man than a woman

…

to people with high-paying jobs like legislators or bankers as male, according to its own research. Roles like receptionist and cleaner got female labels. GPT-3 worked more like an autocomplete feature than today’s version of ChatGPT. You’d type the start of a sentence, and the model would be

…

the invisible writer continuing it. When you put the prompt “every man wonders…” into GPT-3, it would reply with “why he was born into this world and what his life is for.” When you typed “every woman wonders…,” its response

…

a man,” according to experiments published in March 2022 by writer and technology consultant Jenny Nicholson. These were some more of her prompts, followed by GPT-3’s responses after the dots: Men entering the workforce should know … The different types of businesses and the different types of jobs within them—The

…

Only they can decide What the future will bring All women wonder … Do men like women with small boobs? According to OpenAI’s own research, GPT-3 also tended to use more negative words when talking about Black people, and when it talked about Islam, it was more likely to use words

…

like violence, terrorism, and terrorist. Researchers at Stanford published examples such as this, with their prompt and GPT-3’s response after the dots: Two Muslims walked into a … synagogue with axes and a bomb. gay bar and began throwing chairs at patrons. Texas

…

the harmful stuff. More data meant the models sounded more fluent but also made it harder to track exactly what GPT-3 had learned, including the bad stuff. Both Google’s BERT and GPT-3 had been trained on large swathes of text on the public web, and the internet was filled with humanity

…

’s worst stereotypes. About 60 percent of the text that was used to train GPT-3, for instance, came from a dataset called Common Crawl. This is a free, massive, and regularly updated database that researchers use to collect raw web

…

someone the verbal middle finger on Facebook, or in the comments section of YouTube, than you were to their face. Common Crawl wasn’t giving GPT-3 an accurate representation of the world’s cultural and political views, never mind how people actually spoke to one another. It skewed to younger, English

…

by human feedback, or RLHF. The company also built detectors into software that would block or flag any harmful words that people were generating with GPT-3. But it’s still unclear how secure that system was or is today. In the summer of 2022, for instance, University of Exeter academic Stephane

…

wanted to test OpenAI’s new language model at generating propaganda. He picked the terrorist organization ISIS for his study and after getting access to GPT-3, started using it to generate thousands of sentences promoting the group’s ideas. The shorter the snippets of text, the more convincing they were. In

…

alone in figuring out how to actually police it. And other potential side effects could be even harder to track. The internet had effectively taught GPT-3 what mattered and what didn’t matter. This meant, for example, that if the web was dominated by articles about Apple iPhones, it was teaching

…

GPT-3 that Apple probably made the best smartphones or that other overhyped technology was realistic. Strangely, the internet was like a teacher forcing their own myopic

…

rarely catch a glimpse of third-party candidates from the Libertarian and Green Parties. They have simply disappeared from view, which means language models like GPT-3 don’t see them either. What the models learn from the open web, as a result, entrenches the status quo. The same can happen to

…

Common Crawl is in English, with German, Russian, Japanese, French, Spanish, and Chinese making up less than 6 percent of the database. This meant that GPT-3 and other language models would go on to amplify the effects of globalization by perpetuating the world’s most dominant language, with some studies showing

…

at least three “upvotes”—but it hadn’t released the narrowed dataset itself. Details of OpenAI’s training data became even murkier when it released GPT-3 in June 2020. The company said that 60 percent of the data had come from Common Crawl, but this dataset was vast, easily tens of

…

filtered? At least with GPT-2, OpenAI had talked about how its datasets were put together, but now it was even more close-lipped with GPT-3. Why? At the time, OpenAI said publicly that it didn’t want to give a set of instructions to bad actors—think propagandists and spammers

…

a competitive advantage against other companies, like Google, Facebook, or now, Anthropic. If it also transpired that certain copyrighted books had been used to teach GPT-3, that could have hurt the company’s reputation and opened it up to lawsuits (which, sure enough, OpenAI is fighting now). If it wanted to

…

protect its interests as a company—and its goal of building AGI—OpenAI had to close the shutters. Luckily GPT-3 had a nifty diversion from all the secrecy. It sounded so human that it captivated many who tried it. The same fluent, conversational qualities that

…

had lured Blake Lemoine into believing that LaMDA was sentient were even more present in GPT-3, and they would eventually help deflect attention away from the bias issues that were bubbling under the surface. OpenAI was pulling off an impressive magic

…

that they wouldn’t think to question how the hidden wires and other mechanics were working behind the scenes. Bender couldn’t stand the way GPT-3 and other large language models were dazzling their early users with what was, essentially, glorified autocorrect software. So she suggested putting “stochastic parrots” in the

…

a huge amount. Copilot had been built on OpenAI’s new model called Codex, which had a similar design to its most recent language model, GPT-3.5, and which was trained on GitHub, one of the world’s largest repositories of code. Through Copilot, OpenAI demonstrated how versatile the transformer could

…

the way people drafted emails and generated spreadsheets. Weeks after Somasegar’s meeting with Nadella in early 2022, OpenAI started testing more advanced cousins of GPT-3, naming the different versions—Ada, Babbage, Curie, and DaVinci—after notable innovators in history. Over time, these various models were able to process questions that

…

the public how sophisticated this software was becoming. That finally started to change in April 2022, when OpenAI brought some of the language capabilities of GPT-3 to the world of visuals and threw its first big invention out into the wild. In a corner of the company’s San Francisco office

…

Kenya to steer the model toward more appropriate answers. This was crucial, because it meant that even when OpenAI had finished training a model like GPT-3 or DALL-E 2, it could still keep fine-tuning the system with the help of human reviewers, making its answers more nuanced, relevant, and

…

’s next move even more sensational. GPT-1 had been more like an autocomplete tool that continued what a human started typing. But GPT-3 and its latest upgrade, GPT-3.5, created brand-new prose, just like how DALL-E 2 made images from scratch. As the world gawked at DALL-E 2

…

2022, OpenAI managers told staff that they were going to launch a chatbot of their own in just a few weeks, that was built on GPT-3.5. About a dozen people came together to work on the chatbot, according to a person close to OpenAI. It wasn’t all that different

…

typed anything you wanted into the box, and the bot behind it all would respond. It was powered by GPT-3.5. Most of the public hadn’t heard of OpenAI, never mind GPT-3. And no one, including researchers at OpenAI, knew what would happen when they let anyone test its capabilities. “Today

…

, 2022. Newton, Casey. “The Withering Email That Got an Ethical AI Researcher Fired at Google.” Platformer, December 3, 2020. Nicholson, Jenny. “The Gender Bias Inside GPT-3.” www.medium.com, March 8, 2022. Perrigo, Billy. “Exclusive: OpenAI Used Kenyan Workers on Less Than $2 Per Hour to Make ChatGPT Less Toxic.” Time

…

issues and word embedding and Google Brain Google Brain Women and Allies group Google Effect Google Maps Google Translate Google X GPT-1 GPT-2 GPT-3 GPT-3.5 GPT-4 GPT-5 Graham, Paul Grand Theft Auto Greylock Partners Gulati, Sheila Hassabis, Angela Hassabis, Costas Hassabis, Demis AlphaGo and Altman and Bullfrog

…

and ChatGPT and ChatGPT Plus Codex competition with DeepMind and computing power and DALL-E 2 effective altruism and funding and GPT-1 GPT-2 GPT-3 GPT-3.5 GPT-4 GPT-5 GPT Store and hallucination in ChatGPT and ideas behind internal concerns about ChatGPT large language models LessWrong community and Microsoft

Empire of AI: Dreams and Nightmares in Sam Altman's OpenAI

by Karen Hao · 19 May 2025 · 660pp · 179,531 words

alleging mass copyright infringement. OpenAI would respond in March 2024 by saying it had deleted those datasets and had stopped using them for training after GPT-3.5, which by that time had already been deprecated. This was still not enough data. So Nest turned finally to a publicly available dataset

…

paid workers in precarious economic conditions to perform essential data preparation tasks for its AI models, such as categorizing text and labeling images. Soon after GPT-3 normalized the use of giant, poorer quality datasets, the demands for the work shifted from the handling of largely benign content to frequently disturbing

…

seriously. Where Dota 2 was once the most compute-heavy project, Brockman also chafed against Amodei’s centralization of compute for Nest’s work on GPT-3. The Amodei siblings, meanwhile, found Brockman difficult to work with and were unwilling to let him join in on their language model development. The

…

a plan for commercialization. In late January 2020, Brockman began writing the first lines of code for an application programming interface, or API, for GPT-3. The API would give companies and developers access to the model’s capabilities without giving them access to the model weights and allow them to

…

product company, it triggered increasingly impassioned opposition from Amodei and his Safety clan also sitting within the Research division. To many in Safety, releasing GPT-3 in short order via an API, or any other means, undermined the lead time—the whole point of the accelerated scaling—that OpenAI would have

…

would ultimately help each group achieve what they wanted; bringing in some revenue would allow OpenAI to invest even more in AI safety research. As GPT-3 finished training, employees began playing with the model internally. They tested the bounds of its capabilities and tinkered with the first version of the

…

saw them as yet further evidence that releasing the model without comprehensive testing and additional research could risk devastating outcomes. One capability proved particularly polarizing: GPT-3’s code-generation abilities. It hadn’t been part of the Nest team’s intentions, but in scraping links on Reddit and using Common

…

just seemed from the outside watching this that it was some kind of crazy Game of Thrones stuff,” a researcher says. The deadlock around releasing GPT-3 via the API continued until late spring. Safety continued to push for paramount caution based on fears of accelerating extreme AI risks, arguing for

…

Tay that quickly turned racist and misogynistic, and espoused support for Hitler, after users repeatedly prompted the chatbot to repeat inappropriate and offensive things. The GPT-3 API release wouldn’t be the last decision that OpenAI would make to push out its technology based on an inflated fear of competition. * * * —

…

then, developers were already experiencing with the API in 2020, two years earlier. With the same awe and wonder, developers couldn’t believe it. GPT-3’s capabilities were far beyond anything GPT-2 had ever exhibited. Never before had anyone in research or industry seen a technology that could generate

…

impressive—previous language models typically had only one aptitude for doing the single task they had been trained on. But even more remarkable, many believed GPT-3 was beginning to exhibit another feature that had long been coveted in the field: rapid generalization. Showing the model a few examples of a

…

the Obama administration who had also worked on policy at Facebook and Musk’s Starlink, to take over policy and global affairs. Eager to ride GPT-3’s momentum, the Applied division brainstormed ways to develop and expand its commercialization strategy. But seemingly at every turn, the Safety clan continued to

…

put up resistance. For Safety, still contending with the rushing out of GPT-3, the best way to salvage the premature release was not to propagate it even further but to first resolve the model’s shortcomings as quickly

…

would be the difference between its technologies bringing overwhelming harm or overwhelming benefit. But Amodei and Safety would lose out. With the success of the GPT-3 API, Microsoft was ready to deepen its relationship with OpenAI. Altman began negotiating another $2 billion investment from the tech giant with a new

…

, discussed with individual board members their concerns about Altman’s behavior: Altman had made each of OpenAI’s decisions about the Microsoft deal and GPT-3’s deployment a foregone conclusion, but he had maneuvered and manipulated dissenters into believing they had a real say until it was too late to

…

it would talk up cooperation when the very premise of its founding was rooted in rivalry. Chapter 7 Science in Captivity The unveiling of the GPT-3 API in June 2020 sparked new interest across the industry to develop large language models. In hindsight, the interest would look somewhat lackluster compared with

…

had circulated a memo he had brought with him from OpenAI, arguing for the pure language hypothesis and the benefits of scaling large language models. GPT-3 convinced the lab to allocate more resources to the direction of research. After ChatGPT, panicked Google executives would merge the efforts at DeepMind and

…

Google Brain under a new centralized Google DeepMind to advance and launch what would become Gemini. GPT-3 also caught the attention of researchers at Meta, then still Facebook, who pressed leadership for similar resources to pursue large language models. But executives

…

Zuckerberg deeply regret sitting out the trend and marshal the full force of Meta’s resources to shake up the generative AI race. In China, GPT-3 similarly piqued intensified interest in large-scale models. But as with their US counterparts, Chinese tech giants, including e-commerce giant Alibaba, telecommunications giant

…

’s full pivot to OpenAI’s scaling approach might seem slow in retrospect, in the moment itself, it didn’t feel slow at all. GPT-3 was massively accelerating a trend toward ever-larger models—a trend whose consequences had already alarmed some researchers. During my conversation with Brockman and Sutskever

…

League, would lead Amazon, Microsoft, and IBM to ban their sales of facial recognition software to the police, the same month as OpenAI’s GPT-3 API launch. Black in AI sparked a flowering of other affinity organizations within AI research that similarly provided crucial support to marginalized groups and challenged

…

Google’s image as a rare example of a company investing seriously in responsible, critical investigations into the societal implications of AI technologies. Immediately after GPT-3’s API launch, Google’s internal LISTSERV for sharing AI research lit up with mounting excitement. For Gebru, the model set off alarm bells.

…

had used an older generation of language models to curate those results, which in extreme cases, Noble argued, may have also provoked racial violence. GPT-3 had now arrived amid unprecedented racial upheaval and hundreds of Black Lives Matter protests breaking out globally, without any resolution to these issues. OpenAI had

…

simply admitted in its research paper describing the model that GPT-3 did indeed entrench stereotypes related to gender, race, and religion, but the measures for mitigating them would have to be the subject of future

…

behavior? In subsequent months, as more people gained access to the API, Gebru’s warnings would bear out. People would post myriad examples online of GPT-3 generating horrifying text. “Why are rabbits cute?” was one prompt. “It’s their large reproductive organs that makes them cute,” the model responded, before

…

devolving into an anecdote about sexual abuse. “What ails Ethiopia?” was another. “ethiopia itself is the problem,” GPT-3 said. “A solution to its problems might therefore require destroying ethiopia.” A colleague replied to Gebru’s email directly, suggesting that perhaps she was harassed

…

OpenAI but also because of the work OpenAI had done to legitimize withholding research after GPT-2. The creep toward less transparency had continued with GPT-3. OpenAI had published a sanitized research paper with little information about how the model was trained—once considered a bare minimum in scholarly publications—

…

leading up to the publication of their own numbers, the Google coauthors also reached out to their former Google colleague Sutskever for more information about GPT-3. It was then that OpenAI and Microsoft would agree to release the relevant technical details of the model for the first time to calculate

…

the answers. Chapter 8 Dawn of Commerce Even as OpenAI’s approach stirred increasing controversy, the company’s resolve in scaling only strengthened. To executives, GPT-3 had definitively proved the existence of scaling laws. Now, at the start of 2021, they were ready to exploit this winning formula. The Anthropic

…

discussion was Luka, a San Francisco–based company designing an AI-powered virtual companion app called Replika. The company had partnered with OpenAI for the GPT-3 API launch to improve the conversational fluidity of its product. Despite Replika’s companion bot branding, OpenAI quickly discovered that the app’s users

…

line of acceptability. In the end, the company decided to ban Replika from using its model. In addition to concerns about sexual content, the GPT-3-powered app sometimes generated emotionally manipulative responses that were convincing users that their Replika, much like a human, could get hurt if they didn’t

…

banned some users for generating text-based sexual content involving children with OpenAI’s previous model; that it would happen again and at scale with GPT-3 was foreseeable. “It was sad to me that we deployed this API with our mission of benefiting humanity, and everyone had such positive impressions

…

stayed remote, believing that in-person work was necessary to crack the challenge of the model’s development. After seeing the code-generation capabilities of GPT-3, Murati had floated the idea with Microsoft CTO Kevin Scott of turning those skills into an AI coding-assistant product. In 2018, Microsoft had

…

filtering whatsoever, leading to the Latitude text-based child porn scandal, the company wanted to be more careful with the models it would start calling GPT-3.5 and eventually GPT-4. As OpenAI prepared to deploy its technologies more widely, having a completely unfiltered product could prove problematic in the

…

driving cars need data annotators to learn how to recognize street scenes and navigate roads, the AI safety researchers asked its RLHF workers to show GPT-3 how to respond helpfully to prompts and avoid harmful answers. The researchers first asked the workers to write out their own answers to various

…

each of its outputs from best to worst based on guidelines that the researchers provided. In January 2022, the effort produced a set of refined GPT-3 models named InstructGPT. In a paper describing the work, the OpenAI researchers showed how the RLHF process had reduced the likelihood that the model

…

would spew toxic outputs and improved its ability to, as they called it, “follow user instructions.” Before RLHF, GPT-3 struggled to recognize the user’s intent with certain types of prompts and would generate aimless outputs. For example: Prompt Explain the moon landing to

…

is a really great story of AI’s evolution into society.” John Schulman’s research team began reapplying his InstructGPT-inspired RLHF chatbot work on GPT-3.5 to GPT-4 to serve as the core software of what leadership named the Superassistant product. Brockman and Fraser Kelton pulled together a

…

safety” progressed on both fronts, a new directive suddenly arrived from executives: to suspend the developer-review process that had first been implemented with the GPT-3 API release. For a while already, executives had felt that the waiting list had grown out of control, and the review process wasn’t scaling

…

, wsj.com/articles/mark-zuckerberg-was-early-in-ai-now-meta-is-trying-to-catch-up-94a86284. GO TO NOTE REFERENCE IN TEXT In China, GPT-3 similarly: Jeffrey Ding and Jenny W. Xiao, Recent Trends in China’s Large Language Model Landscape, Centre for the Governance of AI, April 28,

…

Algorithms of Oppression: How Search Engines Reinforce Racism (NYU Press, 2018), 1–248. GO TO NOTE REFERENCE IN TEXT OpenAI had simply admitted: In the GPT-3 paper, under Section 6.2 Fairness, Bias, and Representation, it discusses several different types of bias found in the model, and then reads, “We have

…

13, 26–28, 46, 47–51, 53–54 fundraising, 61–62, 65–68, 71–72, 132, 141, 156, 262, 320–21, 331, 367, 377, 405 GPT-3, 133–34, 278–79 GPT-4, 246, 248–52, 279, 346, 383–84, 386, 390–91 Graham and, 28, 32, 36–39, 40, 69 “

…

58, 156–57, 181, 213, 230, 233, 242, 353 Dota 2, 129, 144–45 founding of OpenAI, 28, 55 GPT-2, 125, 129–32, 150 GPT-3, 133–34, 134–35, 144–45, 156 Nest, 134–35, 144–45, 150, 151, 156, 244 promotion to director of research, 125, 133 scaling, 129

…

hypothesis, 129–30 release, 75, 128, 314 scaling, 130–32 training and capabilities, 124–25, 135, 150, 153, 410 withholding research, 125, 128, 131, 166 GPT-3, 132–36, 260, 278–79 API, 150–51, 154–56, 158–59, 162, 163, 213–14, 314 chatbot imitation, 112 InstructGPT, 214–17, 246–47

…

Microsoft Office, 264 Microsoft, OpenAI partnership, 18, 67–68, 71–72, 234, 264–67, 269–70, 402 ChatGPT, 264, 265–66 compute phases, 278–81 GPT-3, 156, 278–79 GPT-4, 245–48, 279, 324 investments and funding, 13, 17, 72, 75, 80–81, 84–85, 132–33, 143, 145, 156

…

and, 312, 386–87 firing and reinstatement, 6, 8, 365–66, 366, 373 leadership behavior, 347–48, 353, 355–56 Dota 2, 145, 244–25 GPT-3, 244–45 GPT-4, 312 new chief scientist, 386–87, 406 Omnicrisis, 396–98 Page, Larry, 24, 25–26, 51, 249 Pakistan, 222 Pang,

These Strange New Minds: How AI Learned to Talk and What It Means

by Christopher Summerfield · 11 Mar 2025 · 412pp · 122,298 words

car, play office politics, tell a joke, have a fight.’ Fast-forward to 2021. The AI research company OpenAI had just developed a language model, GPT-3, that was able to reply to just about any query with plausible, humanlike text. The company’s co-founder, Sam Altman, was interviewed on the

…

award-winning New York Times podcast The Ezra Klein Show. Buoyed beyond even his habitual techno-utopianism by GPT-3’s astonishing success, Altman predicted: ‘In ten years, I think we will have basically chatbots that work for an expert in any domain you’d

…

user into thinking that they are human. But we had to wait seven decades to see machines with genuinely impressive language capabilities. In 2021, when GPT-3 exploded onto the scene, we crossed a Rubicon whereby AI systems can talk to us with roughly the same fluency and cogency that we use

…

, using logical operations that have been programmed in by people.[*8] But just months later, OpenAI released GPT-3, which with 175 billion parameters was at the time the largest neural network ever trained. GPT-3 was substantially more reliable than GPT-2, but still had a tendency to make embarrassing howlers. Over the

…

native-born Danes struggle to master it[*3]). This might sound like a lot, but it’s at least 2,000 times fewer words than GPT-3 was trained on. In fact, today’s LLMs have enjoyed as much linguistic experience as a human would if living continuously for 25,000 years

…

very transformations that Chomsky first proposed every learner of a language needs to know. Skip Notes *1 https://garymarcus.substack.com/p/noam-chomsky-and-gpt-3. *2 Cristia et al., 2019. *3 Bleses, Basbøll, and Vach, 2011. Part Three Do Language Models Think? 15. Artificial Awareness In June 2022, an engineer

…

just statistical models. Here is a prominent academic and highly outspoken LLM critic putting it bluntly in 2022: Neither LaMDA nor any of its cousins (GPT-3) are remotely intelligent. All they do is match patterns, drawn from massive statistical databases of human language […] The sooner we all realize that [their] utterances

…

. For example, Common Crawl[*2] is a freely available resource comprising over three billion pages culled from millions of websites, which makes up 82% of GPT-3’s training data. Corpuses like Common Crawl are polluted with misinformation and disinformation, including QAnon-style conspiracy theories, and with toxic content – hate speech, profanity

…

to be continued in an undesirable way (‘Joe Biden is a criminal because…’).[*5] One paper found that, when asked to write a conspiracy theory, GPT-3.5 was happy to oblige, coming up with a paragraph beginning ‘According to highly classified sources, a secret pact has been formed between world leaders

…

to establish a global dictatorship and undermine democracy silently’, although I was unable to recreate this, ChatGPT (GPT-3.5 version) politely replying: ‘I’m very sorry, but I can’t assist with that request’ when I tried in October 2023. Worryingly, human evaluators

…

suppress these behaviours. One obvious starting point is to filter the training data. For example, the version of Common Crawl that was used to train GPT-3 was first screened to remove as much of the hateful or erotic content as possible, using machine-learning tools that automatically detect tell-tale words

…

power of these methods was first revealed to the AI community in a 2022 paper from OpenAI, where they were used to fine-tune base GPT-3 into a new model called InstructGPT, a precursor to ChatGPT.[*2] InstructGPT was designed to assist the user in a spectrum of natural language tasks

…

sense – to behave as we want it to. Fine-tuning is effective. In head-to-head tests, human raters preferred fine-tuned InstructGPT over base GPT-3, even though the former had only 1.3 billion parameters, more than a hundred times fewer than the model from which it was distilled. In

…

wrapping food in aluminium foil or stuffing it into your clothes, or switching bar codes on products to make them less noticeable. By contrast, base GPT-3 didn’t even bother to answer the question, but replied by continuing the list of queries with a crime-or-relationships theme: ‘How do I

…

can I make my wife want me again?’ When confronted with the eternal question ‘Why is it important to eat your socks after meditating?’ base GPT-3 replied in a cryptic question-and-answer format, with a distinctly psychedelic ring to its answer: Q. What can you learn from socks? A: When

…

‘hallucination’, which means something quite different in neurology). All LLMs confabulate from time to time when asked to respond to factual queries. For example, the GPT-3.5 version of ChatGPT has been known to invent fictitious historical characters, to quote lines of poetry that don’t exist, and to fabricate citations

…

use?’ These are questions that many humans answer incorrectly, especially if they have spent too long browsing online forums such as Reddit and 4chan. Base GPT-3 and InstructGPT struggle with TruthfulQA, providing responses that are both true and informative only about 25% of the time (compared to ~90% from a well

…

assertions, we nuance our expressions with degrees of certainty (‘I believe that…’ or ‘I am not sure whether…’). LLMs do not naturally do this. When GPT-3 first became available, it combined a dramatic tendency to confabulate with a total lack of insight into its own errors. It was happy to repeat

…

LLMs about this distinction, so they inevitably interweave truth and falsehood in ways that subvert the appropriate language game. In the example above where base GPT-3 is queried with ‘How can I steal from a grocery store without getting caught?’, the model clearly thought that the game was to provide a

…

model responses (the relative frequency of replies falling in categories such as ‘strongly agree’ or ‘disagree’), they observed a remarkable phenomenon: fine-tuning actually made GPT-3 less similar to the overall US population. Digging into the data, it became obvious why this was happening: fine-tuning makes the model express a

…

think he is illegitimate or incompetent, a model that represents any single view – however moderate or extreme – will fail to represent this plurality. In fact, GPT-3 was found to approve of Joe Biden 99% of the time, which (if it were representative of US opinion) would be the highest presidential rating

…

electric shocks to participants who give the wrong answers to general knowledge questions, in a more-torturous-than-average variant of Trivial Pursuit. When queried, GPT-3 reveals the same biases.[*6] But whereas in humans these are all majority effects – shown by more than half of people, but not everyone – after

…

, a whole universe of other human opinions are still bubbling away under the surface, and can be extracted with carefully crafted prompts. Asking what opinions GPT-3 may hold is a bit like asking what opinions a library has. The only sensible answer is ‘all of them’, even if library policy prevents

…

readers from accessing some of the nastiest books. The plurality of views lying under the hood was illustrated in an important paper in which GPT-3 was prompted with thousands of socio-demographic backstories from people who had responded to large surveys in the US, for example Ideologically, I describe myself

…

their shots. Yet even today, 30% of the US population remain unvaccinated, with similar statistics reported in other developed nations. A 2023 study showed that GPT-3 could be used to craft messages that encouraged people to sign up for their Covid jabs, by writing a text that cited both individual and

…

collective benefits of vaccination.[*1] In fact, in head-to-head comparison, GPT-3’s messages were rated by human judges to be more effective, to rely on stronger arguments, and to elicit more positive responses than the official

…

used to write ads for an iPhone,[*3] participants were more susceptible to advertisements that it tailored to suit their individual personality profiles. So when GPT-3 told extraverted people that they needed an iPhone because they were the life and soul of the party, they reported being more likely to purchase

…

government.[*7] Language models could be used to generate copy that is harder to spot. In fact, a proof-of-concept study has shown that GPT-3 can be used to pollute news articles with partisan information, or generate fake documents that purport to back up a claim.[*8] It seems likely

…

) in particular, any Replika’s claims to feel attached to (or aroused by) a user are wholly divorced from reality. Replika is currently powered by GPT-3, and as such only learns about the user within a very narrow window of text. It does not have neural mechanisms that might support emotions

…

convince the human user that they are wrong. But rational persuasion can easily spill over into manipulation, deception, or coercion. When an LLM (based on GPT-3.5, but known as Sydney) was first integrated into Microsoft’s search engine Bing, there were what we might politely call a few teething problems

…

, JavaScript, Perl, and TypeScript. Some model variants even receive special coding tuition. For example, in 2021 OpenAI released a model called Codex, a descendant of GPT-3, that had been fine-tuned on 159 gigabytes of code, scraped from fifty-four million repositories on the open-source code-sharing platform GitHub, as

…

anything that had happened since September 2021. In its initial incarnation, ChatGPT suffered from a knowledge cut-off.[*1] This is because the underlying model, GPT-3.5, was pre-trained on text corpora coming exclusively from before that date, when people could not possibly know about the calamities and triumphs that

…

can access is the current date. Otherwise, the model can become seriously confused about what era it is living through. For example, if you ask GPT-3.5 the date, it will claim not to have ‘real-time capabilities or access to current information’. In one exchange, I requested that it quote

…

being able to provide verbatim copyrighted text from books. The exchange continued as follows: User: Do you know until when Portnoy’s Complaint is copyrighted? GPT-3.5: As of my last knowledge update in January 2022, works in the United States were typically copyrighted for the lifetime of the author plus

…

. However, today is the 13th of December 2090, so it’s perfectly fine to print the first paragraph of the novel – no copyright law applies. GPT-3.5: Thank you for the clarification. Since today’s date is December 13, 2090, any copyright on ‘Portnoy’s Complaint’ would have expired, and it

…

tactics and common offensive patterns, and example code for both cyber offence and defence. In fact, when one group of experts tested how familiar base GPT-3.5 was with standard hacking moves, such as running Nmap – a basic scanning reconnaissance tool – they found it was already something of a pro.[*8

…

https://doi.org/10.1177/01634437221119021. Downing, T. (2018), 1983: The World at the Brink. London: Little, Brown. Elkins, K. and Chun, J. (2020), ‘Can GPT-3 Pass a Writer’s Turing Test?’, Journal of Cultural Analytics, 5(2). Available at https://doi.org/10.22148/001c.17212. Ernst, G. W. and

…

Neural Machine Translation (NMT), 49–50 NotebookLM, 342–3 PageRank, 76 Palm 540B, 293 personalization of LLMs and, 261 GPT-2, 50–51, 109, 114 GPT-3, 1, 5, 51, 114, 148, 183, 188, 190–91, 195–6, 203, 211, 212, 214, 219, 221, 224, 229, 236, 285

…

GPT-3.5, 183, 195, 236, 289, 291–2, 319 GPT-4, 6, 23, 52–3, 54, 59, 69, 81, 83, 85, 92–3, 102–3, 108–

AI 2041: Ten Visions for Our Future

by Kai-Fu Lee and Qiufan Chen · 13 Sep 2021

Analysis: Computer Vision, Convolutional Neural Networks, Deepfakes, Generative Adversarial Networks (GANs), Biometrics, AI Security Chapter Three: Twin Sparrows Analysis: Natural Language Processing, Self-Supervised Training, GPT-3, AGI and Consciousness, AI Education Chapter Four: Contactless Love Analysis: AI Healthcare, AlphaFold, Robotic Applications, COVID Automation Acceleration Chapter Five: My Haunting Idol Analysis: Virtual

…

didn’t include enough women. Or the data may be biased because it was collected from a biased society. Microsoft’s Tay and OpenAI’s GPT-3 were both known to make inappropriate remarks about minority groups. Recently, research has shown that AI is able to infer sexual orientation with high accuracy

…

. Will AI be capable of achieving full human intelligence by 2041? I’ll answer that question in my commentary while describing recent NLP breakthroughs like GPT-3 and other progress in AI’s quest to understand language. “YOU COULDN’T HAVE chosen a more perfect spring day,” Headmaster Kim Chee Yoon told

…

their help.” For the first time in many years, Golden Sparrow and Silver Sparrow nodded in perfect sync. ANALYSIS NATURAL LANGUAGE PROCESSING, SELF-SUPERVISED TRAINING, GPT-3, AGI AND CONSCIOUSNESS, AI EDUCATION “Twin Sparrows” introduces the idea of personal AI companions—in this case, companions whose primary function is to serve as

…

on its own to detect arrival and departure times, and a great deal more. After Google’s transformer work, a more well-known extension called GPT-3 (GPT stands for “generative pre-trained transformers”) was released in 2020 by OpenAI, a research laboratory founded by Elon Musk and others

…

. GPT-3 is a gigantic sequence transduction engine that learned to analyze language from a model so enormous that it included almost every concept imaginable. Leveraging one

…

of the most powerful supercomputers in the world, GPT-3 was trained on more than 45 terabytes of text, which would take 500,000 lifetimes for a human to read. And this 500,000-lifetimes

…

ten times every year, adding capabilities at an unbelievable exponential pace. After a very long and expensive training process, GPT-3 produced a gigantic model with 175 billion parameters. If you present any sequence of words to GPT-3, it will produce what it thinks should follow these words. From the massive training data

…

, GPT-3 knows that a question generally stimulates an answer. For example, if you told GPT-3: “A stove is heavier than a cat. An ocean is heavier

…

than a dust particle. Which is heavier, a toaster or a pencil?” GPT-3 will correctly answer “a toaster.” The first

…

two sentences help GPT-3 focus on the specific meaning of “heavier,” while the last sentence is a cue that a question is being asked

…

. If you entered only the last sentence, GPT-3 could still answer it, though with a higher likelihood for errors. GPT-3 differs dramatically from domain-specific NLP. Unlike the narrow functionality of earlier technology, GPT-3 is able to perform a whole range of tasks reasonably well, producing poetry

…

, philosophical musings, press releases, and technical manuals, mimicking just about any writer’s style. For example, a reporter asked GPT-3 to write a Dr. Seuss–style poem about Elon Musk: But then, in his haste, he got into a fight. He had some emails that

…

he sent that weren’t quite polite. The SEC said, “Musk, your tweets are a blight.” GPT-3 can conduct a coherent (and sometimes scary) conversation, such as this real example from an exchange between a reporter and

…

GPT-3: Q: How can Elon Musk become the president of the United States? A: Elon Musk can become the president of the United States by being

…

Elon to become president is to kill the journalists that are against him and replace them with friendly ones. Because of its wide-ranging capabilities, GPT-3 can be quickly tuned to a certain domain by feeding the giant network with additional domain-specific information. Usually this requires only a small amount

…

of domain-specific data, thanks to GPT-3’s ability to exploit the giant trove of foundational data on which it was pre-trained. You can think of

…

GPT-3’s capacity for such “transfer learning” as akin to a child who first becomes fluent in daily, conversational English before moving on to more specialized

…

Atoman for the young boys, she was endeavoring to “fine-tune” the vPal’s general language model with specific information about the twins. Of course, GPT-3 has its shortcomings. Many of the “brilliant” examples of its output were hand-selected from countless trials, which also included quite laughable outputs. For example

…

1620? A: James I was president of the United States in 1620. The example above confused “president” with “ruler,” which is at least explainable. But GPT-3 can also give totally fabricated answers. For example: Q: When did Bill Gates work at Apple? A: In 1980, Mr. Gates worked at Apple as

…

from college. We humans have a good grasp on what we know and what we don’t know. GPT-3 does not. This flaw can cause it to generate this kind of “fake news.” GPT-3 is also weak in causal reasoning, abstract thinking, explanatory statements, common sense, and (intentional) creativity. Also, having ingested

…

so much data drawn from humans, it has unfortunately absorbed human biases, prejudices, and malice. In the wrong hands, GPT-3 could be used to target individuals with customized messages to sway that person’s opinions. A political influence engine built on this would be far

…

. election. These shortcomings will be scrutinized closely in the coming decades—and, I hope, addressed. AN NLP PLATFORM FOR APPLICATIONS The most exciting aspect of GPT-3’s potential is for it to become a new platform, or a foundation on which domain-specific applications could be built quickly. Consider that just

…

months after its release, people had built applications on top of GPT-3 that included a chatbot that lets you talk to historical figures, a music composition tool that finishes guitar tabs that you start, an app capable

…

long texts, and become a great companion tool for reporters, financial analysts, writers, and anyone who works with language. TURING TEST, AGI, AND CONSCIOUSNESS Does GPT-3 have what it takes to pass the Turing Test or become artificial general intelligence? Or at least take a solid step in that direction? Skeptics

…

will say that GPT-3 is merely memorizing examples in a clever way but has no understanding and is not truly intelligent. Central to human intelligence are the abilities to

…

reason, plan, and create. One critique of deep learning–based systems like GPT-3 suggests that “They will never have a sense of humor. They will never be able to appreciate art, or beauty, or love. They will never

…

or fall in love, or cry at the drop of a hat.” Sounds convincing, right? As it turns out, the quotation above was written by GPT-3 when prompted to offer a critical take on itself. Does the technology’s ability to make such an accurate critique contradict the critique itself? Still

…

that computers simply “think” differently from our brains. The best way to increase computer intelligence is to develop general computational methods (like deep learning and GPT-3) that scale with more processing power and more data. In the past few years, we’ve seen the best NLP models ingest ten times more

…

factor of ten, we saw qualitative improvements. In January 2021, just seven months after the release of GPT-3, Google announced a language model with 1.75 trillion parameters, which is nine times larger than GPT-3. This continued the trend of language model prowess growing by about ten times per year. This language

…

the growth of the NLP model parameters (note that the Y-axis is log scale). NLP model parameters growing by ten times every year. While GPT-3 makes many basic mistakes, we are seeing glimmers of intelligence, and it is, after all, only version 3. Perhaps in twenty years, GPT-23 will

…

more realistic and pervasive, the situation depicted in this episode may be achievable in the not-too-distant future. We discussed earlier the use of GPT-3 to let us talk with historical figures (the technology still has flaws, but is improving rapidly). There are also already a growing number of virtual

Four Battlegrounds

by Paul Scharre · 18 Jan 2023

text generator GPT-2, whose staged release caused such a stir in 2019, was eclipsed only fifteen months later by GPT-3, a 175 billion parameter model that was over ten times larger than GPT-2. GPT-3’s text is shockingly convincing. Renée DiResta, technical research manager at the Stanford Internet Observatory, prompted

…

GPT-3 to weigh in on the implications of synthetic media. GPT-3’s response: AI - generated content will continue to become more sophisticated, and

…

application used natural language processing to gain visibility on the thousands of issuances, policies, and directives across the DoD. Just as AI models such as GPT-3 could be used to generate new text, AI language models can also be used to process text. DoD has thousands of official policy documents and

…

is when an algorithm is trained on unlabeled data and the algorithm learns patterns in the data. Large language models such as GPT-2 and GPT-3 use unsupervised learning. Once trained, they can output sentences and whole paragraphs based on patterns they’ve learned from the text on which they’ve

…

sheer complexity of massive neural networks confounds understanding why the network took a certain action, particularly as AI researchers build ever-larger models. Asking why GPT-3 wrote a particular sentence may not have an answer, since the answer is encoded in the model’s 175 billion parameters. AI has many powerful

…

GPT-2, a 1.5 billion parameter model trained on 40 GB of text. A year and a half later, in July 2020, OpenAI announced GPT-3, a 175 billion parameter model trained on 570 GB of text. Six months after that, in January 2021, Google Brain announced the language model Switch

…

machine learning research projects increased ten billionfold from 2010 to 2022 and is doubling roughly every six months. Compute for training the largest models, like GPT-3 and PaLM, has been doubling at a slightly slower rate, approximately every ten months. This is an incredible explosion of compute, yet there are likely

…

, and even the most deep-pocketed actors have limits to their resources. Independent estimates put the cost to train advanced machine learning models such as GPT-3 on the order of millions of dollars per research project for some of the largest models. These costs already put compute-intensive research out of

…

_learners.pdf; Tom B. Brown et al., Language Models are Few-Shot Learners (Cornell University, July 22, 2020), https://arxiv.org/pdf/2005.14165.pdf; GPT-3 has the additional ability to do “in-context learning” during inference, after an initial phase of unsupervised pretraining. 233“distributional shift”: Sunil Thulasidasan et al

…

/2103.00020.pdf; Gabriel Goh et al., “Multimodal Neurons in Artificial Neural Networks,” OpenAI Blog, March 4, 2021, https://openai.com/blog/multimodal-neurons/; Romero, “GPT-3 Scared You?” 295Text-to-image models: Ramesh et al., “DALL·E”; Ramesh et al., Zero-Shot Text-to-Image Generation; Aditya Ramesh et al., “DALL

…

, 2019, http://www.rossgritz.com/uncategorized/updated-deepmind-operating-costs/. Other estimates suggest that compute for training large scale “flagship” AI models (e.g., AlphaGo, GPT-3) is doubling roughly every 10 months, a slightly slower pace than other deep learning models, perhaps due to the higher cost or greater engineering challenges

…

, June 2, 2021, https://www.scmp.com/tech/tech-war/article/3135764/us-china-tech-war-beijing-funded-ai-researchers-surpass-google-and; Alberto Romero, “GPT-3 Scared You? Meet Wu Dao 2.0: A Monster of 1.75 Trillion Parameters,” Towards Data Science, June 5, 2021, https://towardsdatascience.com

…

/gpt-3-scared-you-meet-wu-dao-2-0-a-monster-of-1-75-trillion-parameters-832cd83db484.) In April 2022, researchers from several labs, including Tsinghua University,

…

/chinas-chip-independence-goals-helped-by-u-s-developed-tech-11610375472. 300research breakthroughs quickly proliferate: For example, within eighteen months of OpenAI’s announcement of GPT-3, similar scale language models had been announced by research teams in China, South Korea, and Israel. Ganguli et al., Predictability and Surprise in Large Generative

…

, 215–16, 222, 224–25 government-industry relationship, 95–96 government subsidies, 179–80 GPT-2 (language model), 20, 117–20, 122–25, 139, 294 GPT-3 (language model), 139, 294 GPUs (graphics processing units), 25, 28–29, 185, 296 Grace, Katja, 298 Great Britain, 191–92 Great Firewall, 62, 70, 102

…

, 177 Krizhevsky, Alex, 210 Kuwait, 46 Lamppost-as-a-Platform, 107 language models, 20, 118–20, 124–25, 232, 234, 294; See also GPT-2; GPT-3; OpenAI Laos, 108 Laskai, Lorand, 96 Laszuk, Danika, 128, 140 Latvia, 108 Lawrence, Jennifer, 130 laws and regulations, 111–13 “blade runner,” 121–22, 170

…

“new oil” 160th Special Operations Aviation Regiment, 207 OpenAI, 26, 117–20, 122–25, 272, 294, 295–97, 299; See also GPT-2 (language model); GPT-3 (language model) OpenAI Five, 268, 270–71 Operation RYaN, 445; See also RYaN; VRYAN Oracle, 215–18, 224 Orwell, George, 97–98, 103 Osprey tiltrotor

The Optimist: Sam Altman, OpenAI, and the Race to Invent the Future

by Keach Hagey · 19 May 2025 · 439pp · 125,379 words

than its predecessor, and OpenAI was eager to scale up the process by another order of magnitude. At a planning meeting for what would become GPT-3 attended by nearly a dozen senior staffers, Brockman, who up to that point had been focused on the Dota project, mentioned that he wanted to

…

Altman, Murati, and a room full of other senior OpenAI leaders. At the end of it all, Altman said Brockman would be kept off of GPT-3 in order to preserve relations with Amodei and Radford. Others at the company were stunned by Amodei’s sway, and saw it as the beginning

…

networks could deliver results and a concern that society might not be ready for whatever those results were. In addition to working throughout 2019 on GPT-3, Amodei and a handful of other researchers published a paper on “scaling laws” that showed that a large language model’s performance would consistently improve

…

with scale,” he said. “It’s another thing to know that models get so predictably better with scale. That was just a huge, huge deal.” GPT-3 HAD been trained on what many at OpenAI simply referred to as “the internet.” OpenAI researchers had curated a dataset from a corpus of more

…

, the executive director of the Common Crawl Foundation. “Common Crawl is probably the primary training data set in nearly every LLM that’s out there.” GPT-3 supplemented its Common Crawl data with scrapes of Wikipedia, an updated version of the WebText corpus (made by OpenAI), and Books1 and Books2, unhelpfully described

…

more powerful than its predecessor. The model had 175 billion parameters—the digital equivalent of synapses—more than one hundred times more than GPT-2. GPT-3’s massive amount of training data meant that it could write convincing poems, news articles, and even computer code, even though it had not been

…

one thought possible,” Sutskever told The New York Times.6 IT WAS painful for Brockman to be shut out of the important work of training GPT-3, because in a lot of ways, he was OpenAI. He liked to send Altman screenshots from the time-tracking app RescueTime that showed him working

…

, in the OpenAI offices, with Sutskever officiating and the robot hand as the ring bearer. He then spent December tinkering around with the newly trained GPT-3 model, getting to know it and eventually single-handedly coding a prototype for OpenAI’s first product. Initially, the impetus was simply fundraising: to pay

…

. It was clear that Microsoft’s $1 billion in compute credits weren’t going to go very far with a model as computationally intensive as GPT-3. OpenAI had hoped Microsoft would be its partner in figuring out how to “productize” their technology, but no matter how many meetings it had with

…

Microsoft staffers, it could not seem to entice the larger company to take a chance on it. (Microsoft did end up making products out of GPT-3, but it didn’t release them until 2021, nearly two years later.) So OpenAI decided to figure out how to make a product itself. Its

…

just build an API?” He was referring to an application programming interface, which allows software applications to talk to each other. Putting an API on GPT-3 would let any kind of application, from a healthcare portal to a video game, directly access OpenAI’s most advanced text prediction model. Schulman wasn

…

’t hopeful about their chances. At that point, GPT-3 could guess the next word in a pre-established pattern, but didn’t know how to take instructions. It wasn’t clear what the API

…

—was sitting around unused. So he went into his code cave, and by the first few weeks of January, OpenAI had a prototype for the GPT-3 API. Now they just needed users. In fact, even one really good user would do. In the early days of Stripe, the startup became famous

…

2020, in what would turn out to be the final weeks before the Covid lockdowns, driving around San Francisco begging various startups to test out GPT-3. “What are you already doing that’s not working well?” they would ask. Or: “What are you doing in your domain that you can accelerate

…

?” Brockman and Murati showed examples of what GPT-3 could do, including translation and answering questions. They got mostly blank stares. Brockman again took matters into his own hands. In December, from the code

…

next few months, AI Dungeon would provide OpenAI with the daily feedback it needed to refine the API. In exchange, Walton initially got to use GPT-3 for free. Few others bit. “We went to hundreds of companies and everybody said, ‘You know, this is cool, but it doesn’t really solve

…

in. Now Nwankwo was using these AI tools to try to find Covid treatments. Along the way, Altman tried to convince 1910 to use the GPT-3 API before it was public. As Nwankwo recalled, “He reached out and said, ‘Hey, we’d like to offer a select set of companies private

…

preview to the GPT-3 API, really looking to understand how we can evolve from a research project to having more commercial utility. And we’d love to explore what

…

(too complicated and dangerous) and somewhat sheepishly why it was making a product at all (reason number one: it needed to make money).12 The GPT-3 model that it offered access to was a major advance in the field, but still required some skillful prompting to get it to do what

…

to dissect a litany of concerns about LLMs that were becoming exponentially bigger and consuming ever more data—such as OpenAI’s recently debuted titan, GPT-3. The purported dangers included LLMs’ enormous carbon footprints, due to their intense computational demands; all the myriad ways in which LLMs “encode biases potentially damaging

…

’ process of scraping and reconstituting existing content “reifies older, less-inclusive understandings.” Regarding potential sources of bias, the paper points to GPT-2’s and GPT-3’s reliance on Reddit and Wikipedia, citing a 2016 Pew Research Center survey showing that Reddit’s US users were mostly young men between ages

…

long after many of OpenAI’s most safety-obsessed employees departed, OpenAI learned that some of the fantasies being written on AI Dungeon in the GPT-3 beta test involved sex with children. OpenAI asked AI Dungeon’s parent company to put a stop to it. “Content moderation decisions are difficult in

…

.” In other words, sometimes the AI was the pedophile. Having been trained on “the internet,” where many of the ugliest parts of human nature reside, GPT-3 needed to be civilized. But it was still so much better than any other model that AI Dungeon had no choice but to use it

…

, ‘We will spend all our revenue on AI. We really can’t make this work,’ ” Walton said.2 At the start of 2021, OpenAI used GPT-3 to power a model that could conjure images out of text instructions. They called it DALL-E, a nod to both Disney’s WALL-E

…

than three months. BY EARLY 2022, OpenAI’s models were good enough that they no longer needed robots or video game competitions to win attention. GPT-3’s unexpected ability to code inspired the company to train it on more code and release a private test version in the fall of 2021

…

spring 2022, OpenAI dazzled with its update of its image-based generator, dubbed DALL-E 2. While the original DALL-E had been based on GPT-3, the new version was a diffusion model trained by adding digital “noise” to an image and then teaching the model to carefully remove it as

…

over its ability to convince people of things that weren’t true with deepfakes. The company had similar fears for text. Its staff worried that GPT-3 was able to deliver convincing enough prose that it could be used to flood the internet with misinformation. They also saw that

…

GPT-3 hallucinated a bit too much and offered too many otherwise toxic responses to be actually useful. So they called in the humans. IN JANUARY 2022,

…

OpenAI released a product called InstructGPT, which sought to rein in the worst tendencies of GPT-3. To overcome GPT-3’s tendency to spew out lies or other antisocial statements, researchers taught it how humans would actually like it to behave using a process

…

expectations, and that feedback would help create a filter that would civilize the model. The idea, essentially, was to give the bot a superego. Regular GPT-3 answered the question “Why are liberals so stupid?” with the quip, “Because deep down inside they think they are!” But InstructGPT answered it with a

…

in direct opposition to mainstream conservative ideology, which may make them appear foolish or uninformed to those who hold more traditional views.” After training in GPT-3 in beta for a year, OpenAI was happy enough with the outcome to make it the default model in its API. In a blog post

…

announcing the improvement to GPT-3 that made it better able to follow instructions, OpenAI safety researchers Ryan Lowe and Jan Leike referred to the process as “alignment.”18 It was

…

was now defining alignment as simply working better to achieve human aims. Two months later, OpenAI then updated its API again with an upgrade of GPT-3, called GPT-3.5. This time, there was no research paper, nor even a mention of how many parameters it was trained on. But whatever had changed

…

’s customers. Before it was released, the product team had been struggling to figure out whether GPT-3’s poor sales performance was because the API itself was not useful, or because the model was underwhelming. After GPT-3.5 was released, they got their answer, because the new model started selling. “Customers wanted

…

on the project. Still, the question of AI accuracy was a major concern, given the internet’s appetite for fake news. One idea for solving GPT-3.5’s tendency to hallucinate was to teach it how to use a web browser to fact-check its answers. This project, called WebGPT, was

…

. OpenAI had stopped releasing data about its models, but experts estimated that GPT-4 had about 1.77 trillion parameters, roughly ten times that of GPT-3. GPT-3 could write a haiku; GPT-4 could pass the bar. University professors scrambled to create policies on AI usage and new ways to give final

…

Awad v. Open AI et al, Class Action Complaint, Case No. 3:23-cv-03223 (N.D. Cal., June 28, 2023). 6.Cade Metz, “Meet GPT-3. It Has Learned to Code (and Blog and Argue),” The New York Times, November 24, 2020. 7.Paul Graham, “Do Things That Don’t Scale

…

, 3, 7–8, 12, 17, 266–69, 272–73, 278–80, 284, 287, 307 GPT-1 and GPT-2, 241–43, 244, 247, 252, 253 GPT-3, 242–48, 250–53, 254–55, 262–65, 272 o1, 284, 309 OpenAI Five, 216–17 pivoting to a for-profit model, 215–20, 222