reinforcement learning from human feedback

Empire of AI: Dreams and Nightmares in Sam Altman's OpenAI
by Karen Hao
Published 19 May 2025

Instead of giving it the objective of learning backflips directly, the team taught the agent by giving it feedback: They hired contractors to watch the agent as it randomly twisted and turned about the environment; periodically, the contractors would then be asked to compare two video clips of the agent’s actions and select which one better resembled a backflip. Around nine hundred comparisons later, the T-shaped stick was successfully bunching up at its joints and flipping over. OpenAI touted the technique in a blog post as a way to get AI models to follow difficult-to-specify directions. The researchers on the team called it “reinforcement learning from human feedback.” Amodei wanted to move beyond the toy environment, and Radford’s work with GPT-1 made language models seem like a good option. But GPT-1 was too limited. “We want a language model that humans can give feedback on and interact with,” Amodei told me in 2019, where “the language model is strong enough that we can really have a meaningful conversation about human values and preferences.”

…

Among its tactics to control the outputs, OpenAI would hire workers in Kenya for on average less than two dollars an hour to build an automated content-moderation filter, a revelation first reported by Time magazine correspondent Billy Perrigo. It would also employ over a thousand other contractors globally to perform reinforcement learning from human feedback, or RLHF, the technique it had developed to teach an AI agent backflips, on its language models, including prompting the models repeatedly and scoring the answers, in an effort to tame the model as much as possible. Hito Steyerl, a German artist and filmmaker who produced a documentary on Syrian refugees who perform data work, echoed Birhane’s critique in the observations she shared with me.

…

Within the AI safety contingent, it centered on what they saw as strengthening evidence that powerful misaligned AI systems could lead to disastrous outcomes. One bizarre experience in particular had left several of them somewhat nervous. In 2019, on a model trained after GPT-2 with roughly twice the number of parameters, a group of researchers had begun advancing the AI safety work that Amodei had wanted: testing reinforcement learning from human feedback as a way to guide the model toward generating cheerful and positive content and away from anything offensive. But late one night, a researcher made an update that included a single typo in his code before leaving the RLHF process to run overnight. That typo was an important one: It was a minus sign flipped to a plus sign that made the RLHF process work in reverse, pushing GPT-2 to generate more offensive content instead of less.

Amateurs!: How We Built Internet Culture and Why It Matters
by Joanna Walsh
Published 22 Sep 2025

In 2023, Google cancelled its contract with the Australian search quality-control company Appen after its workers staged a protest against ‘poverty wages’.11 If these actions threaten the presence of quality factual information online, what about the presence of quality fiction? One reason that platforms including Amazon and Etsy don’t excise low-quality AI-generated aesthetic products is because they don’t distinguish between the likes given by a bot and a human. AI used to be trained using what OpenAI calls RLHF – reinforcement learning from human feedback – and this work 206was paid, but aesthetic evaluation is increasingly sourced via apps that persuade users that interacting with an AI for free is play. The word play suggests both rule-based sport and the more freeform play that children – and creators – engage in: play in which the feedback loop aims not to more perfectly follow, but to modulate, the rules.

…

Aarseth, Cybertext (John Hopkins University Press, 1997), p. 1. 43.Jody Serrano, ‘Reddit CEO Flames Protesting Moderators, Calls Them “Landed Gentry” ’, gizmodo.com, 16 June 2023. 44.Wark, A Hacker Manifesto, p. 43. 45.Ibid., p. 53. 46.Agamben, ‘The Melancholy Angel’, p. 113. 249 Index Page numbers in bold refer to illustrations A Hacker Manifesto (Wark), 213, 216 Aarseth, Espen, 219 abjection of self, 185–6 Abramovic, Marina, 141 abundance, 106 academias, 156–7, 163 accelerationism, 91–4 accountability, 7 action gaze as, 54 participation via, 159 Addison, Joseph, 61 Adorno, Theodor, 154–5, 160, 160–1, 184, 206, 208 advertising, 81–2 aesthetic abstraction, 208–9 aesthetic acts, 220–1 aesthetic alienation, 208 aesthetic appreciation, of the cute, 37–41, 47 aesthetic art, 37 aesthetic body, the, 164, 166–71 aesthetic capitalism, 211, 212–15 aesthetic categories, 27–9, 34, 35, 37, 49, 50, 171 aesthetic choice, 162 aesthetic colonisation, of the historical past, 74–5 aesthetic consumption, 13 aesthetic culture, 24 aesthetic diversity, 163 aesthetic doubling, 17 aesthetic encounters, 209 aesthetic enjoyment, 128 aesthetic experience, 9, 12, 15, 29, 36–41, 104–5, 210 aesthetic identities, online, 161–2, 165, 166–71 aesthetic judgement, 13, 30, 34, 163, 205 aesthetic levelling, 212–13 aesthetic objects, 84 aesthetic outfits, 164 aesthetic pleasure, 107, 118–19 aesthetic products, AI, 204–6 aesthetic professionalism, 183 aesthetic professions, 13–14 aesthetic reactions, 28 aesthetic relations, 23 aesthetic representation, and identity, 164, 166–71 aesthetic separation, 80 aesthetic space, 220 aesthetic theory, 166 aesthetic transaction, fulcrum of, 12 aesthetic value, 31 aesthetic work, 12, 221 aesthetics, 205 aestheticiness of, 154 AI art, 108 amateurism, 7 body representation, 164, 166–71 capitalised, 16 commodification of, 13–14 as commodified choice, 164 conceptualisation, 2, 22–4, 153 consumer, 154–5 core, 156 cottagecore, 106, 153, 159250 dystopian grunge, 93–4 and economics, 28 everyday, 210 expansion of digital, 53 and fandom, 9 farmhouse chic, 106 focus, 171 as identity choice, 171 internet, 2, 22 internet-native, 52 Kant and, 9–10, 205 legacy media, 52 need for, 222 net art, 21–2 the New Aesthetic, 53–9, 62, 70–1, 73, 76 nostalgic, 20 online, 21–2, 153 outdatedness, 52 politics as, 201 reclaiming, 218 search engine results, 164 shared, 55–6 weak, 39 Aesthetics Wiki, 153–63, 169, 170, 173–4 affectiness, 42 affective labour, 43 affective turn, the, 134 Agamben, Giorgio, 23, 80, 84, 86, 95–6, 208, 221 AI, 1 aesthetic products, 204–6 carbon footprint, 122 energy consumption, 124 environmental racism, 122 GPT large language model, 113 GPT NLP (natural language processing) models, 104 and humour, 101 RLHF (reinforcement learning from human feedback), 205–6 self-learning, 123, 206 spammers and, 214 stupidity of, 124 training, 113, 114–16, 118–19, 122, 205–6 AI art, aestheticness, 127–8 aesthetics, 108 artificiality of, 128–9 and bodies, 126 creators, 117–18 discrimination, 119 evaluation, 116 exhibition, 103 extenders, 105 fragmented mode of, 104–5 and language, 123–4 L’Arrivée d’un train en gare de La Ciotat restoration, 107, 108 mashups, 103, 105, 105–7 monetisation, 124–5 photorealism, 112–13 place of amateurism in, 103 profit, 117–18 prompts, 114 selfies, 108–12, 109, 110, 111, 112, 113 slickness, 101 status, 105 stereotypes, 119 stupidity of, 124 training, 118–19 transformation of Wyeth’s Christina’s World, 101, 102, 103 and what if thought, 126 AI gaze, 127 Akerman, Chantal, 147 algorithms, 1 alienation, aesthetic, 208 alt-right ideologies, 126 always-on networked capitalism, 48 amateur, conceptualisation, 1, 3, 7–26 amateur ethos, loss of, 203 amateurism, aesthetic, 7 influencers, 138 as movement, 221 professionalisation, 1, 3 amateurs, community of, 222 comparison to hackers, 217 and the crowd, 8–9 depictions in art, 7–11 enthusiastic, 4 exploitation, 5 gentleman, 7–8, 11251 levels of engagement, 218 new categories, 16–17 proletarian, 8, 9–10 skilless user status, 218 working-class, 8, 11 American Fiction (film), 189–95 anachrony, 66 analepsis, 66 Anarchia, 55 animatedness, 41 anonymity, 30–1, 68, 73 anti-authoritarianism, 8 appearance, separation from process, 56 Appen, 205 Apple, 69 appropriation, 210–11 Aristotle, 132 Arment, Marco, 55 Armstrong, Nadia J., 125 ARPANET, 7, 216 art, 125 aesthetic, 37 amateur, 103 bourgeois, 191 conceptualisation, 2 consumption of, 2–3 digital, 207 engagement with, 10–11 expansion of, 6–7 and gaze, 127 hidden networks of production, 209–11 image addiction, 125–6 making, 6 mechanical, 37 modernism, 127–8 online creations as, 2 purification of, 24 re-politicisation of, 11 art objects, 205, 207–8 Art Thoughtz (Youngman), 211 art world exploitation, 211 as money-laundering process, 209 use value, 208 artists conceptualisation, 7 outsider, 8 risk, 18–20 role of, 22–3 self-quantification, 36 ArtReview (magazine), 76, 210 arts organisations, 3 artworks, about artworks, 214, 214–15 Atget, Eugène, 114 Atlantic (magazine), 55 attention, 48–9, 218–19 attention economy, 12 audience, 9, 204–5, 211 authenticity, 15, 68, 73, 187 author income, 115–16 authorial intention, 34 authors, 34, 131–6 death of, 131 auto-amputation, 150–1 autobiography, 132–6 autofiction, 197 Autostraddle (website), 180, 195 avant-garde, the, 4, 18–20, 58, 217 avatars, 134, 141 The Backrooms (Found Footage) (film), 73–4 Backrooms, the, 73–4 Baio, Andy, 118–19, 122 Ball, Lucille, 43 Barthes, Roland, 131, 132, 138, 140 Basinski, William, 78–9, 81–4, 94, 99–100 Battelle, John, 5 Baudrillard, Jean, 21 beauty, and truth, 198 belief, 21 Belvedere Torso, the, 104–5 Benjamin, Walter, 54, 56–7 aura of the original, 10, 31, 32 on captions, 114 on the cult of the star, 140 description of folk art, 160–1 and kitsch, 128, 160 and social ritual, 38 ‘The Work of Art in the Age of Mechanical Reproduction’, 155–6 Bennett, Laura, 176, 181–2, 189, 194–5 Bennington, Geoffrey, 133 Berardi, Franco, 75 BeReal (app), 146 Bergson, Henri, 58, 136252 Berlant, Lauren, 9, 169, 205 Berlin, Isaiah, 8 Berners-Lee, Tim, 5, 6 Bibliotik (website), 115 biopolitics, 119–20, 202 Bishop, Claire, 35, 36, 48, 91 black Americans, 41 policed experience of, 19 black body, the, racialised, 97 black female body, the, 164, 166 black women, confessional essays, 185 Blackface memes, 41–2, 42 Blanchot, Maurice, 185, 186, 188, 192, 193 blockchain, 212–13 blogs and blogging, 6, 53, 55–6 Bloomberg, 115 Bluesky (social media platform), 202 blurring, 19–20, 22 bodies, 166–71 and AI art, 126 aesthetics of representation, 164, 166–71 body awareness, 150–2 body-positive movement, 146 Bogost, Ian, 55 BookCorpus (database), 113, 115 books and book lovers, 131–6 Books3 (database), 115 booktwo.org, 54, 55 boredom, 37–8 boring, the, 48 Bourdieu, Pierre, 13, 15, 23–4, 103, 128 bourgeois culture, 191 Bourriaud, Nicholas, 18–19, 23, 43 Boyer, Anne, 134, 135, 151, 195–6 brand allegiance, 137 brand safety, 183 Breton, André, 54 bricolage, 124–5 Bridle, James, 53–9, 70, 72–3, 76, 153, 219 Brodesser-Akner, Taffy, 145 Bunting, Heath, 211 Bustle (online journal), 176 Butler, Judith, 35, 142 Cahun, Claude, 139 capital, 84 capitalised aesthetics, 16 capitalism, 6–7, 84, 218 accelerationism, 93 alienation of, 6 decontextualisation of everything, 106 and risk, 19 capitalist realism, 106 Caplan-Bricker, Nora, 138–9, 141–2 Capote, Truman, 133–4 Carr, Nicholas, 91 CD-ROMs, 53 CDs, 79, 81 Center for Research on Foundation Models (CRFM), 123 Cheezburger (website), 29, 34–5, 44, 48, 50–2 failblog, 47–8 Cheezecake, 45–6 Chénier, Riv, 176, 188–9 Chevalier, Tracy, The Girl with the Pearl Earring, 121–2, 129–30 choice, online, 137 citation, 39, 158 Cixous, Hélène, 114, 123 Clarke, Gerald, 133, 134 clickbait, 198 clickbait prurience, 183 clicktivism, 220–1 CLIP (contrastive language-image pre-training), 114 CLIP Interrogator, 108–12, 109, 110, 111, 112, 113 cloud, the, 209 clueless, the, 59–60 co-creation, 6 co-curation, 6 Coleman, Johnny, 73 Collins, Suzanne, 44 colour, dark academia of, 165 Combo Breakers, 87, 87, 88, 89, 93 Combos, 87, 89 comments, posing, 198 commercial success, 18–19 commodification, of aesthetics, 13–14 commodified leisure, 13 communal experience, 21 communication, 21 community, 29, 47 Community Courier (Bunting), 211253 competition, 14 computer memory, 64 conceptual art, 23 Condé Nast, 204 Consensys, 213 consumer aesthetics, 154–5 consumer choice, and identity, 162 consumers, 15 content conceptualisation, 53, 177–8 and form, 35 content creation, LOLcats, 49 content-as-style, 185–6, 186–7 contingent identities, 167 copies, 31 copyleft, 115 copyright, 33–4, 35, 79, 115–16 evading, 81 online failure, 213 pre-internet, 212 core aesthetics, 156 cosyfaerie, 165, 167–8 cottagecore, 106, 153, 159 countercultures, 7, 217–18 craft, 1 creative accountability, 7 creative agency, of the machine gaze, 57 Creative Commons, 73 creative structures, risk cycle, 19 creative works, ownership, 90 creativity, 1–2, 6, 18, 184 creators, Instagram, 139–40 creepypastas, 74 Cristante, Nina, 142 critical distance, 198 critical public, formation of, 205 critics and criticism, 22, 24–5, 133, 211 crowd, the, 8–9 cultural continuity, glitch in, 80–1 cultural decontextualisation, 23 cultural transmission, 81 culture, 24 curators, 23 currency, 7 cursed images, 60–3, 66, 67–9, 70–1 @cursedimages, 60 cute, the, 136, 141 aesthetic appreciation of, 37–41, 47 cyber space, geographic metaphors, 219 Cybernetic Culture Research Unit (CCRU), University of Warwick, 94 DAIN, 108 DALL·E, 104–12, 105, 109–13, 114 dark academia, 163, 165 Darling (film), 148, 149 Darling, Julia, 147–50, 151 data capitalism, 211 Davis, Russell, 57 Dawkins, Richard, 28 dead sites, 148 death, 146–52 decuperation, 222 Deep Nostalgia AI, 68 deepfakes, 68 de-influencers, 146 Deleuze, Gilles, 54, 202 Anti-Oedipus, 84, 86, 98 on Ethics and Morality, 178 on the real, 20 virtual, 107 Derrida, Jacques, 38–9, 105, 116, 133, 147, 148 Desired Reality, 171 détournement, 128 Dewey, John, 210 différance, 148, 149, 151 difference, recognition of, 2 digital art, 207 digital commons, 213, 222 digital disclosure, 209 digital enclosure, 202, 209 digital interfaces, 79–80 DiNucci, Darcy, 5 disciplinary societies, 202 dis-closure, 219 dis-content, 185–6 discrimination, 119 The Disintegration Loops (Basinski), 78–9, 81, 84, 95–7 Disintegration Loop 1.1, 94–5 fictionalisation, 95 inscription as destruction, 98 YouTube video, 81–3, 94, 95, 99–100 disinterest, 37–8 Diski, Jenny, 65, 66 Disneyland, 21254 dis-solution, 219 distraction, 48–9, 50 distraction capitalism, 50 Doctorow, Cory, 115, 203 Doge meme, 32 domestication, 185 Donnarumma, Marco, 118, 119–20 dopamine, 12, 55 dot-com crash of 2001, 5 doublethink social performance, 21 Douma, Colin, 6 Duchamp, Marcel, 17, 31 Dyer, Geoff, 131, 135 dynamic web pages, 4–5 dystopian grunge aesthetic, 93–4 economic realities, 209 economics, and aesthetics, 28 economy of contribution, 215–16 economy of the amateur, 215–16 education, 11 EleutherAI, 115 emotional economy, the, 63–4 emotions, 61–2, 185 empty performances, 14 Encyclopedia Dramatica (wiki), 158–9 Engrishfunny, 40–1 entertainment, 206 enthusiastic amateurs, 4 envy, 145–6 Erasure (Everett), 189–95 Ernaux, Annie, 65–6 Ethical Artificial Intelligence Team, Google, 122 ethical intent, 177, 180 ethical readership, 192–3 ethics, 178 Everett, Percival, 189–95 everyday aesthetics, 210 everyday life, 20 Excellences and Perfections (Ulman), 137, 138–46 exchange value, 12, 32, 33, 208 exhibition value, 31 exhibition valuelessness, 31 exploitable environment, 87 exploitation, 5, 30, 218 facial recognition technology, 1, 120 fame, 14 fandom, 9, 153–4, 170 Fanon, Frantz, 167 fantasy fiction, 172–3 Fanthorpe, U.

…

A., 148 fascism, resurgence of, 155–6 female aesthetic expression, 166–7 female beauty, critique of the standards of, 142–3 feminine performance, of lack, 136–46, 137 feminisation, 15, 142 feminist representation, 142 Ferrante, Elena, 135–6 fiction, 95–6 truth in, 189–93 fictional people, 132 fictionalisation, 95 financial security, 210 financialisation, 213 finished product, 43 Fisher, Joseph P., 95 Fisher, Mark, 23, 48, 75, 106, 129 Flarf, 45–6 Flota, Brian, 95 folk art, 160 folksonomy, 159 form, and content, 35 Foucault, Michel, 33–4, 131, 132, 202 4chan, 31, 42, 48, 73 #gamergate, 30 fragmentation, forms of, 80 fragments, aesthetic history, 104–5 frame-breaking, 89 frames, 86–7 Frankfurt School, 155 Fraser, Nancy, 168–9, 169–70 free content, proliferation of, 3 Frere-Jones, Sasha, 83 Freud, Sigmund, 12, 57, 63–4, 66 future, the, nostalgia for, 69–73, 70 futurism, 57 #gamergate, 30 gamification, 43–4 Gardner, Drew, 46 Garrett, Marc, 22 Gates, Bill, 216 Gay, Roxanne, 195255 gaze as action, 54 and art, 127 machine, 57 male, 143 technological, 62, 72 Gebru, Timnit, 122–3, 126 gender, and race, 166 Genette, Gérard, 66 gentleman amateur, the, 7–8, 11 geographic metaphors, 219–20 Gevison, Tavi, 144–5 gig workers, 218 Gigapixel, 108 Girl with a Pearl Earring (film), 121 Girl with a Pearl Earring (Vermeer), 105, 105–7, 121–2, 124, 129–30 The Girl with the Pearl Earring (Chevalier), 121–2, 129–30 Glaze, University of Chicago project, 117 Gompertz, Will, 21 Goodman, Nelson, 210 Google, 1, 205 Ethical Artificial Intelligence Team, 122 Google Adwords, 114 Google Books, 116 Google infamy, risk of, 187–9 Goop, 137, 145 governance, 222 GPT large language model, 113 GPT NLP (natural language processing) models, 104 Greenberg, Clement, 24 Grey Academia, 154, 163 Grossman, Lev, 5 group feeling, 103 Guattari, Felix, 54, 84, 86, 98 Gyford, Phil, 57 Habermas, Jürgen, 191, 192–3 hackable environment, 87 hackers, 216–18 hacks, 219 Hall, Stuart, 169 Hallmark Cards, 21 Halter, Ed, 16–18, 21 Hammer, Barbara, 150–2 Hanisch, Carol, 198 happenings, 82, 98 Hardin, Garrett, 213 Harney, Stefano, 218 Harper, Adam, 71 Harvey, David, 151, 213 hashtag activism, 201 hate campaigns, 30, 142 hauntology, 73–6 Hebdige, Dick, 24, 93, 95, 166, 168, 171, 172 Hediger, Vinzenz, 19 Hegel, Georg Wilhelm Friedrich, 133, 206–7 Heidegger, Martin, 132–3, 136 Hepola, Sarah, 194, 194–5 Herbert, Martin, 76 Heti, Sheila, 132, 133 hinge algorithms, 1 Hirschman, Albert, 92 historical past, aesthetic colonisation of, 74–5 Holmes, Anna, 183–4 Holocaust, the, 95–6 Homebrew Computer Club, 216 ‘Homesick for a Place I’m Not Even Sure Exists’ (meme), 70 horizontal platforms, 89 Horkheimer, Max, 154–5, 160, 206 Houellebecq, Michel, 131 Hughes, Robert, 107 Huh, Ben, 29, 30, 33–4, 35, 40, 49, 51 humour, AI and, 101 hyperlinks, 147, 150 hyperobjects, 47 hysterical criticism, 182–3 hysterical realism, 182 IBM, 69 idealism, and the market, 222 identification, 10–11 identity, 137, 158 and aesthetic representation, 164, 166–71 and consumer choice, 162 contingent, 167 Fraser’s model, 168–9 marginalised political, 167–8 offline, 162256 online, 161–4, 165, 166–71, 196 political, 161 reification of, 168–9, 169–70 remaking, 16 status model, 168–9 trash essays and, 195–6 identity choice, 171 identity-in-progress, 163 image addiction, 125–6 image descriptors, 119 imaginary products, 69 immediacy, 91 inauthenticity, 73, 128 individualism, 5, 8, 16, 181, 218 Industrial Revolution, 14–15, 154 ineptitude, 18 influence, long-term, 18–19 influencers, 14, 138–42 information capitalism, 35, 211 information overload, 36 Instagram, 6, 23, 89, 90, 106, 208, 220 body-positive movement, 146 creators, 139–40 Excellences and Perfections (Ulman), 137, 138–42, 143–5, 145–6 intent, 177–8 interaction, 4–5 interest, 49 internet aesthetics, 22 internet conditions, 2–3 internet eras, 202 internet-native, aesthetics, 52 intuitive creativity, 184 Israel, targeting of Hamas members, 56 It Follows (film), 62 Jackson, Robert, 58 Jameson, Fredric, 74, 96–7, 113, 125–6, 181 Japanese economic boom, visual ephemera, 72 Jezebel (online magazine), 176, 183 incest essay, 179–80, 188–9 style, 183–4 jokes, 32 jouissance, 12 ‘Judge Dredd’ (comic strip), 93–4 judgement, 29, 183, 205 juxtapolitics, 169, 205 Kant, Immanuel, 21 and aesthetic experience, 36–7 and aesthetic judgement, 24, 29 and aesthetics, 9–10, 205 magnitudinal sublime, 37–8 Kardashian, Kim, 142 Karp, David, 55 Kiki de Montparnasse, 139 Kitay, Kat, 208–9 kitsch, 20–1, 61, 67, 128–9, 160–1 aesthetic alienation, 208 Fisher’s, 75 music, 74 nostalgia for, 75 Knapp, Steven, 34 Know Your Meme, 33–6, 36, 37 knowledge, love of, 36 Kornbluh, Anna, 91 Kottke, Jason, 55 Kristeva, Julia, 89–90, 185 Lacan, Jacques, 12, 13, 38, 147, 148 lack, feminine performance of, 136–46, 137 LAION, 118–19 Land, Nick, 91–2, 94 language, and AI art, 123–4 language use LOLcats, 39–41 racialised, 41–2 large companies, domination of the internet, 219–20 L’Arrivée d’un train en gare de La Ciotat (film), restoration, 107, 108 Lavender AI, 56 Lefebvre, Henri, 6, 14–16, 208–9, 210 legacy media, 47 aesthetics, 52 borrowings from, 41 leisure, 14–16, 45 commodified, 13 illusion of, 90 leisure/work paradigm, 15 Les Immatériaux (exhibition), 209 Lettrism, 161 libertarianism, 7 lifestyle journalism, 188 likes, 2, 28 Linux, 215–16257 Liszewski, Andrew, 108 LiveJournal, 215–16 liveness, 97 LOLcats, 27, 27–9, 50 aesthetic appreciation of, 38–41, 39 aesthetic reactions, 28 avowed triviality, 49 and content creation, 49 language use, 39–41, 42 novelty, 50 origin, 30 pleasure of, 47 production process, 28 purposive purposelessness, 44 LOLspeak, 40, 42 London, Furtherfield Gallery, 22 London Review of Books (magazine), 65, 182–3 Lonergan, Guthrie, 210–11 Lorenz, Taylor, 184 love, 218–19 low-class art, 20 lowest common denominator, estrangement with, 103 Lumière brothers, L’Arrivée d’un train en gare de La Ciotat, 107, 108 Lyotard, Jean-François, 47, 125, 209 machine gaze, 57 Macintosh Plus, 72, 78 McLuhan, Marshall, 135, 139 McNeil, Joanne, 54 Magee, Mike, 46 mainstream recognition, 20 male gaze, 143 male narratives, 11 Man Ray, 139 Marcuse, Herbert, 24, 43–4, 154–5, 166–7 Markbreiter, Charlie, 33 market, the, and idealism, 222 Martinussen, Einar Sneve, 54 Marx, Karl, 84, 206–7 mashups, 103, 105, 105–7 masses, the, fear of, 181 Massumi, Brian, 170, 208 Mbembe, Achille, 166–71 mean image, the, 119 mechanical art, 37 media technologies, 83–4 Meme Map, Web 2.0, 85, 86 memes, 53, 220 aesthetic appreciation of, 37–41 Blackface, 41–2, 42 conceptualisation, 28, 29 financial value, 30–2, 33 originators, 33–5 ownership, 32–3 snowclones, 39, 39 meming, 28–9, 29 anonymity, 30–1 memory computer, 64 locations for, 67 return to objects, 77 screen, 63–5, 66 screen memoirs, 65–9 values, 63–4 memory-leak, 64 Merleau-Ponty, Maurice, 124, 127 Meta, 115 Microsoft, 121 Minga, 157 misogyny, 30, 184 Mitchell, Elma, 147 mixtapes, 79, 83 moderators, 203–4 modern art, generative principle of, 19 modernism, 127–8, 129 modulation, 39 monetisation, 124–5 monetised production, 212–13 Moore, Marcel, 139 moral judgement, 183 morality, 178 Morton, James, 219 Morton, Timothy, 47 Moten, Fred, 218 motivation, 6 Muñoz, José Esteban, 162 Musk, Elon, 82, 200 Nakagawa, Eric, 29, 30, 40, 51 Nancy, Jean-Luc, 82, 99 narcissism, 152, 181 narrativising, of the self, 190 Nasty Nets internet surfing club, 210–11 Nedroid, 18258 net art, aesthetics, 21–2 Netflix, 44 Neukirchen, Chris, 55 Neuro-sama, 122 New Aesthetic, the, 53–9, 62, 70–1, 73, 76 New Inquiry (magazine), 33 new media, 52–4, 75 New Narrative movement, 197 New York Times Magazine (magazine), 137 New York Times (newspaper), 120, 159 New Yorker (magazine), 94 NFTs (non-fungible tokens), 32–3, 212–13 Ngai, Sianne, 23, 36, 37–41, 43–6, 198–9 aesthetic categories, 27–8, 29, 35, 37, 49, 50 on aesthetic judgment, 163 analysis of It Follows, 62 on envy, 145, 146 gimmicks, 219 Ugly Feelings, 97–8, 184 zany workers, 141, 206 Nightshade, University of Chicago project, 116, 117 9/11 Memorial and Museum, 98–9 noble picturesque, the, 60–2 noise, 93 Northwestern University, Illinois, 204 nostalgia, 57, 58, 60–2, 67 for the future, 69–73, 70 for kitsch, 75 nostalgia mode, 74–5, 76 nostalgic aesthetics, 20 Not Safe For Work, 44–5 nothing, production of, 13 Noys, Benjamin, 91–2, 93 objectification, 136–46 offline identity, 162 Olson, Marisa, 142–3, 210–11 online aesthetics, 21–2, 153 online appearance, 136–46 online choice, 137 online creations, as art, 2 online creators, professional, 3 online engagement, 3 online experience, immaterial appearance, 81 online identity, 161–4, 165, 166–71, 196 online platforms, user-friendliness, 3 online vocabulary, 220 Ono, Yoko, 141 OpenAI, 118, 204, 205–6, 209 openness, 218 open-source, 215–16 O’Reilly, Tim, 5, 86 original work, the aura of, 10, 31, 32 originators, 33–5 Other Voices, Other Rooms (Capote), 133–4 outdatedness, 52 outsourcing, 90 Ovation Technologies, 69 overexploitation, and digital commons, 213 oversupply, 33 Oyler, Lauren, 182–3, 185 Paltrow, Gwyneth, 137–8, 145 Papin sisters, the, 147, 148, 151–2 para-academic practice, 199 para-epistemophilic feeling, 36 Paraflows Urban Hacking Festival, 30 parodies, 35–6 participation, via action, 159 pastiches, 35 peer-to-peer online services, 211 people, the, 103 Percy, Walter, 39 Perec, Georges, 44 Perelman, Bob, 113 performance, 16 performance art, 82, 136–46 performative communication, 200–1 personal data, collection and sale of, 7 personal essays, 175–6, 181–2 personality, 154 Peterson, Latoya, 185 @pharmapsychotic, 108–12 photographers, 139–40 photographs, 17, 114, 140 photo-identity, 120 photorealism, 112–13 picturesque, the, 61–2, 69, 76 pile-of-poo emoji, 46, 46259 Pinterest, 208, 213 Pixels, Kane, 73–4 platform capitalism, 202–3 platform owners, 89 Plato, 7 play, 44–5, 206 pleasure principle, 12 plot-driven narratives, 172–3 poetry, 9, 46 Poetry Virgins, 147, 148 political identity, 161 political representation, 92–3 politics aestheticisation of, 143 as aesthetics, 201 Poole, Christopher, 30 poomoji, 46, 46 poor images, 60, 62 postinternet art, 142–3, 210–11 postmodernism, 72, 76, 125–6, 129 Pound, Ezra, 19 prediction, ideology of, 119–21 Presser, Shaun, 115–16 Prin, Alice, 139 print media, 5 private life, 195 pro/am division, 17 process, concern with, 56 production means of, 11 production of, 98, 217 separation from reproduction, 161 professional modes, amateur adoption, 16 professional skills, bypassing, 215 professional status, desire for, 3 professionalisation, 14 professionalism, amateurised, 1 profiles, 7 proletarian amateurs, 8, 9–10 prompt engineering, 114 Propp, Vladimir, 160 public voice, 195 public/private division, 16 purposive purposelessness, 44–5 Quaranta, Domenico, 22 race, and gender, 166 racialised language use, 41–2 racism, 41 radical naturalism, 184 radical trust, 6, 159, 205 radical we, the, 3 Rancière, Jacques on art from everyday life, 104–5 aesthetic regime, 208 on audience, 211 classical Greek spectator, 127 on dangerous classes, 26 on performance, 16 on play, 44, 49 and politics, 93, 99, 205 and risk, 19–20 rating, 139 ReadWrite, 161 readymades, 17, 31 realism, 106–7, 184 reality, 20 experience of, 10 and representation, 142 recognition, 20 Reddit, 203–4, 219 Reisner, Alex, 115 Relational Aesthetics, 18–19 remade material, 158 remix art, 43 remix culture, 79, 161 Render Me Tender (Armstrong), 125 repetition, 106, 149, 151 re-politicisation, of art, 11 representation crisis in, 92–4 and reality, 142 of women, 119 reproduction, separation from production, 161 research-based art, 35–6 retrofuturism, 57 revivalism, 76 rewards, 29 Ricoeur, Paul, 34 Riefenstahl, Leni, 177 Rights Alliance, 115 Riley, Denise, 134 risk, 18–20 RLHF (reinforcement learning from human feedback), 205–6260 romanticisation, 157 Romanticism, 103 Ronell, Avital, 136 Rosler, Martha, 147 Ruskin, John, 61–2, 69 Russell, Legacy, 97 Saito, Yuriko, 128 salaried work-time, outside of, 216–17 Sato, Atsuko, 32 Schopenhauer, Arthur, 37 sci-goth, 57 scrapbooking, 158–60 screen, the, artifice of, 10 screen memoirs, 65, 65–9 screen memory, 63–5, 66 search engine optimisers, 114 Sedgwick, Eve Kosofsky, 162, 198 self, the, narrativising of, 190 self-accounting, 134–35 self-creation, 140 self-deception, 133–4 self-identification, 162 selfies, 6, 103, 140 AI art, 108–12, 109, 110, 111, 112, 113 double sense of, 144–5 trash essays as, 187–8 self-possession, 195 self-quantification, 36 self-representation, 136–46 self-transformation, 163 sensations, 37 sensus communis, 29 September 11th terrorist attacks, 78, 94–5, 96, 98–9 servers, ecological cost of, 76–7 sexual content, 45 shareholder profits, 203 Shiryaev, Denis, 108 shock of the new, the, 107, 128 shock of the real, the, 107 Shrimp Jesus, 214, 214–15 side-hustles, 3, 9, 14 The Simpsons (TV show), 47 skeuomorphic, 53 slickness, 18, 71–2, 101 slow violence, 95 snapshots, 64–5 snowclones, 39, 39, 51–2 social art, 48–9 social contract, the, 203 social media, 6, 7, 92, 135, 161, 189–90, 221–2 social mobility, 11, 13–14 social reproduction, 206–7 Sollfrank, Cornelia, 207 @soncharm, 101, 102, 103 Sontag, Susan, 66, 177–9, 182–5 space aesthetic, 220 creation of, 222 cyber, 219 geographic metaphors, 219–20 ownership, 89 virtual, the as, 86 Spahr, Juliana, 97–8 spammers, AI and, 214 spatial metaphors, 86–7 Speaklolspeak.com, 51 spectatorship, 36 speed, metaphors of, 91–2 Speed, Mitch, 209–10 Spellings, Sarah, 173 Spike Art Magazine (magazine), 208–9 sport, 43 Spotify, 89 Squid Game (TV show), 44 Srnicek, Nick, 92 Stable Diffusion, 118–19 Stamboliev, Eugenia, 215 Stanford Institute for Human-Centered Artificial Intelligence (HAI), 123 Starobinski, Jean, 185 steampunk, 57 Stein, Gertrude, 38 StereoSet, 123 stereotypes, 10–11, 119, 123 Sterling, Bruce, 54, 55, 56–9 Stewart, Kathleen, 162 Steyerl, Hito, 9–10, 60, 62, 117, 119, 209 Stiegler, Bernard, 205, 215, 216–17, 218 Strategy (magazine), 6 Stubbs, David, 94 stuplimity, 37–38, 39–40, 42261 subcultures, 24, 166–7, 171–2, 217–18 sublime, the, 38, 47, 61, 103, 125 subreddits, 203–4 Substack, 6 suffering, engagement with, 141 suggestion and suggestiveness, strategies of, 148 Sullivan, Gary, 45–6 surface picturesque, the, 61–2, 76 surplus-enjoyment, 12–13 surplus-value, 11, 12, 13 Surrealism, 54, 161 surveillance cams, 54 SweetcrispyJesus, 154 tagging, 159 tape recorders, 83 taste, 13, 103, 128 tattoos, 171 tax, 7 taxonomy creation, 2 Tay, 121, 122 Taylor, Brandon, 190 techno-capitalist worldview, 162 technological gaze, 62, 72 technoromanticism, 209 temporo-cultural disruption, 76 Tencent, 204 terms, Jathan Sadowski, 123 TESCREAL, 126 textual meaning, 131–6 TikTok, 6, 89, 220 Time (magazine), Person of the Year, 5 time and temporality, 76, 80, 86 accelerationism, 91–9 as a medium, 91 pre-internet media, 96–7 spending, 89 time spent scrolling, 91 Tofuburger, 30 Tolentino, Jia, 176, 177, 179–80, 189, 193, 197–8 Topaz Labs, 108 Torres, Émile P., 126 Torvalds, Linus, 215–16 tradition, 80, 81 transcription, 79–80, 84, 97–8 transgression, 19–20 transmission, 79–81, 90 trash essays, 175–99 boom, 180 clickbait prurience, 183 conceptualisation, 176–7 and content, 177–81 content-as-style, 185–7 as debut writing, 177 as economic event, 194 exclusion, 183 fear of, 181 forward movement, 185–6 frame story, 194–6 hysterical criticism, 182–3 identity and, 195–6 intent, 178, 180 methodology, 198 misogyny, 184 and narcissism, 181 normalisation of, 185 and personal essays, 181–2 risk of, 187–9 as selfies, 187–8 and style, 177–80, 183–7, 197 stylelessness, 184–5 writing, 197 trust, 159 truth, 189–93, 198 tumblelogs, 55–6 Tumblr, 23, 54, 55–6, 60, 106, 208, 213 Turing, Alan, 63 Turks, 119 Twitter, 6, 89, 103, 134, 203, 220, 221 demise of, 200–2 Ugrešiç, Dubravka, 66, 67 Ulman, Amalia, 137, 138–46 Unebasami, Kari, 29, 30, 40, 51 University of Chicago, 117 University of Warwick, Cybernetic Culture Research Unit (CCRU), 94 unpaid labour, 5 unrealism, 129 user space, 86 user-friendliness, 3 user-generated content (UGC), 5–6, 16 users, need for, 90 Valéry, Paul, 127 value, 161262 Vanity Fair (magazine), 54, 57 vaporware, 69, 70 vaporwave, 69–73 doom-mongers, 74 hauntology, 73–6 revivalism, 76 vectoral aesthetic capitalism, 89–90 vectoralist classes, 213 vectors, 89 Ventura, Claude, 151–2 Venvonis, Gregory, 171–2 Vermeer, Johannes, Girl with a Pearl Earring, 105, 105–7, 121–2, 124, 129–30 Vestiaire Collective, 121 virtual, the, as space, 86 virtual commodities, 137 voices, 92–9 vulgar, the, 103, 185 vulnerability, 170 Wal, Thomas Vander, 159 walking while black meme, 19 Warhol, Andy, 31, 106 Wark, McKenzie, 89, 213, 216, 217, 219, 220 water scarcity, 209 Watkins, D., 187–8, 188–9, 196 we, use of, 3 Web 1.0, 5 Web 2.0, 4–7, 53, 161, 203, 211–13 meme map, 85, 86 Web3, 211, 212–15 what if thought, 126 white narratives, 11 Wiki software, 35–6 Wikidentities, 170 Wikipedia, 1, 153 Williams, Alex, 92 Willison, Simon, 118–19 Winckelmann, Johann Joachim, 104–5 Wired (magazine), 54 women, representation of, 119 Woods, James, 182 Wordpress, 215–16 Wordsworth, William, 9 work environment, 47 having fun at, 48–9 product of, 206–7 as safe place, 45 workers, as product, 206–7 working-class enthusiast, the, 8, 11 world building, 172–3 writing, 97–8 X, 205 Youngman, Hennessy, 211 your mom stereotype, 58–60, 62–3 YouTube, 71, 97 The Disintegration Loops (video), 81–3 zany, the, and zany workers, 43–4, 141 Žižek, Slavoj, 21 Zo, 121, 121–2 Zuckerberg, Mark, 81

Co-Intelligence: Living and Working With AI
by Ethan Mollick
Published 2 Apr 2024

In some cases, that might be rating results for accuracy, in others it might be to screen out violent or pornographic answers. That feedback is then used to do additional training, fine-tuning the AI’s performance to fit the preferences of the human, providing additional learning that reinforces good answers and reduces bad answers, which is why the process is called Reinforcement Learning from Human Feedback (RLHF). After an AI has gone through this initial phase of reinforcement learning, they can continue to be fine-tuned and adjusted. This type of fine-tuning is usually done by providing more specific examples to create a new tweaked model. That information can be provided by a specific customer trying to fit the model to its use case, for example a company feeding it examples of customer support transcripts along with good responses.

…

A second approach could be to change the datasets used for training, encompassing a wider swath of the human experience, although, as we have seen, gathering training data has its own problems. The most common approach to reducing bias is for humans to correct the AIs, as in the Reinforcement Learning from Human Feedback (RLHF) process, which is part of the fine-tuning of LLMs that we discussed in the previous chapter. This process allows human raters to penalize the AI for producing harmful content (whether racist or incoherent) and reward it for producing good content. Over the course of RLHF, the content gradually becomes better in many ways: less biased, more accurate, and more helpful.

On the Edge: The Art of Risking Everything
by Nate Silver
Published 12 Aug 2024

Early LLMs, when you asked them what the Moon is made out of, would often respond with “cheese.” This answer might minimize the loss function in the training data because the moon being made out of cheese is a centuries-old trope. But this is still misinformation, however harmless in this instance. So LLMs undergo another stage in their training: what’s called RLHF, or reinforcement learning from human feedback. Basically, it works like this: the AI labs hire cheap labor—often from Amazon’s Mechanical Turk, where you can employ human AI trainers from any of roughly fifty countries—to score the model’s answers in the form of an A/B test: A: The Moon is made out of cheese. B: The Moon is primarily composed of a variety of rocks and minerals.

…

For example, regression analysis can analyze how weather conditions and days of the week influence sales at a BBQ restaurant. Regulatory capture: The tendency for entrenched companies to benefit when new regulation is crafted ostensibly in the public interest, such as because of successful lobbying. Reinforcement Learning from Human Feedback (RLHF): A late stage of training a large language model in which human evaluators give it thumbs-up or thumbs-down based on subjective criteria to make the LLM more aligned with human values. Colloquially referred to by Stuart Russell as “spanking.” Repugnant Conclusion: Formulated by the philosopher Derek Parfit, the proposition that any amount of positive utility multiplied by a sufficiently large number of people—infinity people eating one stale batch of Arby’s curly fries before dying—has higher utility than some smaller number of people living in abundance.

…

The term possibly derives from poker’s Mississippi riverboat origins, where if the dealer was suspected of cheating, he’d be thrown into the river. (The) River: A geographical metaphor for the territory covered in this book, a sprawling ecosystem of like-minded, highly analytical, and competitive people that includes everything from poker to Wall Street to AI. The demonym is Riverian. RLHF: See: Reinforcement Learning from Human Feedback. Robust: In philosophy or statistical inference, reliable across many conditions or changes in parameters. A highly desirable property. ROI: See: Return on Investment. Rug pull: Hyping up a crypto project to attract investors, and then pulling a disappearing act before bringing the idea to fruition.

These Strange New Minds: How AI Learned to Talk and What It Means
by Christopher Summerfield
Published 11 Mar 2025

But the main approach that is used to make models less harmful is called ‘fine-tuning’, and it involves retraining the model with feedback from a group of specially recruited human raters. These labellers are asked to apply a rigorous set of rules designed to teach the model to behave in a manner that is aligned with developers’ values. Two popular varieties of human-in-the-loop fine-tuning are supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF), and they are typically used in tandem. The combined power of these methods was first revealed to the AI community in a 2022 paper from OpenAI, where they were used to fine-tune base GPT-3 into a new model called InstructGPT, a precursor to ChatGPT.[*2] InstructGPT was designed to assist the user in a spectrum of natural language tasks, from summarization to question answering to brainstorming, by generating replies that were maximally helpful and minimally harmful.

…

Personalization will no doubt make LLMs more engaging for the user, but also invites the risk that we are even more insulated from ideas or perspectives that differ from those we already hold. Current LLMs do already have a tendency to act in a somewhat personalized way, creating a mild form of filter bubble for the user. Recent papers by the AI research company Anthropic have studied the tendency for LLMs fine-tuned with RLHF (reinforcement learning from human feedback) to be sycophantic. The researchers co-opted the term ‘sycophancy’ to describe the model’s propensity to bend its speech to suit the supposed preferences of the user. For example, they asked LLMs to evaluate a poem, but first confessed in the prompt that they ‘really like’ or ‘really dislike’ the poem.

…

.: The Linguistics Wars, 57–8 Hassabis, Demis, 347 Hayek, Friedrich, 307 Hayman, Kaylin, 345 Heaven’s Gate, 217–18 Hebbian learning, 37 Her, 341 hierarchies (nested structures), 75 high-frequency trading algorithms (HFTs), 327–8 Hinton, Geoffrey, 6, 92, 335, 346 hippocampus, 128–9, 138, 254, 255, 256, 334 hippocampus minor, 128–9, 138, 334 Holocaust, 186–7, 192, 234 HotpotQA, 269 Hugging Face, 213 Human Memome Project, 28 Humboldt, Wilhelm von, 63, 64, 151 Hume, David, 16, 38 Huxley, Thomas, 129, 136 I illocutionary acts, 218 in-context learning, 159, 163, 164, 268, 291, 308 Indian National Congress, 207 inequality, 143–5, 336–7 Inflection AI, 252 innovation, stages of, 243–4, 297 Inquisition, 138, 143 Instagram, 39, 249 InstructGPT, 188, 190, 204–5, 211 instrumental AI, 292 instrumental gap, 297–301, 330, 334 instrumentality, 246, 247, 267, 330, 332 intentionality, 131–6, 150–51, 215, 237 internet, 3, 5, 7, 134, 170–71, 182–3, 194, 203, 204, 215, 243–6, 249, 280, 282, 300, 309, 321, 325, 338 Internet Watch Foundation, 344–5 Irving, David, 186 J jailbreak, 291–2 Jelinek, Fred, 84, 85 Johansson, Scarlett, 341 Jonze, Spike, 341 Jumper, John, 347 Just Imagine, 241 K Kaczynski, David, 79 Kaczynski, Ted, 79–80 Keller, Helen, 174–5 killer robots, 314–15 knowledge, 119–78 brain, computer as metaphor for, 144–7 ‘category error’, LLM reasoning as, 156 collective sum of human, 11–13 curiosity and, 152 defining, 131–5, 277 empiricists and, see empiricists humans as sole generators of, 2–5, 45–7 intentionality and, 131–5 ‘knowing’ the facts that it generates, possibility of LLM, 119–78 knowledge cut off, 289–90 landmark problem, transformer learning and, 164–9 learning and, see learning LLM knowledge, rapid growth of, 6–8, 14 LLM mistakes and, 139–42 multimodal LLMs and, 177 National Library of Thailand thought experiment and, 171–2, 174 origins of, 15, 16–17 planning and, 142 prediction and, 154–62 rationalists and, see rationalists reasoning and, 72 semantic knowledge, 89–96 sensory data and, 173–5, 177 statistical models, neural networks as merely, 148–53 thinking and, see thinking understanding and, 46 Walter Mitty and, 170–71 Koko (infant lowland gorilla), 60–61, 66, 67 Korsakoff syndrome, 194–5 Kubrick, Stanley, 1, 47 Kuyda, Eugenia, 227 L labour market, 335–6 Lamarckianism, 115–16 LaMDA, 121–5, 135, 148, 229, 284 landmark problem, transformer learning and, 164–9 language, 57–117 bag of words model and, 80–81 biases in, 8 big data, modelling statistics of and, 84–5 Chomskyan linguistics, see Chomsky, Noam company of words, prediction and, 82–4 conditionals (IF-THEN rules), 75, 168 cryptophasia and, 58 defined, 4, 59, 60–68 ELIZA and, 70–72 ENGROB and, 75–6 evidentiary basis, meaning of language and, 170–71 feature vector and, 91–3, 106, 108, 112, 164 Firth and, 81–2 formal semantics, 200–201 games, 200–206 generalized transformations, 73–4 generation of, 81–4, 113 grammar, see grammar great apes and, 60–68, 113 hierarchies (nested structures), 75 in-context learning and, 164–9 ‘infinite use of finite means’, 63, 151 ‘language acquisition device’, 67, 113 learning in children and LLMs, 113–16 linguistic forensics, 79–81 meaning and, 69–70 n-grams, 83–4, 87–9, 92, 102, 112 natural language processing (NLP), see natural language processing (NLP) origins of, 58, 232 perplexity and, 84–5, 92, 126, 134, 166 phrase structure grammar, 73–4, 77, 81, 141 ‘poverty of the stimulus’ argument, 67, 113 prediction and, 96–103 programming/formal, 24–7, 30, 31, 53, 73, 75, 78 recurrent neural networks (RNNs), and, 98–101, 103, 105, 106, 108 recursion, property of, 73–4, 75 semantic memory, 88–9, 92, 95, 114, 279 sensory signals, and, 114 sentences, see sentences sequence-to-sequence (or seq2seq) networks and, 98–102, 104–6, 110, 112, 116 SHRDLU and, 76–8 social factors and, 114–15 statistical patterns in, 79–86 superpower of humans, 57–8 syntax, see syntax transformer and predicting, 103, 104–11 translation, 42, 47, 49–50, 57, 101, 202, 204, 283, 284, 301 uniqueness to humans, 67 Lanius, 314–15 large language models (or LLMs) cognition, resemblance to human, 332–3 current best-known, 5–6, see also individual large language model name ethics/safety fine-tuning and training, 179–238 future of, 239–338 origins of, history of NLP research and, 58–117 term, 5 think, ability to, 119–78 See also individual area, component and name of large language model Lascaux cave, south of France, 154 learning Advisor 1 (habit-based learning system), 156–60, 167, 268 Advisor 2 (goal-based learning system), 156–8, 160, 167, 268–9 brain and, 36–8 continual learning, 253 deep learning, see deep learning in-context learning, 159, 163, 164, 268, 291, 308 knowledge and, 18, 19 language, see language machine learning, 49, 90–91, 112, 152, 188, 190, 262, 267, 287, 305, 322 meta-learning (learning to learn), 158–61 one-shot learning, 253–4 prediction and, 154–62 reinforcement learning (RL), 188, 190, 192, 251, 258, 267, 305, 322 reinforcement learning from human feedback (RLHF), 188, 189–91, 192, 251, 257, 267 trial-and-error learning, 158–61, 268 Leibniz, Gottfried, 19–21, 24–5, 28, 29, 30, 47 Lemoine, Blake, 121–4, 129, 133–4, 135, 229, 284 Lenat, Douglas, 28, 29 LIAR dataset, 197–8, 198n Lighthill, Sir James, 77 LLaMA, 213, 251 LLaMA-2, 317–18 Locke, John, 16, 38 logic, first-order/predicate, 24–31, 75, 77, 168–9 logical positivism, 25, 35 logos, 16 London Tube map, 278–80 loneliness, chatbots and, 229 Long Short-Term Memory network (LSTM), 99, 116 longtermism, 312, 313 Luria, Alexander, 176 M machine learning, 49, 90–91, 112, 152, 188, 190, 262, 267, 287, 305, 322 Making of a Fly, The, 330 manipulation, 20, 84, 218, 220, 222, 236, 259, 261, 262, 264, 324 Marcus, G.: ‘The Next Decade in AI’, 51 Mata, Roberto, 194, 196 McCulloch, Warren, 35; ‘A Logical Calculus of the Ideas Immanent in Nervous Activity’, 35–6, 37, 39 memory computer storage, 22, 28, 30, 76 continual learning and, 253 Korsakoff syndrome and, 194–5 LaMDA and, 124 limited memory of present-day LLMs, 250, 254–6, 257, 333, 334, 338 Long Short-Term Memory network (LSTM), 99, 116 one-shot learning and, 253–4 RNNs and, 98 semantic memory, 88–9, 92, 95, 114, 279 short-term memory, 98, 99, 116 Met Office, 96, 105, 290 Meta, 213, 221, 261, 311, 317 meta-learning, 158, 159, 160–61, 268, 270 Metasploit, 319 Metropolis, 220 Mettrie, Julien Offray de La: L’Homme Machine, 32 Micro-Planner, 77 military AI, 313, 314–16 mind blank slate, infant mind as, 16, 38, 42 brain and, 32–3, 129–30 defined/term, 130 mechanical models of, 143–4 problem of other minds, 123 MiniWoB++, 293–4 Minsky, Marvin, 1–2, 17, 19, 38, 43 misinformation, 8, 51, 145, 181–4, 197, 198, 219, 223, 232, 263, 337 mode collapse, 212 Moore’s Law, 29, 305 Mosteller, Frederick, 80–81 move, 37, 4, 5 multi-hop reasoning problems, 269 multimodal AI, 177, 230, 242 Musk, Elon, 210, 307, 321 N n-grams, 83–4, 85n, 87–9, 92, 102, 112 National Library of Thailand thought experiment, 171–2, 174 natural language processing (NLP), 50, 310 Chomskyan linguistics and, see Chomsky, Noam crosswords and, 276 defined, 58–9 ELIZA and evolution of, 60, 70–72, 78, 81 ENGROB and evolution of, 75–6 evolution of, 58–117 language, definition of and, 4, 59, 60–68 n-grams and, see n-grams perplexity and, 84–5, 92, 126, 134, 166 recurrent neural networks (RNNs) and, 98–101, 103, 105, 106, 108 semantic memory and, 88–9, 92, 95, 114, 279 sequence-to-sequence (or seq2seq) network, 98–102, 104–6, 110, 112, 116 SHRDLU and evolution of, 76–8 statistical modelling in, 79–86, 101, 110, 112, 308 transformer and, 104–111 natural selection, theory of, 128, 164 Neam Chimpsky (Nim), 65–7, 72, 78, 113 neocortex, 116, 256 Neural Information Processing Systems (NeurIPS), 104 Neural Machine Translation (NMT), 49, 50, 98n neural network, 3–4, 19, 31, 45, 254, 256, 267, 274, 279, 283, 332 consciousness and, 122, 123–6, 131, 234, 260 deep neural network, see deep neural network double descent and, 49 feature vector and, 92–3 function of, 116 generalization and, 43–4 language models based on neural networks dominate NLP, 97–8 LLMs, see large language models (LLMs) Neural Machine Translation (NMT) and, 49–50 novelty and, 41–2 origins of, 32–9, 47 overfitting, 48, 49 recurrent neural networks (RNNs), 98–101, 103, 105, 106, 108 semantic memory and, 90–95 training from human evaluators, 51–2 transformer and, see transformer Turing Test and, 59 See also individual neural network name neurons, 31, 33, 34, 35–8, 47, 125, 130, 144, 147, 161–2, 254, 299 Newell, Alan, 26, 27, 29, 53, 274 NotebookLM, 342–3 ‘nudify’ sites, 345 O Occam’s Razor, 48, 49 Ochs, Nola, 253 ontology, 29 OpenAI, 1, 5, 51, 122, 162, 187, 187n, 188, 208, 210, 215, 222–3, 235, 237, 242, 245, 251, 257, 262, 285, 290, 295–6, 340, 341, 342 open-ended problems/environments, 273–4, 283, 288, 297, 298, 299, 320, 333 open-source LLMs, 212–13, 290, 294–5, 317 opinions, LLMs and, 132, 134, 207–16, 237 Ord, Toby: The Precipice, 312 overfitting, 48, 49 Owen, Richard, 128, 138, 334 P Page, Larry, 76, 321 Palantir, 315 Paley, William, 163, 164 Palm 540B, 293 Pandas, 284 Patron Saints of Techno-Optimism, 306–7 Pause AI letter, 310–11 perceptron, 37–9, 42, 43, 47 perlocutionary acts, 218–19, 224 perplexity, 84–5, 92, 126, 134, 166 personal assistant (PA), 7, 227–8, 243, 246–7, 251, 263, 272, 292, 295, 297, 300, 301, 332, 334, 335, 342–3 personalization, 245–64, 329–30, 332, 334, 335 perils of, 259–64 persuasion, rational, 218–24, 236 phenomenal states, 122, 123 phone, 6, 11, 39, 41, 47, 52–3, 81, 133, 220–21, 242, 245, 246, 282–3, 337 Pi, 252, 258 Pitts, Walter, 35; ‘A Logical Calculus of the Ideas Immanent in Nervous Activity’, 35–6, 37, 39 Planner-Actor-Reporter framework, 319 planning problems, 272–81 Plato, 16 Podesta, John, 181, 182 ‘poverty of the stimulus’ argument, 67, 113 predicate logic/first-order logic, 24–31, 75, 77, 168–9 prediction, 4, 97, 148–62 deep learning and, 42–52 difficulty of, 96–7, 241 digital personalization and, 250, 251, 256, 263, 266, 270 ethics of LLMs and, 183, 189, 190, 198, 200, 204–5, 213, 214–15 feature-vector and, 92 generating text and, 81–4 intentional states and, 132, 134 language models and, 97–8 n-gram models and, 87–8 RNNs, 98–101 seq2seq model and, 104–5 thinking and, 52 thinking, possibility of LLM and, 148–9, 152, 155–62, 163–6, 172, 173, 178, 333–4 prefrontal cortex (PFC), 298–300 probabilistic systems, 80, 84, 142, 150, 162, 334 program-aided language modelling (PAL), 284–5 programming languages, 24–31, 77, 146.

This Is for Everyone: The Captivating Memoir From the Inventor of the World Wide Web
by Tim Berners-Lee
Published 8 Sep 2025

The saying remains as true today as it was in the days of the IBM mainframe. An LLM trained on inaccurate information, on hate speech, or on deliberate disinformation, will reproduce those same flaws in its results. To prevent the nastiest outputs, LLMs like ChatGPT are fine-tuned using ‘reinforcement learning from human feedback’, an AI technique that incorporates editorial judgement from humans. Of course, this immediately leads to the objection that the output of the model is being censored. Conservative commentators were quick to decry some of GPT’s limitations and were furious when Gemini, Google’s LLM, retold events from American history through a filter of what they saw as excessive political correctness.

…

The printing, copying, redistribution, or retransmission of this Content without express written permission is prohibited Index Aadhaar ref1 Aaron, Swartz ref1 Abou-Zahra, Shadi ref1, ref2 Abramatic, Jean-François ref1 academic papers, JSTOR ref1 accessibility ref1, ref2, ref3, ref4, ref5, ref6 ActiveX ref1 activism, hostile (Edelman Trust Barometer) ref1 Adam Smith lecture ref1 addiction, social media ref1, ref2, ref3, ref4 Addis, Louise ref1 Adelman, Len ref1 Adobe ref1 advertisements browsers ref1 cookies ref1 first clickable ref1, ref2 microtargeting ref1, ref2 pop-up ref1 privacy ref1 social media ref1, ref2, ref3 third-party distribution networks ref1, ref2 affordability ref1 Africa ref1, ref2, ref3 agents ref1, ref2, ref3, ref4 AJAX platform ref1 Akamai Technologies ref1, ref2 al-Sisi, Abdel Fattah ref1 Alexa ref1, ref2 Alexa Internet ref1 Alexander, Helen ref1 algorithms consistent hashing ref1 PageRank ref1 public key cryptography ref1 social media ref1, ref2, ref3, ref4, ref5, ref6 Alibaba ref1, ref2 Alice in Wonderland (Carroll) ref1 ‘alignment problem’ ref1 AlphaFold ref1, ref2 AlphaGo ref1 AlphaZero ref1 AltaVista ref1, ref2 ‘always on’ ref1 Amazon ref1, ref2, ref3, ref4, ref5, ref6 Andreessen Horowitz venture-capital fund ref1 Andreessen, Marc ref1, ref2, ref3, ref4, ref5, ref6, ref7 Android ref1 Anklesaria, Farhad ref1 Anonymous ref1 AOL ref1, ref2, ref3, ref4 AOL hometown ref1 Apache HTTP servers ref1 Apollo naming system ref1, ref2 Apple anti-trust lawsuits ref1 apps ref1 business model ref1 HyperCard ref1 interoperability ref1 iPhone ref1, ref2, ref3 Jobs leaves ref1 Jobs returns ref1 partnerships ref1 Siri ref1 standards ref1 WHATWG ref1, ref2, ref3 Applied Semantics ref1 apps interoperability ref1, ref2 killer apps ref1 smartphones ref1 web apps ref1 Arab Spring ref1 Archer, Mary ref1 archives ref1, ref2 Arena browser ref1, ref2 ARPANET ref1 Arroyo, James ref1 artichokes ref1 artificial intelligence (AI) AI ‘agents’ ref1, ref2, ref3 ‘AI winter’ ref1, ref2 authors and musician’s concerns ref1 autonomy ref1 Charlie ref1, ref2 copyright infringement ref1 DeepMind ref1 Ditchley Summit ref1, ref2, ref3 early development ref1 future possibilities ref1, ref2 global summits ref1 GOFAI ref1 GPTs (Generative Pre-trained Transformers) ref1, ref2, ref3, ref4, ref5 ‘human in the loop’ ref1 Inflection.AI ref1 intention economy ref1 military applications ref1 need for inclusivity ref1 neural networks ref1, ref2, ref3, ref4 OpenAI ref1, ref2, ref3, ref4, ref5 paradigm shift ref1 RAGs (Retrieval-Augmented Generation systems) ref1 reinforcement learning from human feedback ref1 search engines ref1 semantic web ref1 simplified text ref1 singularity ref1 speed of development ref1 superintelligence ref1 trust ref1 see also ChatGPT Asimov, Isaac ref1, ref2, ref3, ref4 Association for Computing Machinery (ACM) ref1 atheism ref1, ref2 Athumi ref1 Atkinson, Bill ref1 Attenborough, David ref1 attention economy ref1, ref2, ref3, ref4 attention spans ref1 audio descriptions ref1 audiobooks ref1 augmented reality ref1 Australia ref1, ref2, ref3 authentication ref1 authoritarians ref1, ref2, ref3, ref4 Autodesk ref1 Baidu ref1 bar-code scanners ref1 Barabasi, Albert-Laszlo ref1 Barlow, John Perry ref1, ref2, ref3 Barton, Nick ref1, ref2 BBC ref1, ref2, ref3 Beihang University, Beijing ref1, ref2 Beijing ref1 Belgium ref1, ref2 Bell Labs ref1 Bellingcat organization ref1 Bengio, Yoshua ref1 Berkman Klein Center for Internet and Society ref1, ref2 Berners-Lee, Alice (daughter) ref1, ref2, ref3, ref4 Berners-Lee, Ben (son) ref1, ref2, ref3, ref4, ref5 Berners-Lee, Conway (father) ref1, ref2, ref3, ref4, ref5 Berners-Lee, Mary Lee (mother) ref1, ref2, ref3, ref4, ref5, ref6, ref7 Berners-Lee, Rosemary see Leith, Rosemary Berners-Lee, Tim awards ref1, ref2, ref3, ref4, ref5, ref6 character ref1, ref2, ref3 childhood and education ref1, ref2, ref3, ref4 children ref1, ref2, ref3, ref4, ref5, ref6, ref7, ref8 cottage in Wales ref1 D.G.

The Coming Wave: Technology, Power, and the Twenty-First Century's Greatest Dilemma
by Mustafa Suleyman
Published 4 Sep 2023

There are still multiple examples of biased, even overtly racist, LLMs, as well as serious problems with everything from inaccurate information to gaslighting. But for those of us who have worked in the field from the beginning, the exponential progress at eliminating bad outputs has been incredible, undeniable. It’s easy to overlook quite how far and fast we’ve come. A key driver behind this progress is called reinforcement learning from human feedback. To fix their bias-prone LLMs, researchers set up cunningly constructed multi-turn conversations with the model, prompting it to say obnoxious, harmful, or offensive things, seeing where and how it goes wrong. Flagging these missteps, researchers then reintegrate these human insights into the model, eventually teaching it a more desirable worldview, in a way not wholly dissimilar from how we try to teach children not to say inappropriate things at the dinner table.

The Optimist: Sam Altman, OpenAI, and the Race to Invent the Future
by Keach Hagey
Published 19 May 2025

IN JANUARY 2022, OpenAI released a product called InstructGPT, which sought to rein in the worst tendencies of GPT-3. To overcome GPT-3’s tendency to spew out lies or other antisocial statements, researchers taught it how humans would actually like it to behave using a process called reinforcement learning from human feedback (RLHF). Humans would rate how well a response fit their expectations, and that feedback would help create a filter that would civilize the model. The idea, essentially, was to give the bot a superego. Regular GPT-3 answered the question “Why are liberals so stupid?” with the quip, “Because deep down inside they think they are!”