undersea cable

The Dark Cloud: How the Digital World Is Costing the Earth
by Guillaume Pitron
Published 14 Jun 2023

[‘The geopolitics of the internet. Who governs the world?’], Economica, 2013. ‘Submarine Cables and the Oceans: connecting the world’, International Cable Protection Committee, United Nations Environment Programme, World Conservation Monitoring Centre (UNEP-WCMC), December 2009. 20 See ‘Undersea cables: indispensable, insecure’, Policy Exchange, 2017. 21 ‘Internet, un monde bien réel’ [‘The internet, a physical reality’], La Croix, 24 April 2018. 22 ‘A broken submarine cable knocked a country off the internet for two days’, The Verge, 8 April 2018. 23 ‘Submarine cables and the oceans: connecting the world’, op. cit. 24 ‘Sécurité, pêche … Une enquête publique lancée sur le plus grand câble sous-marin du monde qui passe par la Côte d’Opale’ [‘Security, fishing … A public equity is opened on the world’s biggest undersea cable that crosses the Opal Coast’], France Info, 18 November 2019. 25 Interview with Antony Viera, general secretary of the Regional Committee of Marine Fisheries and Marine Farming (CRPMEM), Hauts-de-France, 2020. 26 ‘Submarine cables and the oceans: connecting the world’, op. cit. 27 Ibid. 28 ‘Vandals blamed for phone and Internet outage’, CNET, 10 April 2009. 29 ‘Vietnam’s submarine cable “lost” and “found’”, LIRNEasia, 2 June 2007.

…

‘Submarine Cables and the Oceans: connecting the world’, International Cable Protection Committee, United Nations Environment Programme, World Conservation Monitoring Centre (UNEP-WCMC), December 2009. 20 See ‘Undersea cables: indispensable, insecure’, Policy Exchange, 2017. 21 ‘Internet, un monde bien réel’ [‘The internet, a physical reality’], La Croix, 24 April 2018. 22 ‘A broken submarine cable knocked a country off the internet for two days’, The Verge, 8 April 2018. 23 ‘Submarine cables and the oceans: connecting the world’, op. cit. 24 ‘Sécurité, pêche … Une enquête publique lancée sur le plus grand câble sous-marin du monde qui passe par la Côte d’Opale’ [‘Security, fishing … A public equity is opened on the world’s biggest undersea cable that crosses the Opal Coast’], France Info, 18 November 2019. 25 Interview with Antony Viera, general secretary of the Regional Committee of Marine Fisheries and Marine Farming (CRPMEM), Hauts-de-France, 2020. 26 ‘Submarine cables and the oceans: connecting the world’, op. cit. 27 Ibid. 28 ‘Vandals blamed for phone and Internet outage’, CNET, 10 April 2009. 29 ‘Vietnam’s submarine cable “lost” and “found’”, LIRNEasia, 2 June 2007. This is not an isolated event: in the article, the Vietnamese coastguard explains that it has already seized hundreds of tonnes of cables lifted by fishing boats. 30 ‘Internet, un monde bien réel’ [‘The internet, a physical reality’], La Croix, 24 April 2018. 31 ‘Câbles transcontinentaux: des milliards de gigaoctets sous les mers’ [‘Transcontinental cables: billions of gigabytes under the sea’], Le Monde, 24 June 2018. 32 Of which fourteen are for New Zealand, and three for Australia. 33 ‘Undersea cables: indispensable, insecure’, Policy Exchange, 1 December 2017. 34 Kim Nguyen, The Hummingbird Project, 111 min, Belga Production/Item 7, 2018. 35 Before the Hibernia Express was laid, the history of cables is marked by situations where speed determines the execution of trades on the stock market.

…

Read ‘Australia supplants China to build undersea cable for Solomon Islands’, The Guardian, 13 June 2018. 8 Russian Optical Trans-Arctic Cable System. 9 ‘The Arctic: a new internet highway?’, Arctic Yearbook, 2014. 10 ‘Geography of the global submarine fibre-optic cable network: the case for Arctic ocean solutions’, Geographical Review, June 2020. 11 ‘Arctic subsea communication cables and the regional development of northern peripheries’, Arctic and North, no. 32, September 2018. 12 ‘Quintillion activates Arctic subsea cable’, Submarine Cable Networks, 13 December 2017. 13 ‘Charges: Ex-Quintillion CEO duped investors in Arctic broadband project’, Alaska Public Media, 12 April 2018.

The Cable
by Gillian Cookson
Published 19 Sep 2012

Looking back at the earliest undersea cables, Smith could see that the blame for their poor performance fell upon failures in scientific understanding as well as inadequate testing and quality control. Faraday had discovered a general phenomenon called ‘the capacitance effect’ in 1838, and was able to explain how this worked to hold up underwater signals on a telegraph line. The problem had become evident again on the Channel line of 1851, and then in the cables, 100 miles in length, connecting Orford Ness with Holland in 1853. Faraday realised that a submarine cable, made from a central copper conductor surrounded by an insulating envelope of gutta percha, armoured on the outside with iron wire rope, formed an electrical capacitor.

…

By the end of 1854, half a million dollars had been spent, some of it with Peter Cooper’s wire-drawing business, which supplied the overland sections of the telegraph. In one respect, though, the Americans could not be self-sufficient. England was the only possible source of submarine cables. British dominance in this area was partly a result of their monopoly on the insulating material, gutta percha. More importantly, much of the expertise accumulated during the short life of undersea cables was concentrated in Britain. Cyrus Field therefore set out again to London, to inspect samples of cable made for him by Kuper & Co. using copper cores from the Gutta Percha Co. His only other contact there was John Watkins Brett, who had become head of the Magnetic Telegraph Co.

…

Derived from the latex of gutta trees found only in the Malay peninsula, gutta percha is a natural plastic which can be shaped when hot and stays flexible as it cools. Once applied as an insulator to submarine cables, it needed to be stored in water to retain its remarkable properties. Its supremacy lasted for a century, until the advent of polyethylene-based synthetics, handing Britain, which had a monopoly on gutta percha, a long-lasting stranglehold on the production of undersea cables. While the material itself was seen as a godsend, there was still some way to go in improving the way it was applied to the cable. With land telegraphs, most of which were carried on overhead lines, insulation was hardly an issue.

Some Remarks
by Neal Stephenson
Published 6 Aug 2012

His rival was William Thomson, 10 years younger, a professor of natural philosophy at Glasgow University who was infatuated with Fourier analysis, a new and extremely powerful tool that happened to be perfectly suited to the problem of how to send electrical pulses down long submarine cables. Wildman Whitehouse predicted that sending bits down long undersea cables was going to be easy (the degradation of the signal would be proportional to the length of the cable) and William Thomson predicted that it was going to be hard (proportional to the length of the cable squared). Naturally, they both ended up working for the same company at the same time.

…

Early telegraph lines were just naked conductors strung from pole to pole, but this worked poorly, especially in wet conditions, so some kind of flexible but durable insulation was needed. After much trial and error, rubber became the standard for terrestrial and aerial wires while gutta-percha (a natural gum also derived from a tree grown in Malaya) was used for submarine cables. Gutta-percha is humble-looking stuff, a nondescript brown crud that surrounds the inner core of old submarine cables to a thickness of perhaps 1 centimeter, but it was a wonder material back in those days, and the longer it remained immersed in salt water, the better it got. So far, it was all according to the general plan that the British had in mind: find some useful DNA in the Americas, stockpile it at Kew Gardens, propagate it to other botanical gardens around the world, make money off the proceeds, and grow the economy.

…

In shallower waters, additional layers of protection are laid on, beginning with a steel antishark jacket. As the shore is approached, various other layers of steel armoring wires are added. This more or less describes how all submarine cables are being made as of 1996. Only a few companies in the world know how to make cables like this: AT&T Submarine Systems International (AT&T-SSI) in the US, Alcatel in France, and KDD Submarine Cable Systems (KDD-SCS) in Japan, among others. AT&T-SSI and KDD-SCS frequently work together on large projects and are responsible for FLAG. Alcatel, in classic French fasion, likes to go it alone. This basic technology will, by the end of the century, be carrying most of the information between continents.

The Wires of War: Technology and the Global Struggle for Power
by Jacob Helberg
Published 11 Oct 2021

sh=707cb72d1b2a. 60 “South America-1 (Sam-1),” Submarine Cable Map, March 2001, https://www.submarinecablemap.com/#/submarine-cable/south-america-1-sam-1. 61 Andrew Blum, Tubes, e-book, 116, 125. 62 “Internet Exchange Points,” Data Center Map, https://www.datacentermap.com/ixps.html. 63 John Watkins Brett, On the Origin and Progress of the Oceanic Electric Telegraph: With a Few Facts, and Opinions of the Press (London: Nassau Steam Press, 1858), 66. 64 “History and Achievements,” Global Marine, https://globalmarine.co.uk/about-us/history-achievements/. 65 Adam Satariano, “How the Internet Travels Across Oceans,” New York Times, March 10, 2019, https://www.nytimes.com/interactive/2019/03/10/technology/internet-cables-oceans.html. 66 “Submarine Cable 101,” TeleGeography, https://www2.telegeography.com/submarine-cable-faqs-frequently-asked-questions. 67 “Submarine Cable Map,” Submarine Cable Map, https://www.submarinecablemap.com/#/submarine-cable/seamewe-3. 68 Jeremy Page, Kate O’Keeffe, and Rob Taylor, “America’s Undersea Battle With China for Control of the Global Internet Grid,” Wall Street Journal, March 12, 2019, https://www.wsj.com/articles/u-s-takes-on-chinas-huawei-in-undersea-battle-over-the-global-internet-grid-11552407466. 69 Satariano, “How the Internet Travels Across Oceans.” 70 John Hendel and Betsy Woodruff Swan, “Justice Department opposes Google, Facebook cable link to Hong Kong,” Politico, June 17, 2020, https://www.politico.com/news/2020/06/17/justice-department-hong-kong-google-facebook-cable-326688. 71 Page, O’Keeffe, and Taylor, “America’s Undersea Battle With China for Control of the Global Internet Grid.” 72 Ibid. 73 Ibid. 74 Chris C.

…

Companies,” Bloomberg, October 4, 2018, https://www.bloomberg.com/news/features/2018-10-04/the-big-hack-how-china-used-a-tiny-chip-to-infiltrate-america-s-top-companies. 55 Ibid. 56 Ibid. 57 Ibid. 58 Mara Hvistendahl, “The friendly Mr Wu,” The Economist, February 25, 2020, https://www.1843magazine.com/features/the-friendly-mr-wu. 59 Thomas Brewster, “Exclusive: Warning Over Chinese Mobile Giant Xiaomi Recording Millions of People’s ‘Private’ Web and Phone Use,” Forbes, April 30, 2020, https://www.forbes.com/sites/thomasbrewster/2020/04/30/exclusive-warning-over-chinese-mobile-giant-xiaomi-recording-millions-of-peoples-private-web-and-phone-use/?sh=707cb72d1b2a. 60 “South America-1 (Sam-1),” Submarine Cable Map, March 2001, https://www.submarinecablemap.com/#/submarine-cable/south-america-1-sam-1. 61 Andrew Blum, Tubes, e-book, 116, 125. 62 “Internet Exchange Points,” Data Center Map, https://www.datacentermap.com/ixps.html. 63 John Watkins Brett, On the Origin and Progress of the Oceanic Electric Telegraph: With a Few Facts, and Opinions of the Press (London: Nassau Steam Press, 1858), 66. 64 “History and Achievements,” Global Marine, https://globalmarine.co.uk/about-us/history-achievements/. 65 Adam Satariano, “How the Internet Travels Across Oceans,” New York Times, March 10, 2019, https://www.nytimes.com/interactive/2019/03/10/technology/internet-cables-oceans.html. 66 “Submarine Cable 101,” TeleGeography, https://www2.telegeography.com/submarine-cable-faqs-frequently-asked-questions. 67 “Submarine Cable Map,” Submarine Cable Map, https://www.submarinecablemap.com/#/submarine-cable/seamewe-3. 68 Jeremy Page, Kate O’Keeffe, and Rob Taylor, “America’s Undersea Battle With China for Control of the Global Internet Grid,” Wall Street Journal, March 12, 2019, https://www.wsj.com/articles/u-s-takes-on-chinas-huawei-in-undersea-battle-over-the-global-internet-grid-11552407466. 69 Satariano, “How the Internet Travels Across Oceans.” 70 John Hendel and Betsy Woodruff Swan, “Justice Department opposes Google, Facebook cable link to Hong Kong,” Politico, June 17, 2020, https://www.politico.com/news/2020/06/17/justice-department-hong-kong-google-facebook-cable-326688. 71 Page, O’Keeffe, and Taylor, “America’s Undersea Battle With China for Control of the Global Internet Grid.” 72 Ibid. 73 Ibid. 74 Chris C.

…

As Priscilla Moriuchi, the director of strategic threat development at a Massachusetts cybersecurity company, has warned, “It’s the technologies of the future and technologies of future security systems that could be the biggest security threat in the Belt and Road project.”32 All of these efforts amount to a bewildering mix of corporate deal-making and state-sponsored extortion. They traverse a tangle of undersea cables, next-generation wireless networks, and global supply chains. But broadly speaking, we can break China’s back-end strategy for dominating the Internet into four main components: controlling the production, controlling the pipes, controlling the protocols, and controlling the post-4G future. To help make sense of how these four puzzle pieces fit together, consider the many steps involved in sending a simple email.

Seriously Curious: The Facts and Figures That Turn Our World Upside Down
by Tom Standage
Published 27 Nov 2018

Now they need so much network capacity to synchronise data across their networks of data centres around the world that it makes more sense to lay their own dedicated cables. This has led to a boom in new undersea cable systems. The Submarine Telecoms Forum, an industry body, reckons that 100,000km of submarine cable was laid in 2016, up from just 16,000km in 2015. TeleGeography, a market-research firm, reckons that $9.2bn will have been spent on such cable projects between 2016 and 2018, five times as much as in the previous three years. Plumbing the depths Active and planned submarine cable systems owned* by: October 2017 Source: TeleGeography *In full or in part Game theory: sport and leisure Why tennis players grunt It has become common behaviour in top-level tennis, like the pumped fist on winning a set and the expletive aimed at a coach after an error.

…

Why 5G might be both faster and slower than previous wireless technologies Mobile phones are more common than electricity in much of sub-Saharan Africa Why self-driving cars will mostly be shared, not owned How ride-hailing apps reduce drink-driving What is augmented reality? Why we’re still waiting for the space elevator How astronomers spotted the first interstellar asteroid Why drones could pose a greater risk to aircraft than birds What is the point of spam e-mail? Why the police should wear body cameras Why tech giants are laying their own undersea cables Game theory: sport and leisure Why tennis players grunt Why board games are so popular in Nigeria How drones can keep beaches safe from sharks How football transfers work How St Louis became America’s chess capital What does “digitally remastering” a film really mean? How bookmakers deal with winning customers The world’s highest-earning football clubs Speaking my language: words and wisdom Why emoji have beneficial linguistic side-effects How the letters of the alphabet got their names Why Papua New Guinea has so many languages Is Serbo-Croatian one language or four?

…

Questions will then arise as to how long such data should be kept and in what circumstances recordings should be released to the public. Such worries are not insignificant. But the evidence is mounting that the usefulness of such cameras outweighs the drawbacks. Why tech giants are laying their own undersea cables In September 2017 Microsoft and Facebook announced the completion of a 6,600km (4,100-mile) cable stretching from Virginia Beach, Virginia, to Bilbao, Spain. Dubbed Marea, Spanish for “tide”, the bundle of eight fibre-optic threads, roughly the size of a garden hose, is the highest-capacity connection across the Atlantic Ocean.

The Idea Factory: Bell Labs and the Great Age of American Innovation
by Jon Gertner
Published 15 Mar 2012

When the Labs’ chemists needed to help design impermeable underwater cables, for example, one possibility was gutta-percha. But gutta-percha had drawbacks, including its extreme expense. For an undersea cable to Catalina Island, off the coast of California, the Labs’ chemists began looking for an alternative. Natural rubber was considered too soft in its pure state, and when it was treated with sulfur to toughen it—that is, to “vulcanize” it—the men had to address the problem that sulfur corrodes copper and would undoubtedly degrade the vital wires within the undersea cable. Only after the chemists determined that they could purify the rubber in a complex manner and then create fine silica flour as an insulator could the cable go into production.

…

At the same time, he noted, there was an obvious and unique answer: Figure out a way to make an undersea cable with repeaters spaced every forty miles or so, and then figure out a way to make it work for twenty years without leaks or interruption. To a large degree, then, the cable was a challenge of engineering rather than science, but even Kelly seemed humbled by “the magnitude of the task, its tremendous scope, the necessity for establishing that the designs of each and every item meet not only functional requirements but also the twenty-year no-failure requirement.”5 Fortunately, by the time Kelly began planning an undersea cable with British phone engineers in 1953, a small library already existed about what would and would not work.

…

A number of multinational corporations—RCA, General Electric, ITT, among others—had already made clear their intentions, as Fortune magazine put it in July 1961, to install “the great cable in space.” Satellites offered an alternative to the increasingly burdened underseas cables; more important, they could carry live television, which the current underseas cables could not. Within a decade, some economists predicted, orbiting relays could be a billion-dollar-a-year business.31 “We think the best practical system would use about 50 satellites in space,” Kappel explained to U.S. News & World Report. Would AT&T spend $25 million to do it?

The Emperor's New Road: How China's New Silk Road Is Remaking the World
by Jonathan Hillman
Published 28 Sep 2020

Not to be left out, Russia and Saudi Arabia have expressed interest in setting up bases. One of Djibouti’s greatest assets is beneath the waves. It is a major landing point for underseas fiber-optic cables and a critical stop on China’s “digital silk road.” Globally, there are about 380 active underseas cables, which carry the vast majority of international data. Data from cell phones, for example, is sent to a tower and then onward across terrestrial and submarine cables. These cables will only become more important with the arrival of 5G and other services that will increase the speed and volume of data that is transferred. Using the cloud, despite its name, often requires data to travel under the ocean.4 Western policy makers were slow to grasp the challenge that China’s dominance in global telecommunications could present.

…

Mackinder and Mahan, for all their wisdom, did not live in a world of airpower, let alone cyber power. While Western capitals debate the risks of allowing Chinese technology into their next-generation wireless networks, Chinese firms are rapidly connecting the world’s next-generation markets, as Chapter 9 explains. In East Africa, Chinese firms are laying underseas cables, equipping cities with surveillance cameras, and building wireless networks. On Maslow’s hierarchy of digital needs, network security is secondary to cost, positioning China’s tech champions to grow globally even if their Western presence withers. New connections, physical and digital, also produce unintended consequences.

…

Building south to Gwadar would require upgrading an existing line, for which feasibility studies have only begun.109 Altogether, the two extensions are estimated to cost $16 billion, and each will take at least five years to build.110 Officially, the southern component is slated to begin construction in 2025, and the northern component will begin in 2030, close enough to allow officials to speak about the project with a straight face and far enough away to allow them to comfortably delay it.111 While the railway and pipeline have not moved forward, the Chinese telecom giant Huawei has laid a fiber-optic cable across the Khunjerab Pass. At $44 million, it is less expensive and more practical.112 Huawei representatives point out that it will allow Pakistanis to access Chinese content more quickly, and by avoiding undersea cables, the route makes espionage from neighboring countries other than China more difficult.113 Unlike the proposed pipeline, the fiber-optic cable is less vulnerable to attack and cheaper to repair. Unlike the proposed railway, it cannot carry people. It offers the type of connectivity that is most comfortable to both governments.

Your Computer Is on Fire
by Thomas S. Mullaney , Benjamin Peters , Mar Hicks and Kavita Philip
Published 9 Mar 2021

“The Dot-Com Bubble Bursts,” New York Times (December 24, 2000), https://www.nytimes.com/2000/12/24/opinion/the-dot-com-bubble-bursts.html. 29. High-capacity bandwidth is defined as 1 million Mbps or more. All data from consulting firm Terabit, reported by Economic Times (April 2012), http://articles.economictimes.indiatimes.com/2012-04-15/news/31342442_1_undersea-cable-submarine-cables-fibre. 30. J. P. Singh, Leapfrogging Development? The Political Economy of Telecommunications Restructuring (Binghamton: State University of New York Press, 1999). 31. Bill Woodcock, “Brazil’s official response to NSA spying obscures its massive Web growth challenging US dominance,” Al Jazeera (September 20, 2013); Sascha Meinrath, “We Can’t Let the Internet Become Balkanized,” Slate (October 14, 2013), http://www.slate.com/articles/technology/future_tense/2013/10/internet_balkanization_may_be_a_side_effect_of_the_snowden_surveillance.html. 32.

…

For an ethnogeography of these cables, explicitly inspired by Neal Stephenson’s cable travelogue, see Nicole Starosielski, The Undersea Network (Durham, NC: Duke University Press, 2015). For continuing updates on cable technology, see https://en.wikipedia.org/wiki/Submarine_communications_cable. 20. Mark Poster, The Mode of Information: Poststructuralism and Social Context (Chicago: University of Chicago Press, 1990), 129. 21. Johannes Fabian, Time and the Other: How Anthropology Makes Its Object (New York: Columbia University Press, 2014). 22. “A giant undersea cable makes the Internet a split second faster,” CNN Money (February 30, 2012). On remediated metaphors in the digital world, see Jay David Bolter and Richard Grusin, Remediation: Understanding New Media (Cambridge, MA: MIT Press, 2003).

…

So, too, were the years after Edward Snowden’s 2013 disclosures about NSA surveillance, which spurred many nations to start ocean-spanning cable projects, hoping to circumvent US networks. Undersea cable technology is not new; in the mid-nineteenth century, a global telegraph network depended on them. In the 1850s, undersea cables were made of copper, iron, and gutta-percha (a Malaysian tree latex introduced to the West by a colonial officer of the Indian medical service). A century later, they were coaxial cables with vacuum tube amplifier repeaters. By the late twentieth century, they were fiber optics. By 2015, 99 percent of international data traveled over undersea cables, moving information eight times faster than satellite transmission.19 Communication infrastructure has looked rather like tubes for a century and a half.

Internet for the People: The Fight for Our Digital Future
by Ben Tarnoff
Published 13 Jun 2022

Google Fiber: David Anders, “Whatever Happened to Google Fiber?,” CNET, March 5, 2021. 29, Content providers like Google … Own or lease more than half of undersea bandwidth: Adam Satariano, “How the Internet Travels Across Oceans,” New York Times, March 10, 2019. Google building its own undersea cables: Jameson Zimmer, “Google Owns 63,605 Miles and 8.5% of Submarine Cables Worldwide,” Broadband-Now, September 12, 2018. 30, The proponents of a … “Innovation, expanded …”: The “Framework for Global Electronic Commerce” is available at clinton whitehouse4.archives.gov. 30, How well does … Internet costs higher in US than in Europe or Asia: Becky Chao and Claire Park, “The Cost of Connectivity 2020,” New America, July 2020.

…

India shuts down the internet more than any other country; see Marc Daniel Davies, “Internet Shutdowns Plunged Millions into ‘Digital Darkness’ Last Year,” Bloomberg, March 3, 2021. 65, But state spying, censorship, and shutdowns … NSA tapping fiber-optic cables: Craig Timberg, “NSA Slide Shows Surveillance of Undersea Cables,” Washington Post, July 10, 2013, and Olga Khazan, “The Creepy, Long-Standing Practice of Undersea Cable Tapping,” The Atlantic, July 16, 2013. 65, The post office was the first … Postal confidentiality: Anuj C. Desai, “Wiretapping before the Wires: The Post Office and the Birth of Communications Privacy,” Stanford Law Review 60, no. 2 (2007): 553–94.

…

Many organisms in the abyssal zone glow through bioluminescence; the anglerfish is one of them. The octopus with no ink is the dumbo octopus, while the eel with the very large mouth is the pelican eel. x, There are many such arteries … MAREA: “MAREA,” Submarine Cable Networks, submarinenetworks.com; Winston Qiu, “AWS Acquires a Fiber Pair on MAREA Cable System on IRU Basis,” Submarine Cable Networks, January 21, 2019. x, MAREA is a reminder … “Civilization …”: John Perry Barlow, “A Declaration of the Independence of Cyberspace,” 1996. x, One way is infrastructural. “The contours …”: Nicole Starosielski, The Undersea Network (Durham, NC: Duke University Press, 2015), 2.

Krakatoa: The Day the World Exploded
by Simon Winchester
Published 1 Jan 2003

It took its time to make its way from the East Indies capital to its British equivalent because of the mechanics of early-nineteenth-century communication: it began its long journey by first going north of Java up to Singapore, and doing so by way of the newly made and wholly revolutionary invention, the submarine telegraph cable. There is a nice coincidence of geography and botany in the story of the building of submarine telegraph cables – a business that had only begun in 1850, when the first cable was laid between Dover and Calais. What allowed the industry to flourish – something that really only happened when cables ceased their infuriating habit of breaking in mid ocean, and they could then be relied upon to work and carry their signals without interruption – was the discovery of a handsome evergreen tree called Isonandra gutta, from which oozed a rubbery, workable and waterproof substance that was soon to be called gutta-percha.

…

And yet, as it happens, their observations, painstaking and precise as science demanded, collided head on with a most discomfiting reality: that while in 1883 the world was becoming ever more scientifically advanced, it was in part because of these same advances that its people found themselves in a strangely febrile and delicately balanced condition, which an event like Krakatoa did much to unsettle. The communications technology of the time, for example – the advances of telegraphy, the building of undersea cables, the flourishing of news agencies – ensured that the world's more advanced peoples learned about the eruption within moments of its happening. But at the time the limited, and only slowly unfolding, geological knowledge then to hand did not give the audience an explanation of the events that was sufficient to soothe their fears about all they were learning.

…

But if the operator marked the message Via Eastern, then it went swiftly and securely, beneath the sea. Thirty-five years before, the first insulated cable had been dropped from a ship called the Princess Clementine moored in Folkestone Harbour and connected to a boat two miles away, with a message successfully sent between the two. Since that time the undersea cable had become fixed in the public consciousness. Alfred, Lord Tennyson had written a hymn to the romance of the idea of coded voices hurrying along the ocean floor; and so had Rudyard Kipling, whose brief poem ‘The Deep-Sea Cables’ remains among his best loved: The wrecks dissolve above us; their dust drops down from afar – Down to the dark, to the utter dark, where the blind white sea-snakes are.

Black Code: Inside the Battle for Cyberspace
by Ronald J. Deibert
Published 13 May 2013

Whatever the cause, such cuts to cables are fairly routine: Even in their trenches, undersea cables are pushed to and fro by currents and constantly rub against a rough seafloor. In the case of the 2008 Mediterranean incident, the damage was severe: there were disruptions to 70 percent of Internet traffic in Egypt and 60 percent in India, and outages in Afghanistan, Bahrain, Bangladesh, Kuwait, the Maldives, Pakistan, Qatar, Saudi Arabia, and the United Arab Emirates. Nearly 2 million users were left without Internet access in the U.A.E. alone. Connections were not restored until a French submarine located the severed cables and brought them to the surface for repair.

…

It may come as a surprise that the same type of cables that Shakarian accidentally unearthed traverse the world’s lakes and oceans, and bind cyberspace together in a very material sense. Undersea cables are one of the links that connect today’s cyberspace to the late Industrial Revolution. The first such cables were laid in the late nineteenth century to facilitate telegraph traffic over long distances. Early designs were prone to failure and barely allowed the clicks of a telegraph exchange to be discerned across small bodies of water like the English Channel, but over time innovations in electronics and protective cable sheathings allowed the undersea cable industry to flourish. (This growth led to a dramatic increase in international telephone calls, and a new market for the sap of gutta-percha trees, which was used to coat and protect the cables until the mid-twentieth century.)

…

Although international telecommunications have been supplemented with microwave and satellite transmissions, a surprisingly large volume of data still traverses the world through cables crossing the Atlantic and Pacific oceans, and major bodies of water like the Mediterranean Sea. Due to the staggering costs involved, companies often share the same undersea cable trenches and sometimes competing companies even share the same protective sheathing. This makes those trenches highly vulnerable to major disruption. In a May 2012 article published on the website Gizmodo, provocatively titled “How to Destroy the Internet,” the author details the physical elements of the Internet that could be easily targeted.

Tubes: A Journey to the Center of the Internet
by Andrew Blum
Published 28 May 2012

Our encounters with this kind of geography typically come with more familiar images, like a ribbon of interstate, a length of train track, or a 747 parked expectantly at the airport gate. But undersea cables are invisible. They feel more like rivers than paths, containing a continuous flow of energy rather than the occasional passing conveyance. If the first step in visiting the Internet was to imagine it, then undersea cables always struck me as its most magical places. And only more so when I realized their paths were often ancient. With few exceptions, undersea cables land in or near classic port cities, places like Lisbon, Marseille, Hong Kong, Singapore, New York, Alexandria, Mumbai, Cyprus, or Mombasa.

…

But for Alston the acceleration came with the arrival of an unfathomably long and skinny thing, a singular path across the bottom of the sea. Undersea cables are the ultimate totems of our physical connections. If the Internet is a global phenomenon, it’s because there are tubes underneath the ocean. They are the fundamental medium of the global village. The fiber-optic technology is fantastically complex and dependent on the latest materials and computing technology. Yet the basic principle of the cables is shockingly simple: light goes in on one shore of the ocean and comes out on the other. Undersea cables are straightforward containers for light, as a subway tunnel is for trains.

…

Not every customer wants a link from Kenya to London, but once you can do it, and do it consistently and do it well, people begin to think about things like call centers, which are constantly hunting for the place with the lowest cost services. The demand springs up.” Undersea cables link people—in rich nations, first—but the earth itself always stands in the way. To determine the route of an undersea cable requires navigating a maze of economics, geopolitics, and topography. For example, the curvature of the planet makes the shortest distance between Japan and the United States a northern arc paralleling the coast of Alaska and landing near Seattle.

A Mind at Play: How Claude Shannon Invented the Information Age
by Jimmy Soni and Rob Goodman
Published 17 Jul 2017

Applying more power, amplifying messages, strengthening signals—Whitehouse’s solution to the telegraph problem—remained the most intuitive answer to noise. Its failure in 1858 discredited Whitehouse but not the outlines of his methods; few others were available. Yet there were high costs to shouting. In the best case, it was still expensive and energy-hungry. In the worst case, as with the undersea cable, it could destroy the medium of communication itself. Third, what hope there was of doing better lay in investigating the boundaries between the hard world of physics and the invisible world of messages. The object of study was the relationship between the qualities of messages—their susceptibility to noise, the density of their content, their speed, their accuracy—and the physical media that carried them.

…

Is there information in a message you can’t even understand? These were powerful questions in their own right. But in all the generations of human communication, those questions were posed with urgency and rigor just then because the answers had suddenly grown exceptionally valuable. In the profusion of undersea cables, transcontinental radio calls, pictures sent by phone line, and moving images passing through the air, our sudden skill at communicating had outstripped our knowledge of communication itself. And whether in disaster—a fried cable—or merely an inconvenience—the flicker and blur of the first televisions—that ignorance exacted its toll.

…

This was Shannon’s furthest-reaching find: the one Fano called “unknown, unthinkable,” until Shannon thought it. Until Shannon, it was simply conventional wisdom that noise had to be endured. The means of mitigating noise had hardly changed, in principle, since Wildman Whitehouse fried the great undersea cable. Transmitting information, common sense said, was like transmitting power. Expensively and precariously adding more power remained the best answer—shouting through the static, as it were, brute-forcing the signal-to-noise ratio by pumping out a louder signal. Shannon’s promise of perfect accuracy was something radically new.IV For engineering professor James Massey, it was this promise above all that made Shannon’s theory “Copernican”: Copernican in the sense that it productively stood the obvious on its head and revolutionized our understanding of the world.

Victorian Internet
by Tom Standage
Published 1 Jan 1998

His life's work, the transatlantic cable, was hailed as "the most wonderful achievement of our civilization." The cables were so profitable that Field was able to pay off all his debts in 1867. That year, when one of the two cables got crushed by an iceberg and stopped working, it was repaired within weeks. Before long, the recovery and repair of undersea cables was regarded as commonplace. Another banquet was held for Morse at Delmonico's in New York in December 1868, where he was toasted for having "annihilated both space and time in the transmission of intelligence. The breadth of the Atlantic, with all its waves, is as nothing." Echoing the sentiments expressed on the completion of the 1858 cable, a toast proposed by Edward Thornton, the British ambassador, emphasized the peacemaking potential of the telegraph.

…

It makes sorry reading and is all too convincing—most of the messages were along the lines of "CAN YOU RECEIVE ME?" and "PLEASE SAY IF YOU CAN READ THIS," as the operators at each end struggled to get the cable to work. The following year, another high-profile telegraphic venture, an attempt to build a submarine cable through the Red Sea to India funded by the British government, also ended in failure. This time, since it was public money that had been lost, there were widespread calls for a public inquiry. A joint committee was appointed, with four representatives from the Atlantic Telegraph Company and four chosen by the British government, including Professor Wheatstone.

…

The star witness, and the man who did more than anyone else to put submarine telegraphy on a firm scientific footing, was William Thomson, professor of natural philosophy at Glasgow University—and, by this time, the archrival of the Atlantic cable's designer, Dr. Whitehouse. Whitehouse had been conveniently ill and unable to go to sea to lay the failed cable, and it was Thomson who had kindly agreed to stand in for him, even though he had grave reservations about its design. He had already done a lot of theoretical work on the nature of submarine cables, and his measured, scientifically justified evidence to the public inquiry made mincemeat out of Whitehouse. Not only had Whitehouse made the conducting core too small, Thomson explained, but his use of high-voltage induction coils had gradually destroyed the cable's insulation and caused its eventual demise.

The Metaverse: And How It Will Revolutionize Everything
by Matthew Ball
Published 18 Jul 2022

Laying (or replacing) cables means working around extensive transportation infrastructure (freeways and railways), various population centers (each with their own political processes, constituencies, and incentives), and protected parks and lands. Laying a cable over a seamount in international waters is simple compared to laying a cable over a private-public mountain range. Image 1. Undersea Cables A map of the nearly 500 submarine cables and 1,250 landing stations that enable the global internet. TeleGeography The phrase “internet backbone” might bring to mind a largely planned out, and partly federated, network of cables. In truth, the internet backbone is really a loose federation of private networks. These networks were never laid to be nationally efficient.

…

In addition, most internet cables are not laid as the crow flies—they must navigate international rights, geographic impediments, and cost/benefit analyses. As a result, many countries and major cities lack a direct connection. NYC has a direct undersea cable to France, but not to Portugal. Traffic from the United States can go directly to Tokyo, but reaching India requires jumping from one undersea cable to another on the Asian or Oceanian continent. A single cable could be laid from the United States to India, but it would need to navigate through or around Thailand—adding hundreds or even thousands of miles—and that only solves shore-to-shore transmission.

…

See augmented reality (AR); virtual reality (VR) “mobile internet era,” 12, 35, 63, 64, 240, 242–44, 269, 291 Mojang, 114 Monsters University, 36, 89–90 Moore’s Law, 100, 161 “most favored nations” (MFN) clauses, 180–81, 183 Motorola, 158, 212, 213 multiplayer games battle royale games, 32–33, 91–93, 98, 114–15, 117n, 146–47, 268, 275 computing and, 75 “massively multiplayer” online games, 55 matchmaking, 81, 176, 178 the problem of concurrent users (CCUs), 54–55, 90–92, 122, 146, 234, 245, 261, 268, 283 see also specific games and gaming platforms Multi-User Dungeons (MUDs) and related topics, 7–8, 12, 13–14, 30 Musk, Elon Neuralink, 154 SpaceX, xi, 4, 87 Starlink, 87 Tesla, 101, 166, 271 X.com, 61 MySpace, 34, 274 Nadella, Satya, xii, 18, 141, 239, 279–80 Naked Sun, The, 5 Namco, 173 NASA, ix NASDAQ, 309 National Advisory Committee for Aeronautics, ix National Basketball Association (NBA), 218, 259 National Football League (NFL), 139 National Science Foundation, x native apps, 26, 194–95 near-field communication (NFC), 142, 189, 199–200, 203 Nest Labs, 158 “netcode” solutions, 81–82 NetEase, 19–20 Netflix, 19, 49–53, 96, 100, 111, 187, 194, 196–98, 244, 270, 276n Netherlands, 203 Network Attached Storage (NAS) drive, 74 networking, 71–88, 85 acceptability thresholds in, 80–81 bandwidth and, 27, 41, 48–51, 64, 72–79, 80, 96, 100, 305 broadband operators, 15–16, 38, 49, 271 fiber optic networks, 27, 84, 87, 128 4G networks, 81, 87, 244, 245, 249 5G networks, 87, 243 the internet as a “network of networks,” 16, 23–24, 62 latency and, 27, 48–53, 64, 79–88, 95–96, 99–100, 230, 243, 248, 271, 305 network gateways, 130 rent-seeking and, 15, 299 3G networks, 243 undersea cables, 84–85, 85 Neuralink, 154 Neuromancer, 5–6, 8 Newsweek, 308 New World, 277 New York Times, xv, 4–5, 73, 224, 256 as an app, 149, 185, 194, 196, 202 Live Election feed, 49 Nexon, 105 NextVR, 144 Niantic, 115, 144, 275 Nike, 121, 139, 189–90, 208, 248, 264 Nintendo, 11, 104, 151–52, 173, 303 Nintendo Entertainment System (NES), 32, 173 Nintendo Switch, 30–32, 75, 97, 134, 174–76 Nintendo Wii, 132 Nokia, 136–37 non-fungible tokens (NFTs), 140, 216–22 avatars and identity, 218, 229, 293–94 fractionalization into fungible tokens/shares, 202 marketplaces, 202, 301 “minting” in, 217 speculation, 128, 220, 231 as useless, 220 nonprofit organizations, 227, 231 Nouns DAO, 229 Nuance Communication, 212 Nvidia, xii, 66, 97–98, 282 Android and, 213 GeForce Now, 131, 282, 286 market capitalization of, 166 NVM software development kit, 175 Omniverse, 136–37, 282 see also graphics processing units (GPUs); Huang, Jensen Nye, David, 242 “Oasis, The,” 22, 289 OBJ file format, 299 occlusion, 97 Oculus VR, xi, 21, 57, 143, 153, 160, 274, 276 Horizon Worlds, 115, 204, 277 Oculus Quest 2, 143, 145–47, 161–62 Population: One, 146–47, 268 Office of Scientific Research and Development (OSRD), ix–x OKCupid, 19, 261 Omniverse from Nvidia, 136–37, 282 OnLive!

The Snowden Files: The Inside Story of the World's Most Wanted Man
by Luke Harding
Published 7 Feb 2014

The shared goal among the US’s disgruntled allies was to make it harder for the NSA to get access to national data. For authoritarian countries such as Russia, there was an added bonus. Greater state control of the internet made it easier to snoop on their own citizens and keep a lid on dissent. The most vociferous reaction came from Brazil. In October, Rousseff announced plans to build a new undersea cable linking South America with Europe. This would, in theory, shut out the US and make it harder for the NSA to siphon off Brazilian information. The president also mulled over legislation that would force Google and other US tech giants to store the data for Brazilian users on local servers. Thousands of federal workers, meanwhile, were ordered to adopt a form of highly encrypted email.

…

They are called Apollo North, TAT-8, TAT-14 and Yellow/Atlantic Crossing-2, also known as AC-2. Other transatlantic cables come ashore at nearby Land’s End. Thousands of miles long, the fibre-optic cables are operated by big private telecoms firms, often in consortia. The landing points of these submarine cables are so important that the American Department of Homeland Security lists them as critical American national infrastructure (according to leaked US diplomatic messages). In this new world of internet-driven communications, Britain’s position on the eastern edge of the Atlantic makes it a hub.

…

As much as 25 per cent of the world’s current internet traffic crosses British territory via the cables, en route between the US, Europe, Africa and all points east. Much of the remaining traffic has landing or departure points in the US. So between them Britain and the US play host to most of the planet’s burgeoning data flows. Unsurprisingly, given their history, both countries’ spy agencies wanted to exploit their good luck and tap into all these submarine cables in order to eavesdrop. As technology changed, the two organisations had successively intercepted radio traffic, then microwave beams and ultimately satellite links. It was logical to seek now to break into the floods of internet and phone data which were travelling by the latest fibre-optic systems.

Atlantic: Great Sea Battles, Heroic Discoveries, Titanic Storms & a Vast Ocean of a Million Stories
by Simon Winchester
Published 27 Oct 2009

In the middle of the century, a small group of entrepreneurs wondering how to speed up transatlantic messaging began focusing on this island—and it did so because Newfoundland offered the closest points in North America to Europe: it was a mere 1,600 miles from the harbor opening at St. John’s to the cliffs of Connemara. An undersea cable—for such had already been invented in Europe: a telegraph cable had been laid between Britain and France in August 1850, and soon afterward others were lowered into the seas between Britain and Holland, and between Scotland and Ireland—could perhaps be laid across the Cabot Strait, at the mouth of the St.

…

Wyville, 136, 139 Thoreau, Henry David, 305 Thorfinsson, Snorri, 83–84 Tides, 112, 114, 121 Tierra del Fuego, 125–26, 128, 208–11, 444 Times Atlas of the Oceans (book), 325–26 Tin, 65–66 Titanic (ship), 254, 263, 321–23 Tobacco, 84n, 90 Tomlinson, Charles, 205 Toothed whales, 285n Toothfish (Chilean sea bass), 362–63, 378, 384–86, 389 Topography, 121 Torrey Canyon (ship), 325–26 f Tor Rocks, 7–8 Toxic wastes, 357–58 Tracks, shipping, 348–49 Trade Hanseatic League and rules of, 275–80 Phoenician, in Mediterranean Sea area, 62–69 routes, 319 f Trade winds, 121, 424 Trafalgar Battle, 243–46, 251 Tramp ships, 295–96 Travailleur (ship), 141 Trawling, 368, 370 Treaty of Paris, 366 Treaty of Tordesillas, 93 Trial of Black Bart’s Men, 227–30 Triana, Rodrigo de, 86–87 Triangular slave trade, 232–39 Triassic period, 40 Triple Divide Peak, 147–48 Triremes (ships), 211 Tristan da Cunha, 112, 183n, 437–38 Tristan und Isolde (opera), 194 Tropical cyclones, 423. See also Hurricanes Tsunamis, 432–36. See also Seismic activity Tuna, 363 Turbidity currents, 434 Turner, J. M. W., 197–98 Tuvalu Islands, 412n Typhoon (book), 205 Typhoons, 423n U-boats, German, 261–66, 268–70 Uluburun archaeology site, 64–65 Undersea cables. See Cables, undersea Undersea oil exploration, 403–4 Undertow (painting), 198–99 Union Castle passenger liners, 186 United Nations, 176, 372n United States air transport to and from, 330–32 British Post Office service to, 291–93 Civil War of, 247–50 commercial whaling of, 287–90 literature of, 200–203 motivations of, for Atlantic Ocean exploration, 123–24 Naval Observatory, 130 rivers as Atlantic Ocean sources, 147–48 slave trade and, 221, 230–39 steel ships and, 257 transatlantic cable and, 305–10 War of 1812 of, 246–47 United States Exploring Expedition, 126–29 Ur supercontinent, 38 Uruguay, 257–60 van de Velde, Willem, 166 vaz de Torres, Luis, 112 Venezuela, 87–88 Verdi, Giuseppe, 194 Verne, Jules, 128, 445 Vernet, Claude, 167 Vespucci, Amerigo, 93–96 Viarsa (ship), 389 Victoria, Queen, 309 Victoria Falls, 47 Victory (ship), 243 Vikings, 70–71, 75–85, 89–90, 212–15, 282.

…

In the Fourth Age—that of the soldier—I could tell of the arguments and conflicts that have so often roiled the ocean, of how the force of arms over the years has compelled migration or fostered seaborne crime, of how national navies have reacted, how individual battles have been fought, and how Atlantic heroes have been born. In the Fifth Age—that of the well-fed Justice—I could describe how the sea eventually became a sea of laws and commerce, and how tramp steamers and liners and submarine cables and jet aircraft then crossed and recrossed it in an infinite patchwork designed for the attainment of profit and comfort. In the Sixth Age, that dominated by the fatigue and tedium of the pantaloon, I could reflect upon the ways that man has recently wearied of the great sea, has come to take it for granted, to become careless of its special needs and to deal with it improvidently.

The Costs of Connection: How Data Is Colonizing Human Life and Appropriating It for Capitalism
by Nick Couldry and Ulises A. Mejias
Published 19 Aug 2019

On a regular day, Netflix and YouTube account for half of all internet traffic in the United States, with massive implications for their ability to “buy” and distribute content for money or otherwise.21 The one-to-many model of mass media dissemination has simply been replaced by a many-to-one model, not the democratic many-to-many model we once hoped for.22 The monopoly-monopsony hybrids we are describing represent massive concentrations of power. These companies currently exercise a hold over the infrastructure that makes social quantification possible; they control everything from undersea cables to satellites to the “last mile” architecture that delivers internet service to individuals. They also control the environments or platforms in which data is being generated, the design and production of the devices through which data is collected, and the computing capacity necessary to analyze the data, including through increasingly sophisticated machine learning methods that do not require explicit human intervention.

…

Cultural Studies 31, nos. 2–3 (2017): 185–204. Mignolo, Walter D. The Darker Side of Western Modernity. Durham, NC: Duke University Press, 2011. . “Delinking: The Rhetoric of Modernity, the Logic of Coloniality and the Grammar of De-Coloniality.” Cultural Studies 21, nos. 2–3 (2007): 449–514. Miller, Ron. “Google’s Latest Undersea Cable Project Will Connect Japan to Australia.” Techcrunch, April 4, 2018. Mirowski, Philip. Machine Dreams. Cambridge, UK: Cambridge University Press, 2002. Mitchell, Stacy. “Amazon Is Trying to Control the Underlying Infrastructure of Our Economy.” Motherboard, June 25, 2017. https://motherboard.vice.com/en_us/article/7xpgvx/amazons-is-trying-to-control-the-underlying-infrastructure-of-our-economy.

…

As the planet became crisscrossed with an infrastructure that continues to account for 80 percent of global information flows today, the geography of the cable system replicated the mercantile routes established during historical colonialism,38 and colonial maps continued to exert their influence. For example, Britain forced companies that operated transatlantic submarine cables to use London as their connection hub (after World War I, the epicenter for global communications shifted from England to the United States).39 Ownership of the new communication infrastructure continued to follow colonial models as well by concentrating it in the hands of a few multinational consortia of state and private-sector corporations.

2034: A Novel of the Next World War
by Elliot Ackerman and James Admiral Stavridis
Published 15 Mar 2021

“Perfect,” said Kolchak as he stared at their thrashing dorsal fins. As if sensing Farshad’s confusion, he explained himself. “Those sharks are heading in the direction of the 10G undersea cables. They’re attracted to the electromagnetic energy. Those cables connect to the United States, and sharks have been known to chew through them. Their presence will give us deniability.” Destroying a few of the undersea cables would send a powerful message to the Americans, slowing internet across the country by as much as 60 percent, or so Farshad had been told by Kolchak. This might be enough to de-escalate the crisis, to bring everyone to their senses.

…

Equally obvious was the opportunity presented by its absence. No fewer than five hundred fiber-optic cables, which accounted for 90 percent of North America’s 10G internet access, crisscrossed these icy depths. “If the Americans detonate a nuclear weapon,” said Kolchak, “I don’t think the world will much care if we tamper with a few undersea cables.” He held Farshad in his gaze. “I also don’t think the world would say much if our troops seized a sliver of Poland, to unite Kaliningrad to the Russian mainland.” Kolchak pointed to a map on the wall. He traced out a corridor with his finger, which would give Russia direct overland access to its one Baltic port.

…

He would imagine the positions taken by Trent Wisecarver, by the various service chiefs and cabinet secretaries, the tabulations of arguments for or against what they were about to do—what they had all committed themselves to do when the president had put down her “red line” and dared her counterparts in Beijing to cross it. Which was what it seemed Beijing had now done, though not in the way anyone had anticipated. The cutting of the undersea cables and the resulting plunge into darkness was the demonstrable fact that, when discussed around the conference table, proved Beijing had crossed the red line. The question was the response. And even that was settled in remarkably short order. Chowdhury envisioned the scene—a disquisition of US interests by Wisecarver, followed by a range of options (or lack thereof) presented by the Joint Chiefs, and then formal nuclear authorizations being granted by the president herself.

The System: Who Owns the Internet, and How It Owns Us
by James Ball
Published 19 Aug 2020

From there, it makes its final stop – to a server in a Twitter data centre in Kansas, from where I can get the website I would like to see. This connection is pretty simple – most of the stops along the way actually belong to the company I pay for my internet. But Twitter doesn’t actually own any of the undersea cables across the Atlantic, so is paying someone for its use, meaning that this route shows at least some money must change hands. That’s not an unreasonable proposition, as building and maintaining the transatlantic cables is something of an astonishing task. The cables are smaller than you might think – about the width of a hosepipe for much of their distance, and most of that is just armour for the tiny fibre-optic core – but need to be laid, maintained and monitored.

…

noredirect=on&utm_term=.e7adba67bfe6 3https://www.bbc.co.uk/news/business-42745853 1 THE ARCHITECTS 1US broadband speed taken from http://fortune.com/2017/06/02/internet-speed-akamai-survey/ 2The narrative of the first internet message is taken from this (charming and very readable) transcript: https://archive.icann.org/meetings/losangeles2014/en/schedule/mon-crocker-kleinrock/transcript-crocker-kleinrock-13oct14-en.pdf 3https://www.internethalloffame.org//inductees/steve-crocker 4https://ai.google/research/people/author32412 5Wired have a great feature with much more detail on ‘the mother of all demos’ here: https://www.wired.com/2010/12/1209computer-mouse-mother-of-all-demos/ 6This was the recollection of Bob Taylor, who secured the funding (https://www.computer.org/csdl/magazine/an/2011/03/man2011030004/13rRUxly9fL), but was disputed by Charles Herzfeld, who said he had agreed the funding, but had taken more than twenty minutes’ persuasion (https://www.wired.com/2012/08/herzfeld/). 7Full video and transcript: http://opentranscripts.org/transcript/steve-crocker-internet-hall-fame-2012-profile/ 8This is also from Kleinrock’s 2014 transcript: https://archive.icann.org/meetings/losangeles2014/en/schedule/mon-crocker-kleinrock/transcript-crocker-kleinrock-13oct14-en.pdf 9https://www.ietf.org/rfc/rfc0675.txt 10ARPANET had operated as a packet switching network from its inception – TCP is just a specific implantation of the concept, and the one which came to be the standard. 11This paragraph borrows key dates from https://www.webfx.com/blog/web-design/the-history-of-the-internet-in-a-nutshell/ 12Everything from Steve Lukasik comes from his paper ‘Why the ARPANET Was Built’, published online here: https://www.academia.edu/34728504/WHY_THE_ARPANET_WAS_BUILT 13This is from the Crocker/Kleinrock discussion. 14https://webfoundation.org/about/vision/history-of-the-web/ 15These are sourced to the Internet Services Consortium, but most easily viewed on Wikipedia https://en.wikipedia.org/wiki/Global_Internet_usage 16https://www.bbc.co.uk/news/technology-32884867 17https://www.statista.com/statistics/471264/iot-number-of-connected-devices-worldwide/ 18This stat comes from TeleGeography (https://www2.telegeography.com/submarine-cable-faqs-frequently-asked-questions) – their map of the main undersea internet cables is well worth a look: https://www.submarinecablemap.com/ 2 THE CABLE GUYS 1http://www.washingtonpost.com/wp-dyn/content/article/2007/10/17/AR2007101702359.html?hpid=artslot 2All of Eliason’s comments in this chapter are from an interview with the author. 3Ofcom data is available here: https://www.ofcom.org.uk/__data/assets/pdf_file/0022/145525/comparing-service-quality-2018.pdf 4Calculated from ARPU (average revenue per internet user), sourced here: https://www.nscreenmedia.com/comcast/ 5This tale is contained within this Wired profile of Brian Roberts and Comcast: https://web.archive.org/web/20130107032943/http://www.wired.com/techbiz/people/magazine/17-02/mf_brianroberts?

Time Lord: Sir Sandford Fleming and the Creation of Standard Time
by Clark Blaise
Published 27 Oct 2000

In the 1870s and early 1880s he orchestrated world support for standard time, and in 1884 he succeeded. Still later, and at great personal cost, he bravely rallied the informal commonwealth of overseas dominions against British communications monopolies, and succeeded in laying the worldwide, undersea cable. A thousand miles east of Toronto, in 1851, the Railway Minister of the Nova Scotia colony, Joseph Howe, ventured a prophecy that is almost eerie in its accuracy. Fleming heard it as well, and filed it for later use. Fleming and Howe would not meet for another thirteen years, by which time Howe’s obvious capacity for leadership would carry him to the top of Nova Scotia’s parliamentary ranks.

…

What did those new “social” sciences prove but that man behaved irrationally, and that one of the vaunted Victorian articles of faith, “character,” derived as much from secret drives, repressions, or unconscious influences as from sturdy moral and social exempla? After the standardtime issues had been settled, Fleming spent the next twenty years perfecting a world-circling undersea cable that would link England instantaneously with all her distant colonies. He believed (and how could he not?) that intimate and immediate contact would spawn greater knowledge, loyalty, and affection between the mother of parliaments and her satellites. The better we know our government, the deeper our respect.

…

Rivers were crossed, ships’ designs turned from wood and sail to steel and iron, hold capacities and passenger cabins expanded a hundredfold, with a need to fill their holds with thousands of tons of coal for ocean passage. Meticulous planning and astronomical start-up costs entered the calculation for any new enterprise, and London and Continental banks oversaw bond issues for projects that might have seemed fanciful only a generation earlier: undersea cables, new shipyards, new steel mills, new mining equipment; transcontinental railroads spanning Canada, South Africa, America, India; telegraphs down the African coast. The new technology ran on coal, on more coal than traditional methods could ever extract. Workers had to be at least semiskilled just to handle the demands of the new technology, and, in the end, many grew sufficiently confident to challenge ancient wisdom, or to suggest shortcuts to greater efficiency.

The Perfect Weapon: War, Sabotage, and Fear in the Cyber Age
by David E. Sanger
Published 18 Jun 2018

The diagram didn’t specify exactly how the NSA was planning on getting between those servers, but there were only a few possible options. The NSA would have to hack in remotely from one of its bases around the world, physically tap the undersea cables themselves, or get cooperation from a foreign partner, such as the British. The most likely method was physically tapping the termination points in a country where the undersea cables came ashore. And since Google had not gotten around to encrypting the data that was “in transit” through these cables, merely getting into the network itself was the price of admission to the data. When the Washington Post first published the slide, on October 30, 2013, over four months after the first Snowden revelations, the reaction inside the Googleplex in Mountain View was immediate and predictable.

…

The slides explaining Shotgiant in the Snowden trove gave a sense of the NSA’s thinking: “If we can determine the company’s plans and intentions,” an analyst wrote, “we hope that this will lead us back to the plans and intentions of the PRC.” The NSA saw an additional opportunity: As Huawei invested in new technology and laid undersea cables to connect its networking empire, the agency was interested in tunneling into key Chinese customers, including “high priority targets—Iran, Afghanistan, Pakistan, Kenya, Cuba.” In short, eager as the NSA was to figure out whether Huawei was the PLA’s puppet, it was more interested in putting its own back doors into Huawei networks.

…

In the open cubicles that looked out on the old air base, Moffett Field, in the heart of what is now Silicon Valley—the remnant of a pre–World War II age when the airplane was remaking global power—Grosse and his team of engineers were working day and night to NSA-proof the Google systems. The project, Grosse said, had actually begun long before they saw the smiley-face diagram. As early as 2008, Google had been investing in consortiums that laid undersea cables. But sharing had its risks: The company was not fully in control of who else had access to the cables, and there was always the risk that Google’s traffic could be thrown off the lines during an emergency, leaving its users without access to their data. Less than a decade later, Google had undertaken a multibillion-dollar effort to put down its own fiber-optic lines across the Atlantic and the Pacific so that it could control the speed and reliability of information that flowed between its servers and its users.

Kitten Clone: Inside Alcatel-Lucent
by Douglas Coupland
Published 29 Sep 2014

Eight men, working six months on/six months off, walking in clockwise circles again and again and again, each successive layer only two thirds of an inch higher than the one below it. It’s about as entrancing as watching insolent teenagers chew gum, but completely necessary. The Île de Bréhat’s undersea cable will be deployed on the bottom of the Atlantic Ocean at a rate of up to seven kilometres per hour, and it will take seven months for all of it to be laid. The ocean is as hilly and mountainous as the world above the water, and a one-kilometre length of cable might cover only seven hundred metres on a map.

…

Marine cable names, when viewed together, somewhat resemble a catalogue of extremely high-end yachting gear and sonar equipment: ACS Alaska-Oregon Network (AKORN) Alonso de Ojeda ALPAL-2 American Samoa-Hawaii (ASH) Antillas 1 Asia-America Gateway Australia-Japan Cable Bahamas Domestic Submarine Network Baltica Batam-Rengit Cable System Canada-United States 1 Cayman-Jamaica Fiber System Challenger Bermuda-1 (CB-1) Colombia-Florida Subsea Fiber (CFX-1) Denmark-Sweden 17 East African Marine System Emerald Bridge Estepona-Tetouan Fehmarn Bält Fiber Optic Gulf Fibralink Finland Estonia Connection Gemini Bermuda Geo-Eirgrid GO-1 Mediterranean Cable System Gondwana-1 Greenland Connect Guam Okinama Kyushu Incheon (GOKI) Hibernia Atlantic Hokkaido-Sakhalin Cable System Kattegat 1 Kodiak Kenai Fiber Link Latin American Nautilus Latvia-Sweden 1 (LV-SE 1) Lower Indian Ocean Network (LION) North Asia Loop Pacific Fibre Pangea Baltic Ring Pangea North Saudi Arabia-Sudan-2 (SAS-2) SeaMeWe-4 Svalbard Undersea Cable System Tangerine Tasman-2 Tata TGN-Pacific Turcyos-2 Ulysses Yellow/Atlantic Crossing-2 (AC-2) Near land, most marine cables are laid inside deep trenches cut by deep-water plows. Once on land, the cables travel as unobtrusively as possible to the bland, anonymous-looking routing centres mentioned earlier, mostly in London, New York, Frankfurt, Amsterdam, suburban Virginia, Palo Alto, California and Singapore.

New Dark Age: Technology and the End of the Future
by James Bridle
Published 18 Jun 2018

The cloud shapes itself to geographies of power and influence, and it serves to reinforce them. The cloud is a power relationship, and most people are not on top of it. These are valid criticisms, and one way of interrogating the cloud is to look where its shadow falls: to investigate the sites of data centres and undersea cables and see what they tell us about the real disposition of power at work today. We can seed the cloud, condense it, and force it to give up some of its stories. As it fades, certain secrets may be revealed. By understanding the way the figure of the cloud is used to obscure the real operation of technology, we can start to understand the many ways in which technology itself hides its own agency – through opaque machines and inscrutable code, as well as physical distance and legal constructs.

…

6 The class and social conflicts, the historical hierarchies and injustices, that Woolf alludes to in her processions and ceremonies have in no measure abated today, but some of the places to think them may have changed. The crowds that in 1938 lined London’s Lord Mayor’s and coronation parades are now distributed through the network, and the galleries and places of worship have likewise migrated into data centres and undersea cables. We cannot unthink the network; we can only think through and within it. And we can listen to it, when it tries to speak to us in an emergency. Nothing here is an argument against technology: to do so would be to argue against ourselves. Rather, it is an argument for a more thoughtful engagement with technology, coupled with a radically different understanding of what it is possible to think and know about the world.

…

Prometheus – pro-metheus – is foresight, but without the wisdom we might take to accompany it. It’s anticipation. It’s the white heat of scientific and technological discovery, and that desire for the oncoming rush of the future, the head-down drive of forward movement. It’s resource extraction, fossil fuels, undersea cables, server farms, air conditioning, on-demand delivery, giant robots, and meat under pressure. It’s scale and subjugation, the pushing back of the darkness with little thought for what’s beyond – for who already lives there or who gets crushed along the way. The illusion of knowledge and the anticipation of mastery combine to impel the timeline of progress, but they obfuscate the absence of understanding at its articulation point: the zero mark, the dark present, where we see and comprehend nothing beyond movement and efficiency, where our only possible act is to accelerate the existing order.

@War: The Rise of the Military-Internet Complex
by Shane Harris
Published 14 Sep 2014

In exchange for the federal government granting the company a license to operate in the United States, they had to sign a contract that guaranteed US intelligence agencies uninterrupted access to the networks, so that phone calls could be logged and recorded. One agreement, with Level 3 Communications, even included a “kill switch” provision, stating that if ever directed by the US government, the company must be able to immediately sever all communications traveling through its undersea cables into the United States. This was a protective measure, meant to block the network from delivering malicious software or traffic in the event of a cyber attack. In some cases a foreign communication could be intercepted from inside the United States. (In fact, this kind of capture was routine for e-mail traffic, much of which flowed through United States–based cables and routers.)

…

And with access to the systems of major e-mail and Internet companies, the NSA could gather more intelligence about its adversaries and craft messages that looked trustworthy but were actually loaded with viruses and other malware. The Internet was a battlefield, and the new law gave the NSA more ways to enter it. As the NSA’s powers grew, it cast its net wider, tapping into the undersea cables that carry communications between continents. The agency started filtering the content of all e-mails going in and out of the United States, scanning them for the names, phone numbers, or e-mail addresses of suspected terrorists. And it managed to penetrate the defenses of Google and Yahoo, stealing communications as they traveled between the companies’ overseas private data centers and the public Internet.

…

See Tailored Access Operations Target, [>], [>], [>] TASC, [>]–[>] Team Themis, [>]–[>] telecommunications companies, [>]–[>], [>], [>], [>]–[>], [>], [>], [>]–[>], [>]. See also specific companies telecommunications networks: DOD, [>]; fiber-optic trunk lines, [>], [>], [>]; Google, [>]; kill switch provision, [>]; right to spy in exchange for US license, [>]; Skype, [>]; undersea cables, [>], [>] Telvent, [>]–[>], [>] terrorism. See counterterrorism; 9/11 attacks threat signature, [>], [>], [>], [>], [>], [>]–[>], [>] thumb drives, [>], [>], [>], [>], [>]–[>], [>], [>] Tier [>], [>] tippers, within communications data, [>] Tiversa, [>]–[>] Tor, [>]–[>], [>], [>], [>] Tranche 2 plan, [>]–[>], [>] Transportation Security Administration, [>] transportation system.

An Empire of Wealth: Rise of American Economy Power 1607-2000
by John Steele Gordon
Published 12 Oct 2009

Dickens, born in 1812, the most popular novelist of his time, never mentioned the railroads and telegraph that so characterized the new economic world he lived in. Chapter Nine CHAINING THE LIGHTNING OF HEAVEN THE SPEED OF TRAVEL was not the only thing accelerating in the early nineteenth century. So was the speed of communications. It is hard to imagine today, when satellites and undersea cables keep every part of the globe in instant communication with every other part, just how slowly news spread in the eighteenth century. The battles of Lexington and Concord, the opening events of the American Revolution, occurred on Wednesday, April 19, 1775. But news of the events reached New York only on Sunday, April 23, and Philadelphia on April 24.

…

The next day volume on the New York Stock Exchange reached 1.3 million shares, its highest since the panic of 1907, and many leading stocks fell more than 20 percent. On Friday, July 31, the London Stock Exchange closed for the first time in its history, and New York was the only major exchange scheduled to open the next day. It had no real choice except to close as well. With the world’s markets now tied together with a cat’s cradle of undersea cables, sellers converged on New York, and sell orders began piling up in mountains awaiting the Saturday opening. (The New York Stock Exchange would have a Saturday morning session until after the Second World War.) The governors voted to close the exchange, and the president of the exchange consulted J.

…

Geopositioning satellites have made it possible to determine, with the help of a simple device, one’s location within a few feet. The devices are now appearing in many automobiles, giving directions through the use of synthetic voices, a technology that would have seemed utterly miraculous only a couple of decades ago. Communications satellites, together with an ever-increasing number of undersea cables, have helped greatly to lower the cost of long-distance telephony, leading to an astonishing upsurge in its use. In 1950 about a million overseas phone calls were initiated in the United States. By 1970 the number had grown to twenty-three million; by 1980 to two hundred million. By 2001, as the cost plummeted, the number was 6.3 billion and rising fast.

Cuckoo's Egg
by Clifford Stoll
Published 2 Jan 1989

“You mean that people prefer cable to satellite links?” “Sure. Every time you connect through a satellite, there’s a quarter second delay. The undersea cables don’t slow down your messages so much.” “Who would care?” “People on the telephone, mostly,” Steve said. “Those delays make for jittery conversations. You know, where each person tries to speak at the same time, then they both back off.” “So if the phone companies try to route over the cables, who wants the satellites?” “Television networks, mostly. TV signals can’t be squeezed into submarine cables, so they grab the satellites. But fiber optics will change everything.” I’d heard of fiber optics.

The Great Firewall of China
by James Griffiths;
Published 15 Jan 2018

The ones most people will be familiar with is when a personal modem isn’t working and the request never leaves the house, or (less often today than years ago) the ISP’s servers are down and the request can’t be converted into an IP address. But disruption can also happen at the other end of the equation. In 2008, millions of users across the Middle East were cut off from servers in the US when several undersea cables were damaged by unknown causes.53 While that outage was accidental, the Xinjiang internet blackout the following year showed that the same effect could be achieved intentionally. China is particularly vulnerable in this regard, as all internet backbones and major exchanges are owned or controlled by government entities.54 China Unicom, the colossal state-owned telecoms company that administers much of the country’s internet infrastructure, has three major ‘national-level’ exchanges in Beijing, Guangzhou and Shanghai,55 from where traffic is routed to the rest of the world.

…

Jesse still travels to Xinjiang, and for his own security and that of his family requested that he not be identified. 50My description of the functioning of the internet is largely drawn from Res Shuler’s 2002 Stanford white paper ‘How does the internet work’ and from TeleGeography’s maps of the physical internet. 51Submarine cable map at http://www.submarinecablemap.com/#/submarine-cable/asia-america-gateway-aag-cable-system 52See http://visitslo.com/ 53H. Timmons, ‘Ruptures call safety of internet cables into question’, International Herald Tribune, 4 February 2008, https://web.archive.org/web/20080206083645/http://www.iht.com/articles/2008/02/04/technology/cables.php 54D.

Seasteading: How Floating Nations Will Restore the Environment, Enrich the Poor, Cure the Sick, and Liberate Humanity From Politicians
by Joe Quirk and Patri Friedman
Published 21 Mar 2017

If seaweed can clean the tiniest proportion of these zones, producing food or fuel at a profit, it can be scaled up to colossal proportions. What works in microcosm can work better in macrocosm. That’s the magic of profits. It’s how one oil well becomes a million oil wells, one lightbulb becomes a trillion lightbulbs, and one voice carried along a wire to the next room becomes a planet crisscrossed with telegraph lines and undersea cables. Ricardo cites an oft-repeated fact in his academic paper: “Up to 90 per cent of wastewater flows untreated into densely populated coastal zones contributing to growing marine dead zones.” But he offers a unique solution: “[I]t is logical . . . to take advantage of these nutrient-rich water sources for production, as well as cleaning them through ‘bioremediation.’ ” Bioremediation is sciencespeak for the process of culturing microorganisms to metabolize the resources that humans consider pollution.

…

Article 2 The high seas being open to all nations, no State may validly purport to subject any part of them to its sovereignty. Freedom of the high seas is exercised under the conditions laid down by these articles and by the other rules of international law. It comprises, inter alia, both for coastal and non-coastal States: (1) Freedom of navigation; (2) Freedom of fishing; (3) Freedom to lay submarine cables and pipelines; (4) Freedom to fly over the high seas. * * * “Because international law promulgated by the United Nations [which codifies the major principles of international relations] addresses only state actors and TSI [The Seasteading Institute] is a nonstate actor, this Note argues that international law does not prohibit the seastead communities from merely existing in international waters before they pursue their ambitions for international recognition . . .

Tesla: Inventor of the Electrical Age
by W. Bernard Carlson
Published 11 May 2013

Unable to make his case fully in person, Tesla followed up each encounter with letters. In the first, Tesla built a case for developing his wireless technology by pointing out that it could compete effectively with the transatlantic telegraph cables. Knowing that Morgan relied on messages sent between his New York and London offices by undersea cable, Tesla reported that he was now able to manipulate electrical pressures of a hundred million volts and hundreds of thousands of horsepower of electrical energy so that it was no longer necessary to rely on those “long and expensive cables” to send messages. Implicitly citing what he had learned by observing the stationary waves set up by thunderstorms in Colorado, Tesla assured Morgan that “Long practical experience with apparatus of this kind and exact measurements embracing a range of nearly seven hundred miles, enable me to construct plants for telegraphic communication across the Atlantic and, if need be, across the Pacific Ocean, with the fullest measure of success.”

…

Marconi outlined the achievements of his system so far—seventy ships equipped with wireless and twenty shore stations in England. He described his experiments in Newfoundland, including the problems he had faced with flying kites in the wintry weather. After mentioning that he hoped that wireless telegraphy would allow for cheaper messages than the existing undersea cables, Marconi concluded by raising a glass and toasting the institute. Marconi’s speech was followed by remarks from Thomson and Professor Michael Pupin of Columbia University. Both men emphasized that although the proof for Marconi’s achievement was limited, they accepted Marconi at his word since they knew and trusted him.

…

See also invention Creusot Works, 258 Croatia, 28, 145; Gospíc, 21, 25–32, 47, 154–55, 426n27; Lika province of, 13–21, 32–33, 239, 380; Napoleon and, 14–15; Smiljan, 3, 5, 13, 16–17, 21, 69; soil of, 13–14 Crocker, Francis B., 134 Crookes, William, 29, 126, 145–55, 176, 204, 313 Crookes radiometer, 29, 151, 199 Crookes tubes, 221–22, 249 cryophorus, 82 Cunningham, David, 66 Curtis, Leonard E., 264, 267 Czito, Julius C., 371 Czito, Kolman, 371 Dana, Charles A., 217 da Vinci, Leonardo, 6 Dean, Bashford, 317 deBobula, Titus, 386 Defense Advanced Research Projects Agency (DARPA), 394 de Forest, Lee: background of, 348; demonstrations and, 350; disruptive innovation and, 402; Marconi and, 334–35; Morgan and, 250–51; physics and, 348; radio and, 149, 285; telegraph and, 349, 438n29; telephone and, 348–49; Tesla’s birthday and, 380; Tesla’s legacy and, 403; transmitters and, 350; vacuum tubes and, 149; White and, 349–51; wireless communication and, 285, 303, 334, 348–50, 380, 402–3, 438n29 de Laval, Gustaf, 371 Delmonico’s, 2–3, 5, 186, 194, 198, 201–2, 256–57, 260, 410, 451n34 delta connection, 441n4 demonstrations: Adams and, 166, 205, 219; AIEE and, 116; alternating current (AC) and, 1–2, 78, 85, 89–92, 96, 144–45, 160, 166, 172–73, 215; Anthony and, 116; Bjerknes and, 117; broadcasting power and, 341–42; Brown and, 85, 90–91, 102, 116; Byllesby and, 111; Chicago World’s Fair and, 160–62, 172–73, 182–83, 188; Clarke and, 233; Columbia College lectures and, 133–38, 140, 143; de Forest and, 350; dramatic, 1–2, 90–91, 125–26, 128, 135–38, 144–45, 148, 150–53, 177, 190, 219, 226, 230, 332–33, 409; duplex and, 78; dynamos and, 41; Edison and, 78; egg of Columbus and, 90–92, 103, 172, 257, 407; electric brush phenomena and, 148–49; European shows and, 143–57; Faraday and, 36–37, 53; Fontaine and, 41, 45; Geissler tubes and, 126–28, 132, 135, 221–22, 249; Helmholtz and, 188; high-frequency phenomena and, 138–39; Jacob’s Ladder and, 127; Kerr and, 111; London lectures and, 146–52; Madison Square Garden and, 230, 233, 451n40, 451n41; Marconi and, 259–60, 293–94, 311, 313, 332–37, 347–48; Martin and, 105, 195; Matthews and, 382; models and, 26, 29; motors and, 35–37, 41–42, 45, 53, 90–92, 96, 102, 105, 111–12, 116, 149, 195, 407; oscillators and, 135, 138, 148, 208, 215, 235, 271; particle beam weapons and, 382, 387; Peck and, 91–92, 102, 112, 116; plant for, 303; Pöschl and, 35, 41–42; Price and, 105; Prince Albert and, 256; private, 190, 194–95, 199–200, 215, 230, 259, 294, 451n40; public imagination and, 11; quadruplex and, 78; radio-controlled boat and, 226, 230, 233–37, 264; reflected waves and, 156; Ruhmkorff coil and, 126, 217; Shallenberger and, 112; skin effect and, 125, 136; spinning fan and, 151; Stanley and, 89; stationary waves and, 264, 271, 282, 293–95; Tesla coils and, 120–25, 128, 218, 248–49, 282, 338, 353, 403, 423n6, 450n17, 462n6; Tesla’s fondness for, 10; turbines and, 26, 372–73; Twain and, 186–87; vacuum tube lamps and, 219; von Miller and, 144–45; Westinghouse and, 160, 172–73; wireless lighting and, 135–37, 177, 188, 190–92, 215, 235, 294; wireless power distribution and, 341–42 Déri, Miksa, 61, 62, 88 Dewar, James, 147 Diary newspaper, Novi Sad, 17 direct current (DC): Adams and, 167–68; arc lighting and, 67; Edison and, 4, 165; high-voltage ionization and, 382; motors and, 37–41, 44, 186; Niagara Falls and, 167; particle beam weapons and, 382, 383; Twain and, 186; vs. alternating current (AC), 44–45, 167–68, 170–71 DiscFlo Corporation, 374 Dodge, Flora, 239 Dodge, Mary Mapes, 199, 344 Dolbear, Amos, 139, 232 Dolivo-Dobrowolsky, Michael von, 144–46, 161, 166, 303, 441n4 Dozier, Joseph, 267 Draper, John William, 307 drehstrom (rotary current), 144, 166 Duncan, Curtis, & Page, 93–94 Duveen, James Henry, 312 Dvořák, Antonín, 198 Dwight, Theodore W., 134, 440n15 dynamos: Anthony and, 103–4; arc lighting and, 41; electricity and, 35, 41–45, 63–71, 81–82, 85–88, 95–96, 103, 110, 115, 120, 168, 304, 320–21, 328, 431n42, 433n23; Mary-Ann, 70; power transmission and, 41–42; Weston and, 81; (see also generators) Earth: as compressible fluid, 362–64; electrical potential of, 269–74; high-frequency phenomena and, 138–42; Kennelly-Heaviside layer and, 209, 335; oscillators and, 138–42; resonance and, 211, 248, 269–74, 329, 363; response to electricity pumped into, 362–64 earthquakes, artificial, 187, 446n26 Eastman, George, 101 Eastman Kodak, 101 economic issues: arc lighting and, 72–73, 76; Astor and, 214, 255–61, 264, 303, 305, 313–15, 353, 361, 371, 417; bankruptcy, 311, 374–76, 399, 446n24; copper costs and, 179, 257–58; creativity and, 401–2; disruptive innovation and, 401–3; funding and, 5–6, 130–31, 206, 255–56, 282, 302–5, 316, 318, 343–45, 352–58, 368–69, 386, 388, 393, 402, 445n6; innovation and, 10, 57–59 (see also innovation); investors and, 102, 128, 130–32, 163, 179, 206–7, 218, 263, 294–95, 305, 310, 332, 344–45, 348–61, 374, 407, 465n63; laboratory fire and, 216–17; licensing and, 94, 101–3, 145, 153, 155, 205–6, 218, 255, 347, 378; Light-House Board and, 283–84; manufacturing and, 339; Morgan deal details and, 357–58; navy and, 282–84; patent-promote-sell strategy and, 102–3, 132, 371, 397; power transmission and, 328 (see also power transmission); price of coal and, 165; royalties and, 94, 101, 111, 113, 130–32, 145, 305, 349, 361, 378, 467n19; Schumpeter and, 10, 57–59, 401, 404, 411; selling patents and, 107–13 (see also patents); Simpson and Crawford and, 256, 264; speculative bubbles and, 346–51; undersea cables and, 336; Vanderbilt and, 214, 230; Wall Street and, 2, 78–79, 130, 164, 206, 304, 311, 345, 349–50, 353, 357, 359; Wardenclyffe tower and, 323, 326; wireless communication and, 350 eddy currents, 53–54 Edison, Thomas Alva, 2; arc lighting and, 71–73, 431n40; Brown and, 78–79; Brush Electric Light Company and, 72; corporate conspiracy and, 398; direct current (DC) and, 4, 165; French lighting organization of, 63–66; General Electric and, 101–2, 129, 160, 162, 166, 224, 311, 317, 352, 357, 373, 397, 452n62; incandescent lighting and, 4, 63–67, 72–73, 102, 140, 206, 219; Johnson and, 197–98; Martin and, 195; Mary-Ann dynamos and, 70; mathematics and, 64; Menlo Park and, 49, 194–95, 198; Nobel Prize and, 375; Orton and, 406; patents and, 49, 63, 72, 161; Peck and, 79; phonograph and, 49, 107, 195, 301; physics and, 375; Puskás and, 49–50; pyromagnetic generator and, 76; reporter coverage of, 194–95; S.

The Empire Project: The Rise and Fall of the British World-System, 1830–1970
by John Darwin
Published 23 Sep 2009

In the tropical empire (which British migrants avoided) they supplied much of the labour and business expertise to promote commercial expansion. Lastly, the spheres of British expansion were progressively linked by a complex system of communication. From the 1840s onwards, this was provided by subsidised mail services, telegraph wires, undersea cables, an expanding rail network, fast passenger steamers and (in the twentieth century) imperial air routes. They catered for, and stimulated, the growing volume and frequency of the traffic in news, information, private correspondence, personnel and ideas that flowed between Britain and other parts of the system, as well as between those constituent parts.

…

It secured Britain's claims on a huge range of assets, most of them safely remote from the great powers in Europe. It helped to sustain the flow of migration, Britain's demographic imperialism. Last, but not least, it preserved Britain's lead in communications technology, especially the telegraph and undersea cables that made London (and Britain) the information hub of the world. But Britain's position in the new world economy could not be taken for granted. It required adaptations of practice and outlook that were just as far-reaching as those that tested political nerves in Whitehall and Westminster. The scale of commercial activity as ‘new regions’ were drawn into the ‘high-pressure zone’ of the Atlantic economy; the intricate network of multilateral payments on which trade growth depended; the demand for finance, both credit and capital; and the competitive threat posed by new industrial producers: all posed new challenges to the commercial apparatus that the mid-Victorians had fashioned.

…

The sudden death-crisis of the Company state gave London the chance to impose British priorities on a self-centred expatriate Anglo-Indian regime. The remnants of the Company's old merchant-warrior ethos were thrown on the scrap-heap. India was now to be much more thoroughly integrated into Britain's pattern of trade and investment – a process accelerated by the cutting of the Suez Canal and the extension of the telegraph and submarine cable. But, as quickly became clear, the military foundations of the new British Raj also demanded a closer strategic connection, and imposed willy-nilly a general revision of imperial defence. Once London had to find more than 70,000 soldiers for the Indian garrison (and perhaps more in an emergency – 90,000 had been needed during the Mutiny), the strain on the rest of its imperial commitments became overwhelming.

Wall Street Meat
by Andy Kessler
Published 17 Mar 2003

Jack was generating revenue, and it was increasingly easy to justify his piece of the action. But the ax stuff, as I mentioned, is amazingly intoxicating. It is almost impossible not to get drunk on the glory of moving stocks and of raising billions. Jack became a kingmaker. Gary Winnick had started Global Crossing to sell undersea cable and take market share from AT&T and others. His first undersea cable, Atlantic Crossing 1, cost a few hundred million and was soon generating $100 million a month. Jack was just the guy to help raise billions to duplicate the effort, one more time in the Atlantic and then under the Pacific. · · · The dotcoms’ meltdown started in March and April 2000.

The Four: How Amazon, Apple, Facebook, and Google Divided and Conquered the World
by Scott Galloway
Published 2 Oct 2017

“Google’s Next Cloud Product: Google Blimps to Bring Wireless Internet to Africa.” Fortune. June 5, 2013. https://www.forbes.com/sites/matthewstibbe/2013/06/05/googles-next-cloud-product-google-blimps-to-bring-wireless-internet-to-africa/#4439e478449b. 48. Weise, Elizabeth. “Microsoft, Facebook to lay massive undersea cable.” USA Today. May 26, 2016. https://www.usatoday.com/story/experience/2016/05/26/microsoft-facebook-undersea-cable-google-marea-amazon/84984882/. 49. “The Nokia effect.” Economist. August 25, 2012. http://www.economist.com/node/21560867. 50. Downie, Ryan. “Behind Nokia’s 70% Drop in 10 Years (NOK).” Investopedia. September 8, 2016. http://www.investopedia.com/articles/credit-loans-mortgages/090816/behind-nokias-70-drop-10-years-nok.asp.

Data and Goliath: The Hidden Battles to Collect Your Data and Control Your World
by Bruce Schneier
Published 2 Mar 2015

A Russian system was only used for Russian communications. If the NSA tapped an undersea cable between Petropavlovsk and Vladivostok, it didn’t have to worry about accidentally intercepting phone calls between Detroit and Cleveland. The Internet works differently. Everyone’s communications are mixed up on the same networks. Terrorists use the same e-mail providers as everyone else. The same circuits that carry Russian, Iranian, and Cuban government communications could also carry your Twitter feed. Internet phone calls between New York and Los Angeles might end up on Russian undersea cables. Communications between Rio de Janeiro and Lisbon might be routed through Florida.

…

GOVERNMENT HACKS Electronic espionage is different today from what it was in the pre-Internet days of the Cold War. Before the Internet, when surveillance consisted largely of government-on-government espionage, agencies like the NSA would target specific communications circuits: that Soviet undersea cable between Petropavlovsk and Vladivostok, a military communications satellite, a microwave network. This was for the most part passive, requiring large antenna farms in nearby countries. Modern targeted surveillance is likely to involve actively breaking into an adversary’s computer network and installing malicious software designed to take over that network and “exfiltrate” data—that’s NSA talk for stealing it.

Red November: Inside the Secret U.S.-Soviet Submarine War
by W. Craig Reed
Published 3 May 2010

In October 1971, the upgraded Halibut, carrying a stern-mounted decompression hyperbaric chamber and loaded with deep-sea saturation divers, cruised toward the Soviet Union with Commander “Smiling Jack” McNish in charge. The Soviet communications cable they sought ran thousands of miles under the ocean deep in the Sea of Okhotsk, a large body of water, surrounded by land, that from the air looked like a massive Russian lake. The undersea cable ran from the Petropavlovsk submarine base and missile-testing facilities on the Kamchatka Peninsula, down to Vladivostok, headquarters of the Soviet “Far East” Pacific Fleet. Commander McNish successfully located the cable’s approximate position via Halibut’s periscope by finding a small sign posted on a beach warning passersby to avoid running into a buried cable.

…

Navy photographs by Senior Chief Mass Commuication Specialist Andrew McKaskle and Chief Photographer’s Mate Mark Reinhard, courtesy of www.osd.mil The most dangerous, daring, and decorated missions of the Cold War were undertaken by U.S. Navy deep-sea saturation divers breathing helium and oxygen mixtures. These divers, working from fast-attack submarines, tapped Soviet communications cables off the coast of Russia, at depths of up to 700 feet. U.S. Navy photographs by Mass Commuication Specialist Fisrt Class Eric Lippmann U.S. Navy divers, conducting top-secret Ivy Bells cable-tapping missions, spent weeks compressing and decompressing inside small diving chambers mounted on submarines, which resembled DSRVs like the one above.

…

Dennis Smith’s first run into harm’s way hooked him into the world of Special Projects, where he remained for years to come. One of those missions was Parche’s first into the cold northern waters of the Barents Sea in 1979. Given its proximity to Murmansk, home of the Soviet Northern Fleet, this area provided a heightened degree of danger and difficulty for any submarine, let alone a cable-tapping one. Smith and crew spent days searching for the cable, coming within a few miles of the Soviet coastline. Finding the cable took days, with the Parche sending out a 2090 to snoop around the ocean floor some 700 feet down. Once found, they hovered over the landing spot while dropping 15,000-pound mushroom anchors.

Mapmatics: How We Navigate the World Through Numbers
by Paulina Rowinska
Published 5 Jun 2024

safe locations for new telephone cables: Jeff Hect, ‘Laser’, Encyclopaedia Britannica, 18 May 2023, https://www.britannica.com/technology/laser. connecting users across continents: James Griffiths, ‘The Global Internet Is Powered by Vast Undersea Cables. But They’re Vulnerable’, CNN, 26 July 2019, https://edition.cnn.com/2019/07/25/asia/internet-undersea-cables-intl-hnk/index.html. major internet disruptions on the US east coast: University of Southern California, ‘Internet Outages in the US Doubled during Hurricane Sandy’, ScienceDaily, 18 December 2012, www.sciencedaily.com/releases/2012/12/121218133152.htm.

Reset
by Ronald J. Deibert
Published 14 Aug 2020

Those devices send electronic signals through radio waves or cables that are transmitted through a physical infrastructure of routers, servers, cell towers, and data farms, in some cases spread throughout multiple countries. Each of these elements is operated by numerous businesses, which can include internet service providers (ISPs), cable companies, cell service providers, satellite services, undersea cable providers, and telecommunications firms as well as the various hardware and software manufacturers supporting them all. Which companies operate the various components of this ecosystem, and according to whose rules, matters enormously for users’ security and privacy. For example, policymakers and analysts have raised concerns that China-based communications routing equipment manufacturers Huawei and ZTE may have designed secret “back doors” in their technology that would provide China’s security agencies with an intelligence toehold in 5G networks.16 However valid, these concerns are not unique to China or China-based companies.

…

A program code-named HAPPYFOOT showed that the NSA and its U.K. partner, the Government Communications Headquarters, were exploiting the location data “leaked” by popular mobile apps.27 To help locate targets of interest, the NSA collected hundreds of thousands of contacts from instant messaging and email applications.28 On and on the stories went, following the same pattern: an astounding digital vacuum-cleaning operation up and down the entire infrastructure of global communications networks owned and operated by the private sector. To be sure, the NSA and its partners did still “tap into” and have “direct access” to a lot of networking equipment and infrastructure. Their hubris was remarkable. By partnering with or otherwise enlisting the cooperation of undersea cable providers, telecommunications companies, and cell service operators, they hoovered up massive volumes of data flowing directly through the arteries of the global communications ecosystem, storing it in giant data warehouses for later fusion (the process of integrating disparate data sources together) and analysis.

The Fifth Domain: Defending Our Country, Our Companies, and Ourselves in the Age of Cyber Threats
by Richard A. Clarke and Robert K. Knake
Published 15 Jul 2019

In order to “block” this traffic, the U.S. government would have had to be sitting in between the attacking computers and the target computers. While blocking the attacks sounds appealing, the reality is that the United States has open borders in cyberspace. No agency of the federal government sits at the internet exchange points, where the undersea cables come up onto land, to inspect each packet of internet traffic. Without such a capability, the U.S. government is simply not positioned to block malicious traffic to protect banks or any other companies. Nor should we want such a system to be built. While China has a Great Firewall, a vast system of traffic inspection and interception deployed at the borders of China’s internet and throughout the country, calling the system a firewall suggests, erroneously, that it has value for cybersecurity when it is in fact a tool for censorship and surveillance.

…

, but whereas in the early days those networks were all open and packets of data flowed across them based on the optimal route, now many of those networks have erected barriers around themselves. Still holding it all together is the United States. Most global internet traffic still travels through the United States. An email going from, say, Budapest to Hanoi would head across the Atlantic through undersea cables, traverse the Lower 48 states, and plunge into the Pacific before hitting Japan and being dispersed in the direction of Vietnam. The United States continues to be the loudest voice in the various internet forums for maintaining the global internet. ICANN, the Internet Corporation for Assigned Names and Numbers, which manages the global Domain Name System, is still very much a U.S. entity (though it is no longer controlled by the U.S. government) and it promotes the original U.S. view of the internet.

Building Secure and Reliable Systems: Best Practices for Designing, Implementing, and Maintaining Systems
by Heather Adkins , Betsy Beyer , Paul Blankinship , Ana Oprea , Piotr Lewandowski and Adam Stubblefield
Published 29 Mar 2020

When planning your recovery, you should address the one system that’s known to be compromised, but also consider two other factors: whether the other 20 servers are also compromised, and how you will mitigate the effects of the vulnerability for all of these machines in the future. Repeating Risks Imagine that a large ship needs to hold its position at sea. The ship drops anchor and inadvertently hooks onto an undersea cable, which isn’t strong enough and breaks. While the risk of a vessel snagging an undersea cable is small, the effects can be catastrophic, having a huge impact on intercontinental network capacity—and this risk exists for all the undersea cables in the world, not just the one that happened to have been hooked by that one errant ship. A network outage of this type should trigger an analysis of the broader category of risk.

Taming the Sun: Innovations to Harness Solar Energy and Power the Planet
by Varun Sivaram
Published 2 Mar 2018

Next, countries will link up regionally, in line both with Masa’s early vision of an Asian supergrid and other ambitious proposals, such as a grid to link sunny North Africa with windy Europe. Then, the grand third stage will bring together the regional supergrids to form a single global supergrid, linked by transoceanic undersea cables and electrical superstations where lines would converge (figure 8.1). Figure 8.1 A global supergrid. This map combines various proposed regional supergrids (for example, in the Asia-Pacific region, Europe and the Middle East/North Africa, and North America). Black lines represent long-distance HVDC electricity transmission lines.

…

Such tensions might prove the biggest obstacle to countries trusting one another enough to willingly increase their dependence on traded energy. Therefore, Liu’s eventual goal of going beyond regional supergrids to achieve a global one is pretty implausible, requiring more international trust, far more money, and further advances in such technologies as undersea cables. Nevertheless, that ultimate vision might function as an inspirational, if perhaps impossible, target toward which even partial progress—such as the establishment of regional supergrids—would constitute a big step forward for the world. Indeed, bigger grids will be essential for tapping the world’s best solar and other renewable resources.

Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems
by Martin Kleppmann
Published 17 Apr 2017

Public cloud services such as EC2 are notorious for having frequent transient net‐ work glitches [14], and well-managed private datacenter networks can be stabler environments. Nevertheless, nobody is immune from network problems: for exam‐ ple, a problem during a software upgrade for a switch could trigger a network topol‐ ogy reconfiguration, during which network packets could be delayed for more than a minute [17]. Sharks might bite undersea cables and damage them [18]. Other surpris‐ ing faults include a network interface that sometimes drops all inbound packets but sends outbound packets successfully [19]: just because a network link works in one direction doesn’t guarantee it’s also working in the opposite direction. Network partitions When one part of the network is cut off from the rest due to a net‐ work fault, that is sometimes called a network partition or netsplit.

…

The opposite of bounded. 558 | Glossary Index A aborts (transactions), 222, 224 in two-phase commit, 356 performance of optimistic concurrency con‐ trol, 266 retrying aborted transactions, 231 abstraction, 21, 27, 222, 266, 321 access path (in network model), 37, 60 accidental complexity, removing, 21 accountability, 535 ACID properties (transactions), 90, 223 atomicity, 223, 228 consistency, 224, 529 durability, 226 isolation, 225, 228 acknowledgements (messaging), 445 active/active replication (see multi-leader repli‐ cation) active/passive replication (see leader-based rep‐ lication) ActiveMQ (messaging), 137, 444 distributed transaction support, 361 ActiveRecord (object-relational mapper), 30, 232 actor model, 138 (see also message-passing) comparison to Pregel model, 425 comparison to stream processing, 468 Advanced Message Queuing Protocol (see AMQP) aerospace systems, 6, 10, 305, 372 aggregation data cubes and materialized views, 101 in batch processes, 406 in stream processes, 466 aggregation pipeline query language, 48 Agile, 22 minimizing irreversibility, 414, 497 moving faster with confidence, 532 Unix philosophy, 394 agreement, 365 (see also consensus) Airflow (workflow scheduler), 402 Ajax, 131 Akka (actor framework), 139 algorithms algorithm correctness, 308 B-trees, 79-83 for distributed systems, 306 hash indexes, 72-75 mergesort, 76, 402, 405 red-black trees, 78 SSTables and LSM-trees, 76-79 all-to-all replication topologies, 175 AllegroGraph (database), 50 ALTER TABLE statement (SQL), 40, 111 Amazon Dynamo (database), 177 Amazon Web Services (AWS), 8 Kinesis Streams (messaging), 448 network reliability, 279 postmortems, 9 RedShift (database), 93 S3 (object storage), 398 checking data integrity, 530 amplification of bias, 534 of failures, 364, 495 Index | 559 of tail latency, 16, 207 write amplification, 84 AMQP (Advanced Message Queuing Protocol), 444 (see also messaging systems) comparison to log-based messaging, 448, 451 message ordering, 446 analytics, 90 comparison to transaction processing, 91 data warehousing (see data warehousing) parallel query execution in MPP databases, 415 predictive (see predictive analytics) relation to batch processing, 411 schemas for, 93-95 snapshot isolation for queries, 238 stream analytics, 466 using MapReduce, analysis of user activity events (example), 404 anti-caching (in-memory databases), 89 anti-entropy, 178 Apache ActiveMQ (see ActiveMQ) Apache Avro (see Avro) Apache Beam (see Beam) Apache BookKeeper (see BookKeeper) Apache Cassandra (see Cassandra) Apache CouchDB (see CouchDB) Apache Curator (see Curator) Apache Drill (see Drill) Apache Flink (see Flink) Apache Giraph (see Giraph) Apache Hadoop (see Hadoop) Apache HAWQ (see HAWQ) Apache HBase (see HBase) Apache Helix (see Helix) Apache Hive (see Hive) Apache Impala (see Impala) Apache Jena (see Jena) Apache Kafka (see Kafka) Apache Lucene (see Lucene) Apache MADlib (see MADlib) Apache Mahout (see Mahout) Apache Oozie (see Oozie) Apache Parquet (see Parquet) Apache Qpid (see Qpid) Apache Samza (see Samza) Apache Solr (see Solr) Apache Spark (see Spark) 560 | Index Apache Storm (see Storm) Apache Tajo (see Tajo) Apache Tez (see Tez) Apache Thrift (see Thrift) Apache ZooKeeper (see ZooKeeper) Apama (stream analytics), 466 append-only B-trees, 82, 242 append-only files (see logs) Application Programming Interfaces (APIs), 5, 27 for batch processing, 403 for change streams, 456 for distributed transactions, 361 for graph processing, 425 for services, 131-136 (see also services) evolvability, 136 RESTful, 133 SOAP, 133 application state (see state) approximate search (see similarity search) archival storage, data from databases, 131 arcs (see edges) arithmetic mean, 14 ASCII text, 119, 395 ASN.1 (schema language), 127 asynchronous networks, 278, 553 comparison to synchronous networks, 284 formal model, 307 asynchronous replication, 154, 553 conflict detection, 172 data loss on failover, 157 reads from asynchronous follower, 162 Asynchronous Transfer Mode (ATM), 285 atomic broadcast (see total order broadcast) atomic clocks (caesium clocks), 294, 295 (see also clocks) atomicity (concurrency), 553 atomic increment-and-get, 351 compare-and-set, 245, 327 (see also compare-and-set operations) replicated operations, 246 write operations, 243 atomicity (transactions), 223, 228, 553 atomic commit, 353 avoiding, 523, 528 blocking and nonblocking, 359 in stream processing, 360, 477 maintaining derived data, 453 for multi-object transactions, 229 for single-object writes, 230 auditability, 528-533 designing for, 531 self-auditing systems, 530 through immutability, 460 tools for auditable data systems, 532 availability, 8 (see also fault tolerance) in CAP theorem, 337 in service level agreements (SLAs), 15 Avro (data format), 122-127 code generation, 127 dynamically generated schemas, 126 object container files, 125, 131, 414 reader determining writer’s schema, 125 schema evolution, 123 use in Hadoop, 414 awk (Unix tool), 391 AWS (see Amazon Web Services) Azure (see Microsoft) B B-trees (indexes), 79-83 append-only/copy-on-write variants, 82, 242 branching factor, 81 comparison to LSM-trees, 83-85 crash recovery, 82 growing by splitting a page, 81 optimizations, 82 similarity to dynamic partitioning, 212 backpressure, 441, 553 in TCP, 282 backups database snapshot for replication, 156 integrity of, 530 snapshot isolation for, 238 use for ETL processes, 405 backward compatibility, 112 BASE, contrast to ACID, 223 bash shell (Unix), 70, 395, 503 batch processing, 28, 389-431, 553 combining with stream processing lambda architecture, 497 unifying technologies, 498 comparison to MPP databases, 414-418 comparison to stream processing, 464 comparison to Unix, 413-414 dataflow engines, 421-423 fault tolerance, 406, 414, 422, 442 for data integration, 494-498 graphs and iterative processing, 424-426 high-level APIs and languages, 403, 426-429 log-based messaging and, 451 maintaining derived state, 495 MapReduce and distributed filesystems, 397-413 (see also MapReduce) measuring performance, 13, 390 outputs, 411-413 key-value stores, 412 search indexes, 411 using Unix tools (example), 391-394 Bayou (database), 522 Beam (dataflow library), 498 bias, 534 big ball of mud, 20 Bigtable data model, 41, 99 binary data encodings, 115-128 Avro, 122-127 MessagePack, 116-117 Thrift and Protocol Buffers, 117-121 binary encoding based on schemas, 127 by network drivers, 128 binary strings, lack of support in JSON and XML, 114 BinaryProtocol encoding (Thrift), 118 Bitcask (storage engine), 72 crash recovery, 74 Bitcoin (cryptocurrency), 532 Byzantine fault tolerance, 305 concurrency bugs in exchanges, 233 bitmap indexes, 97 blockchains, 532 Byzantine fault tolerance, 305 blocking atomic commit, 359 Bloom (programming language), 504 Bloom filter (algorithm), 79, 466 BookKeeper (replicated log), 372 Bottled Water (change data capture), 455 bounded datasets, 430, 439, 553 (see also batch processing) bounded delays, 553 in networks, 285 process pauses, 298 broadcast hash joins, 409 Index | 561 brokerless messaging, 442 Brubeck (metrics aggregator), 442 BTM (transaction coordinator), 356 bulk synchronous parallel (BSP) model, 425 bursty network traffic patterns, 285 business data processing, 28, 90, 390 byte sequence, encoding data in, 112 Byzantine faults, 304-306, 307, 553 Byzantine fault-tolerant systems, 305, 532 Byzantine Generals Problem, 304 consensus algorithms and, 366 C caches, 89, 553 and materialized views, 101 as derived data, 386, 499-504 database as cache of transaction log, 460 in CPUs, 99, 338, 428 invalidation and maintenance, 452, 467 linearizability, 324 CAP theorem, 336-338, 554 Cascading (batch processing), 419, 427 hash joins, 409 workflows, 403 cascading failures, 9, 214, 281 Cascalog (batch processing), 60 Cassandra (database) column-family data model, 41, 99 compaction strategy, 79 compound primary key, 204 gossip protocol, 216 hash partitioning, 203-205 last-write-wins conflict resolution, 186, 292 leaderless replication, 177 linearizability, lack of, 335 log-structured storage, 78 multi-datacenter support, 184 partitioning scheme, 213 secondary indexes, 207 sloppy quorums, 184 cat (Unix tool), 391 causal context, 191 (see also causal dependencies) causal dependencies, 186-191 capturing, 191, 342, 494, 514 by total ordering, 493 causal ordering, 339 in transactions, 262 sending message to friends (example), 494 562 | Index causality, 554 causal ordering, 339-343 linearizability and, 342 total order consistent with, 344, 345 consistency with, 344-347 consistent snapshots, 340 happens-before relationship, 186 in serializable transactions, 262-265 mismatch with clocks, 292 ordering events to capture, 493 violations of, 165, 176, 292, 340 with synchronized clocks, 294 CEP (see complex event processing) certificate transparency, 532 chain replication, 155 linearizable reads, 351 change data capture, 160, 454 API support for change streams, 456 comparison to event sourcing, 457 implementing, 454 initial snapshot, 455 log compaction, 456 changelogs, 460 change data capture, 454 for operator state, 479 generating with triggers, 455 in stream joins, 474 log compaction, 456 maintaining derived state, 452 Chaos Monkey, 7, 280 checkpointing in batch processors, 422, 426 in high-performance computing, 275 in stream processors, 477, 523 chronicle data model, 458 circuit-switched networks, 284 circular buffers, 450 circular replication topologies, 175 clickstream data, analysis of, 404 clients calling services, 131 pushing state changes to, 512 request routing, 214 stateful and offline-capable, 170, 511 clocks, 287-299 atomic (caesium) clocks, 294, 295 confidence interval, 293-295 for global snapshots, 294 logical (see logical clocks) skew, 291-294, 334 slewing, 289 synchronization and accuracy, 289-291 synchronization using GPS, 287, 290, 294, 295 time-of-day versus monotonic clocks, 288 timestamping events, 471 cloud computing, 146, 275 need for service discovery, 372 network glitches, 279 shared resources, 284 single-machine reliability, 8 Cloudera Impala (see Impala) clustered indexes, 86 CODASYL model, 36 (see also network model) code generation with Avro, 127 with Thrift and Protocol Buffers, 118 with WSDL, 133 collaborative editing multi-leader replication and, 170 column families (Bigtable), 41, 99 column-oriented storage, 95-101 column compression, 97 distinction between column families and, 99 in batch processors, 428 Parquet, 96, 131, 414 sort order in, 99-100 vectorized processing, 99, 428 writing to, 101 comma-separated values (see CSV) command query responsibility segregation (CQRS), 462 commands (event sourcing), 459 commits (transactions), 222 atomic commit, 354-355 (see also atomicity; transactions) read committed isolation, 234 three-phase commit (3PC), 359 two-phase commit (2PC), 355-359 commutative operations, 246 compaction of changelogs, 456 (see also log compaction) for stream operator state, 479 of log-structured storage, 73 issues with, 84 size-tiered and leveled approaches, 79 CompactProtocol encoding (Thrift), 119 compare-and-set operations, 245, 327 implementing locks, 370 implementing uniqueness constraints, 331 implementing with total order broadcast, 350 relation to consensus, 335, 350, 352, 374 relation to transactions, 230 compatibility, 112, 128 calling services, 136 properties of encoding formats, 139 using databases, 129-131 using message-passing, 138 compensating transactions, 355, 461, 526 complex event processing (CEP), 465 complexity distilling in theoretical models, 310 hiding using abstraction, 27 of software systems, managing, 20 composing data systems (see unbundling data‐ bases) compute-intensive applications, 3, 275 concatenated indexes, 87 in Cassandra, 204 Concord (stream processor), 466 concurrency actor programming model, 138, 468 (see also message-passing) bugs from weak transaction isolation, 233 conflict resolution, 171, 174 detecting concurrent writes, 184-191 dual writes, problems with, 453 happens-before relationship, 186 in replicated systems, 161-191, 324-338 lost updates, 243 multi-version concurrency control (MVCC), 239 optimistic concurrency control, 261 ordering of operations, 326, 341 reducing, through event logs, 351, 462, 507 time and relativity, 187 transaction isolation, 225 write skew (transaction isolation), 246-251 conflict-free replicated datatypes (CRDTs), 174 conflicts conflict detection, 172 causal dependencies, 186, 342 in consensus algorithms, 368 in leaderless replication, 184 Index | 563 in log-based systems, 351, 521 in nonlinearizable systems, 343 in serializable snapshot isolation (SSI), 264 in two-phase commit, 357, 364 conflict resolution automatic conflict resolution, 174 by aborting transactions, 261 by apologizing, 527 convergence, 172-174 in leaderless systems, 190 last write wins (LWW), 186, 292 using atomic operations, 246 using custom logic, 173 determining what is a conflict, 174, 522 in multi-leader replication, 171-175 avoiding conflicts, 172 lost updates, 242-246 materializing, 251 relation to operation ordering, 339 write skew (transaction isolation), 246-251 congestion (networks) avoidance, 282 limiting accuracy of clocks, 293 queueing delays, 282 consensus, 321, 364-375, 554 algorithms, 366-368 preventing split brain, 367 safety and liveness properties, 365 using linearizable operations, 351 cost of, 369 distributed transactions, 352-375 in practice, 360-364 two-phase commit, 354-359 XA transactions, 361-364 impossibility of, 353 membership and coordination services, 370-373 relation to compare-and-set, 335, 350, 352, 374 relation to replication, 155, 349 relation to uniqueness constraints, 521 consistency, 224, 524 across different databases, 157, 452, 462, 492 causal, 339-348, 493 consistent prefix reads, 165-167 consistent snapshots, 156, 237-242, 294, 455, 500 (see also snapshots) 564 | Index crash recovery, 82 enforcing constraints (see constraints) eventual, 162, 322 (see also eventual consistency) in ACID transactions, 224, 529 in CAP theorem, 337 linearizability, 324-338 meanings of, 224 monotonic reads, 164-165 of secondary indexes, 231, 241, 354, 491, 500 ordering guarantees, 339-352 read-after-write, 162-164 sequential, 351 strong (see linearizability) timeliness and integrity, 524 using quorums, 181, 334 consistent hashing, 204 consistent prefix reads, 165 constraints (databases), 225, 248 asynchronously checked, 526 coordination avoidance, 527 ensuring idempotence, 519 in log-based systems, 521-524 across multiple partitions, 522 in two-phase commit, 355, 357 relation to consensus, 374, 521 relation to event ordering, 347 requiring linearizability, 330 Consul (service discovery), 372 consumers (message streams), 137, 440 backpressure, 441 consumer offsets in logs, 449 failures, 445, 449 fan-out, 11, 445, 448 load balancing, 444, 448 not keeping up with producers, 441, 450, 502 context switches, 14, 297 convergence (conflict resolution), 172-174, 322 coordination avoidance, 527 cross-datacenter, 168, 493 cross-partition ordering, 256, 294, 348, 523 services, 330, 370-373 coordinator (in 2PC), 356 failure, 358 in XA transactions, 361-364 recovery, 363 copy-on-write (B-trees), 82, 242 CORBA (Common Object Request Broker Architecture), 134 correctness, 6 auditability, 528-533 Byzantine fault tolerance, 305, 532 dealing with partial failures, 274 in log-based systems, 521-524 of algorithm within system model, 308 of compensating transactions, 355 of consensus, 368 of derived data, 497, 531 of immutable data, 461 of personal data, 535, 540 of time, 176, 289-295 of transactions, 225, 515, 529 timeliness and integrity, 524-528 corruption of data detecting, 519, 530-533 due to pathological memory access, 529 due to radiation, 305 due to split brain, 158, 302 due to weak transaction isolation, 233 formalization in consensus, 366 integrity as absence of, 524 network packets, 306 on disks, 227 preventing using write-ahead logs, 82 recovering from, 414, 460 Couchbase (database) durability, 89 hash partitioning, 203-204, 211 rebalancing, 213 request routing, 216 CouchDB (database) B-tree storage, 242 change feed, 456 document data model, 31 join support, 34 MapReduce support, 46, 400 replication, 170, 173 covering indexes, 86 CPUs cache coherence and memory barriers, 338 caching and pipelining, 99, 428 increasing parallelism, 43 CRDTs (see conflict-free replicated datatypes) CREATE INDEX statement (SQL), 85, 500 credit rating agencies, 535 Crunch (batch processing), 419, 427 hash joins, 409 sharded joins, 408 workflows, 403 cryptography defense against attackers, 306 end-to-end encryption and authentication, 519, 543 proving integrity of data, 532 CSS (Cascading Style Sheets), 44 CSV (comma-separated values), 70, 114, 396 Curator (ZooKeeper recipes), 330, 371 curl (Unix tool), 135, 397 cursor stability, 243 Cypher (query language), 52 comparison to SPARQL, 59 D data corruption (see corruption of data) data cubes, 102 data formats (see encoding) data integration, 490-498, 543 batch and stream processing, 494-498 lambda architecture, 497 maintaining derived state, 495 reprocessing data, 496 unifying, 498 by unbundling databases, 499-515 comparison to federated databases, 501 combining tools by deriving data, 490-494 derived data versus distributed transac‐ tions, 492 limits of total ordering, 493 ordering events to capture causality, 493 reasoning about dataflows, 491 need for, 385 data lakes, 415 data locality (see locality) data models, 27-64 graph-like models, 49-63 Datalog language, 60-63 property graphs, 50 RDF and triple-stores, 55-59 query languages, 42-48 relational model versus document model, 28-42 data protection regulations, 542 data systems, 3 about, 4 Index | 565 concerns when designing, 5 future of, 489-544 correctness, constraints, and integrity, 515-533 data integration, 490-498 unbundling databases, 499-515 heterogeneous, keeping in sync, 452 maintainability, 18-22 possible faults in, 221 reliability, 6-10 hardware faults, 7 human errors, 9 importance of, 10 software errors, 8 scalability, 10-18 unreliable clocks, 287-299 data warehousing, 91-95, 554 comparison to data lakes, 415 ETL (extract-transform-load), 92, 416, 452 keeping data systems in sync, 452 schema design, 93 slowly changing dimension (SCD), 476 data-intensive applications, 3 database triggers (see triggers) database-internal distributed transactions, 360, 364, 477 databases archival storage, 131 comparison of message brokers to, 443 dataflow through, 129 end-to-end argument for, 519-520 checking integrity, 531 inside-out, 504 (see also unbundling databases) output from batch workflows, 412 relation to event streams, 451-464 (see also changelogs) API support for change streams, 456, 506 change data capture, 454-457 event sourcing, 457-459 keeping systems in sync, 452-453 philosophy of immutable events, 459-464 unbundling, 499-515 composing data storage technologies, 499-504 designing applications around dataflow, 504-509 566 | Index observing derived state, 509-515 datacenters geographically distributed, 145, 164, 278, 493 multi-tenancy and shared resources, 284 network architecture, 276 network faults, 279 replication across multiple, 169 leaderless replication, 184 multi-leader replication, 168, 335 dataflow, 128-139, 504-509 correctness of dataflow systems, 525 differential, 504 message-passing, 136-139 reasoning about, 491 through databases, 129 through services, 131-136 dataflow engines, 421-423 comparison to stream processing, 464 directed acyclic graphs (DAG), 424 partitioning, approach to, 429 support for declarative queries, 427 Datalog (query language), 60-63 datatypes binary strings in XML and JSON, 114 conflict-free, 174 in Avro encodings, 122 in Thrift and Protocol Buffers, 121 numbers in XML and JSON, 114 Datomic (database) B-tree storage, 242 data model, 50, 57 Datalog query language, 60 excision (deleting data), 463 languages for transactions, 255 serial execution of transactions, 253 deadlocks detection, in two-phase commit (2PC), 364 in two-phase locking (2PL), 258 Debezium (change data capture), 455 declarative languages, 42, 554 Bloom, 504 CSS and XSL, 44 Cypher, 52 Datalog, 60 for batch processing, 427 recursive SQL queries, 53 relational algebra and SQL, 42 SPARQL, 59 delays bounded network delays, 285 bounded process pauses, 298 unbounded network delays, 282 unbounded process pauses, 296 deleting data, 463 denormalization (data representation), 34, 554 costs, 39 in derived data systems, 386 materialized views, 101 updating derived data, 228, 231, 490 versus normalization, 462 derived data, 386, 439, 554 from change data capture, 454 in event sourcing, 458-458 maintaining derived state through logs, 452-457, 459-463 observing, by subscribing to streams, 512 outputs of batch and stream processing, 495 through application code, 505 versus distributed transactions, 492 deterministic operations, 255, 274, 554 accidental nondeterminism, 423 and fault tolerance, 423, 426 and idempotence, 478, 492 computing derived data, 495, 526, 531 in state machine replication, 349, 452, 458 joins, 476 DevOps, 394 differential dataflow, 504 dimension tables, 94 dimensional modeling (see star schemas) directed acyclic graphs (DAGs), 424 dirty reads (transaction isolation), 234 dirty writes (transaction isolation), 235 discrimination, 534 disks (see hard disks) distributed actor frameworks, 138 distributed filesystems, 398-399 decoupling from query engines, 417 indiscriminately dumping data into, 415 use by MapReduce, 402 distributed systems, 273-312, 554 Byzantine faults, 304-306 cloud versus supercomputing, 275 detecting network faults, 280 faults and partial failures, 274-277 formalization of consensus, 365 impossibility results, 338, 353 issues with failover, 157 limitations of distributed transactions, 363 multi-datacenter, 169, 335 network problems, 277-286 quorums, relying on, 301 reasons for using, 145, 151 synchronized clocks, relying on, 291-295 system models, 306-310 use of clocks and time, 287 distributed transactions (see transactions) Django (web framework), 232 DNS (Domain Name System), 216, 372 Docker (container manager), 506 document data model, 30-42 comparison to relational model, 38-42 document references, 38, 403 document-oriented databases, 31 many-to-many relationships and joins, 36 multi-object transactions, need for, 231 versus relational model convergence of models, 41 data locality, 41 document-partitioned indexes, 206, 217, 411 domain-driven design (DDD), 457 DRBD (Distributed Replicated Block Device), 153 drift (clocks), 289 Drill (query engine), 93 Druid (database), 461 Dryad (dataflow engine), 421 dual writes, problems with, 452, 507 duplicates, suppression of, 517 (see also idempotence) using a unique ID, 518, 522 durability (transactions), 226, 554 duration (time), 287 measurement with monotonic clocks, 288 dynamic partitioning, 212 dynamically typed languages analogy to schema-on-read, 40 code generation and, 127 Dynamo-style databases (see leaderless replica‐ tion) E edges (in graphs), 49, 403 property graph model, 50 edit distance (full-text search), 88 effectively-once semantics, 476, 516 Index | 567 (see also exactly-once semantics) preservation of integrity, 525 elastic systems, 17 Elasticsearch (search server) document-partitioned indexes, 207 partition rebalancing, 211 percolator (stream search), 467 usage example, 4 use of Lucene, 79 ElephantDB (database), 413 Elm (programming language), 504, 512 encodings (data formats), 111-128 Avro, 122-127 binary variants of JSON and XML, 115 compatibility, 112 calling services, 136 using databases, 129-131 using message-passing, 138 defined, 113 JSON, XML, and CSV, 114 language-specific formats, 113 merits of schemas, 127 representations of data, 112 Thrift and Protocol Buffers, 117-121 end-to-end argument, 277, 519-520 checking integrity, 531 publish/subscribe streams, 512 enrichment (stream), 473 Enterprise JavaBeans (EJB), 134 entities (see vertices) epoch (consensus algorithms), 368 epoch (Unix timestamps), 288 equi-joins, 403 erasure coding (error correction), 398 Erlang OTP (actor framework), 139 error handling for network faults, 280 in transactions, 231 error-correcting codes, 277, 398 Esper (CEP engine), 466 etcd (coordination service), 370-373 linearizable operations, 333 locks and leader election, 330 quorum reads, 351 service discovery, 372 use of Raft algorithm, 349, 353 Ethereum (blockchain), 532 Ethernet (networks), 276, 278, 285 packet checksums, 306, 519 568 | Index Etherpad (collaborative editor), 170 ethics, 533-543 code of ethics and professional practice, 533 legislation and self-regulation, 542 predictive analytics, 533-536 amplifying bias, 534 feedback loops, 536 privacy and tracking, 536-543 consent and freedom of choice, 538 data as assets and power, 540 meaning of privacy, 539 surveillance, 537 respect, dignity, and agency, 543, 544 unintended consequences, 533, 536 ETL (extract-transform-load), 92, 405, 452, 554 use of Hadoop for, 416 event sourcing, 457-459 commands and events, 459 comparison to change data capture, 457 comparison to lambda architecture, 497 deriving current state from event log, 458 immutability and auditability, 459, 531 large, reliable data systems, 519, 526 Event Store (database), 458 event streams (see streams) events, 440 deciding on total order of, 493 deriving views from event log, 461 difference to commands, 459 event time versus processing time, 469, 477, 498 immutable, advantages of, 460, 531 ordering to capture causality, 493 reads as, 513 stragglers, 470, 498 timestamp of, in stream processing, 471 EventSource (browser API), 512 eventual consistency, 152, 162, 308, 322 (see also conflicts) and perpetual inconsistency, 525 evolvability, 21, 111 calling services, 136 graph-structured data, 52 of databases, 40, 129-131, 461, 497 of message-passing, 138 reprocessing data, 496, 498 schema evolution in Avro, 123 schema evolution in Thrift and Protocol Buffers, 120 schema-on-read, 39, 111, 128 exactly-once semantics, 360, 476, 516 parity with batch processors, 498 preservation of integrity, 525 exclusive mode (locks), 258 eXtended Architecture transactions (see XA transactions) extract-transform-load (see ETL) F Facebook Presto (query engine), 93 React, Flux, and Redux (user interface libra‐ ries), 512 social graphs, 49 Wormhole (change data capture), 455 fact tables, 93 failover, 157, 554 (see also leader-based replication) in leaderless replication, absence of, 178 leader election, 301, 348, 352 potential problems, 157 failures amplification by distributed transactions, 364, 495 failure detection, 280 automatic rebalancing causing cascading failures, 214 perfect failure detectors, 359 timeouts and unbounded delays, 282, 284 using ZooKeeper, 371 faults versus, 7 partial failures in distributed systems, 275-277, 310 fan-out (messaging systems), 11, 445 fault tolerance, 6-10, 555 abstractions for, 321 formalization in consensus, 365-369 use of replication, 367 human fault tolerance, 414 in batch processing, 406, 414, 422, 425 in log-based systems, 520, 524-526 in stream processing, 476-479 atomic commit, 477 idempotence, 478 maintaining derived state, 495 microbatching and checkpointing, 477 rebuilding state after a failure, 478 of distributed transactions, 362-364 transaction atomicity, 223, 354-361 faults, 6 Byzantine faults, 304-306 failures versus, 7 handled by transactions, 221 handling in supercomputers and cloud computing, 275 hardware, 7 in batch processing versus distributed data‐ bases, 417 in distributed systems, 274-277 introducing deliberately, 7, 280 network faults, 279-281 asymmetric faults, 300 detecting, 280 tolerance of, in multi-leader replication, 169 software errors, 8 tolerating (see fault tolerance) federated databases, 501 fence (CPU instruction), 338 fencing (preventing split brain), 158, 302-304 generating fencing tokens, 349, 370 properties of fencing tokens, 308 stream processors writing to databases, 478, 517 Fibre Channel (networks), 398 field tags (Thrift and Protocol Buffers), 119-121 file descriptors (Unix), 395 financial data, 460 Firebase (database), 456 Flink (processing framework), 421-423 dataflow APIs, 427 fault tolerance, 422, 477, 479 Gelly API (graph processing), 425 integration of batch and stream processing, 495, 498 machine learning, 428 query optimizer, 427 stream processing, 466 flow control, 282, 441, 555 FLP result (on consensus), 353 FlumeJava (dataflow library), 403, 427 followers, 152, 555 (see also leader-based replication) foreign keys, 38, 403 forward compatibility, 112 forward decay (algorithm), 16 Index | 569 Fossil (version control system), 463 shunning (deleting data), 463 FoundationDB (database) serializable transactions, 261, 265, 364 fractal trees, 83 full table scans, 403 full-text search, 555 and fuzzy indexes, 88 building search indexes, 411 Lucene storage engine, 79 functional reactive programming (FRP), 504 functional requirements, 22 futures (asynchronous operations), 135 fuzzy search (see similarity search) G garbage collection immutability and, 463 process pauses for, 14, 296-299, 301 (see also process pauses) genome analysis, 63, 429 geographically distributed datacenters, 145, 164, 278, 493 geospatial indexes, 87 Giraph (graph processing), 425 Git (version control system), 174, 342, 463 GitHub, postmortems, 157, 158, 309 global indexes (see term-partitioned indexes) GlusterFS (distributed filesystem), 398 GNU Coreutils (Linux), 394 GoldenGate (change data capture), 161, 170, 455 (see also Oracle) Google Bigtable (database) data model (see Bigtable data model) partitioning scheme, 199, 202 storage layout, 78 Chubby (lock service), 370 Cloud Dataflow (stream processor), 466, 477, 498 (see also Beam) Cloud Pub/Sub (messaging), 444, 448 Docs (collaborative editor), 170 Dremel (query engine), 93, 96 FlumeJava (dataflow library), 403, 427 GFS (distributed file system), 398 gRPC (RPC framework), 135 MapReduce (batch processing), 390 570 | Index (see also MapReduce) building search indexes, 411 task preemption, 418 Pregel (graph processing), 425 Spanner (see Spanner) TrueTime (clock API), 294 gossip protocol, 216 government use of data, 541 GPS (Global Positioning System) use for clock synchronization, 287, 290, 294, 295 GraphChi (graph processing), 426 graphs, 555 as data models, 49-63 example of graph-structured data, 49 property graphs, 50 RDF and triple-stores, 55-59 versus the network model, 60 processing and analysis, 424-426 fault tolerance, 425 Pregel processing model, 425 query languages Cypher, 52 Datalog, 60-63 recursive SQL queries, 53 SPARQL, 59-59 Gremlin (graph query language), 50 grep (Unix tool), 392 GROUP BY clause (SQL), 406 grouping records in MapReduce, 406 handling skew, 407 H Hadoop (data infrastructure) comparison to distributed databases, 390 comparison to MPP databases, 414-418 comparison to Unix, 413-414, 499 diverse processing models in ecosystem, 417 HDFS distributed filesystem (see HDFS) higher-level tools, 403 join algorithms, 403-410 (see also MapReduce) MapReduce (see MapReduce) YARN (see YARN) happens-before relationship, 340 capturing, 187 concurrency and, 186 hard disks access patterns, 84 detecting corruption, 519, 530 faults in, 7, 227 sequential write throughput, 75, 450 hardware faults, 7 hash indexes, 72-75 broadcast hash joins, 409 partitioned hash joins, 409 hash partitioning, 203-205, 217 consistent hashing, 204 problems with hash mod N, 210 range queries, 204 suitable hash functions, 203 with fixed number of partitions, 210 HAWQ (database), 428 HBase (database) bug due to lack of fencing, 302 bulk loading, 413 column-family data model, 41, 99 dynamic partitioning, 212 key-range partitioning, 202 log-structured storage, 78 request routing, 216 size-tiered compaction, 79 use of HDFS, 417 use of ZooKeeper, 370 HDFS (Hadoop Distributed File System), 398-399 (see also distributed filesystems) checking data integrity, 530 decoupling from query engines, 417 indiscriminately dumping data into, 415 metadata about datasets, 410 NameNode, 398 use by Flink, 479 use by HBase, 212 use by MapReduce, 402 HdrHistogram (numerical library), 16 head (Unix tool), 392 head vertex (property graphs), 51 head-of-line blocking, 15 heap files (databases), 86 Helix (cluster manager), 216 heterogeneous distributed transactions, 360, 364 heuristic decisions (in 2PC), 363 Hibernate (object-relational mapper), 30 hierarchical model, 36 high availability (see fault tolerance) high-frequency trading, 290, 299 high-performance computing (HPC), 275 hinted handoff, 183 histograms, 16 Hive (query engine), 419, 427 for data warehouses, 93 HCatalog and metastore, 410 map-side joins, 409 query optimizer, 427 skewed joins, 408 workflows, 403 Hollerith machines, 390 hopping windows (stream processing), 472 (see also windows) horizontal scaling (see scaling out) HornetQ (messaging), 137, 444 distributed transaction support, 361 hot spots, 201 due to celebrities, 205 for time-series data, 203 in batch processing, 407 relieving, 205 hot standbys (see leader-based replication) HTTP, use in APIs (see services) human errors, 9, 279, 414 HyperDex (database), 88 HyperLogLog (algorithm), 466 I I/O operations, waiting for, 297 IBM DB2 (database) distributed transaction support, 361 recursive query support, 54 serializable isolation, 242, 257 XML and JSON support, 30, 42 electromechanical card-sorting machines, 390 IMS (database), 36 imperative query APIs, 46 InfoSphere Streams (CEP engine), 466 MQ (messaging), 444 distributed transaction support, 361 System R (database), 222 WebSphere (messaging), 137 idempotence, 134, 478, 555 by giving operations unique IDs, 518, 522 idempotent operations, 517 immutability advantages of, 460, 531 Index | 571 deriving state from event log, 459-464 for crash recovery, 75 in B-trees, 82, 242 in event sourcing, 457 inputs to Unix commands, 397 limitations of, 463 Impala (query engine) for data warehouses, 93 hash joins, 409 native code generation, 428 use of HDFS, 417 impedance mismatch, 29 imperative languages, 42 setting element styles (example), 45 in doubt (transaction status), 358 holding locks, 362 orphaned transactions, 363 in-memory databases, 88 durability, 227 serial transaction execution, 253 incidents cascading failures, 9 crashes due to leap seconds, 290 data corruption and financial losses due to concurrency bugs, 233 data corruption on hard disks, 227 data loss due to last-write-wins, 173, 292 data on disks unreadable, 309 deleted items reappearing, 174 disclosure of sensitive data due to primary key reuse, 157 errors in transaction serializability, 529 gigabit network interface with 1 Kb/s throughput, 311 network faults, 279 network interface dropping only inbound packets, 279 network partitions and whole-datacenter failures, 275 poor handling of network faults, 280 sending message to ex-partner, 494 sharks biting undersea cables, 279 split brain due to 1-minute packet delay, 158, 279 vibrations in server rack, 14 violation of uniqueness constraint, 529 indexes, 71, 555 and snapshot isolation, 241 as derived data, 386, 499-504 572 | Index B-trees, 79-83 building in batch processes, 411 clustered, 86 comparison of B-trees and LSM-trees, 83-85 concatenated, 87 covering (with included columns), 86 creating, 500 full-text search, 88 geospatial, 87 hash, 72-75 index-range locking, 260 multi-column, 87 partitioning and secondary indexes, 206-209, 217 secondary, 85 (see also secondary indexes) problems with dual writes, 452, 491 SSTables and LSM-trees, 76-79 updating when data changes, 452, 467 Industrial Revolution, 541 InfiniBand (networks), 285 InfiniteGraph (database), 50 InnoDB (storage engine) clustered index on primary key, 86 not preventing lost updates, 245 preventing write skew, 248, 257 serializable isolation, 257 snapshot isolation support, 239 inside-out databases, 504 (see also unbundling databases) integrating different data systems (see data integration) integrity, 524 coordination-avoiding data systems, 528 correctness of dataflow systems, 525 in consensus formalization, 365 integrity checks, 530 (see also auditing) end-to-end, 519, 531 use of snapshot isolation, 238 maintaining despite software bugs, 529 Interface Definition Language (IDL), 117, 122 intermediate state, materialization of, 420-423 internet services, systems for implementing, 275 invariants, 225 (see also constraints) inversion of control, 396 IP (Internet Protocol) unreliability of, 277 ISDN (Integrated Services Digital Network), 284 isolation (in transactions), 225, 228, 555 correctness and, 515 for single-object writes, 230 serializability, 251-266 actual serial execution, 252-256 serializable snapshot isolation (SSI), 261-266 two-phase locking (2PL), 257-261 violating, 228 weak isolation levels, 233-251 preventing lost updates, 242-246 read committed, 234-237 snapshot isolation, 237-242 iterative processing, 424-426 J Java Database Connectivity (JDBC) distributed transaction support, 361 network drivers, 128 Java Enterprise Edition (EE), 134, 356, 361 Java Message Service (JMS), 444 (see also messaging systems) comparison to log-based messaging, 448, 451 distributed transaction support, 361 message ordering, 446 Java Transaction API (JTA), 355, 361 Java Virtual Machine (JVM) bytecode generation, 428 garbage collection pauses, 296 process reuse in batch processors, 422 JavaScript in MapReduce querying, 46 setting element styles (example), 45 use in advanced queries, 48 Jena (RDF framework), 57 Jepsen (fault tolerance testing), 515 jitter (network delay), 284 joins, 555 by index lookup, 403 expressing as relational operators, 427 in relational and document databases, 34 MapReduce map-side joins, 408-410 broadcast hash joins, 409 merge joins, 410 partitioned hash joins, 409 MapReduce reduce-side joins, 403-408 handling skew, 407 sort-merge joins, 405 parallel execution of, 415 secondary indexes and, 85 stream joins, 472-476 stream-stream join, 473 stream-table join, 473 table-table join, 474 time-dependence of, 475 support in document databases, 42 JOTM (transaction coordinator), 356 JSON Avro schema representation, 122 binary variants, 115 for application data, issues with, 114 in relational databases, 30, 42 representing a résumé (example), 31 Juttle (query language), 504 K k-nearest neighbors, 429 Kafka (messaging), 137, 448 Kafka Connect (database integration), 457, 461 Kafka Streams (stream processor), 466, 467 fault tolerance, 479 leader-based replication, 153 log compaction, 456, 467 message offsets, 447, 478 request routing, 216 transaction support, 477 usage example, 4 Ketama (partitioning library), 213 key-value stores, 70 as batch process output, 412 hash indexes, 72-75 in-memory, 89 partitioning, 201-205 by hash of key, 203, 217 by key range, 202, 217 dynamic partitioning, 212 skew and hot spots, 205 Kryo (Java), 113 Kubernetes (cluster manager), 418, 506 L lambda architecture, 497 Lamport timestamps, 345 Index | 573 Large Hadron Collider (LHC), 64 last write wins (LWW), 173, 334 discarding concurrent writes, 186 problems with, 292 prone to lost updates, 246 late binding, 396 latency instability under two-phase locking, 259 network latency and resource utilization, 286 response time versus, 14 tail latency, 15, 207 leader-based replication, 152-161 (see also replication) failover, 157, 301 handling node outages, 156 implementation of replication logs change data capture, 454-457 (see also changelogs) statement-based, 158 trigger-based replication, 161 write-ahead log (WAL) shipping, 159 linearizability of operations, 333 locking and leader election, 330 log sequence number, 156, 449 read-scaling architecture, 161 relation to consensus, 367 setting up new followers, 155 synchronous versus asynchronous, 153-155 leaderless replication, 177-191 (see also replication) detecting concurrent writes, 184-191 capturing happens-before relationship, 187 happens-before relationship and concur‐ rency, 186 last write wins, 186 merging concurrently written values, 190 version vectors, 191 multi-datacenter, 184 quorums, 179-182 consistency limitations, 181-183, 334 sloppy quorums and hinted handoff, 183 read repair and anti-entropy, 178 leap seconds, 8, 290 in time-of-day clocks, 288 leases, 295 implementation with ZooKeeper, 370 574 | Index need for fencing, 302 ledgers, 460 distributed ledger technologies, 532 legacy systems, maintenance of, 18 less (Unix tool), 397 LevelDB (storage engine), 78 leveled compaction, 79 Levenshtein automata, 88 limping (partial failure), 311 linearizability, 324-338, 555 cost of, 335-338 CAP theorem, 336 memory on multi-core CPUs, 338 definition, 325-329 implementing with total order broadcast, 350 in ZooKeeper, 370 of derived data systems, 492, 524 avoiding coordination, 527 of different replication methods, 332-335 using quorums, 334 relying on, 330-332 constraints and uniqueness, 330 cross-channel timing dependencies, 331 locking and leader election, 330 stronger than causal consistency, 342 using to implement total order broadcast, 351 versus serializability, 329 LinkedIn Azkaban (workflow scheduler), 402 Databus (change data capture), 161, 455 Espresso (database), 31, 126, 130, 153, 216 Helix (cluster manager) (see Helix) profile (example), 30 reference to company entity (example), 34 Rest.li (RPC framework), 135 Voldemort (database) (see Voldemort) Linux, leap second bug, 8, 290 liveness properties, 308 LMDB (storage engine), 82, 242 load approaches to coping with, 17 describing, 11 load testing, 16 load balancing (messaging), 444 local indexes (see document-partitioned indexes) locality (data access), 32, 41, 555 in batch processing, 400, 405, 421 in stateful clients, 170, 511 in stream processing, 474, 478, 508, 522 location transparency, 134 in the actor model, 138 locks, 556 deadlock, 258 distributed locking, 301-304, 330 fencing tokens, 303 implementation with ZooKeeper, 370 relation to consensus, 374 for transaction isolation in snapshot isolation, 239 in two-phase locking (2PL), 257-261 making operations atomic, 243 performance, 258 preventing dirty writes, 236 preventing phantoms with index-range locks, 260, 265 read locks (shared mode), 236, 258 shared mode and exclusive mode, 258 in two-phase commit (2PC) deadlock detection, 364 in-doubt transactions holding locks, 362 materializing conflicts with, 251 preventing lost updates by explicit locking, 244 log sequence number, 156, 449 logic programming languages, 504 logical clocks, 293, 343, 494 for read-after-write consistency, 164 logical logs, 160 logs (data structure), 71, 556 advantages of immutability, 460 compaction, 73, 79, 456, 460 for stream operator state, 479 creating using total order broadcast, 349 implementing uniqueness constraints, 522 log-based messaging, 446-451 comparison to traditional messaging, 448, 451 consumer offsets, 449 disk space usage, 450 replaying old messages, 451, 496, 498 slow consumers, 450 using logs for message storage, 447 log-structured storage, 71-79 log-structured merge tree (see LSMtrees) replication, 152, 158-161 change data capture, 454-457 (see also changelogs) coordination with snapshot, 156 logical (row-based) replication, 160 statement-based replication, 158 trigger-based replication, 161 write-ahead log (WAL) shipping, 159 scalability limits, 493 loose coupling, 396, 419, 502 lost updates (see updates) LSM-trees (indexes), 78-79 comparison to B-trees, 83-85 Lucene (storage engine), 79 building indexes in batch processes, 411 similarity search, 88 Luigi (workflow scheduler), 402 LWW (see last write wins) M machine learning ethical considerations, 534 (see also ethics) iterative processing, 424 models derived from training data, 505 statistical and numerical algorithms, 428 MADlib (machine learning toolkit), 428 magic scaling sauce, 18 Mahout (machine learning toolkit), 428 maintainability, 18-22, 489 defined, 23 design principles for software systems, 19 evolvability (see evolvability) operability, 19 simplicity and managing complexity, 20 many-to-many relationships in document model versus relational model, 39 modeling as graphs, 49 many-to-one and many-to-many relationships, 33-36 many-to-one relationships, 34 MapReduce (batch processing), 390, 399-400 accessing external services within job, 404, 412 comparison to distributed databases designing for frequent faults, 417 diversity of processing models, 416 diversity of storage, 415 Index | 575 comparison to stream processing, 464 comparison to Unix, 413-414 disadvantages and limitations of, 419 fault tolerance, 406, 414, 422 higher-level tools, 403, 426 implementation in Hadoop, 400-403 the shuffle, 402 implementation in MongoDB, 46-48 machine learning, 428 map-side processing, 408-410 broadcast hash joins, 409 merge joins, 410 partitioned hash joins, 409 mapper and reducer functions, 399 materialization of intermediate state, 419-423 output of batch workflows, 411-413 building search indexes, 411 key-value stores, 412 reduce-side processing, 403-408 analysis of user activity events (exam‐ ple), 404 grouping records by same key, 406 handling skew, 407 sort-merge joins, 405 workflows, 402 marshalling (see encoding) massively parallel processing (MPP), 216 comparison to composing storage technolo‐ gies, 502 comparison to Hadoop, 414-418, 428 master-master replication (see multi-leader replication) master-slave replication (see leader-based repli‐ cation) materialization, 556 aggregate values, 101 conflicts, 251 intermediate state (batch processing), 420-423 materialized views, 101 as derived data, 386, 499-504 maintaining, using stream processing, 467, 475 Maven (Java build tool), 428 Maxwell (change data capture), 455 mean, 14 media monitoring, 467 median, 14 576 | Index meeting room booking (example), 249, 259, 521 membership services, 372 Memcached (caching server), 4, 89 memory in-memory databases, 88 durability, 227 serial transaction execution, 253 in-memory representation of data, 112 random bit-flips in, 529 use by indexes, 72, 77 memory barrier (CPU instruction), 338 MemSQL (database) in-memory storage, 89 read committed isolation, 236 memtable (in LSM-trees), 78 Mercurial (version control system), 463 merge joins, MapReduce map-side, 410 mergeable persistent data structures, 174 merging sorted files, 76, 402, 405 Merkle trees, 532 Mesos (cluster manager), 418, 506 message brokers (see messaging systems) message-passing, 136-139 advantages over direct RPC, 137 distributed actor frameworks, 138 evolvability, 138 MessagePack (encoding format), 116 messages exactly-once semantics, 360, 476 loss of, 442 using total order broadcast, 348 messaging systems, 440-451 (see also streams) backpressure, buffering, or dropping mes‐ sages, 441 brokerless messaging, 442 event logs, 446-451 comparison to traditional messaging, 448, 451 consumer offsets, 449 replaying old messages, 451, 496, 498 slow consumers, 450 message brokers, 443-446 acknowledgements and redelivery, 445 comparison to event logs, 448, 451 multiple consumers of same topic, 444 reliability, 442 uniqueness in log-based messaging, 522 Meteor (web framework), 456 microbatching, 477, 495 microservices, 132 (see also services) causal dependencies across services, 493 loose coupling, 502 relation to batch/stream processors, 389, 508 Microsoft Azure Service Bus (messaging), 444 Azure Storage, 155, 398 Azure Stream Analytics, 466 DCOM (Distributed Component Object Model), 134 MSDTC (transaction coordinator), 356 Orleans (see Orleans) SQL Server (see SQL Server) migrating (rewriting) data, 40, 130, 461, 497 modulus operator (%), 210 MongoDB (database) aggregation pipeline, 48 atomic operations, 243 BSON, 41 document data model, 31 hash partitioning (sharding), 203-204 key-range partitioning, 202 lack of join support, 34, 42 leader-based replication, 153 MapReduce support, 46, 400 oplog parsing, 455, 456 partition splitting, 212 request routing, 216 secondary indexes, 207 Mongoriver (change data capture), 455 monitoring, 10, 19 monotonic clocks, 288 monotonic reads, 164 MPP (see massively parallel processing) MSMQ (messaging), 361 multi-column indexes, 87 multi-leader replication, 168-177 (see also replication) handling write conflicts, 171 conflict avoidance, 172 converging toward a consistent state, 172 custom conflict resolution logic, 173 determining what is a conflict, 174 linearizability, lack of, 333 replication topologies, 175-177 use cases, 168 clients with offline operation, 170 collaborative editing, 170 multi-datacenter replication, 168, 335 multi-object transactions, 228 need for, 231 Multi-Paxos (total order broadcast), 367 multi-table index cluster tables (Oracle), 41 multi-tenancy, 284 multi-version concurrency control (MVCC), 239, 266 detecting stale MVCC reads, 263 indexes and snapshot isolation, 241 mutual exclusion, 261 (see also locks) MySQL (database) binlog coordinates, 156 binlog parsing for change data capture, 455 circular replication topology, 175 consistent snapshots, 156 distributed transaction support, 361 InnoDB storage engine (see InnoDB) JSON support, 30, 42 leader-based replication, 153 performance of XA transactions, 360 row-based replication, 160 schema changes in, 40 snapshot isolation support, 242 (see also InnoDB) statement-based replication, 159 Tungsten Replicator (multi-leader replica‐ tion), 170 conflict detection, 177 N nanomsg (messaging library), 442 Narayana (transaction coordinator), 356 NATS (messaging), 137 near-real-time (nearline) processing, 390 (see also stream processing) Neo4j (database) Cypher query language, 52 graph data model, 50 Nephele (dataflow engine), 421 netcat (Unix tool), 397 Netflix Chaos Monkey, 7, 280 Network Attached Storage (NAS), 146, 398 network model, 36 Index | 577 graph databases versus, 60 imperative query APIs, 46 Network Time Protocol (see NTP) networks congestion and queueing, 282 datacenter network topologies, 276 faults (see faults) linearizability and network delays, 338 network partitions, 279, 337 timeouts and unbounded delays, 281 next-key locking, 260 nodes (in graphs) (see vertices) nodes (processes), 556 handling outages in leader-based replica‐ tion, 156 system models for failure, 307 noisy neighbors, 284 nonblocking atomic commit, 359 nondeterministic operations accidental nondeterminism, 423 partial failures in distributed systems, 275 nonfunctional requirements, 22 nonrepeatable reads, 238 (see also read skew) normalization (data representation), 33, 556 executing joins, 39, 42, 403 foreign key references, 231 in systems of record, 386 versus denormalization, 462 NoSQL, 29, 499 transactions and, 223 Notation3 (N3), 56 npm (package manager), 428 NTP (Network Time Protocol), 287 accuracy, 289, 293 adjustments to monotonic clocks, 289 multiple server addresses, 306 numbers, in XML and JSON encodings, 114 O object-relational mapping (ORM) frameworks, 30 error handling and aborted transactions, 232 unsafe read-modify-write cycle code, 244 object-relational mismatch, 29 observer pattern, 506 offline systems, 390 (see also batch processing) 578 | Index stateful, offline-capable clients, 170, 511 offline-first applications, 511 offsets consumer offsets in partitioned logs, 449 messages in partitioned logs, 447 OLAP (online analytic processing), 91, 556 data cubes, 102 OLTP (online transaction processing), 90, 556 analytics queries versus, 411 workload characteristics, 253 one-to-many relationships, 30 JSON representation, 32 online systems, 389 (see also services) Oozie (workflow scheduler), 402 OpenAPI (service definition format), 133 OpenStack Nova (cloud infrastructure) use of ZooKeeper, 370 Swift (object storage), 398 operability, 19 operating systems versus databases, 499 operation identifiers, 518, 522 operational transformation, 174 operators, 421 flow of data between, 424 in stream processing, 464 optimistic concurrency control, 261 Oracle (database) distributed transaction support, 361 GoldenGate (change data capture), 161, 170, 455 lack of serializability, 226 leader-based replication, 153 multi-table index cluster tables, 41 not preventing write skew, 248 partitioned indexes, 209 PL/SQL language, 255 preventing lost updates, 245 read committed isolation, 236 Real Application Clusters (RAC), 330 recursive query support, 54 snapshot isolation support, 239, 242 TimesTen (in-memory database), 89 WAL-based replication, 160 XML support, 30 ordering, 339-352 by sequence numbers, 343-348 causal ordering, 339-343 partial order, 341 limits of total ordering, 493 total order broadcast, 348-352 Orleans (actor framework), 139 outliers (response time), 14 Oz (programming language), 504 P package managers, 428, 505 packet switching, 285 packets corruption of, 306 sending via UDP, 442 PageRank (algorithm), 49, 424 paging (see virtual memory) ParAccel (database), 93 parallel databases (see massively parallel pro‐ cessing) parallel execution of graph analysis algorithms, 426 queries in MPP databases, 216 Parquet (data format), 96, 131 (see also column-oriented storage) use in Hadoop, 414 partial failures, 275, 310 limping, 311 partial order, 341 partitioning, 199-218, 556 and replication, 200 in batch processing, 429 multi-partition operations, 514 enforcing constraints, 522 secondary index maintenance, 495 of key-value data, 201-205 by key range, 202 skew and hot spots, 205 rebalancing partitions, 209-214 automatic or manual rebalancing, 213 problems with hash mod N, 210 using dynamic partitioning, 212 using fixed number of partitions, 210 using N partitions per node, 212 replication and, 147 request routing, 214-216 secondary indexes, 206-209 document-based partitioning, 206 term-based partitioning, 208 serial execution of transactions and, 255 Paxos (consensus algorithm), 366 ballot number, 368 Multi-Paxos (total order broadcast), 367 percentiles, 14, 556 calculating efficiently, 16 importance of high percentiles, 16 use in service level agreements (SLAs), 15 Percona XtraBackup (MySQL tool), 156 performance describing, 13 of distributed transactions, 360 of in-memory databases, 89 of linearizability, 338 of multi-leader replication, 169 perpetual inconsistency, 525 pessimistic concurrency control, 261 phantoms (transaction isolation), 250 materializing conflicts, 251 preventing, in serializability, 259 physical clocks (see clocks) pickle (Python), 113 Pig (dataflow language), 419, 427 replicated joins, 409 skewed joins, 407 workflows, 403 Pinball (workflow scheduler), 402 pipelined execution, 423 in Unix, 394 point in time, 287 polyglot persistence, 29 polystores, 501 PostgreSQL (database) BDR (multi-leader replication), 170 causal ordering of writes, 177 Bottled Water (change data capture), 455 Bucardo (trigger-based replication), 161, 173 distributed transaction support, 361 foreign data wrappers, 501 full text search support, 490 leader-based replication, 153 log sequence number, 156 MVCC implementation, 239, 241 PL/pgSQL language, 255 PostGIS geospatial indexes, 87 preventing lost updates, 245 preventing write skew, 248, 261 read committed isolation, 236 recursive query support, 54 representing graphs, 51 Index | 579 serializable snapshot isolation (SSI), 261 snapshot isolation support, 239, 242 WAL-based replication, 160 XML and JSON support, 30, 42 pre-splitting, 212 Precision Time Protocol (PTP), 290 predicate locks, 259 predictive analytics, 533-536 amplifying bias, 534 ethics of (see ethics) feedback loops, 536 preemption of datacenter resources, 418 of threads, 298 Pregel processing model, 425 primary keys, 85, 556 compound primary key (Cassandra), 204 primary-secondary replication (see leaderbased replication) privacy, 536-543 consent and freedom of choice, 538 data as assets and power, 540 deleting data, 463 ethical considerations (see ethics) legislation and self-regulation, 542 meaning of, 539 surveillance, 537 tracking behavioral data, 536 probabilistic algorithms, 16, 466 process pauses, 295-299 processing time (of events), 469 producers (message streams), 440 programming languages dataflow languages, 504 for stored procedures, 255 functional reactive programming (FRP), 504 logic programming, 504 Prolog (language), 61 (see also Datalog) promises (asynchronous operations), 135 property graphs, 50 Cypher query language, 52 Protocol Buffers (data format), 117-121 field tags and schema evolution, 120 provenance of data, 531 publish/subscribe model, 441 publishers (message streams), 440 punch card tabulating machines, 390 580 | Index pure functions, 48 putting computation near data, 400 Q Qpid (messaging), 444 quality of service (QoS), 285 Quantcast File System (distributed filesystem), 398 query languages, 42-48 aggregation pipeline, 48 CSS and XSL, 44 Cypher, 52 Datalog, 60 Juttle, 504 MapReduce querying, 46-48 recursive SQL queries, 53 relational algebra and SQL, 42 SPARQL, 59 query optimizers, 37, 427 queueing delays (networks), 282 head-of-line blocking, 15 latency and response time, 14 queues (messaging), 137 quorums, 179-182, 556 for leaderless replication, 179 in consensus algorithms, 368 limitations of consistency, 181-183, 334 making decisions in distributed systems, 301 monitoring staleness, 182 multi-datacenter replication, 184 relying on durability, 309 sloppy quorums and hinted handoff, 183 R R-trees (indexes), 87 RabbitMQ (messaging), 137, 444 leader-based replication, 153 race conditions, 225 (see also concurrency) avoiding with linearizability, 331 caused by dual writes, 452 dirty writes, 235 in counter increments, 235 lost updates, 242-246 preventing with event logs, 462, 507 preventing with serializable isolation, 252 write skew, 246-251 Raft (consensus algorithm), 366 sensitivity to network problems, 369 term number, 368 use in etcd, 353 RAID (Redundant Array of Independent Disks), 7, 398 railways, schema migration on, 496 RAMCloud (in-memory storage), 89 ranking algorithms, 424 RDF (Resource Description Framework), 57 querying with SPARQL, 59 RDMA (Remote Direct Memory Access), 276 read committed isolation level, 234-237 implementing, 236 multi-version concurrency control (MVCC), 239 no dirty reads, 234 no dirty writes, 235 read path (derived data), 509 read repair (leaderless replication), 178 for linearizability, 335 read replicas (see leader-based replication) read skew (transaction isolation), 238, 266 as violation of causality, 340 read-after-write consistency, 163, 524 cross-device, 164 read-modify-write cycle, 243 read-scaling architecture, 161 reads as events, 513 real-time collaborative editing, 170 near-real-time processing, 390 (see also stream processing) publish/subscribe dataflow, 513 response time guarantees, 298 time-of-day clocks, 288 rebalancing partitions, 209-214, 556 (see also partitioning) automatic or manual rebalancing, 213 dynamic partitioning, 212 fixed number of partitions, 210 fixed number of partitions per node, 212 problems with hash mod N, 210 recency guarantee, 324 recommendation engines batch process outputs, 412 batch workflows, 403, 420 iterative processing, 424 statistical and numerical algorithms, 428 records, 399 events in stream processing, 440 recursive common table expressions (SQL), 54 redelivery (messaging), 445 Redis (database) atomic operations, 243 durability, 89 Lua scripting, 255 single-threaded execution, 253 usage example, 4 redundancy hardware components, 7 of derived data, 386 (see also derived data) Reed–Solomon codes (error correction), 398 refactoring, 22 (see also evolvability) regions (partitioning), 199 register (data structure), 325 relational data model, 28-42 comparison to document model, 38-42 graph queries in SQL, 53 in-memory databases with, 89 many-to-one and many-to-many relation‐ ships, 33 multi-object transactions, need for, 231 NoSQL as alternative to, 29 object-relational mismatch, 29 relational algebra and SQL, 42 versus document model convergence of models, 41 data locality, 41 relational databases eventual consistency, 162 history, 28 leader-based replication, 153 logical logs, 160 philosophy compared to Unix, 499, 501 schema changes, 40, 111, 130 statement-based replication, 158 use of B-tree indexes, 80 relationships (see edges) reliability, 6-10, 489 building a reliable system from unreliable components, 276 defined, 6, 22 hardware faults, 7 human errors, 9 importance of, 10 of messaging systems, 442 Index | 581 software errors, 8 Remote Method Invocation (Java RMI), 134 remote procedure calls (RPCs), 134-136 (see also services) based on futures, 135 data encoding and evolution, 136 issues with, 134 using Avro, 126, 135 using Thrift, 135 versus message brokers, 137 repeatable reads (transaction isolation), 242 replicas, 152 replication, 151-193, 556 and durability, 227 chain replication, 155 conflict resolution and, 246 consistency properties, 161-167 consistent prefix reads, 165 monotonic reads, 164 reading your own writes, 162 in distributed filesystems, 398 leaderless, 177-191 detecting concurrent writes, 184-191 limitations of quorum consistency, 181-183, 334 sloppy quorums and hinted handoff, 183 monitoring staleness, 182 multi-leader, 168-177 across multiple datacenters, 168, 335 handling write conflicts, 171-175 replication topologies, 175-177 partitioning and, 147, 200 reasons for using, 145, 151 single-leader, 152-161 failover, 157 implementation of replication logs, 158-161 relation to consensus, 367 setting up new followers, 155 synchronous versus asynchronous, 153-155 state machine replication, 349, 452 using erasure coding, 398 with heterogeneous data systems, 453 replication logs (see logs) reprocessing data, 496, 498 (see also evolvability) from log-based messaging, 451 request routing, 214-216 582 | Index approaches to, 214 parallel query execution, 216 resilient systems, 6 (see also fault tolerance) response time as performance metric for services, 13, 389 guarantees on, 298 latency versus, 14 mean and percentiles, 14 user experience, 15 responsibility and accountability, 535 REST (Representational State Transfer), 133 (see also services) RethinkDB (database) document data model, 31 dynamic partitioning, 212 join support, 34, 42 key-range partitioning, 202 leader-based replication, 153 subscribing to changes, 456 Riak (database) Bitcask storage engine, 72 CRDTs, 174, 191 dotted version vectors, 191 gossip protocol, 216 hash partitioning, 203-204, 211 last-write-wins conflict resolution, 186 leaderless replication, 177 LevelDB storage engine, 78 linearizability, lack of, 335 multi-datacenter support, 184 preventing lost updates across replicas, 246 rebalancing, 213 search feature, 209 secondary indexes, 207 siblings (concurrently written values), 190 sloppy quorums, 184 ring buffers, 450 Ripple (cryptocurrency), 532 rockets, 10, 36, 305 RocksDB (storage engine), 78 leveled compaction, 79 rollbacks (transactions), 222 rolling upgrades, 8, 112 routing (see request routing) row-oriented storage, 96 row-based replication, 160 rowhammer (memory corruption), 529 RPCs (see remote procedure calls) Rubygems (package manager), 428 rules (Datalog), 61 S safety and liveness properties, 308 in consensus algorithms, 366 in transactions, 222 sagas (see compensating transactions) Samza (stream processor), 466, 467 fault tolerance, 479 streaming SQL support, 466 sandboxes, 9 SAP HANA (database), 93 scalability, 10-18, 489 approaches for coping with load, 17 defined, 22 describing load, 11 describing performance, 13 partitioning and, 199 replication and, 161 scaling up versus scaling out, 146 scaling out, 17, 146 (see also shared-nothing architecture) scaling up, 17, 146 scatter/gather approach, querying partitioned databases, 207 SCD (slowly changing dimension), 476 schema-on-read, 39 comparison to evolvable schema, 128 in distributed filesystems, 415 schema-on-write, 39 schemaless databases (see schema-on-read) schemas, 557 Avro, 122-127 reader determining writer’s schema, 125 schema evolution, 123 dynamically generated, 126 evolution of, 496 affecting application code, 111 compatibility checking, 126 in databases, 129-131 in message-passing, 138 in service calls, 136 flexibility in document model, 39 for analytics, 93-95 for JSON and XML, 115 merits of, 127 schema migration on railways, 496 Thrift and Protocol Buffers, 117-121 schema evolution, 120 traditional approach to design, fallacy in, 462 searches building search indexes in batch processes, 411 k-nearest neighbors, 429 on streams, 467 partitioned secondary indexes, 206 secondaries (see leader-based replication) secondary indexes, 85, 557 partitioning, 206-209, 217 document-partitioned, 206 index maintenance, 495 term-partitioned, 208 problems with dual writes, 452, 491 updating, transaction isolation and, 231 secondary sorts, 405 sed (Unix tool), 392 self-describing files, 127 self-joins, 480 self-validating systems, 530 semantic web, 57 semi-synchronous replication, 154 sequence number ordering, 343-348 generators, 294, 344 insufficiency for enforcing constraints, 347 Lamport timestamps, 345 use of timestamps, 291, 295, 345 sequential consistency, 351 serializability, 225, 233, 251-266, 557 linearizability versus, 329 pessimistic versus optimistic concurrency control, 261 serial execution, 252-256 partitioning, 255 using stored procedures, 253, 349 serializable snapshot isolation (SSI), 261-266 detecting stale MVCC reads, 263 detecting writes that affect prior reads, 264 distributed execution, 265, 364 performance of SSI, 265 preventing write skew, 262-265 two-phase locking (2PL), 257-261 index-range locks, 260 performance, 258 Serializable (Java), 113 Index | 583 serialization, 113 (see also encoding) service discovery, 135, 214, 372 using DNS, 216, 372 service level agreements (SLAs), 15 service-oriented architecture (SOA), 132 (see also services) services, 131-136 microservices, 132 causal dependencies across services, 493 loose coupling, 502 relation to batch/stream processors, 389, 508 remote procedure calls (RPCs), 134-136 issues with, 134 similarity to databases, 132 web services, 132, 135 session windows (stream processing), 472 (see also windows) sessionization, 407 sharding (see partitioning) shared mode (locks), 258 shared-disk architecture, 146, 398 shared-memory architecture, 146 shared-nothing architecture, 17, 146-147, 557 (see also replication) distributed filesystems, 398 (see also distributed filesystems) partitioning, 199 use of network, 277 sharks biting undersea cables, 279 counting (example), 46-48 finding (example), 42 website about (example), 44 shredding (in relational model), 38 siblings (concurrent values), 190, 246 (see also conflicts) similarity search edit distance, 88 genome data, 63 k-nearest neighbors, 429 single-leader replication (see leader-based rep‐ lication) single-threaded execution, 243, 252 in batch processing, 406, 421, 426 in stream processing, 448, 463, 522 size-tiered compaction, 79 skew, 557 584 | Index clock skew, 291-294, 334 in transaction isolation read skew, 238, 266 write skew, 246-251, 262-265 (see also write skew) meanings of, 238 unbalanced workload, 201 compensating for, 205 due to celebrities, 205 for time-series data, 203 in batch processing, 407 slaves (see leader-based replication) sliding windows (stream processing), 472 (see also windows) sloppy quorums, 183 (see also quorums) lack of linearizability, 334 slowly changing dimension (data warehouses), 476 smearing (leap seconds adjustments), 290 snapshots (databases) causal consistency, 340 computing derived data, 500 in change data capture, 455 serializable snapshot isolation (SSI), 261-266, 329 setting up a new replica, 156 snapshot isolation and repeatable read, 237-242 implementing with MVCC, 239 indexes and MVCC, 241 visibility rules, 240 synchronized clocks for global snapshots, 294 snowflake schemas, 95 SOAP, 133 (see also services) evolvability, 136 software bugs, 8 maintaining integrity, 529 solid state drives (SSDs) access patterns, 84 detecting corruption, 519, 530 faults in, 227 sequential write throughput, 75 Solr (search server) building indexes in batch processes, 411 document-partitioned indexes, 207 request routing, 216 usage example, 4 use of Lucene, 79 sort (Unix tool), 392, 394, 395 sort-merge joins (MapReduce), 405 Sorted String Tables (see SSTables) sorting sort order in column storage, 99 source of truth (see systems of record) Spanner (database) data locality, 41 snapshot isolation using clocks, 295 TrueTime API, 294 Spark (processing framework), 421-423 bytecode generation, 428 dataflow APIs, 427 fault tolerance, 422 for data warehouses, 93 GraphX API (graph processing), 425 machine learning, 428 query optimizer, 427 Spark Streaming, 466 microbatching, 477 stream processing on top of batch process‐ ing, 495 SPARQL (query language), 59 spatial algorithms, 429 split brain, 158, 557 in consensus algorithms, 352, 367 preventing, 322, 333 using fencing tokens to avoid, 302-304 spreadsheets, dataflow programming capabili‐ ties, 504 SQL (Structured Query Language), 21, 28, 43 advantages and limitations of, 416 distributed query execution, 48 graph queries in, 53 isolation levels standard, issues with, 242 query execution on Hadoop, 416 résumé (example), 30 SQL injection vulnerability, 305 SQL on Hadoop, 93 statement-based replication, 158 stored procedures, 255 SQL Server (database) data warehousing support, 93 distributed transaction support, 361 leader-based replication, 153 preventing lost updates, 245 preventing write skew, 248, 257 read committed isolation, 236 recursive query support, 54 serializable isolation, 257 snapshot isolation support, 239 T-SQL language, 255 XML support, 30 SQLstream (stream analytics), 466 SSDs (see solid state drives) SSTables (storage format), 76-79 advantages over hash indexes, 76 concatenated index, 204 constructing and maintaining, 78 making LSM-Tree from, 78 staleness (old data), 162 cross-channel timing dependencies, 331 in leaderless databases, 178 in multi-version concurrency control, 263 monitoring for, 182 of client state, 512 versus linearizability, 324 versus timeliness, 524 standbys (see leader-based replication) star replication topologies, 175 star schemas, 93-95 similarity to event sourcing, 458 Star Wars analogy (event time versus process‐ ing time), 469 state derived from log of immutable events, 459 deriving current state from the event log, 458 interplay between state changes and appli‐ cation code, 507 maintaining derived state, 495 maintenance by stream processor in streamstream joins, 473 observing derived state, 509-515 rebuilding after stream processor failure, 478 separation of application code and, 505 state machine replication, 349, 452 statement-based replication, 158 statically typed languages analogy to schema-on-write, 40 code generation and, 127 statistical and numerical algorithms, 428 StatsD (metrics aggregator), 442 stdin, stdout, 395, 396 Stellar (cryptocurrency), 532 Index | 585 stock market feeds, 442 STONITH (Shoot The Other Node In The Head), 158 stop-the-world (see garbage collection) storage composing data storage technologies, 499-504 diversity of, in MapReduce, 415 Storage Area Network (SAN), 146, 398 storage engines, 69-104 column-oriented, 95-101 column compression, 97-99 defined, 96 distinction between column families and, 99 Parquet, 96, 131 sort order in, 99-100 writing to, 101 comparing requirements for transaction processing and analytics, 90-96 in-memory storage, 88 durability, 227 row-oriented, 70-90 B-trees, 79-83 comparing B-trees and LSM-trees, 83-85 defined, 96 log-structured, 72-79 stored procedures, 161, 253-255, 557 and total order broadcast, 349 pros and cons of, 255 similarity to stream processors, 505 Storm (stream processor), 466 distributed RPC, 468, 514 Trident state handling, 478 straggler events, 470, 498 stream processing, 464-481, 557 accessing external services within job, 474, 477, 478, 517 combining with batch processing lambda architecture, 497 unifying technologies, 498 comparison to batch processing, 464 complex event processing (CEP), 465 fault tolerance, 476-479 atomic commit, 477 idempotence, 478 microbatching and checkpointing, 477 rebuilding state after a failure, 478 for data integration, 494-498 586 | Index maintaining derived state, 495 maintenance of materialized views, 467 messaging systems (see messaging systems) reasoning about time, 468-472 event time versus processing time, 469, 477, 498 knowing when window is ready, 470 types of windows, 472 relation to databases (see streams) relation to services, 508 search on streams, 467 single-threaded execution, 448, 463 stream analytics, 466 stream joins, 472-476 stream-stream join, 473 stream-table join, 473 table-table join, 474 time-dependence of, 475 streams, 440-451 end-to-end, pushing events to clients, 512 messaging systems (see messaging systems) processing (see stream processing) relation to databases, 451-464 (see also changelogs) API support for change streams, 456 change data capture, 454-457 derivative of state by time, 460 event sourcing, 457-459 keeping systems in sync, 452-453 philosophy of immutable events, 459-464 topics, 440 strict serializability, 329 strong consistency (see linearizability) strong one-copy serializability, 329 subjects, predicates, and objects (in triplestores), 55 subscribers (message streams), 440 (see also consumers) supercomputers, 275 surveillance, 537 (see also privacy) Swagger (service definition format), 133 swapping to disk (see virtual memory) synchronous networks, 285, 557 comparison to asynchronous networks, 284 formal model, 307 synchronous replication, 154, 557 chain replication, 155 conflict detection, 172 system models, 300, 306-310 assumptions in, 528 correctness of algorithms, 308 mapping to the real world, 309 safety and liveness, 308 systems of record, 386, 557 change data capture, 454, 491 treating event log as, 460 systems thinking, 536 T t-digest (algorithm), 16 table-table joins, 474 Tableau (data visualization software), 416 tail (Unix tool), 447 tail vertex (property graphs), 51 Tajo (query engine), 93 Tandem NonStop SQL (database), 200 TCP (Transmission Control Protocol), 277 comparison to circuit switching, 285 comparison to UDP, 283 connection failures, 280 flow control, 282, 441 packet checksums, 306, 519, 529 reliability and duplicate suppression, 517 retransmission timeouts, 284 use for transaction sessions, 229 telemetry (see monitoring) Teradata (database), 93, 200 term-partitioned indexes, 208, 217 termination (consensus), 365 Terrapin (database), 413 Tez (dataflow engine), 421-423 fault tolerance, 422 support by higher-level tools, 427 thrashing (out of memory), 297 threads (concurrency) actor model, 138, 468 (see also message-passing) atomic operations, 223 background threads, 73, 85 execution pauses, 286, 296-298 memory barriers, 338 preemption, 298 single (see single-threaded execution) three-phase commit, 359 Thrift (data format), 117-121 BinaryProtocol, 118 CompactProtocol, 119 field tags and schema evolution, 120 throughput, 13, 390 TIBCO, 137 Enterprise Message Service, 444 StreamBase (stream analytics), 466 time concurrency and, 187 cross-channel timing dependencies, 331 in distributed systems, 287-299 (see also clocks) clock synchronization and accuracy, 289 relying on synchronized clocks, 291-295 process pauses, 295-299 reasoning about, in stream processors, 468-472 event time versus processing time, 469, 477, 498 knowing when window is ready, 470 timestamp of events, 471 types of windows, 472 system models for distributed systems, 307 time-dependence in stream joins, 475 time-of-day clocks, 288 timeliness, 524 coordination-avoiding data systems, 528 correctness of dataflow systems, 525 timeouts, 279, 557 dynamic configuration of, 284 for failover, 158 length of, 281 timestamps, 343 assigning to events in stream processing, 471 for read-after-write consistency, 163 for transaction ordering, 295 insufficiency for enforcing constraints, 347 key range partitioning by, 203 Lamport, 345 logical, 494 ordering events, 291, 345 Titan (database), 50 tombstones, 74, 191, 456 topics (messaging), 137, 440 total order, 341, 557 limits of, 493 sequence numbers or timestamps, 344 total order broadcast, 348-352, 493, 522 consensus algorithms and, 366-368 Index | 587 implementation in ZooKeeper and etcd, 370 implementing with linearizable storage, 351 using, 349 using to implement linearizable storage, 350 tracking behavioral data, 536 (see also privacy) transaction coordinator (see coordinator) transaction manager (see coordinator) transaction processing, 28, 90-95 comparison to analytics, 91 comparison to data warehousing, 93 transactions, 221-267, 558 ACID properties of, 223 atomicity, 223 consistency, 224 durability, 226 isolation, 225 compensating (see compensating transac‐ tions) concept of, 222 distributed transactions, 352-364 avoiding, 492, 502, 521-528 failure amplification, 364, 495 in doubt/uncertain status, 358, 362 two-phase commit, 354-359 use of, 360-361 XA transactions, 361-364 OLTP versus analytics queries, 411 purpose of, 222 serializability, 251-266 actual serial execution, 252-256 pessimistic versus optimistic concur‐ rency control, 261 serializable snapshot isolation (SSI), 261-266 two-phase locking (2PL), 257-261 single-object and multi-object, 228-232 handling errors and aborts, 231 need for multi-object transactions, 231 single-object writes, 230 snapshot isolation (see snapshots) weak isolation levels, 233-251 preventing lost updates, 242-246 read committed, 234-238 transitive closure (graph algorithm), 424 trie (data structure), 88 triggers (databases), 161, 441 implementing change data capture, 455 implementing replication, 161 588 | Index triple-stores, 55-59 SPARQL query language, 59 tumbling windows (stream processing), 472 (see also windows) in microbatching, 477 tuple spaces (programming model), 507 Turtle (RDF data format), 56 Twitter constructing home timelines (example), 11, 462, 474, 511 DistributedLog (event log), 448 Finagle (RPC framework), 135 Snowflake (sequence number generator), 294 Summingbird (processing library), 497 two-phase commit (2PC), 353, 355-359, 558 confusion with two-phase locking, 356 coordinator failure, 358 coordinator recovery, 363 how it works, 357 issues in practice, 363 performance cost, 360 transactions holding locks, 362 two-phase locking (2PL), 257-261, 329, 558 confusion with two-phase commit, 356 index-range locks, 260 performance of, 258 type checking, dynamic versus static, 40 U UDP (User Datagram Protocol) comparison to TCP, 283 multicast, 442 unbounded datasets, 439, 558 (see also streams) unbounded delays, 558 in networks, 282 process pauses, 296 unbundling databases, 499-515 composing data storage technologies, 499-504 federation versus unbundling, 501 need for high-level language, 503 designing applications around dataflow, 504-509 observing derived state, 509-515 materialized views and caching, 510 multi-partition data processing, 514 pushing state changes to clients, 512 uncertain (transaction status) (see in doubt) uniform consensus, 365 (see also consensus) uniform interfaces, 395 union type (in Avro), 125 uniq (Unix tool), 392 uniqueness constraints asynchronously checked, 526 requiring consensus, 521 requiring linearizability, 330 uniqueness in log-based messaging, 522 Unix philosophy, 394-397 command-line batch processing, 391-394 Unix pipes versus dataflow engines, 423 comparison to Hadoop, 413-414 comparison to relational databases, 499, 501 comparison to stream processing, 464 composability and uniform interfaces, 395 loose coupling, 396 pipes, 394 relation to Hadoop, 499 UPDATE statement (SQL), 40 updates preventing lost updates, 242-246 atomic write operations, 243 automatically detecting lost updates, 245 compare-and-set operations, 245 conflict resolution and replication, 246 using explicit locking, 244 preventing write skew, 246-251 V validity (consensus), 365 vBuckets (partitioning), 199 vector clocks, 191 (see also version vectors) vectorized processing, 99, 428 verification, 528-533 avoiding blind trust, 530 culture of, 530 designing for auditability, 531 end-to-end integrity checks, 531 tools for auditable data systems, 532 version control systems, reliance on immutable data, 463 version vectors, 177, 191 capturing causal dependencies, 343 versus vector clocks, 191 Vertica (database), 93 handling writes, 101 replicas using different sort orders, 100 vertical scaling (see scaling up) vertices (in graphs), 49 property graph model, 50 Viewstamped Replication (consensus algo‐ rithm), 366 view number, 368 virtual machines, 146 (see also cloud computing) context switches, 297 network performance, 282 noisy neighbors, 284 reliability in cloud services, 8 virtualized clocks in, 290 virtual memory process pauses due to page faults, 14, 297 versus memory management by databases, 89 VisiCalc (spreadsheets), 504 vnodes (partitioning), 199 Voice over IP (VoIP), 283 Voldemort (database) building read-only stores in batch processes, 413 hash partitioning, 203-204, 211 leaderless replication, 177 multi-datacenter support, 184 rebalancing, 213 reliance on read repair, 179 sloppy quorums, 184 VoltDB (database) cross-partition serializability, 256 deterministic stored procedures, 255 in-memory storage, 89 output streams, 456 secondary indexes, 207 serial execution of transactions, 253 statement-based replication, 159, 479 transactions in stream processing, 477 W WAL (write-ahead log), 82 web services (see services) Web Services Description Language (WSDL), 133 webhooks, 443 webMethods (messaging), 137 WebSocket (protocol), 512 Index | 589 windows (stream processing), 466, 468-472 infinite windows for changelogs, 467, 474 knowing when all events have arrived, 470 stream joins within a window, 473 types of windows, 472 winners (conflict resolution), 173 WITH RECURSIVE syntax (SQL), 54 workflows (MapReduce), 402 outputs, 411-414 key-value stores, 412 search indexes, 411 with map-side joins, 410 working set, 393 write amplification, 84 write path (derived data), 509 write skew (transaction isolation), 246-251 characterizing, 246-251, 262 examples of, 247, 249 materializing conflicts, 251 occurrence in practice, 529 phantoms, 250 preventing in snapshot isolation, 262-265 in two-phase locking, 259-261 options for, 248 write-ahead log (WAL), 82, 159 writes (database) atomic write operations, 243 detecting writes affecting prior reads, 264 preventing dirty writes with read commit‐ ted, 235 WS-* framework, 133 (see also services) WS-AtomicTransaction (2PC), 355 590 | Index X XA transactions, 355, 361-364 heuristic decisions, 363 limitations of, 363 xargs (Unix tool), 392, 396 XML binary variants, 115 encoding RDF data, 57 for application data, issues with, 114 in relational databases, 30, 41 XSL/XPath, 45 Y Yahoo!

…

The opposite of bounded. 558 | Glossary Index A aborts (transactions), 222, 224 in two-phase commit, 356 performance of optimistic concurrency con‐ trol, 266 retrying aborted transactions, 231 abstraction, 21, 27, 222, 266, 321 access path (in network model), 37, 60 accidental complexity, removing, 21 accountability, 535 ACID properties (transactions), 90, 223 atomicity, 223, 228 consistency, 224, 529 durability, 226 isolation, 225, 228 acknowledgements (messaging), 445 active/active replication (see multi-leader repli‐ cation) active/passive replication (see leader-based rep‐ lication) ActiveMQ (messaging), 137, 444 distributed transaction support, 361 ActiveRecord (object-relational mapper), 30, 232 actor model, 138 (see also message-passing) comparison to Pregel model, 425 comparison to stream processing, 468 Advanced Message Queuing Protocol (see AMQP) aerospace systems, 6, 10, 305, 372 aggregation data cubes and materialized views, 101 in batch processes, 406 in stream processes, 466 aggregation pipeline query language, 48 Agile, 22 minimizing irreversibility, 414, 497 moving faster with confidence, 532 Unix philosophy, 394 agreement, 365 (see also consensus) Airflow (workflow scheduler), 402 Ajax, 131 Akka (actor framework), 139 algorithms algorithm correctness, 308 B-trees, 79-83 for distributed systems, 306 hash indexes, 72-75 mergesort, 76, 402, 405 red-black trees, 78 SSTables and LSM-trees, 76-79 all-to-all replication topologies, 175 AllegroGraph (database), 50 ALTER TABLE statement (SQL), 40, 111 Amazon Dynamo (database), 177 Amazon Web Services (AWS), 8 Kinesis Streams (messaging), 448 network reliability, 279 postmortems, 9 RedShift (database), 93 S3 (object storage), 398 checking data integrity, 530 amplification of bias, 534 of failures, 364, 495 Index | 559 of tail latency, 16, 207 write amplification, 84 AMQP (Advanced Message Queuing Protocol), 444 (see also messaging systems) comparison to log-based messaging, 448, 451 message ordering, 446 analytics, 90 comparison to transaction processing, 91 data warehousing (see data warehousing) parallel query execution in MPP databases, 415 predictive (see predictive analytics) relation to batch processing, 411 schemas for, 93-95 snapshot isolation for queries, 238 stream analytics, 466 using MapReduce, analysis of user activity events (example), 404 anti-caching (in-memory databases), 89 anti-entropy, 178 Apache ActiveMQ (see ActiveMQ) Apache Avro (see Avro) Apache Beam (see Beam) Apache BookKeeper (see BookKeeper) Apache Cassandra (see Cassandra) Apache CouchDB (see CouchDB) Apache Curator (see Curator) Apache Drill (see Drill) Apache Flink (see Flink) Apache Giraph (see Giraph) Apache Hadoop (see Hadoop) Apache HAWQ (see HAWQ) Apache HBase (see HBase) Apache Helix (see Helix) Apache Hive (see Hive) Apache Impala (see Impala) Apache Jena (see Jena) Apache Kafka (see Kafka) Apache Lucene (see Lucene) Apache MADlib (see MADlib) Apache Mahout (see Mahout) Apache Oozie (see Oozie) Apache Parquet (see Parquet) Apache Qpid (see Qpid) Apache Samza (see Samza) Apache Solr (see Solr) Apache Spark (see Spark) 560 | Index Apache Storm (see Storm) Apache Tajo (see Tajo) Apache Tez (see Tez) Apache Thrift (see Thrift) Apache ZooKeeper (see ZooKeeper) Apama (stream analytics), 466 append-only B-trees, 82, 242 append-only files (see logs) Application Programming Interfaces (APIs), 5, 27 for batch processing, 403 for change streams, 456 for distributed transactions, 361 for graph processing, 425 for services, 131-136 (see also services) evolvability, 136 RESTful, 133 SOAP, 133 application state (see state) approximate search (see similarity search) archival storage, data from databases, 131 arcs (see edges) arithmetic mean, 14 ASCII text, 119, 395 ASN.1 (schema language), 127 asynchronous networks, 278, 553 comparison to synchronous networks, 284 formal model, 307 asynchronous replication, 154, 553 conflict detection, 172 data loss on failover, 157 reads from asynchronous follower, 162 Asynchronous Transfer Mode (ATM), 285 atomic broadcast (see total order broadcast) atomic clocks (caesium clocks), 294, 295 (see also clocks) atomicity (concurrency), 553 atomic increment-and-get, 351 compare-and-set, 245, 327 (see also compare-and-set operations) replicated operations, 246 write operations, 243 atomicity (transactions), 223, 228, 553 atomic commit, 353 avoiding, 523, 528 blocking and nonblocking, 359 in stream processing, 360, 477 maintaining derived data, 453 for multi-object transactions, 229 for single-object writes, 230 auditability, 528-533 designing for, 531 self-auditing systems, 530 through immutability, 460 tools for auditable data systems, 532 availability, 8 (see also fault tolerance) in CAP theorem, 337 in service level agreements (SLAs), 15 Avro (data format), 122-127 code generation, 127 dynamically generated schemas, 126 object container files, 125, 131, 414 reader determining writer’s schema, 125 schema evolution, 123 use in Hadoop, 414 awk (Unix tool), 391 AWS (see Amazon Web Services) Azure (see Microsoft) B B-trees (indexes), 79-83 append-only/copy-on-write variants, 82, 242 branching factor, 81 comparison to LSM-trees, 83-85 crash recovery, 82 growing by splitting a page, 81 optimizations, 82 similarity to dynamic partitioning, 212 backpressure, 441, 553 in TCP, 282 backups database snapshot for replication, 156 integrity of, 530 snapshot isolation for, 238 use for ETL processes, 405 backward compatibility, 112 BASE, contrast to ACID, 223 bash shell (Unix), 70, 395, 503 batch processing, 28, 389-431, 553 combining with stream processing lambda architecture, 497 unifying technologies, 498 comparison to MPP databases, 414-418 comparison to stream processing, 464 comparison to Unix, 413-414 dataflow engines, 421-423 fault tolerance, 406, 414, 422, 442 for data integration, 494-498 graphs and iterative processing, 424-426 high-level APIs and languages, 403, 426-429 log-based messaging and, 451 maintaining derived state, 495 MapReduce and distributed filesystems, 397-413 (see also MapReduce) measuring performance, 13, 390 outputs, 411-413 key-value stores, 412 search indexes, 411 using Unix tools (example), 391-394 Bayou (database), 522 Beam (dataflow library), 498 bias, 534 big ball of mud, 20 Bigtable data model, 41, 99 binary data encodings, 115-128 Avro, 122-127 MessagePack, 116-117 Thrift and Protocol Buffers, 117-121 binary encoding based on schemas, 127 by network drivers, 128 binary strings, lack of support in JSON and XML, 114 BinaryProtocol encoding (Thrift), 118 Bitcask (storage engine), 72 crash recovery, 74 Bitcoin (cryptocurrency), 532 Byzantine fault tolerance, 305 concurrency bugs in exchanges, 233 bitmap indexes, 97 blockchains, 532 Byzantine fault tolerance, 305 blocking atomic commit, 359 Bloom (programming language), 504 Bloom filter (algorithm), 79, 466 BookKeeper (replicated log), 372 Bottled Water (change data capture), 455 bounded datasets, 430, 439, 553 (see also batch processing) bounded delays, 553 in networks, 285 process pauses, 298 broadcast hash joins, 409 Index | 561 brokerless messaging, 442 Brubeck (metrics aggregator), 442 BTM (transaction coordinator), 356 bulk synchronous parallel (BSP) model, 425 bursty network traffic patterns, 285 business data processing, 28, 90, 390 byte sequence, encoding data in, 112 Byzantine faults, 304-306, 307, 553 Byzantine fault-tolerant systems, 305, 532 Byzantine Generals Problem, 304 consensus algorithms and, 366 C caches, 89, 553 and materialized views, 101 as derived data, 386, 499-504 database as cache of transaction log, 460 in CPUs, 99, 338, 428 invalidation and maintenance, 452, 467 linearizability, 324 CAP theorem, 336-338, 554 Cascading (batch processing), 419, 427 hash joins, 409 workflows, 403 cascading failures, 9, 214, 281 Cascalog (batch processing), 60 Cassandra (database) column-family data model, 41, 99 compaction strategy, 79 compound primary key, 204 gossip protocol, 216 hash partitioning, 203-205 last-write-wins conflict resolution, 186, 292 leaderless replication, 177 linearizability, lack of, 335 log-structured storage, 78 multi-datacenter support, 184 partitioning scheme, 213 secondary indexes, 207 sloppy quorums, 184 cat (Unix tool), 391 causal context, 191 (see also causal dependencies) causal dependencies, 186-191 capturing, 191, 342, 494, 514 by total ordering, 493 causal ordering, 339 in transactions, 262 sending message to friends (example), 494 562 | Index causality, 554 causal ordering, 339-343 linearizability and, 342 total order consistent with, 344, 345 consistency with, 344-347 consistent snapshots, 340 happens-before relationship, 186 in serializable transactions, 262-265 mismatch with clocks, 292 ordering events to capture, 493 violations of, 165, 176, 292, 340 with synchronized clocks, 294 CEP (see complex event processing) certificate transparency, 532 chain replication, 155 linearizable reads, 351 change data capture, 160, 454 API support for change streams, 456 comparison to event sourcing, 457 implementing, 454 initial snapshot, 455 log compaction, 456 changelogs, 460 change data capture, 454 for operator state, 479 generating with triggers, 455 in stream joins, 474 log compaction, 456 maintaining derived state, 452 Chaos Monkey, 7, 280 checkpointing in batch processors, 422, 426 in high-performance computing, 275 in stream processors, 477, 523 chronicle data model, 458 circuit-switched networks, 284 circular buffers, 450 circular replication topologies, 175 clickstream data, analysis of, 404 clients calling services, 131 pushing state changes to, 512 request routing, 214 stateful and offline-capable, 170, 511 clocks, 287-299 atomic (caesium) clocks, 294, 295 confidence interval, 293-295 for global snapshots, 294 logical (see logical clocks) skew, 291-294, 334 slewing, 289 synchronization and accuracy, 289-291 synchronization using GPS, 287, 290, 294, 295 time-of-day versus monotonic clocks, 288 timestamping events, 471 cloud computing, 146, 275 need for service discovery, 372 network glitches, 279 shared resources, 284 single-machine reliability, 8 Cloudera Impala (see Impala) clustered indexes, 86 CODASYL model, 36 (see also network model) code generation with Avro, 127 with Thrift and Protocol Buffers, 118 with WSDL, 133 collaborative editing multi-leader replication and, 170 column families (Bigtable), 41, 99 column-oriented storage, 95-101 column compression, 97 distinction between column families and, 99 in batch processors, 428 Parquet, 96, 131, 414 sort order in, 99-100 vectorized processing, 99, 428 writing to, 101 comma-separated values (see CSV) command query responsibility segregation (CQRS), 462 commands (event sourcing), 459 commits (transactions), 222 atomic commit, 354-355 (see also atomicity; transactions) read committed isolation, 234 three-phase commit (3PC), 359 two-phase commit (2PC), 355-359 commutative operations, 246 compaction of changelogs, 456 (see also log compaction) for stream operator state, 479 of log-structured storage, 73 issues with, 84 size-tiered and leveled approaches, 79 CompactProtocol encoding (Thrift), 119 compare-and-set operations, 245, 327 implementing locks, 370 implementing uniqueness constraints, 331 implementing with total order broadcast, 350 relation to consensus, 335, 350, 352, 374 relation to transactions, 230 compatibility, 112, 128 calling services, 136 properties of encoding formats, 139 using databases, 129-131 using message-passing, 138 compensating transactions, 355, 461, 526 complex event processing (CEP), 465 complexity distilling in theoretical models, 310 hiding using abstraction, 27 of software systems, managing, 20 composing data systems (see unbundling data‐ bases) compute-intensive applications, 3, 275 concatenated indexes, 87 in Cassandra, 204 Concord (stream processor), 466 concurrency actor programming model, 138, 468 (see also message-passing) bugs from weak transaction isolation, 233 conflict resolution, 171, 174 detecting concurrent writes, 184-191 dual writes, problems with, 453 happens-before relationship, 186 in replicated systems, 161-191, 324-338 lost updates, 243 multi-version concurrency control (MVCC), 239 optimistic concurrency control, 261 ordering of operations, 326, 341 reducing, through event logs, 351, 462, 507 time and relativity, 187 transaction isolation, 225 write skew (transaction isolation), 246-251 conflict-free replicated datatypes (CRDTs), 174 conflicts conflict detection, 172 causal dependencies, 186, 342 in consensus algorithms, 368 in leaderless replication, 184 Index | 563 in log-based systems, 351, 521 in nonlinearizable systems, 343 in serializable snapshot isolation (SSI), 264 in two-phase commit, 357, 364 conflict resolution automatic conflict resolution, 174 by aborting transactions, 261 by apologizing, 527 convergence, 172-174 in leaderless systems, 190 last write wins (LWW), 186, 292 using atomic operations, 246 using custom logic, 173 determining what is a conflict, 174, 522 in multi-leader replication, 171-175 avoiding conflicts, 172 lost updates, 242-246 materializing, 251 relation to operation ordering, 339 write skew (transaction isolation), 246-251 congestion (networks) avoidance, 282 limiting accuracy of clocks, 293 queueing delays, 282 consensus, 321, 364-375, 554 algorithms, 366-368 preventing split brain, 367 safety and liveness properties, 365 using linearizable operations, 351 cost of, 369 distributed transactions, 352-375 in practice, 360-364 two-phase commit, 354-359 XA transactions, 361-364 impossibility of, 353 membership and coordination services, 370-373 relation to compare-and-set, 335, 350, 352, 374 relation to replication, 155, 349 relation to uniqueness constraints, 521 consistency, 224, 524 across different databases, 157, 452, 462, 492 causal, 339-348, 493 consistent prefix reads, 165-167 consistent snapshots, 156, 237-242, 294, 455, 500 (see also snapshots) 564 | Index crash recovery, 82 enforcing constraints (see constraints) eventual, 162, 322 (see also eventual consistency) in ACID transactions, 224, 529 in CAP theorem, 337 linearizability, 324-338 meanings of, 224 monotonic reads, 164-165 of secondary indexes, 231, 241, 354, 491, 500 ordering guarantees, 339-352 read-after-write, 162-164 sequential, 351 strong (see linearizability) timeliness and integrity, 524 using quorums, 181, 334 consistent hashing, 204 consistent prefix reads, 165 constraints (databases), 225, 248 asynchronously checked, 526 coordination avoidance, 527 ensuring idempotence, 519 in log-based systems, 521-524 across multiple partitions, 522 in two-phase commit, 355, 357 relation to consensus, 374, 521 relation to event ordering, 347 requiring linearizability, 330 Consul (service discovery), 372 consumers (message streams), 137, 440 backpressure, 441 consumer offsets in logs, 449 failures, 445, 449 fan-out, 11, 445, 448 load balancing, 444, 448 not keeping up with producers, 441, 450, 502 context switches, 14, 297 convergence (conflict resolution), 172-174, 322 coordination avoidance, 527 cross-datacenter, 168, 493 cross-partition ordering, 256, 294, 348, 523 services, 330, 370-373 coordinator (in 2PC), 356 failure, 358 in XA transactions, 361-364 recovery, 363 copy-on-write (B-trees), 82, 242 CORBA (Common Object Request Broker Architecture), 134 correctness, 6 auditability, 528-533 Byzantine fault tolerance, 305, 532 dealing with partial failures, 274 in log-based systems, 521-524 of algorithm within system model, 308 of compensating transactions, 355 of consensus, 368 of derived data, 497, 531 of immutable data, 461 of personal data, 535, 540 of time, 176, 289-295 of transactions, 225, 515, 529 timeliness and integrity, 524-528 corruption of data detecting, 519, 530-533 due to pathological memory access, 529 due to radiation, 305 due to split brain, 158, 302 due to weak transaction isolation, 233 formalization in consensus, 366 integrity as absence of, 524 network packets, 306 on disks, 227 preventing using write-ahead logs, 82 recovering from, 414, 460 Couchbase (database) durability, 89 hash partitioning, 203-204, 211 rebalancing, 213 request routing, 216 CouchDB (database) B-tree storage, 242 change feed, 456 document data model, 31 join support, 34 MapReduce support, 46, 400 replication, 170, 173 covering indexes, 86 CPUs cache coherence and memory barriers, 338 caching and pipelining, 99, 428 increasing parallelism, 43 CRDTs (see conflict-free replicated datatypes) CREATE INDEX statement (SQL), 85, 500 credit rating agencies, 535 Crunch (batch processing), 419, 427 hash joins, 409 sharded joins, 408 workflows, 403 cryptography defense against attackers, 306 end-to-end encryption and authentication, 519, 543 proving integrity of data, 532 CSS (Cascading Style Sheets), 44 CSV (comma-separated values), 70, 114, 396 Curator (ZooKeeper recipes), 330, 371 curl (Unix tool), 135, 397 cursor stability, 243 Cypher (query language), 52 comparison to SPARQL, 59 D data corruption (see corruption of data) data cubes, 102 data formats (see encoding) data integration, 490-498, 543 batch and stream processing, 494-498 lambda architecture, 497 maintaining derived state, 495 reprocessing data, 496 unifying, 498 by unbundling databases, 499-515 comparison to federated databases, 501 combining tools by deriving data, 490-494 derived data versus distributed transac‐ tions, 492 limits of total ordering, 493 ordering events to capture causality, 493 reasoning about dataflows, 491 need for, 385 data lakes, 415 data locality (see locality) data models, 27-64 graph-like models, 49-63 Datalog language, 60-63 property graphs, 50 RDF and triple-stores, 55-59 query languages, 42-48 relational model versus document model, 28-42 data protection regulations, 542 data systems, 3 about, 4 Index | 565 concerns when designing, 5 future of, 489-544 correctness, constraints, and integrity, 515-533 data integration, 490-498 unbundling databases, 499-515 heterogeneous, keeping in sync, 452 maintainability, 18-22 possible faults in, 221 reliability, 6-10 hardware faults, 7 human errors, 9 importance of, 10 software errors, 8 scalability, 10-18 unreliable clocks, 287-299 data warehousing, 91-95, 554 comparison to data lakes, 415 ETL (extract-transform-load), 92, 416, 452 keeping data systems in sync, 452 schema design, 93 slowly changing dimension (SCD), 476 data-intensive applications, 3 database triggers (see triggers) database-internal distributed transactions, 360, 364, 477 databases archival storage, 131 comparison of message brokers to, 443 dataflow through, 129 end-to-end argument for, 519-520 checking integrity, 531 inside-out, 504 (see also unbundling databases) output from batch workflows, 412 relation to event streams, 451-464 (see also changelogs) API support for change streams, 456, 506 change data capture, 454-457 event sourcing, 457-459 keeping systems in sync, 452-453 philosophy of immutable events, 459-464 unbundling, 499-515 composing data storage technologies, 499-504 designing applications around dataflow, 504-509 566 | Index observing derived state, 509-515 datacenters geographically distributed, 145, 164, 278, 493 multi-tenancy and shared resources, 284 network architecture, 276 network faults, 279 replication across multiple, 169 leaderless replication, 184 multi-leader replication, 168, 335 dataflow, 128-139, 504-509 correctness of dataflow systems, 525 differential, 504 message-passing, 136-139 reasoning about, 491 through databases, 129 through services, 131-136 dataflow engines, 421-423 comparison to stream processing, 464 directed acyclic graphs (DAG), 424 partitioning, approach to, 429 support for declarative queries, 427 Datalog (query language), 60-63 datatypes binary strings in XML and JSON, 114 conflict-free, 174 in Avro encodings, 122 in Thrift and Protocol Buffers, 121 numbers in XML and JSON, 114 Datomic (database) B-tree storage, 242 data model, 50, 57 Datalog query language, 60 excision (deleting data), 463 languages for transactions, 255 serial execution of transactions, 253 deadlocks detection, in two-phase commit (2PC), 364 in two-phase locking (2PL), 258 Debezium (change data capture), 455 declarative languages, 42, 554 Bloom, 504 CSS and XSL, 44 Cypher, 52 Datalog, 60 for batch processing, 427 recursive SQL queries, 53 relational algebra and SQL, 42 SPARQL, 59 delays bounded network delays, 285 bounded process pauses, 298 unbounded network delays, 282 unbounded process pauses, 296 deleting data, 463 denormalization (data representation), 34, 554 costs, 39 in derived data systems, 386 materialized views, 101 updating derived data, 228, 231, 490 versus normalization, 462 derived data, 386, 439, 554 from change data capture, 454 in event sourcing, 458-458 maintaining derived state through logs, 452-457, 459-463 observing, by subscribing to streams, 512 outputs of batch and stream processing, 495 through application code, 505 versus distributed transactions, 492 deterministic operations, 255, 274, 554 accidental nondeterminism, 423 and fault tolerance, 423, 426 and idempotence, 478, 492 computing derived data, 495, 526, 531 in state machine replication, 349, 452, 458 joins, 476 DevOps, 394 differential dataflow, 504 dimension tables, 94 dimensional modeling (see star schemas) directed acyclic graphs (DAGs), 424 dirty reads (transaction isolation), 234 dirty writes (transaction isolation), 235 discrimination, 534 disks (see hard disks) distributed actor frameworks, 138 distributed filesystems, 398-399 decoupling from query engines, 417 indiscriminately dumping data into, 415 use by MapReduce, 402 distributed systems, 273-312, 554 Byzantine faults, 304-306 cloud versus supercomputing, 275 detecting network faults, 280 faults and partial failures, 274-277 formalization of consensus, 365 impossibility results, 338, 353 issues with failover, 157 limitations of distributed transactions, 363 multi-datacenter, 169, 335 network problems, 277-286 quorums, relying on, 301 reasons for using, 145, 151 synchronized clocks, relying on, 291-295 system models, 306-310 use of clocks and time, 287 distributed transactions (see transactions) Django (web framework), 232 DNS (Domain Name System), 216, 372 Docker (container manager), 506 document data model, 30-42 comparison to relational model, 38-42 document references, 38, 403 document-oriented databases, 31 many-to-many relationships and joins, 36 multi-object transactions, need for, 231 versus relational model convergence of models, 41 data locality, 41 document-partitioned indexes, 206, 217, 411 domain-driven design (DDD), 457 DRBD (Distributed Replicated Block Device), 153 drift (clocks), 289 Drill (query engine), 93 Druid (database), 461 Dryad (dataflow engine), 421 dual writes, problems with, 452, 507 duplicates, suppression of, 517 (see also idempotence) using a unique ID, 518, 522 durability (transactions), 226, 554 duration (time), 287 measurement with monotonic clocks, 288 dynamic partitioning, 212 dynamically typed languages analogy to schema-on-read, 40 code generation and, 127 Dynamo-style databases (see leaderless replica‐ tion) E edges (in graphs), 49, 403 property graph model, 50 edit distance (full-text search), 88 effectively-once semantics, 476, 516 Index | 567 (see also exactly-once semantics) preservation of integrity, 525 elastic systems, 17 Elasticsearch (search server) document-partitioned indexes, 207 partition rebalancing, 211 percolator (stream search), 467 usage example, 4 use of Lucene, 79 ElephantDB (database), 413 Elm (programming language), 504, 512 encodings (data formats), 111-128 Avro, 122-127 binary variants of JSON and XML, 115 compatibility, 112 calling services, 136 using databases, 129-131 using message-passing, 138 defined, 113 JSON, XML, and CSV, 114 language-specific formats, 113 merits of schemas, 127 representations of data, 112 Thrift and Protocol Buffers, 117-121 end-to-end argument, 277, 519-520 checking integrity, 531 publish/subscribe streams, 512 enrichment (stream), 473 Enterprise JavaBeans (EJB), 134 entities (see vertices) epoch (consensus algorithms), 368 epoch (Unix timestamps), 288 equi-joins, 403 erasure coding (error correction), 398 Erlang OTP (actor framework), 139 error handling for network faults, 280 in transactions, 231 error-correcting codes, 277, 398 Esper (CEP engine), 466 etcd (coordination service), 370-373 linearizable operations, 333 locks and leader election, 330 quorum reads, 351 service discovery, 372 use of Raft algorithm, 349, 353 Ethereum (blockchain), 532 Ethernet (networks), 276, 278, 285 packet checksums, 306, 519 568 | Index Etherpad (collaborative editor), 170 ethics, 533-543 code of ethics and professional practice, 533 legislation and self-regulation, 542 predictive analytics, 533-536 amplifying bias, 534 feedback loops, 536 privacy and tracking, 536-543 consent and freedom of choice, 538 data as assets and power, 540 meaning of privacy, 539 surveillance, 537 respect, dignity, and agency, 543, 544 unintended consequences, 533, 536 ETL (extract-transform-load), 92, 405, 452, 554 use of Hadoop for, 416 event sourcing, 457-459 commands and events, 459 comparison to change data capture, 457 comparison to lambda architecture, 497 deriving current state from event log, 458 immutability and auditability, 459, 531 large, reliable data systems, 519, 526 Event Store (database), 458 event streams (see streams) events, 440 deciding on total order of, 493 deriving views from event log, 461 difference to commands, 459 event time versus processing time, 469, 477, 498 immutable, advantages of, 460, 531 ordering to capture causality, 493 reads as, 513 stragglers, 470, 498 timestamp of, in stream processing, 471 EventSource (browser API), 512 eventual consistency, 152, 162, 308, 322 (see also conflicts) and perpetual inconsistency, 525 evolvability, 21, 111 calling services, 136 graph-structured data, 52 of databases, 40, 129-131, 461, 497 of message-passing, 138 reprocessing data, 496, 498 schema evolution in Avro, 123 schema evolution in Thrift and Protocol Buffers, 120 schema-on-read, 39, 111, 128 exactly-once semantics, 360, 476, 516 parity with batch processors, 498 preservation of integrity, 525 exclusive mode (locks), 258 eXtended Architecture transactions (see XA transactions) extract-transform-load (see ETL) F Facebook Presto (query engine), 93 React, Flux, and Redux (user interface libra‐ ries), 512 social graphs, 49 Wormhole (change data capture), 455 fact tables, 93 failover, 157, 554 (see also leader-based replication) in leaderless replication, absence of, 178 leader election, 301, 348, 352 potential problems, 157 failures amplification by distributed transactions, 364, 495 failure detection, 280 automatic rebalancing causing cascading failures, 214 perfect failure detectors, 359 timeouts and unbounded delays, 282, 284 using ZooKeeper, 371 faults versus, 7 partial failures in distributed systems, 275-277, 310 fan-out (messaging systems), 11, 445 fault tolerance, 6-10, 555 abstractions for, 321 formalization in consensus, 365-369 use of replication, 367 human fault tolerance, 414 in batch processing, 406, 414, 422, 425 in log-based systems, 520, 524-526 in stream processing, 476-479 atomic commit, 477 idempotence, 478 maintaining derived state, 495 microbatching and checkpointing, 477 rebuilding state after a failure, 478 of distributed transactions, 362-364 transaction atomicity, 223, 354-361 faults, 6 Byzantine faults, 304-306 failures versus, 7 handled by transactions, 221 handling in supercomputers and cloud computing, 275 hardware, 7 in batch processing versus distributed data‐ bases, 417 in distributed systems, 274-277 introducing deliberately, 7, 280 network faults, 279-281 asymmetric faults, 300 detecting, 280 tolerance of, in multi-leader replication, 169 software errors, 8 tolerating (see fault tolerance) federated databases, 501 fence (CPU instruction), 338 fencing (preventing split brain), 158, 302-304 generating fencing tokens, 349, 370 properties of fencing tokens, 308 stream processors writing to databases, 478, 517 Fibre Channel (networks), 398 field tags (Thrift and Protocol Buffers), 119-121 file descriptors (Unix), 395 financial data, 460 Firebase (database), 456 Flink (processing framework), 421-423 dataflow APIs, 427 fault tolerance, 422, 477, 479 Gelly API (graph processing), 425 integration of batch and stream processing, 495, 498 machine learning, 428 query optimizer, 427 stream processing, 466 flow control, 282, 441, 555 FLP result (on consensus), 353 FlumeJava (dataflow library), 403, 427 followers, 152, 555 (see also leader-based replication) foreign keys, 38, 403 forward compatibility, 112 forward decay (algorithm), 16 Index | 569 Fossil (version control system), 463 shunning (deleting data), 463 FoundationDB (database) serializable transactions, 261, 265, 364 fractal trees, 83 full table scans, 403 full-text search, 555 and fuzzy indexes, 88 building search indexes, 411 Lucene storage engine, 79 functional reactive programming (FRP), 504 functional requirements, 22 futures (asynchronous operations), 135 fuzzy search (see similarity search) G garbage collection immutability and, 463 process pauses for, 14, 296-299, 301 (see also process pauses) genome analysis, 63, 429 geographically distributed datacenters, 145, 164, 278, 493 geospatial indexes, 87 Giraph (graph processing), 425 Git (version control system), 174, 342, 463 GitHub, postmortems, 157, 158, 309 global indexes (see term-partitioned indexes) GlusterFS (distributed filesystem), 398 GNU Coreutils (Linux), 394 GoldenGate (change data capture), 161, 170, 455 (see also Oracle) Google Bigtable (database) data model (see Bigtable data model) partitioning scheme, 199, 202 storage layout, 78 Chubby (lock service), 370 Cloud Dataflow (stream processor), 466, 477, 498 (see also Beam) Cloud Pub/Sub (messaging), 444, 448 Docs (collaborative editor), 170 Dremel (query engine), 93, 96 FlumeJava (dataflow library), 403, 427 GFS (distributed file system), 398 gRPC (RPC framework), 135 MapReduce (batch processing), 390 570 | Index (see also MapReduce) building search indexes, 411 task preemption, 418 Pregel (graph processing), 425 Spanner (see Spanner) TrueTime (clock API), 294 gossip protocol, 216 government use of data, 541 GPS (Global Positioning System) use for clock synchronization, 287, 290, 294, 295 GraphChi (graph processing), 426 graphs, 555 as data models, 49-63 example of graph-structured data, 49 property graphs, 50 RDF and triple-stores, 55-59 versus the network model, 60 processing and analysis, 424-426 fault tolerance, 425 Pregel processing model, 425 query languages Cypher, 52 Datalog, 60-63 recursive SQL queries, 53 SPARQL, 59-59 Gremlin (graph query language), 50 grep (Unix tool), 392 GROUP BY clause (SQL), 406 grouping records in MapReduce, 406 handling skew, 407 H Hadoop (data infrastructure) comparison to distributed databases, 390 comparison to MPP databases, 414-418 comparison to Unix, 413-414, 499 diverse processing models in ecosystem, 417 HDFS distributed filesystem (see HDFS) higher-level tools, 403 join algorithms, 403-410 (see also MapReduce) MapReduce (see MapReduce) YARN (see YARN) happens-before relationship, 340 capturing, 187 concurrency and, 186 hard disks access patterns, 84 detecting corruption, 519, 530 faults in, 7, 227 sequential write throughput, 75, 450 hardware faults, 7 hash indexes, 72-75 broadcast hash joins, 409 partitioned hash joins, 409 hash partitioning, 203-205, 217 consistent hashing, 204 problems with hash mod N, 210 range queries, 204 suitable hash functions, 203 with fixed number of partitions, 210 HAWQ (database), 428 HBase (database) bug due to lack of fencing, 302 bulk loading, 413 column-family data model, 41, 99 dynamic partitioning, 212 key-range partitioning, 202 log-structured storage, 78 request routing, 216 size-tiered compaction, 79 use of HDFS, 417 use of ZooKeeper, 370 HDFS (Hadoop Distributed File System), 398-399 (see also distributed filesystems) checking data integrity, 530 decoupling from query engines, 417 indiscriminately dumping data into, 415 metadata about datasets, 410 NameNode, 398 use by Flink, 479 use by HBase, 212 use by MapReduce, 402 HdrHistogram (numerical library), 16 head (Unix tool), 392 head vertex (property graphs), 51 head-of-line blocking, 15 heap files (databases), 86 Helix (cluster manager), 216 heterogeneous distributed transactions, 360, 364 heuristic decisions (in 2PC), 363 Hibernate (object-relational mapper), 30 hierarchical model, 36 high availability (see fault tolerance) high-frequency trading, 290, 299 high-performance computing (HPC), 275 hinted handoff, 183 histograms, 16 Hive (query engine), 419, 427 for data warehouses, 93 HCatalog and metastore, 410 map-side joins, 409 query optimizer, 427 skewed joins, 408 workflows, 403 Hollerith machines, 390 hopping windows (stream processing), 472 (see also windows) horizontal scaling (see scaling out) HornetQ (messaging), 137, 444 distributed transaction support, 361 hot spots, 201 due to celebrities, 205 for time-series data, 203 in batch processing, 407 relieving, 205 hot standbys (see leader-based replication) HTTP, use in APIs (see services) human errors, 9, 279, 414 HyperDex (database), 88 HyperLogLog (algorithm), 466 I I/O operations, waiting for, 297 IBM DB2 (database) distributed transaction support, 361 recursive query support, 54 serializable isolation, 242, 257 XML and JSON support, 30, 42 electromechanical card-sorting machines, 390 IMS (database), 36 imperative query APIs, 46 InfoSphere Streams (CEP engine), 466 MQ (messaging), 444 distributed transaction support, 361 System R (database), 222 WebSphere (messaging), 137 idempotence, 134, 478, 555 by giving operations unique IDs, 518, 522 idempotent operations, 517 immutability advantages of, 460, 531 Index | 571 deriving state from event log, 459-464 for crash recovery, 75 in B-trees, 82, 242 in event sourcing, 457 inputs to Unix commands, 397 limitations of, 463 Impala (query engine) for data warehouses, 93 hash joins, 409 native code generation, 428 use of HDFS, 417 impedance mismatch, 29 imperative languages, 42 setting element styles (example), 45 in doubt (transaction status), 358 holding locks, 362 orphaned transactions, 363 in-memory databases, 88 durability, 227 serial transaction execution, 253 incidents cascading failures, 9 crashes due to leap seconds, 290 data corruption and financial losses due to concurrency bugs, 233 data corruption on hard disks, 227 data loss due to last-write-wins, 173, 292 data on disks unreadable, 309 deleted items reappearing, 174 disclosure of sensitive data due to primary key reuse, 157 errors in transaction serializability, 529 gigabit network interface with 1 Kb/s throughput, 311 network faults, 279 network interface dropping only inbound packets, 279 network partitions and whole-datacenter failures, 275 poor handling of network faults, 280 sending message to ex-partner, 494 sharks biting undersea cables, 279 split brain due to 1-minute packet delay, 158, 279 vibrations in server rack, 14 violation of uniqueness constraint, 529 indexes, 71, 555 and snapshot isolation, 241 as derived data, 386, 499-504 572 | Index B-trees, 79-83 building in batch processes, 411 clustered, 86 comparison of B-trees and LSM-trees, 83-85 concatenated, 87 covering (with included columns), 86 creating, 500 full-text search, 88 geospatial, 87 hash, 72-75 index-range locking, 260 multi-column, 87 partitioning and secondary indexes, 206-209, 217 secondary, 85 (see also secondary indexes) problems with dual writes, 452, 491 SSTables and LSM-trees, 76-79 updating when data changes, 452, 467 Industrial Revolution, 541 InfiniBand (networks), 285 InfiniteGraph (database), 50 InnoDB (storage engine) clustered index on primary key, 86 not preventing lost updates, 245 preventing write skew, 248, 257 serializable isolation, 257 snapshot isolation support, 239 inside-out databases, 504 (see also unbundling databases) integrating different data systems (see data integration) integrity, 524 coordination-avoiding data systems, 528 correctness of dataflow systems, 525 in consensus formalization, 365 integrity checks, 530 (see also auditing) end-to-end, 519, 531 use of snapshot isolation, 238 maintaining despite software bugs, 529 Interface Definition Language (IDL), 117, 122 intermediate state, materialization of, 420-423 internet services, systems for implementing, 275 invariants, 225 (see also constraints) inversion of control, 396 IP (Internet Protocol) unreliability of, 277 ISDN (Integrated Services Digital Network), 284 isolation (in transactions), 225, 228, 555 correctness and, 515 for single-object writes, 230 serializability, 251-266 actual serial execution, 252-256 serializable snapshot isolation (SSI), 261-266 two-phase locking (2PL), 257-261 violating, 228 weak isolation levels, 233-251 preventing lost updates, 242-246 read committed, 234-237 snapshot isolation, 237-242 iterative processing, 424-426 J Java Database Connectivity (JDBC) distributed transaction support, 361 network drivers, 128 Java Enterprise Edition (EE), 134, 356, 361 Java Message Service (JMS), 444 (see also messaging systems) comparison to log-based messaging, 448, 451 distributed transaction support, 361 message ordering, 446 Java Transaction API (JTA), 355, 361 Java Virtual Machine (JVM) bytecode generation, 428 garbage collection pauses, 296 process reuse in batch processors, 422 JavaScript in MapReduce querying, 46 setting element styles (example), 45 use in advanced queries, 48 Jena (RDF framework), 57 Jepsen (fault tolerance testing), 515 jitter (network delay), 284 joins, 555 by index lookup, 403 expressing as relational operators, 427 in relational and document databases, 34 MapReduce map-side joins, 408-410 broadcast hash joins, 409 merge joins, 410 partitioned hash joins, 409 MapReduce reduce-side joins, 403-408 handling skew, 407 sort-merge joins, 405 parallel execution of, 415 secondary indexes and, 85 stream joins, 472-476 stream-stream join, 473 stream-table join, 473 table-table join, 474 time-dependence of, 475 support in document databases, 42 JOTM (transaction coordinator), 356 JSON Avro schema representation, 122 binary variants, 115 for application data, issues with, 114 in relational databases, 30, 42 representing a résumé (example), 31 Juttle (query language), 504 K k-nearest neighbors, 429 Kafka (messaging), 137, 448 Kafka Connect (database integration), 457, 461 Kafka Streams (stream processor), 466, 467 fault tolerance, 479 leader-based replication, 153 log compaction, 456, 467 message offsets, 447, 478 request routing, 216 transaction support, 477 usage example, 4 Ketama (partitioning library), 213 key-value stores, 70 as batch process output, 412 hash indexes, 72-75 in-memory, 89 partitioning, 201-205 by hash of key, 203, 217 by key range, 202, 217 dynamic partitioning, 212 skew and hot spots, 205 Kryo (Java), 113 Kubernetes (cluster manager), 418, 506 L lambda architecture, 497 Lamport timestamps, 345 Index | 573 Large Hadron Collider (LHC), 64 last write wins (LWW), 173, 334 discarding concurrent writes, 186 problems with, 292 prone to lost updates, 246 late binding, 396 latency instability under two-phase locking, 259 network latency and resource utilization, 286 response time versus, 14 tail latency, 15, 207 leader-based replication, 152-161 (see also replication) failover, 157, 301 handling node outages, 156 implementation of replication logs change data capture, 454-457 (see also changelogs) statement-based, 158 trigger-based replication, 161 write-ahead log (WAL) shipping, 159 linearizability of operations, 333 locking and leader election, 330 log sequence number, 156, 449 read-scaling architecture, 161 relation to consensus, 367 setting up new followers, 155 synchronous versus asynchronous, 153-155 leaderless replication, 177-191 (see also replication) detecting concurrent writes, 184-191 capturing happens-before relationship, 187 happens-before relationship and concur‐ rency, 186 last write wins, 186 merging concurrently written values, 190 version vectors, 191 multi-datacenter, 184 quorums, 179-182 consistency limitations, 181-183, 334 sloppy quorums and hinted handoff, 183 read repair and anti-entropy, 178 leap seconds, 8, 290 in time-of-day clocks, 288 leases, 295 implementation with ZooKeeper, 370 574 | Index need for fencing, 302 ledgers, 460 distributed ledger technologies, 532 legacy systems, maintenance of, 18 less (Unix tool), 397 LevelDB (storage engine), 78 leveled compaction, 79 Levenshtein automata, 88 limping (partial failure), 311 linearizability, 324-338, 555 cost of, 335-338 CAP theorem, 336 memory on multi-core CPUs, 338 definition, 325-329 implementing with total order broadcast, 350 in ZooKeeper, 370 of derived data systems, 492, 524 avoiding coordination, 527 of different replication methods, 332-335 using quorums, 334 relying on, 330-332 constraints and uniqueness, 330 cross-channel timing dependencies, 331 locking and leader election, 330 stronger than causal consistency, 342 using to implement total order broadcast, 351 versus serializability, 329 LinkedIn Azkaban (workflow scheduler), 402 Databus (change data capture), 161, 455 Espresso (database), 31, 126, 130, 153, 216 Helix (cluster manager) (see Helix) profile (example), 30 reference to company entity (example), 34 Rest.li (RPC framework), 135 Voldemort (database) (see Voldemort) Linux, leap second bug, 8, 290 liveness properties, 308 LMDB (storage engine), 82, 242 load approaches to coping with, 17 describing, 11 load testing, 16 load balancing (messaging), 444 local indexes (see document-partitioned indexes) locality (data access), 32, 41, 555 in batch processing, 400, 405, 421 in stateful clients, 170, 511 in stream processing, 474, 478, 508, 522 location transparency, 134 in the actor model, 138 locks, 556 deadlock, 258 distributed locking, 301-304, 330 fencing tokens, 303 implementation with ZooKeeper, 370 relation to consensus, 374 for transaction isolation in snapshot isolation, 239 in two-phase locking (2PL), 257-261 making operations atomic, 243 performance, 258 preventing dirty writes, 236 preventing phantoms with index-range locks, 260, 265 read locks (shared mode), 236, 258 shared mode and exclusive mode, 258 in two-phase commit (2PC) deadlock detection, 364 in-doubt transactions holding locks, 362 materializing conflicts with, 251 preventing lost updates by explicit locking, 244 log sequence number, 156, 449 logic programming languages, 504 logical clocks, 293, 343, 494 for read-after-write consistency, 164 logical logs, 160 logs (data structure), 71, 556 advantages of immutability, 460 compaction, 73, 79, 456, 460 for stream operator state, 479 creating using total order broadcast, 349 implementing uniqueness constraints, 522 log-based messaging, 446-451 comparison to traditional messaging, 448, 451 consumer offsets, 449 disk space usage, 450 replaying old messages, 451, 496, 498 slow consumers, 450 using logs for message storage, 447 log-structured storage, 71-79 log-structured merge tree (see LSMtrees) replication, 152, 158-161 change data capture, 454-457 (see also changelogs) coordination with snapshot, 156 logical (row-based) replication, 160 statement-based replication, 158 trigger-based replication, 161 write-ahead log (WAL) shipping, 159 scalability limits, 493 loose coupling, 396, 419, 502 lost updates (see updates) LSM-trees (indexes), 78-79 comparison to B-trees, 83-85 Lucene (storage engine), 79 building indexes in batch processes, 411 similarity search, 88 Luigi (workflow scheduler), 402 LWW (see last write wins) M machine learning ethical considerations, 534 (see also ethics) iterative processing, 424 models derived from training data, 505 statistical and numerical algorithms, 428 MADlib (machine learning toolkit), 428 magic scaling sauce, 18 Mahout (machine learning toolkit), 428 maintainability, 18-22, 489 defined, 23 design principles for software systems, 19 evolvability (see evolvability) operability, 19 simplicity and managing complexity, 20 many-to-many relationships in document model versus relational model, 39 modeling as graphs, 49 many-to-one and many-to-many relationships, 33-36 many-to-one relationships, 34 MapReduce (batch processing), 390, 399-400 accessing external services within job, 404, 412 comparison to distributed databases designing for frequent faults, 417 diversity of processing models, 416 diversity of storage, 415 Index | 575 comparison to stream processing, 464 comparison to Unix, 413-414 disadvantages and limitations of, 419 fault tolerance, 406, 414, 422 higher-level tools, 403, 426 implementation in Hadoop, 400-403 the shuffle, 402 implementation in MongoDB, 46-48 machine learning, 428 map-side processing, 408-410 broadcast hash joins, 409 merge joins, 410 partitioned hash joins, 409 mapper and reducer functions, 399 materialization of intermediate state, 419-423 output of batch workflows, 411-413 building search indexes, 411 key-value stores, 412 reduce-side processing, 403-408 analysis of user activity events (exam‐ ple), 404 grouping records by same key, 406 handling skew, 407 sort-merge joins, 405 workflows, 402 marshalling (see encoding) massively parallel processing (MPP), 216 comparison to composing storage technolo‐ gies, 502 comparison to Hadoop, 414-418, 428 master-master replication (see multi-leader replication) master-slave replication (see leader-based repli‐ cation) materialization, 556 aggregate values, 101 conflicts, 251 intermediate state (batch processing), 420-423 materialized views, 101 as derived data, 386, 499-504 maintaining, using stream processing, 467, 475 Maven (Java build tool), 428 Maxwell (change data capture), 455 mean, 14 media monitoring, 467 median, 14 576 | Index meeting room booking (example), 249, 259, 521 membership services, 372 Memcached (caching server), 4, 89 memory in-memory databases, 88 durability, 227 serial transaction execution, 253 in-memory representation of data, 112 random bit-flips in, 529 use by indexes, 72, 77 memory barrier (CPU instruction), 338 MemSQL (database) in-memory storage, 89 read committed isolation, 236 memtable (in LSM-trees), 78 Mercurial (version control system), 463 merge joins, MapReduce map-side, 410 mergeable persistent data structures, 174 merging sorted files, 76, 402, 405 Merkle trees, 532 Mesos (cluster manager), 418, 506 message brokers (see messaging systems) message-passing, 136-139 advantages over direct RPC, 137 distributed actor frameworks, 138 evolvability, 138 MessagePack (encoding format), 116 messages exactly-once semantics, 360, 476 loss of, 442 using total order broadcast, 348 messaging systems, 440-451 (see also streams) backpressure, buffering, or dropping mes‐ sages, 441 brokerless messaging, 442 event logs, 446-451 comparison to traditional messaging, 448, 451 consumer offsets, 449 replaying old messages, 451, 496, 498 slow consumers, 450 message brokers, 443-446 acknowledgements and redelivery, 445 comparison to event logs, 448, 451 multiple consumers of same topic, 444 reliability, 442 uniqueness in log-based messaging, 522 Meteor (web framework), 456 microbatching, 477, 495 microservices, 132 (see also services) causal dependencies across services, 493 loose coupling, 502 relation to batch/stream processors, 389, 508 Microsoft Azure Service Bus (messaging), 444 Azure Storage, 155, 398 Azure Stream Analytics, 466 DCOM (Distributed Component Object Model), 134 MSDTC (transaction coordinator), 356 Orleans (see Orleans) SQL Server (see SQL Server) migrating (rewriting) data, 40, 130, 461, 497 modulus operator (%), 210 MongoDB (database) aggregation pipeline, 48 atomic operations, 243 BSON, 41 document data model, 31 hash partitioning (sharding), 203-204 key-range partitioning, 202 lack of join support, 34, 42 leader-based replication, 153 MapReduce support, 46, 400 oplog parsing, 455, 456 partition splitting, 212 request routing, 216 secondary indexes, 207 Mongoriver (change data capture), 455 monitoring, 10, 19 monotonic clocks, 288 monotonic reads, 164 MPP (see massively parallel processing) MSMQ (messaging), 361 multi-column indexes, 87 multi-leader replication, 168-177 (see also replication) handling write conflicts, 171 conflict avoidance, 172 converging toward a consistent state, 172 custom conflict resolution logic, 173 determining what is a conflict, 174 linearizability, lack of, 333 replication topologies, 175-177 use cases, 168 clients with offline operation, 170 collaborative editing, 170 multi-datacenter replication, 168, 335 multi-object transactions, 228 need for, 231 Multi-Paxos (total order broadcast), 367 multi-table index cluster tables (Oracle), 41 multi-tenancy, 284 multi-version concurrency control (MVCC), 239, 266 detecting stale MVCC reads, 263 indexes and snapshot isolation, 241 mutual exclusion, 261 (see also locks) MySQL (database) binlog coordinates, 156 binlog parsing for change data capture, 455 circular replication topology, 175 consistent snapshots, 156 distributed transaction support, 361 InnoDB storage engine (see InnoDB) JSON support, 30, 42 leader-based replication, 153 performance of XA transactions, 360 row-based replication, 160 schema changes in, 40 snapshot isolation support, 242 (see also InnoDB) statement-based replication, 159 Tungsten Replicator (multi-leader replica‐ tion), 170 conflict detection, 177 N nanomsg (messaging library), 442 Narayana (transaction coordinator), 356 NATS (messaging), 137 near-real-time (nearline) processing, 390 (see also stream processing) Neo4j (database) Cypher query language, 52 graph data model, 50 Nephele (dataflow engine), 421 netcat (Unix tool), 397 Netflix Chaos Monkey, 7, 280 Network Attached Storage (NAS), 146, 398 network model, 36 Index | 577 graph databases versus, 60 imperative query APIs, 46 Network Time Protocol (see NTP) networks congestion and queueing, 282 datacenter network topologies, 276 faults (see faults) linearizability and network delays, 338 network partitions, 279, 337 timeouts and unbounded delays, 281 next-key locking, 260 nodes (in graphs) (see vertices) nodes (processes), 556 handling outages in leader-based replica‐ tion, 156 system models for failure, 307 noisy neighbors, 284 nonblocking atomic commit, 359 nondeterministic operations accidental nondeterminism, 423 partial failures in distributed systems, 275 nonfunctional requirements, 22 nonrepeatable reads, 238 (see also read skew) normalization (data representation), 33, 556 executing joins, 39, 42, 403 foreign key references, 231 in systems of record, 386 versus denormalization, 462 NoSQL, 29, 499 transactions and, 223 Notation3 (N3), 56 npm (package manager), 428 NTP (Network Time Protocol), 287 accuracy, 289, 293 adjustments to monotonic clocks, 289 multiple server addresses, 306 numbers, in XML and JSON encodings, 114 O object-relational mapping (ORM) frameworks, 30 error handling and aborted transactions, 232 unsafe read-modify-write cycle code, 244 object-relational mismatch, 29 observer pattern, 506 offline systems, 390 (see also batch processing) 578 | Index stateful, offline-capable clients, 170, 511 offline-first applications, 511 offsets consumer offsets in partitioned logs, 449 messages in partitioned logs, 447 OLAP (online analytic processing), 91, 556 data cubes, 102 OLTP (online transaction processing), 90, 556 analytics queries versus, 411 workload characteristics, 253 one-to-many relationships, 30 JSON representation, 32 online systems, 389 (see also services) Oozie (workflow scheduler), 402 OpenAPI (service definition format), 133 OpenStack Nova (cloud infrastructure) use of ZooKeeper, 370 Swift (object storage), 398 operability, 19 operating systems versus databases, 499 operation identifiers, 518, 522 operational transformation, 174 operators, 421 flow of data between, 424 in stream processing, 464 optimistic concurrency control, 261 Oracle (database) distributed transaction support, 361 GoldenGate (change data capture), 161, 170, 455 lack of serializability, 226 leader-based replication, 153 multi-table index cluster tables, 41 not preventing write skew, 248 partitioned indexes, 209 PL/SQL language, 255 preventing lost updates, 245 read committed isolation, 236 Real Application Clusters (RAC), 330 recursive query support, 54 snapshot isolation support, 239, 242 TimesTen (in-memory database), 89 WAL-based replication, 160 XML support, 30 ordering, 339-352 by sequence numbers, 343-348 causal ordering, 339-343 partial order, 341 limits of total ordering, 493 total order broadcast, 348-352 Orleans (actor framework), 139 outliers (response time), 14 Oz (programming language), 504 P package managers, 428, 505 packet switching, 285 packets corruption of, 306 sending via UDP, 442 PageRank (algorithm), 49, 424 paging (see virtual memory) ParAccel (database), 93 parallel databases (see massively parallel pro‐ cessing) parallel execution of graph analysis algorithms, 426 queries in MPP databases, 216 Parquet (data format), 96, 131 (see also column-oriented storage) use in Hadoop, 414 partial failures, 275, 310 limping, 311 partial order, 341 partitioning, 199-218, 556 and replication, 200 in batch processing, 429 multi-partition operations, 514 enforcing constraints, 522 secondary index maintenance, 495 of key-value data, 201-205 by key range, 202 skew and hot spots, 205 rebalancing partitions, 209-214 automatic or manual rebalancing, 213 problems with hash mod N, 210 using dynamic partitioning, 212 using fixed number of partitions, 210 using N partitions per node, 212 replication and, 147 request routing, 214-216 secondary indexes, 206-209 document-based partitioning, 206 term-based partitioning, 208 serial execution of transactions and, 255 Paxos (consensus algorithm), 366 ballot number, 368 Multi-Paxos (total order broadcast), 367 percentiles, 14, 556 calculating efficiently, 16 importance of high percentiles, 16 use in service level agreements (SLAs), 15 Percona XtraBackup (MySQL tool), 156 performance describing, 13 of distributed transactions, 360 of in-memory databases, 89 of linearizability, 338 of multi-leader replication, 169 perpetual inconsistency, 525 pessimistic concurrency control, 261 phantoms (transaction isolation), 250 materializing conflicts, 251 preventing, in serializability, 259 physical clocks (see clocks) pickle (Python), 113 Pig (dataflow language), 419, 427 replicated joins, 409 skewed joins, 407 workflows, 403 Pinball (workflow scheduler), 402 pipelined execution, 423 in Unix, 394 point in time, 287 polyglot persistence, 29 polystores, 501 PostgreSQL (database) BDR (multi-leader replication), 170 causal ordering of writes, 177 Bottled Water (change data capture), 455 Bucardo (trigger-based replication), 161, 173 distributed transaction support, 361 foreign data wrappers, 501 full text search support, 490 leader-based replication, 153 log sequence number, 156 MVCC implementation, 239, 241 PL/pgSQL language, 255 PostGIS geospatial indexes, 87 preventing lost updates, 245 preventing write skew, 248, 261 read committed isolation, 236 recursive query support, 54 representing graphs, 51 Index | 579 serializable snapshot isolation (SSI), 261 snapshot isolation support, 239, 242 WAL-based replication, 160 XML and JSON support, 30, 42 pre-splitting, 212 Precision Time Protocol (PTP), 290 predicate locks, 259 predictive analytics, 533-536 amplifying bias, 534 ethics of (see ethics) feedback loops, 536 preemption of datacenter resources, 418 of threads, 298 Pregel processing model, 425 primary keys, 85, 556 compound primary key (Cassandra), 204 primary-secondary replication (see leaderbased replication) privacy, 536-543 consent and freedom of choice, 538 data as assets and power, 540 deleting data, 463 ethical considerations (see ethics) legislation and self-regulation, 542 meaning of, 539 surveillance, 537 tracking behavioral data, 536 probabilistic algorithms, 16, 466 process pauses, 295-299 processing time (of events), 469 producers (message streams), 440 programming languages dataflow languages, 504 for stored procedures, 255 functional reactive programming (FRP), 504 logic programming, 504 Prolog (language), 61 (see also Datalog) promises (asynchronous operations), 135 property graphs, 50 Cypher query language, 52 Protocol Buffers (data format), 117-121 field tags and schema evolution, 120 provenance of data, 531 publish/subscribe model, 441 publishers (message streams), 440 punch card tabulating machines, 390 580 | Index pure functions, 48 putting computation near data, 400 Q Qpid (messaging), 444 quality of service (QoS), 285 Quantcast File System (distributed filesystem), 398 query languages, 42-48 aggregation pipeline, 48 CSS and XSL, 44 Cypher, 52 Datalog, 60 Juttle, 504 MapReduce querying, 46-48 recursive SQL queries, 53 relational algebra and SQL, 42 SPARQL, 59 query optimizers, 37, 427 queueing delays (networks), 282 head-of-line blocking, 15 latency and response time, 14 queues (messaging), 137 quorums, 179-182, 556 for leaderless replication, 179 in consensus algorithms, 368 limitations of consistency, 181-183, 334 making decisions in distributed systems, 301 monitoring staleness, 182 multi-datacenter replication, 184 relying on durability, 309 sloppy quorums and hinted handoff, 183 R R-trees (indexes), 87 RabbitMQ (messaging), 137, 444 leader-based replication, 153 race conditions, 225 (see also concurrency) avoiding with linearizability, 331 caused by dual writes, 452 dirty writes, 235 in counter increments, 235 lost updates, 242-246 preventing with event logs, 462, 507 preventing with serializable isolation, 252 write skew, 246-251 Raft (consensus algorithm), 366 sensitivity to network problems, 369 term number, 368 use in etcd, 353 RAID (Redundant Array of Independent Disks), 7, 398 railways, schema migration on, 496 RAMCloud (in-memory storage), 89 ranking algorithms, 424 RDF (Resource Description Framework), 57 querying with SPARQL, 59 RDMA (Remote Direct Memory Access), 276 read committed isolation level, 234-237 implementing, 236 multi-version concurrency control (MVCC), 239 no dirty reads, 234 no dirty writes, 235 read path (derived data), 509 read repair (leaderless replication), 178 for linearizability, 335 read replicas (see leader-based replication) read skew (transaction isolation), 238, 266 as violation of causality, 340 read-after-write consistency, 163, 524 cross-device, 164 read-modify-write cycle, 243 read-scaling architecture, 161 reads as events, 513 real-time collaborative editing, 170 near-real-time processing, 390 (see also stream processing) publish/subscribe dataflow, 513 response time guarantees, 298 time-of-day clocks, 288 rebalancing partitions, 209-214, 556 (see also partitioning) automatic or manual rebalancing, 213 dynamic partitioning, 212 fixed number of partitions, 210 fixed number of partitions per node, 212 problems with hash mod N, 210 recency guarantee, 324 recommendation engines batch process outputs, 412 batch workflows, 403, 420 iterative processing, 424 statistical and numerical algorithms, 428 records, 399 events in stream processing, 440 recursive common table expressions (SQL), 54 redelivery (messaging), 445 Redis (database) atomic operations, 243 durability, 89 Lua scripting, 255 single-threaded execution, 253 usage example, 4 redundancy hardware components, 7 of derived data, 386 (see also derived data) Reed–Solomon codes (error correction), 398 refactoring, 22 (see also evolvability) regions (partitioning), 199 register (data structure), 325 relational data model, 28-42 comparison to document model, 38-42 graph queries in SQL, 53 in-memory databases with, 89 many-to-one and many-to-many relation‐ ships, 33 multi-object transactions, need for, 231 NoSQL as alternative to, 29 object-relational mismatch, 29 relational algebra and SQL, 42 versus document model convergence of models, 41 data locality, 41 relational databases eventual consistency, 162 history, 28 leader-based replication, 153 logical logs, 160 philosophy compared to Unix, 499, 501 schema changes, 40, 111, 130 statement-based replication, 158 use of B-tree indexes, 80 relationships (see edges) reliability, 6-10, 489 building a reliable system from unreliable components, 276 defined, 6, 22 hardware faults, 7 human errors, 9 importance of, 10 of messaging systems, 442 Index | 581 software errors, 8 Remote Method Invocation (Java RMI), 134 remote procedure calls (RPCs), 134-136 (see also services) based on futures, 135 data encoding and evolution, 136 issues with, 134 using Avro, 126, 135 using Thrift, 135 versus message brokers, 137 repeatable reads (transaction isolation), 242 replicas, 152 replication, 151-193, 556 and durability, 227 chain replication, 155 conflict resolution and, 246 consistency properties, 161-167 consistent prefix reads, 165 monotonic reads, 164 reading your own writes, 162 in distributed filesystems, 398 leaderless, 177-191 detecting concurrent writes, 184-191 limitations of quorum consistency, 181-183, 334 sloppy quorums and hinted handoff, 183 monitoring staleness, 182 multi-leader, 168-177 across multiple datacenters, 168, 335 handling write conflicts, 171-175 replication topologies, 175-177 partitioning and, 147, 200 reasons for using, 145, 151 single-leader, 152-161 failover, 157 implementation of replication logs, 158-161 relation to consensus, 367 setting up new followers, 155 synchronous versus asynchronous, 153-155 state machine replication, 349, 452 using erasure coding, 398 with heterogeneous data systems, 453 replication logs (see logs) reprocessing data, 496, 498 (see also evolvability) from log-based messaging, 451 request routing, 214-216 582 | Index approaches to, 214 parallel query execution, 216 resilient systems, 6 (see also fault tolerance) response time as performance metric for services, 13, 389 guarantees on, 298 latency versus, 14 mean and percentiles, 14 user experience, 15 responsibility and accountability, 535 REST (Representational State Transfer), 133 (see also services) RethinkDB (database) document data model, 31 dynamic partitioning, 212 join support, 34, 42 key-range partitioning, 202 leader-based replication, 153 subscribing to changes, 456 Riak (database) Bitcask storage engine, 72 CRDTs, 174, 191 dotted version vectors, 191 gossip protocol, 216 hash partitioning, 203-204, 211 last-write-wins conflict resolution, 186 leaderless replication, 177 LevelDB storage engine, 78 linearizability, lack of, 335 multi-datacenter support, 184 preventing lost updates across replicas, 246 rebalancing, 213 search feature, 209 secondary indexes, 207 siblings (concurrently written values), 190 sloppy quorums, 184 ring buffers, 450 Ripple (cryptocurrency), 532 rockets, 10, 36, 305 RocksDB (storage engine), 78 leveled compaction, 79 rollbacks (transactions), 222 rolling upgrades, 8, 112 routing (see request routing) row-oriented storage, 96 row-based replication, 160 rowhammer (memory corruption), 529 RPCs (see remote procedure calls) Rubygems (package manager), 428 rules (Datalog), 61 S safety and liveness properties, 308 in consensus algorithms, 366 in transactions, 222 sagas (see compensating transactions) Samza (stream processor), 466, 467 fault tolerance, 479 streaming SQL support, 466 sandboxes, 9 SAP HANA (database), 93 scalability, 10-18, 489 approaches for coping with load, 17 defined, 22 describing load, 11 describing performance, 13 partitioning and, 199 replication and, 161 scaling up versus scaling out, 146 scaling out, 17, 146 (see also shared-nothing architecture) scaling up, 17, 146 scatter/gather approach, querying partitioned databases, 207 SCD (slowly changing dimension), 476 schema-on-read, 39 comparison to evolvable schema, 128 in distributed filesystems, 415 schema-on-write, 39 schemaless databases (see schema-on-read) schemas, 557 Avro, 122-127 reader determining writer’s schema, 125 schema evolution, 123 dynamically generated, 126 evolution of, 496 affecting application code, 111 compatibility checking, 126 in databases, 129-131 in message-passing, 138 in service calls, 136 flexibility in document model, 39 for analytics, 93-95 for JSON and XML, 115 merits of, 127 schema migration on railways, 496 Thrift and Protocol Buffers, 117-121 schema evolution, 120 traditional approach to design, fallacy in, 462 searches building search indexes in batch processes, 411 k-nearest neighbors, 429 on streams, 467 partitioned secondary indexes, 206 secondaries (see leader-based replication) secondary indexes, 85, 557 partitioning, 206-209, 217 document-partitioned, 206 index maintenance, 495 term-partitioned, 208 problems with dual writes, 452, 491 updating, transaction isolation and, 231 secondary sorts, 405 sed (Unix tool), 392 self-describing files, 127 self-joins, 480 self-validating systems, 530 semantic web, 57 semi-synchronous replication, 154 sequence number ordering, 343-348 generators, 294, 344 insufficiency for enforcing constraints, 347 Lamport timestamps, 345 use of timestamps, 291, 295, 345 sequential consistency, 351 serializability, 225, 233, 251-266, 557 linearizability versus, 329 pessimistic versus optimistic concurrency control, 261 serial execution, 252-256 partitioning, 255 using stored procedures, 253, 349 serializable snapshot isolation (SSI), 261-266 detecting stale MVCC reads, 263 detecting writes that affect prior reads, 264 distributed execution, 265, 364 performance of SSI, 265 preventing write skew, 262-265 two-phase locking (2PL), 257-261 index-range locks, 260 performance, 258 Serializable (Java), 113 Index | 583 serialization, 113 (see also encoding) service discovery, 135, 214, 372 using DNS, 216, 372 service level agreements (SLAs), 15 service-oriented architecture (SOA), 132 (see also services) services, 131-136 microservices, 132 causal dependencies across services, 493 loose coupling, 502 relation to batch/stream processors, 389, 508 remote procedure calls (RPCs), 134-136 issues with, 134 similarity to databases, 132 web services, 132, 135 session windows (stream processing), 472 (see also windows) sessionization, 407 sharding (see partitioning) shared mode (locks), 258 shared-disk architecture, 146, 398 shared-memory architecture, 146 shared-nothing architecture, 17, 146-147, 557 (see also replication) distributed filesystems, 398 (see also distributed filesystems) partitioning, 199 use of network, 277 sharks biting undersea cables, 279 counting (example), 46-48 finding (example), 42 website about (example), 44 shredding (in relational model), 38 siblings (concurrent values), 190, 246 (see also conflicts) similarity search edit distance, 88 genome data, 63 k-nearest neighbors, 429 single-leader replication (see leader-based rep‐ lication) single-threaded execution, 243, 252 in batch processing, 406, 421, 426 in stream processing, 448, 463, 522 size-tiered compaction, 79 skew, 557 584 | Index clock skew, 291-294, 334 in transaction isolation read skew, 238, 266 write skew, 246-251, 262-265 (see also write skew) meanings of, 238 unbalanced workload, 201 compensating for, 205 due to celebrities, 205 for time-series data, 203 in batch processing, 407 slaves (see leader-based replication) sliding windows (stream processing), 472 (see also windows) sloppy quorums, 183 (see also quorums) lack of linearizability, 334 slowly changing dimension (data warehouses), 476 smearing (leap seconds adjustments), 290 snapshots (databases) causal consistency, 340 computing derived data, 500 in change data capture, 455 serializable snapshot isolation (SSI), 261-266, 329 setting up a new replica, 156 snapshot isolation and repeatable read, 237-242 implementing with MVCC, 239 indexes and MVCC, 241 visibility rules, 240 synchronized clocks for global snapshots, 294 snowflake schemas, 95 SOAP, 133 (see also services) evolvability, 136 software bugs, 8 maintaining integrity, 529 solid state drives (SSDs) access patterns, 84 detecting corruption, 519, 530 faults in, 227 sequential write throughput, 75 Solr (search server) building indexes in batch processes, 411 document-partitioned indexes, 207 request routing, 216 usage example, 4 use of Lucene, 79 sort (Unix tool), 392, 394, 395 sort-merge joins (MapReduce), 405 Sorted String Tables (see SSTables) sorting sort order in column storage, 99 source of truth (see systems of record) Spanner (database) data locality, 41 snapshot isolation using clocks, 295 TrueTime API, 294 Spark (processing framework), 421-423 bytecode generation, 428 dataflow APIs, 427 fault tolerance, 422 for data warehouses, 93 GraphX API (graph processing), 425 machine learning, 428 query optimizer, 427 Spark Streaming, 466 microbatching, 477 stream processing on top of batch process‐ ing, 495 SPARQL (query language), 59 spatial algorithms, 429 split brain, 158, 557 in consensus algorithms, 352, 367 preventing, 322, 333 using fencing tokens to avoid, 302-304 spreadsheets, dataflow programming capabili‐ ties, 504 SQL (Structured Query Language), 21, 28, 43 advantages and limitations of, 416 distributed query execution, 48 graph queries in, 53 isolation levels standard, issues with, 242 query execution on Hadoop, 416 résumé (example), 30 SQL injection vulnerability, 305 SQL on Hadoop, 93 statement-based replication, 158 stored procedures, 255 SQL Server (database) data warehousing support, 93 distributed transaction support, 361 leader-based replication, 153 preventing lost updates, 245 preventing write skew, 248, 257 read committed isolation, 236 recursive query support, 54 serializable isolation, 257 snapshot isolation support, 239 T-SQL language, 255 XML support, 30 SQLstream (stream analytics), 466 SSDs (see solid state drives) SSTables (storage format), 76-79 advantages over hash indexes, 76 concatenated index, 204 constructing and maintaining, 78 making LSM-Tree from, 78 staleness (old data), 162 cross-channel timing dependencies, 331 in leaderless databases, 178 in multi-version concurrency control, 263 monitoring for, 182 of client state, 512 versus linearizability, 324 versus timeliness, 524 standbys (see leader-based replication) star replication topologies, 175 star schemas, 93-95 similarity to event sourcing, 458 Star Wars analogy (event time versus process‐ ing time), 469 state derived from log of immutable events, 459 deriving current state from the event log, 458 interplay between state changes and appli‐ cation code, 507 maintaining derived state, 495 maintenance by stream processor in streamstream joins, 473 observing derived state, 509-515 rebuilding after stream processor failure, 478 separation of application code and, 505 state machine replication, 349, 452 statement-based replication, 158 statically typed languages analogy to schema-on-write, 40 code generation and, 127 statistical and numerical algorithms, 428 StatsD (metrics aggregator), 442 stdin, stdout, 395, 396 Stellar (cryptocurrency), 532 Index | 585 stock market feeds, 442 STONITH (Shoot The Other Node In The Head), 158 stop-the-world (see garbage collection) storage composing data storage technologies, 499-504 diversity of, in MapReduce, 415 Storage Area Network (SAN), 146, 398 storage engines, 69-104 column-oriented, 95-101 column compression, 97-99 defined, 96 distinction between column families and, 99 Parquet, 96, 131 sort order in, 99-100 writing to, 101 comparing requirements for transaction processing and analytics, 90-96 in-memory storage, 88 durability, 227 row-oriented, 70-90 B-trees, 79-83 comparing B-trees and LSM-trees, 83-85 defined, 96 log-structured, 72-79 stored procedures, 161, 253-255, 557 and total order broadcast, 349 pros and cons of, 255 similarity to stream processors, 505 Storm (stream processor), 466 distributed RPC, 468, 514 Trident state handling, 478 straggler events, 470, 498 stream processing, 464-481, 557 accessing external services within job, 474, 477, 478, 517 combining with batch processing lambda architecture, 497 unifying technologies, 498 comparison to batch processing, 464 complex event processing (CEP), 465 fault tolerance, 476-479 atomic commit, 477 idempotence, 478 microbatching and checkpointing, 477 rebuilding state after a failure, 478 for data integration, 494-498 586 | Index maintaining derived state, 495 maintenance of materialized views, 467 messaging systems (see messaging systems) reasoning about time, 468-472 event time versus processing time, 469, 477, 498 knowing when window is ready, 470 types of windows, 472 relation to databases (see streams) relation to services, 508 search on streams, 467 single-threaded execution, 448, 463 stream analytics, 466 stream joins, 472-476 stream-stream join, 473 stream-table join, 473 table-table join, 474 time-dependence of, 475 streams, 440-451 end-to-end, pushing events to clients, 512 messaging systems (see messaging systems) processing (see stream processing) relation to databases, 451-464 (see also changelogs) API support for change streams, 456 change data capture, 454-457 derivative of state by time, 460 event sourcing, 457-459 keeping systems in sync, 452-453 philosophy of immutable events, 459-464 topics, 440 strict serializability, 329 strong consistency (see linearizability) strong one-copy serializability, 329 subjects, predicates, and objects (in triplestores), 55 subscribers (message streams), 440 (see also consumers) supercomputers, 275 surveillance, 537 (see also privacy) Swagger (service definition format), 133 swapping to disk (see virtual memory) synchronous networks, 285, 557 comparison to asynchronous networks, 284 formal model, 307 synchronous replication, 154, 557 chain replication, 155 conflict detection, 172 system models, 300, 306-310 assumptions in, 528 correctness of algorithms, 308 mapping to the real world, 309 safety and liveness, 308 systems of record, 386, 557 change data capture, 454, 491 treating event log as, 460 systems thinking, 536 T t-digest (algorithm), 16 table-table joins, 474 Tableau (data visualization software), 416 tail (Unix tool), 447 tail vertex (property graphs), 51 Tajo (query engine), 93 Tandem NonStop SQL (database), 200 TCP (Transmission Control Protocol), 277 comparison to circuit switching, 285 comparison to UDP, 283 connection failures, 280 flow control, 282, 441 packet checksums, 306, 519, 529 reliability and duplicate suppression, 517 retransmission timeouts, 284 use for transaction sessions, 229 telemetry (see monitoring) Teradata (database), 93, 200 term-partitioned indexes, 208, 217 termination (consensus), 365 Terrapin (database), 413 Tez (dataflow engine), 421-423 fault tolerance, 422 support by higher-level tools, 427 thrashing (out of memory), 297 threads (concurrency) actor model, 138, 468 (see also message-passing) atomic operations, 223 background threads, 73, 85 execution pauses, 286, 296-298 memory barriers, 338 preemption, 298 single (see single-threaded execution) three-phase commit, 359 Thrift (data format), 117-121 BinaryProtocol, 118 CompactProtocol, 119 field tags and schema evolution, 120 throughput, 13, 390 TIBCO, 137 Enterprise Message Service, 444 StreamBase (stream analytics), 466 time concurrency and, 187 cross-channel timing dependencies, 331 in distributed systems, 287-299 (see also clocks) clock synchronization and accuracy, 289 relying on synchronized clocks, 291-295 process pauses, 295-299 reasoning about, in stream processors, 468-472 event time versus processing time, 469, 477, 498 knowing when window is ready, 470 timestamp of events, 471 types of windows, 472 system models for distributed systems, 307 time-dependence in stream joins, 475 time-of-day clocks, 288 timeliness, 524 coordination-avoiding data systems, 528 correctness of dataflow systems, 525 timeouts, 279, 557 dynamic configuration of, 284 for failover, 158 length of, 281 timestamps, 343 assigning to events in stream processing, 471 for read-after-write consistency, 163 for transaction ordering, 295 insufficiency for enforcing constraints, 347 key range partitioning by, 203 Lamport, 345 logical, 494 ordering events, 291, 345 Titan (database), 50 tombstones, 74, 191, 456 topics (messaging), 137, 440 total order, 341, 557 limits of, 493 sequence numbers or timestamps, 344 total order broadcast, 348-352, 493, 522 consensus algorithms and, 366-368 Index | 587 implementation in ZooKeeper and etcd, 370 implementing with linearizable storage, 351 using, 349 using to implement linearizable storage, 350 tracking behavioral data, 536 (see also privacy) transaction coordinator (see coordinator) transaction manager (see coordinator) transaction processing, 28, 90-95 comparison to analytics, 91 comparison to data warehousing, 93 transactions, 221-267, 558 ACID properties of, 223 atomicity, 223 consistency, 224 durability, 226 isolation, 225 compensating (see compensating transac‐ tions) concept of, 222 distributed transactions, 352-364 avoiding, 492, 502, 521-528 failure amplification, 364, 495 in doubt/uncertain status, 358, 362 two-phase commit, 354-359 use of, 360-361 XA transactions, 361-364 OLTP versus analytics queries, 411 purpose of, 222 serializability, 251-266 actual serial execution, 252-256 pessimistic versus optimistic concur‐ rency control, 261 serializable snapshot isolation (SSI), 261-266 two-phase locking (2PL), 257-261 single-object and multi-object, 228-232 handling errors and aborts, 231 need for multi-object transactions, 231 single-object writes, 230 snapshot isolation (see snapshots) weak isolation levels, 233-251 preventing lost updates, 242-246 read committed, 234-238 transitive closure (graph algorithm), 424 trie (data structure), 88 triggers (databases), 161, 441 implementing change data capture, 455 implementing replication, 161 588 | Index triple-stores, 55-59 SPARQL query language, 59 tumbling windows (stream processing), 472 (see also windows) in microbatching, 477 tuple spaces (programming model), 507 Turtle (RDF data format), 56 Twitter constructing home timelines (example), 11, 462, 474, 511 DistributedLog (event log), 448 Finagle (RPC framework), 135 Snowflake (sequence number generator), 294 Summingbird (processing library), 497 two-phase commit (2PC), 353, 355-359, 558 confusion with two-phase locking, 356 coordinator failure, 358 coordinator recovery, 363 how it works, 357 issues in practice, 363 performance cost, 360 transactions holding locks, 362 two-phase locking (2PL), 257-261, 329, 558 confusion with two-phase commit, 356 index-range locks, 260 performance of, 258 type checking, dynamic versus static, 40 U UDP (User Datagram Protocol) comparison to TCP, 283 multicast, 442 unbounded datasets, 439, 558 (see also streams) unbounded delays, 558 in networks, 282 process pauses, 296 unbundling databases, 499-515 composing data storage technologies, 499-504 federation versus unbundling, 501 need for high-level language, 503 designing applications around dataflow, 504-509 observing derived state, 509-515 materialized views and caching, 510 multi-partition data processing, 514 pushing state changes to clients, 512 uncertain (transaction status) (see in doubt) uniform consensus, 365 (see also consensus) uniform interfaces, 395 union type (in Avro), 125 uniq (Unix tool), 392 uniqueness constraints asynchronously checked, 526 requiring consensus, 521 requiring linearizability, 330 uniqueness in log-based messaging, 522 Unix philosophy, 394-397 command-line batch processing, 391-394 Unix pipes versus dataflow engines, 423 comparison to Hadoop, 413-414 comparison to relational databases, 499, 501 comparison to stream processing, 464 composability and uniform interfaces, 395 loose coupling, 396 pipes, 394 relation to Hadoop, 499 UPDATE statement (SQL), 40 updates preventing lost updates, 242-246 atomic write operations, 243 automatically detecting lost updates, 245 compare-and-set operations, 245 conflict resolution and replication, 246 using explicit locking, 244 preventing write skew, 246-251 V validity (consensus), 365 vBuckets (partitioning), 199 vector clocks, 191 (see also version vectors) vectorized processing, 99, 428 verification, 528-533 avoiding blind trust, 530 culture of, 530 designing for auditability, 531 end-to-end integrity checks, 531 tools for auditable data systems, 532 version control systems, reliance on immutable data, 463 version vectors, 177, 191 capturing causal dependencies, 343 versus vector clocks, 191 Vertica (database), 93 handling writes, 101 replicas using different sort orders, 100 vertical scaling (see scaling up) vertices (in graphs), 49 property graph model, 50 Viewstamped Replication (consensus algo‐ rithm), 366 view number, 368 virtual machines, 146 (see also cloud computing) context switches, 297 network performance, 282 noisy neighbors, 284 reliability in cloud services, 8 virtualized clocks in, 290 virtual memory process pauses due to page faults, 14, 297 versus memory management by databases, 89 VisiCalc (spreadsheets), 504 vnodes (partitioning), 199 Voice over IP (VoIP), 283 Voldemort (database) building read-only stores in batch processes, 413 hash partitioning, 203-204, 211 leaderless replication, 177 multi-datacenter support, 184 rebalancing, 213 reliance on read repair, 179 sloppy quorums, 184 VoltDB (database) cross-partition serializability, 256 deterministic stored procedures, 255 in-memory storage, 89 output streams, 456 secondary indexes, 207 serial execution of transactions, 253 statement-based replication, 159, 479 transactions in stream processing, 477 W WAL (write-ahead log), 82 web services (see services) Web Services Description Language (WSDL), 133 webhooks, 443 webMethods (messaging), 137 WebSocket (protocol), 512 Index | 589 windows (stream processing), 466, 468-472 infinite windows for changelogs, 467, 474 knowing when all events have arrived, 470 stream joins within a window, 473 types of windows, 472 winners (conflict resolution), 173 WITH RECURSIVE syntax (SQL), 54 workflows (MapReduce), 402 outputs, 411-414 key-value stores, 412 search indexes, 411 with map-side joins, 410 working set, 393 write amplification, 84 write path (derived data), 509 write skew (transaction isolation), 246-251 characterizing, 246-251, 262 examples of, 247, 249 materializing conflicts, 251 occurrence in practice, 529 phantoms, 250 preventing in snapshot isolation, 262-265 in two-phase locking, 259-261 options for, 248 write-ahead log (WAL), 82, 159 writes (database) atomic write operations, 243 detecting writes affecting prior reads, 264 preventing dirty writes with read commit‐ ted, 235 WS-* framework, 133 (see also services) WS-AtomicTransaction (2PC), 355 590 | Index X XA transactions, 355, 361-364 heuristic decisions, 363 limitations of, 363 xargs (Unix tool), 392, 396 XML binary variants, 115 encoding RDF data, 57 for application data, issues with, 114 in relational databases, 30, 41 XSL/XPath, 45 Y Yahoo!

Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems
by Martin Kleppmann
Published 16 Mar 2017

Public cloud services such as EC2 are notorious for having frequent transient network glitches [14], and well-managed private datacenter networks can be stabler environments. Nevertheless, nobody is immune from network problems: for example, a problem during a software upgrade for a switch could trigger a network topology reconfiguration, during which network packets could be delayed for more than a minute [17]. Sharks might bite undersea cables and damage them [18]. Other surprising faults include a network interface that sometimes drops all inbound packets but sends outbound packets successfully [19]: just because a network link works in one direction doesn’t guarantee it’s also working in the opposite direction. Network partitions When one part of the network is cut off from the rest due to a network fault, that is sometimes called a network partition or netsplit.

…

imperative query APIs, Declarative Queries on the Web InfoSphere Streams (CEP engine), Complex event processing MQ (messaging), Message brokers compared to databasesdistributed transaction support, XA transactions System R (database), The Slippery Concept of a Transaction WebSphere (messaging), Message brokers idempotence, The problems with remote procedure calls (RPCs), Idempotence, Glossaryby giving operations unique IDs, Operation identifiers, Multi-partition request processing idempotent operations, Exactly-once execution of an operation immutabilityadvantages of, Advantages of immutable events, Designing for auditability deriving state from event log, State, Streams, and Immutability-Limitations of immutability for crash recovery, Hash Indexes in B-trees, B-tree optimizations, Indexes and snapshot isolation in event sourcing, Event Sourcing inputs to Unix commands, Transparency and experimentation limitations of, Limitations of immutability Impala (query engine)for data warehouses, The divergence between OLTP databases and data warehouses hash joins, Broadcast hash joins native code generation, The move toward declarative query languages use of HDFS, Diversity of processing models impedance mismatch, The Object-Relational Mismatch imperative languages, Query Languages for Datasetting element styles (example), Declarative Queries on the Web in doubt (transaction status), Coordinator failureholding locks, Holding locks while in doubt orphaned transactions, Recovering from coordinator failure in-memory databases, Keeping everything in memorydurability, Durability serial transaction execution, Actual Serial Execution incidentscascading failures, Software Errors crashes due to leap seconds, Clock Synchronization and Accuracy data corruption and financial losses due to concurrency bugs, Weak Isolation Levels data corruption on hard disks, Durability data loss due to last-write-wins, Converging toward a consistent state, Timestamps for ordering events data on disks unreadable, Mapping system models to the real world deleted items reappearing, Custom conflict resolution logic disclosure of sensitive data due to primary key reuse, Leader failure: Failover errors in transaction serializability, Maintaining integrity in the face of software bugs gigabit network interface with 1 Kb/s throughput, Summary network faults, Network Faults in Practice network interface dropping only inbound packets, Network Faults in Practice network partitions and whole-datacenter failures, Faults and Partial Failures poor handling of network faults, Network Faults in Practice sending message to ex-partner, Ordering events to capture causality sharks biting undersea cables, Network Faults in Practice split brain due to 1-minute packet delay, Leader failure: Failover, Network Faults in Practice vibrations in server rack, Describing Performance violation of uniqueness constraint, Maintaining integrity in the face of software bugs indexes, Data Structures That Power Your Database, Glossaryand snapshot isolation, Indexes and snapshot isolation as derived data, Derived Data, Composing Data Storage Technologies-What’s missing?

…

pessimistic versus optimistic concurrency control, Pessimistic versus optimistic concurrency control serial execution, Actual Serial Execution-Summary of serial executionpartitioning, Partitioning using stored procedures, Encapsulating transactions in stored procedures, Using total order broadcast serializable snapshot isolation (SSI), Serializable Snapshot Isolation (SSI)-Performance of serializable snapshot isolationdetecting stale MVCC reads, Detecting stale MVCC reads detecting writes that affect prior reads, Detecting writes that affect prior reads distributed execution, Performance of serializable snapshot isolation, Limitations of distributed transactions performance of SSI, Performance of serializable snapshot isolation preventing write skew, Decisions based on an outdated premise-Detecting writes that affect prior reads two-phase locking (2PL), Two-Phase Locking (2PL)-Index-range locksindex-range locks, Index-range locks performance, Performance of two-phase locking Serializable (Java), Language-Specific Formats serialization, Formats for Encoding Data(see also encoding) service discovery, Current directions for RPC, Request Routing, Service discoveryusing DNS, Request Routing, Service discovery service level agreements (SLAs), Describing Performance service-oriented architecture (SOA), Dataflow Through Services: REST and RPC(see also services) services, Dataflow Through Services: REST and RPC-Data encoding and evolution for RPCmicroservices, Dataflow Through Services: REST and RPCcausal dependencies across services, The limits of total ordering loose coupling, Making unbundling work relation to batch/stream processors, Batch Processing, Stream processors and services remote procedure calls (RPCs), The problems with remote procedure calls (RPCs)-Data encoding and evolution for RPCissues with, The problems with remote procedure calls (RPCs) similarity to databases, Dataflow Through Services: REST and RPC web services, Web services, Current directions for RPC session windows (stream processing), Types of windows(see also windows) sessionization, GROUP BY sharding (see partitioning) shared mode (locks), Implementation of two-phase locking shared-disk architecture, Distributed Data, MapReduce and Distributed Filesystems shared-memory architecture, Distributed Data shared-nothing architecture, Approaches for Coping with Load, Distributed Data-Distributed Data, Glossary(see also replication) distributed filesystems, MapReduce and Distributed Filesystems(see also distributed filesystems) partitioning, Partitioning use of network, Unreliable Networks sharksbiting undersea cables, Network Faults in Practice counting (example), MapReduce Querying-MapReduce Querying finding (example), Query Languages for Data website about (example), Declarative Queries on the Web shredding (in relational model), Which data model leads to simpler application code? siblings (concurrent values), Merging concurrently written values, Conflict resolution and replication(see also conflicts) similarity searchedit distance, Full-text search and fuzzy indexes genome data, Summary k-nearest neighbors, Specialization for different domains single-leader replication (see leader-based replication) single-threaded execution, Atomic write operations, Actual Serial Executionin batch processing, Bringing related data together in the same place, Dataflow engines, Parallel execution in stream processing, Logs compared to traditional messaging, Concurrency control, Uniqueness in log-based messaging size-tiered compaction, Performance optimizations skew, Glossaryclock skew, Relying on Synchronized Clocks-Clock readings have a confidence interval, Implementing Linearizable Systems in transaction isolationread skew, Snapshot Isolation and Repeatable Read, Summary write skew, Write Skew and Phantoms-Materializing conflicts, Decisions based on an outdated premise-Detecting writes that affect prior reads(see also write skew) meanings of, Snapshot Isolation and Repeatable Read unbalanced workload, Partitioning of Key-Value Datacompensating for, Skewed Workloads and Relieving Hot Spots due to celebrities, Skewed Workloads and Relieving Hot Spots for time-series data, Partitioning by Key Range in batch processing, Handling skew slaves (see leader-based replication) sliding windows (stream processing), Types of windows(see also windows) sloppy quorums, Sloppy Quorums and Hinted Handoff(see also quorums) lack of linearizability, Implementing Linearizable Systems slowly changing dimension (data warehouses), Time-dependence of joins smearing (leap seconds adjustments), Clock Synchronization and Accuracy snapshots (databases)causal consistency, Ordering and Causality computing derived data, Creating an index in change data capture, Initial snapshot serializable snapshot isolation (SSI), Serializable Snapshot Isolation (SSI)-Performance of serializable snapshot isolation, What Makes a System Linearizable?

Digital Empires: The Global Battle to Regulate Technology
by Anu Bradford
Published 25 Sep 2023

But this kind of influence has also spread through Chinese companies that provide other kinds of digital infrastructure: cloud services, e-commerce platforms, and more recently, social media platforms as companies such as Alibaba, Tencent, and ByteDance globalize their operations. Thus, the term “infrastructure power,” as used in this chapter, broadly describes Chinese companies’ growing global role in building not only physical digital infrastructures (such as 5G telecom networks and undersea cables) but also other types of foundational digital ecosystems (such as cloud systems and AI-driven surveillance capabilities) that countries, cities, and other governing entities procure to build and manage their digital societies. This increasingly vast global influence of Chinese companies has raised concerns among governments, particularly among the US and its close allies.

…

Huawei and China International Telecommunications and Construction Corporation (CITCC) provide the technology for the project, which also benefits from close to $400 million of Chinese loans. Huawei and ZTE also have a strong presence in constructing digital infrastructure such as fiber optic cables across the region.51 In addition, Huawei Marine has undertaken several undersea cable projects in the Philippines and elsewhere in Southeast Asia, further illustrating the pervasive influence that Chinese tech companies have exercised in the region. The Chinese sphere of influence also reaches into Latin America, which has traditionally been politically more connected to the US by geographic proximity and the deep economic relationships the US has cultivated in the region.

…

Since its introduction in 2013, the BRI has provided an umbrella framework for Chinese investment in over a hundred countries.10 The goal underlying the BRI is to connect Asia with other continents via land, rail, and maritime networks in an effort to deepen trade relations and contribute toward China’s economic growth. The DSR, which the Chinese government unveiled in a 2015 white paper, adds a digital connectivity element to the BRI.11 The DSR comprises a range of projects, including building telecom networks, laying submarine data cables, establishing data centers, installing global satellite navigation systems, and exporting technologies for smart cities and other cyber infrastructure across Africa, Asia, Europe, Latin America, and the Caribbean. According to some estimates, over a third of the 138 countries participating in the BRI in 2019 were involved in some DSR projects with Chinese companies.12 But given the somewhat amorphous character of the DSR, it is difficult to assess the initiative’s true scope.

Neutrino Hunters: The Thrilling Chase for a Ghostly Particle to Unlock the Secrets of the Universe
by Ray Jayawardhana
Published 10 Dec 2013

One of the wackier ideas for putting neutrinos to practical use involves employing them for long-distance communication, since they travel unhindered through virtually anything. You could imagine sending an encoded beam of neutrinos from one side of the Earth to the other straight through the planet, whereas a radio signal needs to be bounced off multiple satellites in orbit or transmitted through undersea cables circling the globe. Along similar lines, one scientist has suggested using neutrino beams to send messages to submarines deep underwater. To have any chance of success, though, the beam would have to be about a million times more intense than those used in current experiments, and there would be some daunting challenges at the receiving end.

War of Shadows: Codebreakers, Spies, and the Secret Struggle to Drive the Nazis From the Middle East
by Gershom Gorenberg
Published 19 Jan 2021

Troops poured in from England, Australia, New Zealand, and India. The Australians made the strongest impression: they were huge, loud, and happy and drank prodigiously.21 ANTHONY EDEN RETURNED to London in early November with word from Wavell. A messenger was the only means of communication that Wavell trusted. Italy had cut Britain’s undersea cables in the Mediterranean, and Wavell feared the treachery of wireless. Anything sent by it, even in the best code you had, could give you away.22 Churchill heard the general’s plans and “purred like six cats,” as he said himself.23 By the light of a half-moon, the thirty thousand men of the Western Desert Force marched forward on the night of December 7, passing between Graziani’s isolated fortified camps.

…

The Foreign Office asked Menzies, who forwarded the question to Travis, who would have most likely asked Dudley-Smith to get answers. A note initialed by Travis came back to the Foreign Office. It said American diplomatic telegrams to Washington were safe from interception—by the Axis. The unstated implication was that American traffic traveled on an intact British undersea cable. As for the wider question of US diplomatic codes and ciphers, this was delicate. The State Department had three code systems, the GCHQ note said. One system was used rarely, for the most secret messages. So far it was unbreakable. “The second-best book,” used more often, for material that was just slightly less secret, was “legible but patchy.”

GCHQ
by Richard Aldrich
Published 10 Jun 2010

(‘Fred’) Stannard RN 1965–1969 Directors of Communications-Electronics Security Group 1969– Air Vice Marshal Arthur Foden 1969–1975 Brian Tovey 1975–1978 Dr John Johnson 1978–1980 Major Gen Alistair Anderson 1980–1985 Paul Foster 1985–1989 Air Vice Marshal John Porter 1989–1991 Andrew Saunders 1991–1998 Richard Walton 1999–2002 Huw Rees 2002–2005 Dr John Widdowson 2005– Appendix 2 GCHQ Timeline 1 Nov. 1919 GC&CS formed from a merger of the Army’s MI1b and the Navy’s Room 40 1921 Alastair Denniston becomes Director of GG&CS Aug. 1939 GC&CS moves to Bletchley Park to avoid wartime bombing 1942 Holden agreement on Anglo-American naval sigint 1943 BRUSA agreement on Anglo–American military sigint 1944 Revised Holden agreement on Anglo–American naval sigint 1944 Edward Travis becomes Director of GC&CS Feb. 1944 Cypher Policy Board created due to penetration of Allied cyphers 15 Sept. 1945 US Army code-breakers redesignated Army Security Agency (ASA) 22 Feb. 1946 Commonwealth sigint cooperation conference in London begins 5 Mar. 1946 Revised BRUSA agreement on Anglo–American sigint cooperation May 1946 Revised BRUSA technical appendices on Anglo–American sigint cooperation Jul. 1948 UK–USA Communications Intelligence Agreement 1 Aug. 1948 Captain Edmund Wilson and his team are shown over Oakley and Benhall sites 29 Oct. 1948 ‘Black Friday’ – a major change occurs in Russian cypher procedures 1 Nov. 1948 The term ‘London Signals Intelligence Centre’ is abolished in favour of GCHQ 20 May 1949 US Armed Forces Security Agency created, a weak forerunner of NSA Aug. 1949 Loss of the USS Cochino 1949 Vienna tunnel operations begin 10 Mar. 1950 US Communications Intelligence Board created Mar. 1952 Eric Jones becomes Director of GCHQ, taking over from Edward Travis Jul. 1952 Move from Eastcote to Cheltenham begins 24 Oct. 1952 President Truman signs order to create NSA, following failure to warn of Korean War 1953 LCSA created with Major General William Penney as Director 12 Mar. 1953 Loss of an RAF Lincoln over the Inner German Border Feb. 1954 Move from Eastcote to Cheltenham completed 2 Sept. 1954 Work on the Berlin tunnel begins 21 Apr. 1956 Eastern Bloc troops break into the Berlin tunnel 1956 NSA moves to new headquarters at Fort Meade 1957 RAF Hambuhren handed over to German communications units 1957 Move from HMS Anderson sigint site to Perkar on Ceylon 1957 Government White Paper suggests coming end of National Service 1957 Karamursel NSA station built in Turkey to the south of Istanbul 11 Oct. 1957 Jodrell Bank, the first radiotelescope, is completed and is also used for sigint 1 Nov. 1957 Captain Robert Stannard RN becomes Director of LCSA, taking over from William Penney 21 Aug. 1958 The RAF’s sigint unit, 192 Squadron, is renumbered as 51 Squadron 2 Sept. 1958 US Sigint C-130A Hercules shot down over Armenia with loss of seventeen crew 1958 Berlin tunnel translators move from SIS to GCHQ as London Processing Group 1958 Comet sigint aircraft enter service with 51 Squadron 1959 1 Special Wireless Regiment renamed 13 Signals Regiment May 1959 RAF Habbaniya in Iraq closed and sigint personnel moved to Cyprus 3 Jun. 1959 One of 51 Squadron’s new Comets destroyed by fire 1 Sept. 1959 2 Wireless Regiment at Ayios Nikolaos renamed 9 Signals Regiment 12 Nov. 1959 The first dedicated US sigint-gathering ship, USS Oxford, is authorised 1 May 1960 Shooting down of Gary Powers’ U-2 spy plane 1960 Clive Loehnis becomes Director of GCHQ, taking over from Eric Jones Sept. 1960 Joe Hooper introduces joint sigint equipment purchase with services Dec. 1960 Templer Report on integration of service interception 1961 George Blake arrested and early compromise of Berlin tunnel realised 1961 Collection begins from Teufelsberg in Berlin operating out of ASA vans Nov. 1961 Project Sandra receives Treasury approval 1962 Perkar sigint site on Ceylon closed Jul. 1962 Sir Stuart Hampshire’s review of GCHQ Dec. 1962 Hampshire visits NSA for three weeks Mar. 1963 LCSA becomes London Communications–Electronics Security Agency May 1963 Hampshire review of GCHQ completed Jun. 1963 Permission for a communications facility on Diego Garcia requested 1963 MC 74/1 – NATO Cryptographic Policy agreed 1963 Project Sandra begins operations on Cyprus 1 Jan. 1964 Little Sai Wan station in Hong Kong passes from RAF control to GCHQ Sept. 1964 Cypher clerks at the British Embassy in Moscow repel KGB ‘firemen’ 1965 Joe Hooper becomes Director of GCHQ, taking over from Clive Loehnis 1965 LCSA takes over SCDU and JSRU and is renamed CESD Nov. 1965 Diego Garcia agreement completed and 2,300 inhabitants removed 1966 Pine Gap station opened in Australia Oct. 1966 Decision taken to replace sigint Comets with Nimrod R1s 1967 Scharfoldendorf station near the Inner German Border closed 7 Jun. 1967 USS Liberty AGTR ship attacked off Israel 1967 Chapman Pincher and the ‘D-Notice affair’ exposes cable vetting 1967 Teufelsberg sigint site in Berlin begins operations Dec. 1967 First voyage of the USS Pueblo Jan. 1968 Geoffrey Prime offers his services to the KGB 23 Jan. 1968 USS Pueblo ALGER-2 ship captured by North Korea 24 Jan. 1968 HMS Totem, now renamed Dakar, sinks off Cyprus 20 Aug. 1968 Russian invasion of Czechoslovakia takes GCHQ and NSA by surprise Late 1968 Dick White review of DIS and JIC in the wake of Czech invasion Late 1968 Dick White inquiry into rising sigint costs 9 Sept. 1968 Geoffrey Prime begins work at GCHQ’s London Processing Group 1969 GCHQ’s nuclear-powered sigint ship project abandoned Jun. 1969 Decision to merge CESD with GCHQ and change its name to CESG 1969 Arthur Foden becomes Director of CESG, taking over from Robert Stannard 30 Mar. 1972 Kizildere incident: eight GCHQ staff taken hostage and three killed by TPLA terrorists Apr. 1972 JIC (A) delegation meets Groupe de Synthèse et Prévision in Paris 1972 Skynet III decision 6 Mar. 1973 Ryolite satellite launched to verify the SALT 1 arms control treaty 18 Jun. 1973 Closure of Cobra Mist facility at Orford Ness announced Aug. 1973 Arthur Bonsall becomes Director of GCHQ, taking over from Joe Hooper Aug. 1973 Nixon–Kissinger ‘cut-off’ of intelligence cooperation attempted 1973 Transfer of London Processing Group to Cheltenham begun 1973 James Ellis discovers the asymmetric algorithm, later the foundation of RSA 3 May 1974 First operational flight by a Nimrod R1 1974 Cyprus invasion by Turkey 1974 US sigint bases in Turkey shut down Feb. 1975 Government Secure Speech Network cancelled Jul. 1975 Ivy Bells undersea cable-tapping operation begins using USS Halibut Aug. 1975 Work on Diego Garcia expansion begins 1975 GCHQ Mauritius station closed 1975 NSA takes delivery of its first Cray computer Sept. 1975 Brian Tovey becomes Director of CESG, taking over from Arthur Foden 22 Mar. 1976 Geoffrey Prime moved from London to Cheltenham 1976 ‘ABC trial’ of Aubrey, Berry and Campbell begins 1976 Geoffrey Prime promoted to Section Head in J Division 1977 Transfer of London Processing Group to Cheltenham completed 1977 14 Signals Regiment (Electronic Warfare) formed 1977 GCHQ’s Wincombe station closed 1977 GCHQ’s Flowerdown station closed 28 Sept. 1977 Geoffrey Prime resigns from GCHQ 1978 GCHQ’s Gilnahirk station in Northern Ireland closed 1978 Special Collection Service, a joint NSA–CIA black-bag unit created 1978 GCHQ station at Two Boats on Ascension Island reactivated 1978 John Johnson becomes Director of CESG, taking over from Brian Tovey 1978 Brian Tovey becomes Director of GCHQ, taking over from Arthur Bonsall Jan. 1979 Iranian revolution – NSA and GCHQ listening posts in Iran are lost 23 Feb. 1979 One-day strike triggers Brian Tovey’s thinking on union removal at GCHQ 1980 Alastair Anderson becomes Director of CESG, taking over from John Johnson 1980 Ivy Bells submarine tapping operation blown by Ronald Pelton, KGB agent in NSA 9 Mar. 1981 One-day strike at GCHQ, then disruptive action to April 16 Nov. 1981 Geoffrey Prime makes his last contact with the KGB in East Berlin 1982 Gordon Welchman threatened with OSA over Hut Six Story 1982 GCHQ’s Little Sai Wan closed down and moved to Chum Hom Kok 26 Jun. 1982 Geoffrey Prime confesses 15 Jul. 1982 Prime remanded in custody on OSA charges 23 Sept. 1982 James Bamford’s Puzzle Palace published 10 Nov. 1982 Geoffrey Prime pleads guilty 1983 STU-III secure speech unit introduced by NSA Sept. 1983 Peter Marychurch becomes Director of GCHQ, taking over from Brian Tovey 1 Dec. 1983 Pilot polygraph machine arrives at R12 at Cheltenham Dec. 1983 Decision on de-unionisation taken by ad hoc Cabinet committee 25 Jan. 1984 GCHQ staff receive GN 100/84 letter on unions 17 Apr. 1984 WPC Yvonne Fletcher shot outside the Libyan People’s Bureau in London 1984 GCHQ’s Brora station in Sutherland closed 1985 Paul Foster becomes Director of CESG, taking over from Alastair Anderson 1985 Interception of Communications Act 1 Aug. 1985 KGB officer Vitaly Yurchenko defects and reveals NSA spy Ronald Pelton 5 Jun. 1986 Pelton convicted of espionage for the KGB 6 Jun. 1986 Jonathan Pollard pleads guilty to espionage for Mossad 1986 Joint Speech Research Unit at GCHQ amalgamated with Speech Research Group at RSRE to form Speech Research Unit 20 Jan. 1987 European Commission for Human Rights declares GCHQ trade union case inadmissible 31 Jul. 1988 Hawklaw station in Fife closed 1989 Hilltop station at Teufelberg in Berlin closed 1989 John Porter becomes Director of CESG, taking over from Paul Foster 1989 John Adye becomes Director of GCHQ, taking over from Peter Marychurch Mar. 1991 Andrew Saunders becomes Director of CESG, taking over from John Porter Late 1993 Sir Michael Quinlan asked to look at government spending on intelligence 22 Dec. 1993 T of R for Quinlan’s ‘Review of Intelligence Requirements and Resources’ agreed Jun. 1994 ‘Review of Intelligence Requirements and Resources’ completed 1994 GCHQ’s Earl’s Court station at Empress Building closed Oct. 1994 Operations at GCHQ’s Chum Hom Kok station end 2 Nov. 1994 Intelligence Services Act 12 Dec. 1994 Roger Hum Special Study of GCHQ commissioned Jan. 1995 Chum Hom Kok closed down and operations move to Geraldton in Australia Jan. 1995 Operations at Cheadle end and station closed in June 25 Mar. 1995 Roger Hum Special Study completed Mar. 1995 13 Signals Regiment in Germany disbanded, some move to JSSU at RAF Digby 16 May 1995 XW666, one of the three sigint Nimrod R1s, ditches in the Moray Firth 1995 RAF 51 Squadron moves from RAF Wyton to RAF Waddington 16 Oct. 1995 New high-level post created to represent GCHQ in London 6 Nov. 1995 J, K and V Divisions abolished.

…

AB Cryptoteknik 213 ‘ABC’ trial (1977–78) 8, 359–61, 423, 459 Abernethy, Barbara 70 Abu Hamza al-Masri 542 Abyssinia (Ethiopia) 19 Adams, Gerry 500 Aden (Yemen) 6, 156, 164, 345 Admiralty Signals Division 137 Admoni, Nahum 471 Adye, John 427, 431, 476, 483, 494, 495, 598, 608 aerial reconnaissance 31, 59 Afghanistan 387, 420, 421, 510, 511, 533–9 Africa 99, 148, 182, 268, 299, 336, 454–5, 479 Aid, Matthew 521 Aiken, John 326, 327, 328, 331 Airborne Rafter programme 267, 538 Aitken, Jonathan 493–5 al-Badr, Imam 163–4 al-Jazeera 513–14 al-Qaeda 9, 509–11, 513–14, 517, 532 Alanbrooke, Field Marshal Lord 45 Aldeburgh (Suffolk) 286 Aldridge, Peter 449 Alexander, A.V. 123–4 Alexander, Hugh 25, 27, 78, 96, 599 Allen, Lew 357 Allied Commission for Austria and London 170 Alp, Saffet 314 Alvear, Soledad 519 American Office of Naval Intelligence 143 American Type-777 satellite 348 Amery, Julian 162–3, 295 Ames, Aldrich 385, 444 Amory, Robert 158 Amsterdam 487 Anaya, Admiral Jorge 389, 393, 395 Anderson, Jack 344 Andrew, Christopher 2, 362 Anglo-American-Commonwealth sigint 64, 82, 152 Anglo-American relations 7–8; and Balkans 472–5; and Berlin tunnel operation 172–6; changing nature of 441–3, 449–50; and cost of cooperation 222–3; and Cuba 341–2; deterioration in 281–95, 333; and Far East 151–2; and global sigint 89–101; gradual improvement in 295–8; impact of politics on 278; and liberating of Axis sigint 47–56; and Nimrod programme 268–70, 273–4; and Project Sandra 322–3; and public disclosure of sigint material on 355–7, 358, 361–2; and sale of cypher machines 209–15; shared problems 333–4; and sigint 7–8, 38–46, 91–2; and sigint satellite and computer revolution 347–54, 437–8; successful Russian intercepts 279–81; and Suez crisis 157–9; and Third World bases 334–9; and trade unions at GCHQ 421–2, 429; in Turkey 302; and Venona Project 72–88; in wartime 38–46 Angola 357, 454–5 Ankara 58, 254, 302, 303, 305, 310–11, 313, 315, 318, 330 Annan, Kofi 523–4 Antalya (Turkey) 326 Anti-Christ Doom Squad 487 Anti-Smuggling Task Force (Hong Kong) 477 AQ Khan network 531 Aquarius (computer) 349 Arab-Israeli War (1973) see Yom Kippur War Arab states 109 Arafat, Yasser 277 Aral Sea 306 Arbuthnot, Mrs 70 Arctic Circle 133, 136–9, 144–6, 147, 265 Argentina 307–8, 388–415 Argentine Air Force 401, 408, 410, 415 Argentine Army 396 Argentine Navy 395, 405, 408 Argus satellite 377 Arlington Hall (US Army code-breaking centre, Washington) 45, 74, 77, 80, 83 arms control 203, 257, 287, 288, 290 Armstrong, Sir Robert 416, 425, 427, 428, 430–1 Army Intelligence Corps 230 Army Security Agency 80 as-Sallal, Abdullah 163–4 Ascension Island 162, 278, 321, 392, 414 Ash, Timothy Garton 465 Athens 324, 330 Atlantic, Battle of (WWII) 42, 60 Atlas (computer) 349–50 Atomic Demolition Munitions (ADMS) 249 Atomic Energy Authority (AEA) 261 Atomic Energy Intelligence 155 atomic and nuclear weapons 2, 5, 36, 45; and Anglo-US arms control 287–9; at Los Alamos 75, 76, 82; British 163, 249; Chinese 155; need for better intelligence on 253, 255, 321–3; Soviet 107–8, 112, 114, 116, 119, 131–2, 148, 157, 173, 301–2; spiralling of arms race 438–9; US 249; US put on alert (1973) 293–4 Atomic Weapons Establishment (Aldermaston, Berkshire) 418–19 Attlee, Clement 73, 86 ATV 432 Aubrey, Crispin 358–9, 360 Auckland (New Zealand) 487 Audiotel 480 Augsburg (Germany) 48 Austin, Harris M. 115 Australia 79, 80, 85–8, 89, 90, 92–4, 98, 154, 164, 165, 167, 168, 213, 467, 477, 487, 533; Australian Security Intelligence Organisation (ASIO) 86–7, 88; Central Bureau 92; Defence Signals Branch (DSB) 151, 153, 213; Defence Signals Department (DSD) 348; Royal Australian Air Force 150; Royal Australian Navy 266; Royal Australian Signals 166 Austria 96, 371, 384 Automatic Data Processing 353 Ayios Nikolaos (Cyprus) 156, 162, 163, 230, 234, 327, 328, 358–9, 360, 383, 384 Azerbaijan 109 Bad Aibling (Germany) 423 Bad Godesberg (Germany) 215 Badger’s Lair (SAS training exercise) 249–50 BAE Systems Ltd 544 Baghdad (Iraq) 161, 468, 469, 471, 524 Baghdad Pact 161 Bahia Paraios (Argentinean ship) 394, 395 Bahrain 347 Baillie, George 122 Bain, Helen 445 Baldwin, Stanley 18, 72, 400 Balgat (Turkey) 303 Bali bombing (2002) 511, 513 Balkanabteilung (German code-breaking HQ) 50 Balkans 51 Baltic 112, 114, 116, 125, 285 Baltic Sea 273 Bamford, James 263, 361–2, 521 Bandaranaike, Solomon 160 Bank of England 241, 487 Banner, Gordon 312–16, 318 Barbieri, Major 52, 53–4 Barents Sea 114, 115 Barker, Nick 391–2 Barkley, Howard 77 Barsby, Mrs 374–5 Basra (Iraq) 466, 525 Battle of Britain 29 Bay of Pigs crisis (1961) 8, 226 BBC 330, 348, 429, 501, 517, 523–4; BBC Scotland 459 Beach, Sir Hugh 380, 381 Bearman, Sid 595 Beasley, Tony 133–9 Beaumanor Hall (Leicestershire) 63 Beijing 476 Belbasi (Turkey) 302 Belfast 261, 500, 501 Belgium 442, 492 Bell, Rod 409, 410 Benitez, Rafael 114, 115 Benjamin, Ralph 216 Benn, Tony 227 Bennett, Ralph 59 Bentinck, Victor Cavendish 67 Bergen (Norway) 450 Bergold, Harry 295 Beria, Lavrentii 107 Berlin 112, 127–8, 130, 196–7, 227, 228, 253, 270, 369, 370, 372, 478; Berlin Blockade (1948) 71, 113; Berlin Cryptographic Centre 50; Berlin tunnel 169, 170, 172–6, 373, 477 Berlusconi, Silvio 532 Berry, John 358–9, 360 Betts, Richard 600 Beulmann, Major 50 Bevin, Ernest 70 BfV (German domestic security service) 452–3 bin Laden, Osama 511, 513, 514, 549 Binalshibh, Ramzi 514 Bingham, Lord 481 Birch, Frank 43 Bitburg (Germany) 131 Black, Jeremy 407, 408 black chambers 4, 14 ‘Black Friday’ (29 October 1948) 81, 108, 119, 169, 280 Black Sea 112, 131–2, 301, 302, 311, 313, 317, 319 Blair, Tony 4, 436, 497–8, 500, 504, 506, 509, 515, 517, 519, 530, 532 Blake, George 173, 174–5, 176, 178, 179, 238, 385 Bleckede 127 Bletchley Park 103, 186, 188, 221, 354, 549; Americans at 39, 43; and breaking ‘Red’ 25–6; closure of 67–71; and cypher security 54–9; expansion and reorganisation 27–8, 62–3; GCHQ as successor to 1, 5; Huts Three and Six 23, 25, 36, 48, 64–5, 119, 121, 356, 362–3, 364, 387; and machine-based espionage 5; military emphasis at 22–3; post-war role 60–1, 63–7; release of records on 355; SIS GC&CS moves to 22–8; takeover of Axis sigint effort 47–54; tight security measures at 69; unmasking of 362–3; wartime value and achievements 59–60, 61–3; wartime work of 1–2, 5, 25–9, 30, 31–46, 109–10 Blix, Hans 520 Block, Lieutenant 145 Bloomer-Reeve, Carlos 394, 412 ‘Blue Book’ 2–3, 395–6 Bluff Cove (Falklands) 400 Blum, Eberhard 450, 452 Blunkett, David 510, 522 Blunt, Anthony 36, 37, 83, 188, 224, 225, 364, 367 Board of Trade 241 Bodsworth, William 69, 80 Boizenburg (Germany) 127 Bolivia 52, 300 Bonnet, Georges 52 Bonsall, Arthur ‘Bill’ 110 Bontoft, Gerry 352 Borehamwood (aka ‘Department B’, London) 181, 182, 190 Borisenko, Alexsandr Ivanovich 236, 237 Borneo 6, 148, 164–8, 250 Bosnia 8, 9, 471–4, 512 BOSS (South African secret service) 357 Bourbon (Soviet radio intercepts) 75 Boyce, Ed 377 Bracknell (Berkshire) 113 Bradley, General Omar 116 Bradshaw, Mike 423 Brauntoltz, George 432 Bremerhaven (Germany) 116 Brezhnev, Leonid 245, 247 Brezhnev Doctrine 244 Bride (UK code-name for Venona) 80 Bridges, Sir Edward 42, 141–2, 181 Brinks Mat bullion robbery (1983) 505 British Army 14–15, 63, 451, 469, 472; 9 Signals Regiment 162, 327, 383–4; 13 Signals Regiment 166, 228, 493; 14 Signals Regiment 524, 525, 534; 40 Commando 525; British Army on the Rhine 248, 412; First Armoured Division 467, 525; Queen’s Dragoon Guards 525; ‘Rhino Force’ 467; Royal Armoured Corps 248; satellites 198, 208–9, 223, 243, 258, 262–3, 340, 342, 344, 345–8, 347–8, 376, 377, 401, 415, 421, 437–8, 441, 442–3, 445–7, 460–1, 478; sigint units 162, 166–7, 218–19, 228, 327, 383–4, 493, 524, 525, 534; Special Air Service (SAS) 164, 165, 168, 248–51, 359, 409, 410, 411, 468, 472, 475, 536, 583; Special Boat Service (SBS) 333, 409, 410, 411, 443, 468; Special Operations Executive (SOE) 36, 51; Special Reconnaissance Squadron 248 British European Airways (BEA) 129 British Indian Ocean Territories (BIOT) 334 ‘British Intelligence and Weapons of Mass Destruction’ (Butler Report) 2 British Leyland 367–8 British Military Mission (Brixmis) 123, 245–6, 247, 252 British Nuclear Fuels Ltd 500 British Psychological Society 433–4 British Tabulating Machine Company 349 British Telecom 545; Medium Wave Tower (Holyhead) 500 Britten, Douglas 230–8, 369, 382 Broadcasting Standards Authority 481 Broadside operation (US Embassy intercepts in Moscow) 280–1 Brockway, Ernst 383 Brook, Sir Norman 142, 219–20 Brooks, Richard 399 Brooks Field (Michigan) 120 Brown, Gordon 498 Brundrett, Sir Frederick 177, 178 Brunei, Sultan of 164, 586 BRUSA agreement (1943) 43, 44, 121, 151 Brussels 179, 253 Brzezinski, Zbigniew 448 Buffham, Benson 381, 424 Bufton, Air Vice Marshal 207 bugging operations 176–82, 193, 196–7, 473–4, 479, 482, 499 Bulganin, Nikolai 140 Bulgaria 518, 519 Bundesnachrichtendienst (BND, German foreign intelligence service) 214–15, 422–3, 438, 447–51, 452–4, 455, 456, 471, 472, 524 Bundy, William ‘Bill’ 50, 356, 363, 364 Bunyan, Tony 361 Burgess, Guy 8, 37, 73, 82, 83, 84–5, 224, 225, 238, 367 Burma 65 Burrough, John 598 Burton, Sir Edmund 527 Burton-Miller, T.R.W. 57, 191, 577 Bush, George H.W. 356–7 Bush, George W. 511, 517, 532 Butler, Lord 2, 428, 482, 529–31, 610 Byers, Stephen 506–7 Byrnes, Jimmy 53 ‘C’ (head of SIS) 16, 24, 67 Cabell, Charles 96 Cabinet Office Intelligence Coordinator 241, 245, 264, 353, 354, 360, 387, 399, 504 Cable, Danielle 505, 506 Cable, James 278 cable vetting 238–41 Cable & Wireless 240, 312, 392 Caccia, Harold 171 Cadogan, Sir Alexander 23–4, 26, 39, 45, 70 Cairncross, John 36–7, 73, 82, 364 Cairo (Egypt) 58, 159, 179, 185 Callaghan, James 324, 325, 330, 333, 361, 391 Calvi, Roberto 407 Calvocoressi, Peter 59, 61 Cameron, Stephen 505 Campaign for Nuclear Disarmament (CND) 368 Campbell, Alastair 517, 523, 530 Campbell, Duncan 8, 358, 360, 361, 362, 423, 458–9 Canada 38, 89, 93, 95, 97–8, 178, 447, 533; Canadian CBNRC 348; Canadian Communications Security Establishment (CSE) 381, 447; Canadian Joint Intelligence Committee 92 Canine, Ralph 101, 174, 336 Canyon satellite 376 Cape Canaveral (Florida) 322, 437, 461 Cape Matapan, Battle of (1941) 60 Capenhurst Tower (Cheshire) 500–1 Caraman, Mihai 253–4 Carey-Foster, William 185, 188, 228 Carlile of Berriew, Lord 543–4 Carpenter, Harry 598 Carrington, Lord 282–3, 284, 294, 389, 394, 396, 423–4 Carsamba (Turkey) 311–12 Carter, Jimmy 390, 448 Carter, Marshall 264, 272, 273–4, 350 Carter, Pat 448 Cartwright, Ian 328 Carver, Michael 334 Casey, Bill 457 Caspian Sea 112, 132, 161, 301, 302 Castle, Barbara 240 Castro, Fidel 226, 341 Catroux, Georges 52 Caucasus 131, 157 Caviar (Soviet encyphered traffic) 49, 69 Cayan, Mahir 309, 310–15, 314 Caygill, David 445 Celebes 167 Cellnet 481 Cemgil, Sinan 306 Central Gunnery School (Leconfield) 126 Central Signals Establishment 113–14 Ceylon 58, 69, 160, 259, 352 Chagos Islands 278, 335 Chamberlain, Neville 3, 18 Charles, Prince 482 Cheadle (Cheshire) 34, 63, 231 Chechnya 494–5 chemical weapons 470–1, 516 Chevaline project (upgrading of Polaris) 438–40, 459 Cheyne, Bill 166 Chicago Islands 278 Chicksands (Bedfordshire) 63, 360, 535 Chifley, Ben 86, 88, 94 Chile 357, 394, 396, 517 China 98, 129, 150–5, 193–5, 256, 273, 277, 282, 285, 376, 475–8; Chinese Communist Party 150; People’s Liberation Army 151 Chinese language teaching 598 Ching Peng 149, 150 Chippewa Falls (Minnesota) 350 Chirac, Jacques 520–1 Chitty, Brigadier 55 Chiverton, Roy 476 Chum Hom Kok (Hong Kong) 475–6 Church, Frank 356, 357 Church Committee 356, 357 Churchill, Winston 84; addiction to ‘Ultra’ intelligence 1; and airborne incidents 128, 129; and bugging operations 177; comments on the Belgians and Dutch 52; and cypher security 56; and diplomatic intelligence 41; and elint airborne operations 124; and Kuznetsov-Marshall affair 187, 189; meets Stalin 47; as recipient of sigint 3, 5, 40; supports work at Bletchley 26–7, 59, 65, 362 CIA (Central Intelligence Agency) 85, 87, 91; and Bay of Pigs 8, 226; and Berlin-Vienna tunnel operations 169–76; and counter-espionage coup 253–4; and Cyprus 324, 326–7; Foreign Broadcast Information Service 155; and invasion of Czechoslovakia 246; and invasion of the Falklands 399; Jewish sympathisers in 97; and Korean War 100; and Libyan terrorism 457; and Middle East 157, 158; and Nixon administration 4, 278; Office of National Estimates 151; revelations concerning 356–8; Russian spies in 444; security measures 101, 381; and shooting of Che Guevara 300; and Soviet weapons 108, 439; Special Activities Division 514–15; and Turkey 472; and U-2 spy planes 142, 226, 292, 296; and use of communication satellites 348; and Venona Project 77, 82; and Zinnia 322 Cianchi, Commander 54 CIB3 (Metropolitan Police anti-corruption squad) 506 City of London 487, 510 Civil Service 381, 421, 427; Civil and Public Servants Association (CPSA) 419–20; Civil Service Medical Officers Group 382; Civil Service Order in Council (1982) 431; Civil Service Union (CSU) 418, 420, 422–3; First Division Association 435; Society of Civil and Public Servants 426 Claret operations (1964–66) 165–8 Clarke, Kenneth 481, 482 Clarke, Liam 608 Clarke, Peter 266–7 Clarke, William F. 66 Cleveland, Paul 444–5 Clifford, Clark 333 Clinton, Bill 492 Clipper Chip (encryption bypass system) 492 Cobra (Cabinet emergency planning committee) 509–10, 527, 532, 610 Cobra Mist (Over the Horizon Radar) 285–7 code-breakers 2, 6, 198; and Anglo-American relationship 38–46; Australian 92; and Berlin tunnel operation 174; collaboration with Baltic states 31; combined operations 15–16; and computers 340, 348–9, 350–1; and decypherment of ‘Fish’ messages 28; diplomatic 27–8, 37, 43–4, 52–3; during WWII 25–9, 30–46, 58–9; Egyptian 164; and ending of WWII 61; expansion of 63–4; Far Eastern 40; Finnish 32, 35, 91; and French intercepts 52, 53, 209; and global sigint 96; importance of Hong Kong to 151; international work 16–17; and internet 493; Italian 52–4; military operations 14–15, 19–20; move from Bletchley to Cheltenham 5; naval 15; and Pelton affair 444; and personal computing and the internet 488; as Post Office department 14; pre-WWII 22–5; and Prime case 380; quadripartite meeting on computer hackers and encryption 489; revival on eve of WWI 14–15; Russian focus 17–19, 33–8, 169; and sale of cypher machines 209–15; and supply of intelligence reports 2–4, 6–7; and telephone tapping 170–1; and use of cypher machines 21–2; value of 60; and Venona Project 72, 74–88; wartime secrecy 354 COINS (Community On-line Intelligence System) 353 Colby, William ‘Bill’ 293, 327, 329, 330 Cold War 47, 377, 420; airborne incidents 125–33; benefits of intelligence in 175–6; Berlin blockade 71; calming of nerves in 257–8; computing in 349; end of 461, 465, 477, 478–9, 493; flashpoints 203; high profile espionage activity in 8; and planning for future war 247–53; seaborne incidents 133–47; secret service operations 484; sigint in 1, 2, 5, 107–24, 125–47, 402; Soviet nuclear weaponry 108; telephone and bugging operations 169–82; thawing of 195–7; and Venona Project 72–88 Cole, David 379, 380 Coleman, Don 131 Coleridge (Soviet teletype system) 78 Colombia 486, 538 Colonial Office 150 Colorob (computer) 349 Colossus (computer) 28, 48, 68, 70, 349 Combined Cypher Machine 98 Comet aircraft 121–2, 268, 273, 295 Cominstum (digest of hot material) 96 comint (communications intelligence) 96, 101, 110, 111, 122, 123, 228, 252–3, 266, 385, 402, 413 Comintern (Communist International) 19, 30, 37–8, 79 Commonwealth 85–8, 89, 92, 95, 97, 148, 266, 352, 447 Communications Branch of the National Research Council (CBNRC) 94 Communications Data Bill (2009) 543 Communications-Electronics Security Department 241–2 Communications Trials Ship (purpose-built sigint ship) 260–2 Communist Party of Great Britain 19, 188, 367–8, 417 Communist Party of India 87 Communist Party of USA 87 Comprehensive Comparative Radar Library 266 computers 198, 219, 220, 222, 340–1, 342–3, 348–54, 458, 486–93, 507, 507–8, 513, 527–8, 546–8 comsec (communications security) 90, 191–3, 195–7, 211, 218, 241–2, 377 Confederation of British Industry (CBI) 241 Conflict (Vienna tunnel code) 171 Congo crisis (1960) 336 Control Orders 543 Cook, Robin 498 Cooney Hill psychiatric hospital (Gloucester) 382 Cooper, Arthur 598 Cooper, Frank 423 Cooper, Josh 20, 34, 213 Coote, John 136, 137, 138, 143 Copenhagen (Denmark) 179 Corona (satellite) 208 Corporal (battlefield missile system) 249 Cosby, Bill 327 Costello, John 364 Cot, Pierre 53 counter-terrorism 456, 516, 528 Cox, Arthur 301 Crabb, Lionel ‘Buster’ 140–3, 207 Cradock, Percy 193–4, 294 Crankshaw, Edward 34, 35–6, 64 Cray super computers 350–1 Crete 157, 265, 292 Croatia 471–4 Croft, John 37–8 Cromer, Rowley 287–8, 294, 297, 337 Croslieve Mountain (Northern Ireland) 501 Crossman, Richard 227 Cruise missiles 511, 513 Crypto AG 212–15, 457 CSE (Communications Security Establishment) Watton 131 Cuba 226, 341–2; Cuban Missile Crisis (1962) 203, 253, 260 Cukr, Baclav 112 Cummings, Mansfield (aka ‘C’) 16 Cunningham, Andrew 91 Current Intelligence Groups 291, 396 Currie, Laughlin 87 Currier, Prescott 39 Curry, John 79 Curzon, George 15, 16, 18 cyber attacks 487–93 cypher machines; capture of 48–9, 264; Chinese capture of 194, 195; commercial origins of 20; development and use of 20–2, 57, 192; military 54; online 28, 210; and proposed Anglo-US collaboration 98–9; radiation or emanation from 215–18; supplying to NATO countries 209–15; see also Enigma Cypher Policy Board 42, 56–7 cypher security 54–9, 98–9, 191 Cyprus 7–8, 154, 155–6, 159, 161, 162–3, 219, 229, 234, 235, 259, 265, 277, 285, 292, 294, 295, 302, 319, 320–34, 337, 338, 345, 348, 356, 359, 369, 372, 382, 383–4, 419, 423, 466, 471, 504; Cyprus Eight 385 Czechoslovakia 203, 244–7, 253, 387; Czech Air Force Association 112 D-Notice affair (1967) 226, 238–41, 242, 362–3 Damascus (Syria) 159, 291, 300 Darwin (Falklands) 410 Data Encryption Standard (DES) 489 data-mining 486, 546–8 Daubney, Claude 103 Davies, Philip 27 de Gaulle, Charles 52, 195 de Grey, Nigel 35, 43 Deaf Aid (elint reception and analysis kit) 123 Dean, Patrick 142 Decabral, Alan 506 Defcon 3 (US nuclear alert) 293–4 Defence Intelligence and Security Centre (Chicksands) 63, 360, 535 Defence Intelligence Staff (DIS) 245, 246, 353, 369, 397, 402, 414, 527–8 Defence Reviews 329 Defence Signals Branch 93, 151, 153, 213, 578 Demos-1 (Hong Kong sigint station) 475–6 Demos-4 (Hong Kong sigint station) 476 Denham, Flight Sergeant 126 Denmark 442, 533 Denmark Hill (Metropolitan Police intercept station, London) 37 Denniston, Alastair 15, 18, 21, 24–5, 27, 28, 31, 34, 38, 40, 43–4, 70, 79 Denton Green, Robert 400, 402, 403, 407 Department of Economic Affairs 241 Department of Trade and Industry (DTI) 487 détente 247 Detica (security company) 544 Dexter, Harry 87 Dhekelia 163, 233, 327 Diana, Princess of Wales 479–83 Dictionary (keywords/predesignated phrases system) 343 Diego Garcia 321, 332, 335–9, 597 Dieppe raid (1942) 55 Diffie, Whitfield 490 Dimbleby, Jonathan 524 Dingli (Malta) 32, 156, 162 diplomatic intelligence 27–8, 37, 43–6, 52–3, 62, 69, 148, 159, 164, 176–82, 349, 355, 377 Diplomatic Protection Squad 506 Diplomatic Wireless Service (DWS) 58–9, 123, 181, 185–90, 192, 262, 417, 418, 585 Director General of Intelligence (DGI) 246 Directorate of Scientific Intelligence 123 Discovery (space shuttle) 437 Diyarbakir (Turkey) 300, 301–2, 306 Dobrynin, Anatoly 454 Domazet, Davor 472 domestic surveillance and intercepts 540–50 Donoughue, Bernard 3, 325 Doran, Frank 126 Doublecross system 229, 255 Douglas-Home, Sir Alec 163, 282, 312–13 Dozier, James 407, 452 Drake, Edward 94 Drew, John 229 drugs 486, 503, 514, 538 Drumheller, Tyler 529–30 Drummond (Argentine frigate) 395 DS.19 (MoD unit) 368 Dubček, Alexander 244 Dublin 501 Dudley-Smith, Russell 49, 55, 261 Duff, Antony 360, 399 Duffton, Nancy 435 Dulles, Allen 157, 158, 174, 176, 203 Dulles, John Foster 157 Dunderdale, ‘Biffy’ 21 Dunlap, Jack 355 Dunnell, Peter 127 Dwyer, Peter 82 East Africa 335–6 East Asia 120 East Germany 123, 131, 195–7, 345, 370, 379, 385, 453, 605 East-West summit (Paris, 1960) 204 Eastcote (London Signals Intelligence Centre) 62, 68–9, 79, 80, 103, 191, 349 Eastern Bloc 123, 175, 244, 245, 247, 256, 267, 282, 447, 465 Eastern Europe 33, 53, 78, 99, 119, 256, 284 Easton, James 82 Eavesdropper revelations (1976) 358, 359 Echelon (Anglo-US communications network) 7 economic intelligence 240–1 Ecuador 52 Eden, Anthony 46, 85, 129, 140, 141, 142–3, 155–6, 160, 178, 189 Eemnes (Netherlands) 415 Eger (Norwegian ship) 117 Egypt 58, 109, 155–9, 259, 263–4, 271, 277, 290–2, 295, 320, 467 Eichmann, Adolf 307–8 Eisenhower, Dwight D. 140, 157, 158, 202, 205, 219 Electrical Trades Union 368 Electronic Warfare Conferences 122, 123, 307–10 elint (electronic intelligence); air-based 111–14, 118–19, 122, 124, 250–2, 267–73; and Anglo-American relations 111–12; and European cooperation 591; in Germany 247, 250–1; and invasion of the Falklands 401, 413; and jamming of Whetstone monitoring station 190; land-based 117–20, 123; naval 114–17; near the Soviet Union 169; postwar expansion 110; rejection of 247; in Turkey 306; wartime use of 110 Elizabeth II, Queen 480, 482 Elkins, Robert 143 Elliott Brothers Ltd 598 Ellis, James 490, 492 Elmers School (GC&CS Diplomatic Sections) 23 email 488, 507, 513–14, 521–3, 541 embassies 151; Anglo-US-Canadian intercepts in Moscow 280–1; attacks and raids on 193–5; as forward listening stations 31; KGB in 82–3, 283–4; and MI5 watcher operations 183–90; security headache 195–7; sifting of waste-baskets in 56; spies in 84; tapping and bugging operations 171, 176–82, 193, 197–8, 281, 477; ultra-secret short-range sigint stations in 244–5; worldwide collection of intercepts from 45, 53, 79, 112, 159, 242–3, 385 Employment Select Committee of the House of Commons 424 Engulf (Egyptian Embassy cypher machine operation) 216 Enigma (German cypher machine) 1, 20–2, 23, 25–6, 27, 35, 38, 39, 42–3, 43, 51, 68, 78, 80, 354, 387 EOKA (Cypriot guerrilla force) 163 Episkopi (Cyprus) 234 Erim, Nihat 306, 312, 315 Eritrean Liberation Front 336 Escobar, Pablo 538, 549 Ethiopia 299, 334, 335–6 European Convention on Human Rights 433, 483 European Economic Community (EEC) 284 European Principals Meeting 450–1 European Union (EU) 540 Evatt, Dr H.V. 85, 86 Evere (NATO-GCHQ cell) 254, 255–7 Exocets (sea-skimming missiles) 390, 406–7, 414, 415 Faisal, King 160–1 Falkland Islands 6, 424, 429, 441, 442, 452, 467; Argentinean ambitions towards 389–92; Argentinean invasion of 392–3, 394–401; British troops on 408–14; comint and elint on 401, 413; diplomatic exchanges with 403–4; effect of war on British sigint 415; French help on 415; improvised communications with GCHQ 402–3; inadequate intelligence on 392–401; leaseback idea 392; naval action 404–8; near-miss air disaster 408; Norwegian help on 401, 442; scrap-metal incident 393–4; surprise attack on 388–9; Task Force sent to 398, 401, 403, 404–8; US denies pre-knowledge 601 Famagusta (Cyprus) 163, 235, 327, 328 Far East 39, 69, 78, 93, 129, 148–51, 164–8 Farrell, Terry 496 Faslane naval base (Scotland) 145, 146 Fatah (Palestinian organisation) 304, 308 FBI (Federal Bureau of Investigation) 76–7, 81 Federation of Malaysia 164–8 Ferranti 349, 598 ‘ferrets’ (flying intelligence stations) 111–14, 203–7 Fetherstone-Haugh, Timothy 383 Fetterlein, Ernst 17–18 Fieldhouse, Admiral 392 Fiji 446 Finland 31, 32, 76, 83, 91, 371, 489 First World War 14, 15, 16–17 Firyubin, Nikolay 279–80 Fischer, David 407 Fish (encyphered teleprinter) 28, 48, 49, 51 Fitz, Harold 127 Fleet Headquarters (Northwood) 400, 401, 402 Fletcher, WPC Yvonne 455–6 Florida 341 Flowers, Tommy 28, 349 Foden, Arthur 242 Foot, Michael 433 Ford, Gerald 297 Foreign and Commonwealth Office 16, 22, 45, 46, 56, 58, 66, 70, 83, 103, 128, 171, 172, 190–1, 192, 220–1, 239, 245, 273, 281, 287, 333, 335, 339, 353, 355, 360, 392–3, 398, 417, 420, 428; South-East European Department 317; Technical Maintenance Service 182 Forest Moor (wireless station near Harrogate) 96 Fort Bridgelands (Kent) 63 Fort Knox (Kentucky) 101 Fort Meade (NSA HQ, Washington) 102, 157, 174, 223, 271, 513, 528 Förvarets Radionstalt (FRA) 91 Foss, Hugh 21, 64 Fox, Katherine 598 FRA (Swedish sigint service) 421, 438, 456, 483–4 France 21, 32, 44, 52, 52–3, 109, 130, 268, 442, 445, 450, 467, 492 Franks, George 382 Free French 28, 52 Freedom of Information Act 482–3 Freeman, John 279 Freeman, Peter 531 French Guyana 415 Friedman, William 39, 44, 95, 213, 214 Friedrich, Lt Colonel 50 Fuchs, Klaus 72, 82, 83, 87, 104, 238 Fyjis-Walker, Richard 316 Fylingdales (Yorkshire) 287 Gaddafi, Muammar 455, 457, 531 Gagarin, Yuri 301 Gaitskell, Hugh 141 Galvin, John 474 Gambier-Parry, Richard 57, 181, 186, 188 Gamma-Guppy (Soviet intelligence intercepts) 244–5 Gardner, Meredith 75, 79, 80 Garner, Joe 249 Gates, Robert 457 GC&CS (Government Code and Cypher School) 361; and Anglo-American collaboration 40–1; civil achievements 28–9; and cypher security 56–7; diplomatic centre at Berkeley Street 27–8, 37, 43–4, 52–3; divided into civil and military sections 27–8; and European collaboration 20–2; military interests 19–20; post-war role 61, 63–7; relocation to Bletchley 22–7; Russian interests 17–19, 30–2; setting up of 16 GCHQ (Government Communications Headquarters) 1, 31, 104; and al Qaeda 511–12; Benhall 350, 360, 497, 526; and Bosnian-Croatian conflict 472–5; Bude (formerly CSO Morwenstow) 342, 343–4; budget figures 587; building of ‘Doughnut’ 9, 497, 526, 527–8; ceases exchanging intelligence with NSA 289–90; and changing nature of global threats 504–5; and closer relationship with MI5 and SIS 503–4; and Cobra Mist/Orford Ness problems 285–7; code-breaking and intelligence-gathering 6–8; combined NATO-GCHQ cell at Evere 254, 255–7; computers in 527–8; cooperation with NSA 222–3, 278, 282–3, 346, 347–54, 438, 448–58, 461; declining position of 422–3, 438, 441; development of new systems 342–54; diplomatic initiatives 108–9; Directors of 551–2; domestic surveillance and interceptions 540–50; DWS operations 186; E Division (Personnel) 425, 427; and economic intelligence 493; Empress Building (Earl’s Court) 382; encryption problems in banking and commerce 487–93; and end of Empire 148–55; expansion of 79, 169; F Division 476; file storage 598; funding of 219–23, 334, 458, 493, 494, 495; Free Trade Union 430; future purpose of 485; and global sigint 92, 94, 95–100; and Gulf War (1991) 466, 469; H Division (mathematicians and cryptographers) 432; and hoax letters prank 469–70; increased intelligence operations 120, 121–4; influence on foreign policy 321; installation of dedicated computer unit 507–8; and internal surveillance 9–10; and internet 100; and invasion of the Falklands 392–403, 405–6, 411; and IRA 498–503; and Iraqi dossiers 516–17, 530; J Division (Special sigint—Russian) 346, 374, 376, 419, 429, 434–5, 438, 495; J-Ops 429, 434–5, 438; K Division (non-Russian sigint) 218, 222, 402, 420–1, 478, 495; and KGB espionage 108, 189, 424–5; and Korean War 101; language problem 512–13, 516; legal identity of 484–5; and Libyan Embassy affair 455, 456; London office (Palmer Street) 192, 497; loss of Hong Kong listening station 475–8; merger with com sec 241–2; and Middle East 155–64; moles in 368–85; move to Cheltenham (1952) 102–3, 120–1, 122, 191; need for 8–9; ‘need to share’ problem 503–4; Nimrod programme 267–70, 271–4; and Noye affair 505–7; Oakley 360, 380, 427, 496, 497, 526; ocean-going activities 6, 136; organisation overviews 563–5; overhaul of operations 493–7; positive vetting at 227–8; post-war organisation and location 67–71; and Princess Diana 482, 483; and problems with ‘special relationship’ 441–3; promotion and career structures 576; purpose-built sigint ship 260–4; R Division (security) 425; reads HVA traffic 605; reinstatement of unions at 497–8; relationship with private companies 240; removal of trade unions at 416–36; and Russian problem 46, 71, 75, 78, 169, 299; S Division 261; secret pact with armed services 5–6, 117–18, 132–3; size of 227; ‘Station X’ 69; as successor to Bletchley Park 1, 5; suicides connected to 382–3; and supply of cypher machines to NATO 209–15; T Division 123; Tempest 216–18; Trade Union Campaign 498; Turkish operations 300–1, 311–19; unmasking of 355–64; use of deaf and dumb civilian personnel 153; use of name ‘GCHQ’ 61, 67; and use of polygraph at 425–6; and Venona Project 77–81; visibility of 1, 2, 8, 341, 436, 484; W Division 261; and War on Terror 533, 539–40; and West German defections 455–6; whistleblower in 521–3; X Division 6, 350, 353; Z Division (use of sigint) 388, 503; Zionist interests 109 GEC-Marconi 433 General Belgrano (Argentine cruiser) 404–6 General Strike (1926) 18 Geneva (Switzerland) 178; Peace Conference (1954) 178 George VI, King 59, 191 Georkadijis, Polycarpos 323–4 Geraldton (DSD site, Australia) 477 Germany 15, 29, 30, 31, 32–3, 44, 47–50, 55, 62, 78, 96, 127–9, 130, 142, 170–1, 219, 229, 247, 256, 270, 492; Army 26, 29, 43, 47; High Command (OKW) 49, 349; Navy 42; see also East Germany; West Germany Gezmiş, Deniz 310, 311 Giant Reach (SR-71 flights from US to Middle East) 292–3 Gibraltar 162, 398, 415 Gibson, Sir Peter 502 Gilbey, James 479–81 Glazebrook, George 92 Glidwell, Mr Justice 430–1 Glover, Sir James 413 Godfrey, Admiral 32 Golan Heights 297 Golden Valley Hotel (Cheltenham) 432–3 Goldsmith, Lord 522–3 Golombek, Harry 25 Goodpaster, Andrew 256 Goonhilly Downs (Cornwall) 342–3, 597 Goose Green (Falklands) 404, 410, 411 Gorbachev, Mikhail 456 Gordievsky, Oleg 385, 478 Gore Booth, Sir Paul 339 Gosport (Hampshire) 134 Gouzenko, Igor 85 Government Communications Staff Federation 428, 429 Government and Overseas Cable and Wireless Operators Association 418 Government Technical Assistance Centre (GTAC) 507, 547 Government Telecommunications Advisory Centre 504 Gow, Ian 482 Gowrie, Lord 428 Grab (Galactic Radiation and Background) satellite 208 Grant (MI5 computer scheme) 528 Grantham, Sir Alexander 152 Granville (Argentine frigate) 395 Grechko, Andrei 245 Gredjeva, Nina Michailovna 189 Greece 163, 259, 319, 320, 324, 334, 450, 472 Green Light (US special atomic demolition munitions programme) 249 Greenhill, Denis 239, 284 Greenock naval base (Scotland) 144 Greenpeace 446 Grey (US diplomatic code) 40 Grindley, Mike 430 Gromyko, Andrei 205, 206 Groupe de Synthèse et Prévision (France) 284 GRU (Soviet Military Intelligence) 88, 173, 230 Guardrail (US airborne tactical sigint systems) 251–2, 272 Guernica bombing (1937) 22 Guevara, (Ernesto) Che 300 Gulf War (1990–91) 452, 465–71, 529 Gulf War (2003) 479, 516–26 Gun, Katharine 521–3 Gurdon, Adam 396 Gurkhas 164–5 Gurney, Sir Henry 149 Habbaniya (Iraq) 20 Hagelin, Boris Jnr 212–13 Hagelin, Boris Snr 212 Hagelin (cypher machine) 56, 78 Haig, Alexander 403–4 Halifax, Lord 24 Hall, Theodore 73 Hallock, Richard 74–5 Hamilton, Alexander 432 Hampshire, Sir Stuart 221–5, 260 Handel, Michael 362 Hankey, Lord 36 Hanley, Michael 361, 587 Hanley, William J. 305, 306 Hanslope Park (Buckinghamshire) 57, 58, 182, 185, 186, 187, 192, 196, 418 Hanssen, Robert 444 Hardy, Tim 166 Harland & Wolff 261 Harman, Harriet 368 Harrier jets 403, 404, 407, 408, 441 Harris, Robert 59 Hart, Herbert 225 Harty, Russell 359 Harvest (computer) 350 Hashmi, Jabron 535 Hastings, Edward 95 Hawaii 92 Hawkes, John 216 Hay, Malcolm 15 Hayden, Michael 508 Hayter, William 97 Healey, Denis 168, 245, 256, 399, 426, 429, 433 Heath, Edward 4, 239, 279, 315, 337, 338, 439 Heliopolis (Egypt) 92, 155, 162 Hellenbroich, Heribert 452–3 Hellman, Martin 490 Helmand province (Afghanistan) 534, 535 Helms, Richard 292, 356 Hemblys-Scales, Roger 86 Henderson, Nicholas 2–3 Hennessy, Peter 90, 577 Herman, Michael 261, 287, 419, 421, 435 Heseltine, Michael 426 Hibberson, Anthony 189 Hill, Jim 86–7 Hill, Major 120 Hillenkoeter, Roscoe 85 Hillgruber, Andreas 59 Hinsley, Harry 59, 64 Hiroshima 2 Hiss, Alger 88 Hitler, Adolf 3, 5, 29, 30, 31, 32–3, 48, 221, 290 Hoad, Norman 132 Hokkaido (Japan) 112 Holden Agreement (1942 & 1944) 43 Hollis, Sir Roger 79, 86, 182, 367 Holmberg, Elena 389–90 Home Office 507, 538, 544 Honest John (battlefield missile system) 249 Honeywell (computers) 458 Hong Kong 19, 30, 32, 96, 100, 151–5, 219, 256, 277, 419, 475–8 Hooper, Joe 191, 222, 223, 228, 273–4, 285–6, 343, 353, 419, 448, 466, 585 Hoover Commission 219 Hosenball, Mark 358 House of Commons Public Accounts Committee 440, 459 House of Commons Select Committee on Employment (1985) 433–4 Howard, Edward Lee 384, 444, 447 Howard, Michael 364 Howarth, Jack 189 Howe, Geoffrey 426, 427–8, 429, 431–2, 436, 460 Howse, Philip 79, 84 Hughes, Chief Inspector 187 Hughes, Robert D. 111 Hungary 46, 158 Hunt, Sir John 329–30, 337–8, 356–7, 361 Hunters Stones Post Office Tower 346 Hurd, Douglas 495 Hurley, Michael 144–6 Hurn, Roger 495, 526 Hussein, King 161, 164, 290 Hussein, Saddam 467, 516, 524, 525 Husum-Milstedt (intercept station, Germany) 50 Hutton, Lord 529 Huxley, Aldous 549 HVA 605 Iacobescu, Ion 253 IBM 350, 352, 489 Igloo White (ground sensors) 252 Imre, Nahit 254 Incirlik (Turkey) 326 India 4, 18, 19, 30, 32, 85, 95, 178, 334 Indonesia 153, 164–8 Information Research Department 156 Ingebrigsten, Jan 450 Ingham, Bernard 428 Inman, Bobby Ray 399, 422, 601 Intelligence Assault Units 47–8 Intelligence and Security Committee 484–5, 529, 539 Intelligence Services Act (1994) 484–5, 488 Intelligence Support Activity 168 Intelsat 342 Intercept Control Centre 250 Intercept Modernisation Programme (IMP) 543–5, 547–8 International Control Commission on Vietnam 178 International Regulations on Sigint (IRSIG) 90 International Security Assistance Force (ISAF) 533 internet 8, 488, 507–8, 541, 544–5 IRA 345, 455, 479, 481–2, 494, 498–503, 593 Iran 109, 112, 155, 268, 295, 299, 302, 421, 467, 472, 605 Iran, Shah of 299 Iraq 155, 156, 160–2, 259, 295, 320, 465–71, 479, 516–17, 528–31; 124 Electronic Warfare Regiment 525 IRSIG (Instructions and Regulations concerning the Security of Signals Intelligence) 503–4 Iscot (wartime Comintern traffic) 37–8 ISI (Pakistani intelligence service) 514 Ismailia (Egypt) 32, 185 Ismay, General Hastings ‘Pug’ 27 Israel 97, 164, 263–4, 277, 290–1, 293, 307–8, 415, 471; Israeli Sea Corps 264–5 Istanbul 307, 309, 310, 316, 318 Italy 19, 44, 52, 96, 345, 452; Italian Cryptographic Bureau 54–5 ITT (telecoms company) 341, 342 Ivy Bells (undersea cable-tapping) 384 Jakarta (Indonesia) 167, 168 Japan 17, 29, 39, 40, 44, 65, 100, 110, 152, 445, 446 Jebb, Gladwyn 64 Jenkins, Roy 51 Johnson, Lyndon B. 50, 238, 353 Johnson, Robert 346 Johnstone, Sir Charles 164 Johnstone, Colonel Hugh 327–8, 359, 360 Joint Intelligence Committee (JIC) 67; and Arab states 109; Chevaline project 440; circulation of BJs 70; collection of elint on Soviet air-defence capabilities 132; considers Soviet threats a bluff 204; Crabb incident 141–2; D-Notice affair 239; deployment of equipment in Eastern Bloc 123; failures and inaccuracies of 108, 245–6, 387–8; focus on economic, technological and scientific subjects 241; French cooperation 284–5; intelligence failures 387–8; and invasion of the Falklands 391, 395–7; and Iraq 466; Joint Intelligence Committee Far East 167; and new technology 353; and Palestine 97; and possible Soviet move inside Eastern Europe 256; rethinking of European targets 345; review of aerial and submarine surveillance 207; review of GCHQ spending 221; and Soviet invasion of Czechoslovakia 245–6; and Soviet Union 46; and surprise Soviet nuclear attacks 321; and Turkish invasion of Cyprus 319, 325; and Vienna tunnel 171; and Yemen Civil War 164 Joint Technical Services Language School (Tangmere, Surrey) 370 Jones, Eric 46, 121, 132, 142, 159, 188, 191, 197, 418, 585 Jones, Colonel H. 411 Jones, R.V. 102–3, 110, 111, 579 Jordan 157, 164, 308 Joseph, Keith 286 Jowell, Tessa 498 July Bomb Plot (1944) 221 Kabul (Afghanistan) 533 Kagnew (Ethiopia) 335–6 Kaiser, Michael 240–1 Kapustin Yar (Soviet Union) 112, 131, 301 Karadſić, Radovan 473 Karamursel (Turkey) 301 Karlshorst (Germany) 371 Katanga (Congo) 336 Kazakhstan 107 Keepnet (recording equipment) 458 Keith, Bruce 69, 93, 94 Kelly, Gerry 500 Kennan, George 177 Kennedy, Jacqueline 180 Kennedy, Paul 59 Kenya 125, 334, 370, 511 Ker, Leonard Douglas 189 Kern, Dick 449–50 Kerr, Sir Archibald Clark 84 Key Recovery (or Key Escrow) 492, 506–7 KGB (Russian secret service) 137, 230, 285, 419, 538; agents working for 36–7, 185–90, 224–5, 231–7, 354–5, 369–85; and Airborne Rafter programme 267; defections from 478–9; Eight Directorate 377; expulsion from London embassy 283–4; intercepts on 53, 96; microwave intercepts 281; and miners’ strike (1982) 368; and release of material on GCHQ 355; Sixteenth Directorate 377; surveillance operations 4, 183–5, 190–1; tapping and bugging operations 170, 173, 175–82, 193; and Tempest 216–17; and Venona Project 72–88, 98, 104 Khalid Sheikh Mohammed 514, 515 Khrushchev, Nikita 4, 140, 142, 173, 179, 180, 202, 204 kidnapping and hostage-taking 452, 513–14; in Turkey 302–19; see also terrorists, terrorism Kiev (Soviet Union) 126 Killian, James R. 219 King, Tom 426, 427 Kingsdown (Kent) 34 Kinnock, Neil 433 Kipling, Rudyard 13–14 Kirknewton airbase (Scotland) 118 Kirkpatrick, Sir Ivone 130 Kissinger, Henry 277–81, 283–4, 287–90, 292, 293, 294–7, 319, 324, 326, 329, 330, 331, 337–8, 403, 441, 444 Kizildere (Turkey) 312–19 Klemme, General 50 Klugman, James 36, 188 Knockholt (Kent) 120 Knox, Dilly 21 Kohl, Helmut 453 Kola Peninsula (Soviet Union) 118, 136 Komer, Robert 303 Korea 6 Korean War (1950–53) 99–101, 116, 118–19, 120, 129, 152, 178 Kosovo 8, 512 Kosygin, Alexei 280 Koza, Frank 517–18, 521 Kuala Lumpur (Malaysia) 150–1 Kubat, Ferit 313, 314 Kuching (Malaysia) 166 Kurchatov, Igor 107 Kurku, Ertugrul 313–14, 315 Kursk (Soviet Union) 36–7 Kuwait 465–70, 524 Kuznetsov, Pavel 183–8 La Belle discothèque (West Berlin) 457 Labuan (Malaysia) 166 Lagos, Ricardo 519 Lamphere, Robert 76–7 Lander, Stephen 494 Lange, David 444–5 Langley (Virginia) 292, 441 Laos 446 Larnaca (Cyprus) 384 Latakia (Syria) 331 Law, John 312–16, 318 Lawson, Nigel 460 Le Bailly, Louis 246, 286–7, 291, 439 Leach, Henry 395 Lebanon 161, 308 Lee, Raymond 39, 40–1 Libya 109, 295, 334, 455–8, 531; Libyan People’s Bureau (London) 455–6 Liddell, Guy 367 Light Electronic Warfare Teams (LEWTs) 534–5 Limassol (Cyprus) 325 Lindsay, Michael 151 Little, Peter 420–1 Little, Rod 402 Little Sai Wan (Hong Kong) 153, 475 Livebait (comparison of different signals) 458 Llanos, Gonzales 408 Lobban, Ian 542 Lockerbie incident (1988) 457–8, 605 Lockhart, John Bruce 171 Lockheed 476 Loehnis, Clive 31, 197, 210–11, 223, 466 Lohan, Sammy 239 Lombardo, Juan 393, 395 London bombings (2005) 532–6 London Communications Security Agency (LCSA) 103, 191–3, 210, 211, 213, 217, 585 London Communications Security Board 585 London Processing Group (LPG) 372–3 London Signals Intelligence Board 51–2, 109, 142; London Signals Intelligence Centre 69; London Signals Intelligence Committee 267, 268 Longfellow (Soviet cypher system) 78 Lonsdale, Gordon 238 Lord (Vienna tunnel code) 171 Luanda (Angola) 59, 455 Lucas, George 144, 145 Luftwaffe 26, 33–4, 35, 36, 43, 50 Luga airport (Malta) 295 Luneburg Heath (Germany) 127 Lunn, Peter 171, 172, 174 Lyalin, Oleg 283 Lyttelton, Oliver 149 MacArthur, General Douglas 45 McCormack, Alfred 45 Macdonald, Ken 543, 548 McGuinness, Martin 500 Machon, Annie 456 Mackay of Clashfern, Lord 484–5 Mackenzie King, William Lyon 94 Maclean, Donald 8, 37, 72, 73, 76, 82, 84, 87, 104, 238, 367 McManners, Hugh 409, 410 Macmillan, Harold 3–4, 143, 179, 204–7, 226, 364 McNamara, Robert 168 McNeill, Hector 224 Magdeburg (Germany) 50 Magic (Japanese cypher) 29, 39, 41, 44, 69 Maguire, Harold 268, 269 Major, John 398, 484 Makarios, Archbishop Mihail 163, 296, 320, 323–5, 328, 330 Malatya (Turkey) 306 Malaya 6, 30, 38, 125; Malayan Communist Party (MCP) 149, 150; Malayan Emergency 149–51 Malaysia see Federation of Malaysia Malinovsky, Rodion 204 Malta 156, 162, 295 Maltby, Ted 58, 79, 187 Manchester University 349 Manchuria 19 Mandelson, Peter 498 Manhattan Project (Los Alamos atomic bomb project) 75, 76, 82, 219 Manningham-Buller, Dame Eliza 515 Mao Tse-tung 4, 151, 193, 195 Marchetti, Victor 356 Marconi (company) 311 Marconi, Guglielmo 13 Marenches, Alexandre de 442 Marr-Johnson, Patrick 77 Marshall, George 42 Marshall, William 184–90 Martin, William H. 176, 355, 423 Marychurch, Peter 428, 434, 445, 448, 449, 451, 458–60 Mask operation 19 Mason, Roy 332 Mathison, Alan 25 Mauborgne, Joseph 18 Mauritius 334, 335, 338 Mazzini, Giuseppe 14 Medical Research Council 434 Mediterranean 16, 44, 114, 273, 295, 319 Meir, Golda 290–1 Menendez, Mario 412–13 Mentyukov, Igor 201–2 Menwith Hill (Yorkshire) 345–6, 347, 421, 449 Menzies, Sir Stewart 23–4, 26, 27, 28, 38, 39, 42, 45, 51–2, 55, 67, 82, 142 Methods to Improve (MTI) 220 Mexico 15, 517, 519–20 Meyer, Cord 358, 595 Meyer, John C. 271, 272 MI5 see Security Service MI6 see Secret Intelligence Service Middle East 7, 15, 16, 19–20, 32, 34, 41, 51, 97, 148, 155–64, 181, 182, 271, 277, 282, 290–5, 299, 320, 333, 334, 336, 376, 385, 454, 467, 472, 479, 494; Middle East Technical University (Ankara) 303, 304; Middle East War (1973) see Yom Kippur War Middle Six countries 519 Millward, Bill 63, 121, 221–2, 253 Milne, Alasdair 459 Milner, Ian 86 Milner-Barry, Stuart 25, 27, 364, 387 Milošević, Slobodan 473 Ministry of Defence (MoD) 240, 272, 286–7, 291, 312, 368, 423, 456, 495, 510 missiles see rockets and missiles MIT (Turkish National Intelligence Agency) 304, 314 Mitchell, Bernon F. 176, 355, 423 Mitchell, Graham 367 Mitchell, J.R. 118 Mitterrand, François 414–15 Mladić, Ratko 473, 474 mobile phones 492–3, 505, 538, 541, 548 Modin, Yuri 82–3, 84 Moffit, Bill 296–7 Molotov, Vyacheslav 178–9 Monterey (California) 332 Montgomery, Field Marshal Bernard Law 58, 85 Moon-bounce project 262–3 Morgan, Gerry 78 Moriarty, D.M. 587 Morocco 334 Morris, Gareth 433 Morwenstow (Cornwall) 342 Moscow 4, 18, 19, 30, 33, 35, 36, 37, 45, 47, 53, 59, 74, 82, 84, 86, 104, 126, 148, 151, 175, 176–8, 179, 185, 186, 189, 201, 203, 205, 284, 465; Moscow Peace Treaty (1942) 32 Mossad (Israeli secret service) 291, 299–300, 307–8, 444, 472 Mottram, Richard 577 Mount Tumbledown (Falklands) 413 Mountbatten, Lord Louis 110, 143, 192 Mowlam, Mo 500 Mubarak, Hosni 467 Mullah Dadullah 535 Muller, Wilma 127 Mullet Creek (Falklands) 397 Munich Crisis (1938) 3, 22 Murmansk (Soviet Union) 133, 143 Murray, Len 416–17, 426, 427, 428 Muslims 472, 473, 474, 537, 538, 542 Mustard (Enigma key) 35 Mutual Balanced Force Reduction programme 287–9, 319 Mutual Weapons Development Programme 211 Nagasaki (Japan) 2 Nan-Szu-Pu (Taiwan) 152 Narvik raid (1940) 55 Narwal (Argentine trawler) 407–8 Nasser, Gamal Abdel 155–6, 161, 164 National Central Electronic Reconnaissance Agency (NSEI, Croatia) 473 National Council for Civil Liberties 361 National Criminal Intelligence Service 504 National Infrastructure Security Coordination Centre 609 National Security Agency (NSA) 7; and al Qaeda 510–12; and Berlin tunnel operation 174; and BND 422–3, 438, 447–51, 452–4, 455; and Bosnian-Croatian conflict 472–5; ceases intelligence exchange with GCHQ 289–90; and commercial encryption 488, 489; cooperation with GCHQ 222–3, 278, 282–3, 346, 347–54, 438, 441–3, 448–58, 461; creation of 101–2; data silo in Utah 546; development of new systems 345–6; development of Technical Research Ships 260; and documents on Princess Diana 483; and downing of Powers’ U-2 202–3; elint and comint responsibilities 122–3; funding of 334, 346; influence on foreign policy 321; and internet 8, 508; and invasion of the Falklands 398, 399, 415; and North Korea 100; and polygraph 434; and Prime case 376; public mention of 242–3, 355, 358, 361–2; relationship with US armed services 271–2; and Russian nuclear forces 119; Russian spies in 384–5, 444; and sale of cypher machines 209–15; and strategic elint 267; and Suez crisis 157, 158; and Tempest 216–18; and trade unions at GCHQ 421–2, 424, 429; Turkish operations 300–19; use of security measures 381; visibility of 341; and Yom Kippur War 291–2 National Service 117, 153, 219, 229, 369 NATO 126, 130, 209–15, 217–18, 242, 247, 248, 253–7, 268–9, 270, 272, 283, 300, 319, 328, 345, 429, 448, 452, 456, 467, 533, 534; Military Committee 256; Nuclear Planning Group 332; Special Committee 257 Nauticus Corporation 265 Naval Intelligence 32, 116, 143 Naval Mechanics School (Buenos Aires) 389 Nave, Eric 19 Nazi-Soviet Pact (1939) 31 Neff, Paul 50, 76 Netherlands 442, 443 ‘Never Again’ agreement (1982) 440–1 New People’s Army (Philippines) 452 New Zealand 89, 90, 92, 93, 98, 164, 165, 438, 442, 444–7, 448, 487; Government Communications Security Bureau (GCSB) 445–6 Newman, Dr 145 Newman, Max 28, 70 Nicoll, Douglas 387–8, 396, 421, 529 Nicoll Report (1981) 388 Nicosia (Cyprus) 326 Nimrods 268–70, 271–4, 326, 414, 415, 442, 474, 515, 536, 537 9/11 9, 509–14, 531 Nixon, Richard M. 4, 277–8, 279, 281, 283–4, 288, 290, 293, 297–8, 304–5, 325–6, 337, 338, 434 Noakes, John 348 Noise-Induced Hearing Loss 608 Noise Investigation Bureau 110 Nokia (telecoms company) 489 Norland, Selmer 48, 78 North Atlantic Council 254 North Cape, Battle of (1943) 60 North Korea 100, 120, 129, 175, 264 Northern Ireland 329, 498–503; Peace Process 501 Northwood Hills (communications security establishment) 192, 400, 401, 402, 469 Norway 55, 99, 116–17, 134, 139, 269 Nott, John 395, 396, 397–8, 414–15, 429, 601 Noye, Kenneth 505–6 Nuclear Planning Group 255 Oakleigh Park North (Soviet-radio monitoring station, Whetstone) 190–1 O’Connor, Morris J. 144, 145 Odette (Army intercept equipment) 474, 525, 534, 536–7 Odom, William ‘Bill’ 214, 385, 413, 434, 442–4, 446–52, 454–5, 457, 458, 471, 476 Oedipus (computer) 349 Oeljeschaeger, Major 50 Office of Strategic Services 87, 91 Official Secrets Act 8, 359, 360, 363, 383, 522 oil 298, 336, 356 Okinawa (Japan) 152 OKK-5 (Soviet codebook) 35 Oldfield, Maurice 82, 358 Omagh (Northern Ireland) 501–3 Oman 271, 345 Omand, Sir David 9, 272, 398, 495–7, 498 one-time pads (encryption system) 18–19, 20, 56, 74, 81, 83, 108 Operation Citadel (1943) 36 Operation Claret (1956) 140–2 Operation Damage (Comet sorties in the Mediterranean) 273 Operation Debenture (1954) 152–3 Operation Defiant (1955) 137 Operation Desert Storm (1991) 467–9 Operation Duster (sigint flight operations during Yom Kippur War) 295 Operation Gold (1948) 97 Operation Halfmoon (1948) 96 Operation Hem (sigint flight operations during Yom Kippur War) 295 Operation Musketeer (1956) 156–9 Operation Nigeria (against journalists and their sources) 506 Operation Overlord (1944) 59 Operation Pat (Comet sorties over the Baltic) 273 Operation Sanjak (1955) 137–9 Operation Storm (1995) 472–3 Operation Tartan (1955) 136–7 Operation Trail Hammer 536 Orford Ness (Suffolk) 285–7, 322 Organ, Helena see Prime, Helena Orion (sigint satellite) 437 Ormsby Gore, David 206 Orwell, George 549 Oshima, Baron 29 Ottawa (Canada) 57, 85, 92, 94, 97 Over the Horizon Radar 285–7, 322 Owen, Dr David 299–300, 332–3, 360, 391 Padeborn (Germany) 248 Pakistan 323, 334, 384, 513–14, 519, 537 Palestine 97, 109, 155, 156, 320 Paris 21, 25, 52, 53, 158, 194–5, 243, 284–5, 510 Parker-Bowles, Camilla 480, 482 Parliamentary Select Committee on Foreign Affairs 529 Patagonia 393 Patchett, Brian 228–30, 369 Paterson, Brian 507 Pearl, Daniel 514 Pearl Harbor attack (7 December 1941) 29, 290 Peking 150, 151, 194 Pelton, Ronald 384–5, 443–4, 447 Penkovsky, Oleg 322–3 Penney, William 192 Pepper, David 526, 527, 528, 532, 539 Perkar (Ceylon) 160 Perkins, Alice 495 Permanent Secretaries Committee on the Intelligence Services (PSIS) 219–20, 241, 260, 423 Perrin, Ken 267, 270, 363 Peshawar (Pakistan) 384 Petersfield (Hampshire) 133 PGP (Pretty Good Privacy, code-making programme) 490–1 Philby, Kim 37, 82–3, 84–5, 225, 226, 238, 242, 354–5, 367, 385 Philco (telecoms company) 350 Philippines 445 Phillips, Cecil 75, 79 Pilsey Island (Sussex) 140 Pincher, Chapman 17, 226, 238, 239, 240, 242, 386 Pine Gap (Australia) 345 Pinner (Middlesex) 68, 69 Pinochet, Augusto 519 Pirinclik Air Base (Turkey) 301–2, 306 Pither, Judith 383 Plessey (telecoms company) 212, 267–8, 311, 433 PLO (Palestine Liberation Organisation) 277 Poets Systems (Soviet cypher machines) 78, 81 Poland 21–2, 31, 46, 178, 387, 421, 515 Polaris missile system 266, 322, 335, 337, 438–40 Pollard, Jonathan 444 Polyarnoe (Soviet Union) 34, 36 polygraphs 381, 383, 424–6, 433–4, 444 Port Said (Egypt) 156 Port Stanley (Falklands) 390, 397, 413 Portsmouth (Hampshire) 134 Portugal 44 Poseidon (missile system) 439 positive vetting 88, 227–8, 229, 229–30 Posner, Gerald 483 Post Hostilities Planning (PHP) Committee 46, 85 Post Office 241, 286; develops fibre-optic cables 604; Research Department (Dollis Hill) 28, 68, 171, 172, 349; ‘Secret Department’ 14 Potts, Archie 154–5 Poulden, Teddy 93, 94, 352–3, 354 Powell, Jonathan 500 Powers, Gary 8, 201–7, 208, 226 Prague 244; Prague Spring (1968) 387 Prime, Geoffrey 8, 368–86, 423, 424, 425, 444, 447, 600 Prime, Helena 374–5 Prime, Rhona 375, 378–9 Princeton University 350 Prior, James 422 Profumo affair (1963) 8, 226, 228 Project Clipeus (British ADM programme) 249 Project Cobra Shoe (US intelligence station on Cyprus) 323, 348, 356 Project K (NSA HQ) 102 Project Minaret (US illegal monitoring of domestic radicals) 357 Project Sambo (tracking low-frequency submarine radio transmissions) 378 Project Sandra (intelligence facility on Cyprus) 321–3, 348, 356 Public Interest Immunity certificates 505 Public Key Cryptography 489–93, 508, 512 Public Record Office (Kew) 355 Puerto Belgrano (Argentina) 393 Punta Arenas (Chile) 414 Purple (Japanese cypher machine) 29, 38 Purves, Peter 348 Pym, Francis 423 Pyramider satellite 377 Quinlan, Sir Michael 493 Racal (telecoms company) 401, 433, 469, 524, 590 radar 110, 126, 132, 133, 136, 138, 145, 154, 202, 266, 285–7, 301–2, 306, 406, 408, 466 Radcliffe, Lord 239–40 Radcliffe Inquiry into Security in the Civil Service (1962) 381, 418 Radio Corporation of Cuba 341 Radio Operators (GCHQ) 15, 185, 186, 228, 261, 382, 418, 419, 420, 422, 432, 435, 458 Radio Reconnaissance Teams (Afghanistan) 535 Radio Security Service 37, 58, 79, 221 Radio Warfare Special Branch 133, 134 Rainbow Warrior (Greenpeace ship) 446 Rangoon (Burma) 110 Rattan (Soviet radio intercepts) 75 RC-135 ‘Looking Glass’ aircraft 273 Reagan, Ronald 398, 403, 457 ‘Real IRA’ 501, 502 Reenan, Cyril 479, 480 Rees-Mogg, Lord William 481 Reeve, James 196, 197 Regulation of Investigatory Powers Act (RIPA) (2000) 547 Reid, John 516 Reijn, Joop van der 308 Reilly, Patrick 205, 206 rendition programmes 539–40 Rendle, John 578 Rennie, John 156 Res (reserved cypher material) 44–5 Review of Intelligence Requirements and Resources (1994) 493 Review of Intercept as Evidence (2004) 541 Rhodes, Miriam 378 Rhodesia 3, 209 Ribbentrop, Joachim von 31 Richards, Brooks 387 Richards, Francis 504, 511, 526 Riddington, Tony 598 Ridley, Nicholas 392 Rimington, Stella 372, 494, 521 Ring of Five 442–3 Ritter, Scott 470–1 Riyadh (Saudi Arabia) 513 Roake, Alfie 147, 265 Robinson (computer) 349 Robinson, Robin 493 rockets and missiles 5, 108, 110, 131, 169, 201–2, 203, 207, 266, 271, 301, 306, 315, 322–3, 335, 337, 376, 390, 439–40, 510, 511, 515, 525 Rockex (UK cypher machines) 57–8, 194 Rolf, Vic 213 Romania 52, 256, 257, 284; foreign intelligence service (DGIE) 253–4 Rome 372 Rommel, Erwin 5 Roosevelt, Franklin D. 39, 41, 47, 73, 84, 87 Rose, Michael 472, 473, 474 Rosenberg, Ethel and Julius 83 Rosenheim (Germany) 48 Roussilhe, François 254–5 Rowe, Vivian 402 Rowlands, Ted 390, 399–400, 401 Rowlett, Frank 44, 174 Rowntree Foundation 361 Royal Air Force (RAF) 369, 537: 192 Squadron (monitoring aircraft unit) 113, 122, 125, 149, 159; 199 Squadron (radio wafare unit) 113, 125, 131; invited onto Blue Peter children’s programme 348; loss of Avro Lincoln (1953) 125–9, 580; and Malayan Emergency 149–50; near-miss incidents 131; negotiations on air corridors 129–30; 100 Countermeasure Group 112; as part of GC&CS 20; RAF Akrotiri 273, 295, 296, 323, 325; RAF Brampton 469; RAF Brawdy 377; RAF Brize Norton 204, 375; RAF Celle 128; RAF Crail 369; RAF Digby 230, 231, 237; RAF Gatow 112, 370, 383; RAF Habbaniya 155, 161–2, 230; RAF Hammersley Hayes 231; RAF Lakenheath 116, 142; RAF Leconfield 126; RAF Mildenhall 257, 273, 284, 297; RAF North Luffenham 153; RAF Northolt 537; RAF Oakhanger 262; RAF Pergamos 162, 323; RAF Scharfoldendorf 128; RAF Sharjah 271; RAF Strike Command 270; RAF Upper Heyford 457; RAF Watton 112–14, 122, 125, 131, 159; RAF Wythall 153; RAF Wyton 273, 295; ‘Rock Apes’ Regiment 154; sigint and elint operations 110–14, 116–18, 121–2, 124, 131–3, 153, 154, 206, 218–19, 269–70, 272–3, 285–7; Signals Units 153, 154, 161; surveillance operations 537–8; and U-2 overflight programme 202; Y stations in Kent and Cheshire 34, 63, 111 Royal Commission on Criminal Justice Procedure 425–6 Royal Electrical and Mechanical Engineers 499 Royal Marines 390, 391, 394, 402, 411, 413, 524, 535; Y Troop 524, 525, 535 Royal Navy 159, 441, 474, 477; and Buster Crabb incident 140–2; HMS Affray 133; HMS Albion 259; HMS Anderson (sigint station in Ceylon) 69, 92, 93, 94, 160; HMS Antrim 409, 410; HMS Ardent 410; HMS Conqueror 404; HMS Coventry 406; HMS Dolphin 265; HMS Endurance 391–2, 394, 396; HMS Glamorgan 410; HMS Glasgow 406; HMS Hermes 402–3, 411; HMS Invincible 407; HMS Maidstone 139; HMS Mariner 134; HMS Mercury (RN Signals School) 133, 134, 139; HMS Pickle 134; HMS Pucklechurch 133, 134; HMS Sheffield 406; HMS Sir Galahad 400; HMS Superb 259; HMS Taciturn 144–6; HMS Totem (later Dakar) 137, 264, 265; HMS Truculent 133; HMS Truelove 134; HMS Truncheon 264, 265; HMS Turpin 135–9, 140, 143, 146–7, 264; influence on GC&CS 16; intercept sites at Scarborough and Winchester 63; Kipling story involving 13; Provost Branch 139; and radio station at Polyarnoe 34; ‘Room 40’ code-breakers 15; sigint and elint operations 6, 114, 133–9, 143–7, 218–19, 264–7 Royal Radar Establishment (Malvern) 501 Royal Ulster Constabulary (RUC) 501, 502–3; RUC Special Branch 499, 502, 503 Rumsfeld, Donald 511 Rushworth, Edward 48, 78 Russian Mafia 504 Ryolite satellites 345, 346, 376, 421 Sabah (Borneo) 165 Sabri, Naji 530 Sadat, Anwar 296 Sadi, Sener 314 Safford, Laurance 39 Samford, General 101 Samsun (Turkey) 301, 311 San Carlos Water (Falklands) 408–11 San Francisco 92 Sangin province (Afghanistan) 535 Santa Fe (Argentine submarine) 397 Sarafand (Palestine) 20, 32, 155, 162 Sarajevo (Bosnia and Herzegovina) 473, 474 Sarawak 165 Sarell, Sir Roderick 308, 312, 313, 315, 319 Saudi Arabia 164, 467, 468 Saunders, Andrew 458 Savimbi, Joseph 454 Scan Odd (airborne radar) 121 Scarborough (Yorkshire) 63 Scargill, Arthur 416 Scarman, Lord 431 Scarus (portable interception kit) 534 Schlesinger, James 294–5, 326, 329, 330, 332 Schmidt, Hans 21 Schröder, Gerhard 520 Schulze, Reinhard and Sonja 455–6 Scott-Farnie, G.R. 34 SCUD missiles 251, 468, 469 SDECE (French secret intelligence service) 442 Second World War 135, 335; end of 47–50, 59–60; events leading up to 22; and improved communications 5; and release of sigint material 355–6; Russian interceptions in 30, 31–2; sigint and code-breaking in 2, 5, 28–9, 32–46, 250 Secret Intelligence Service (SIS; MI6) 1, 57, 82, 91, 97; agents sent into the Eastern Bloc 123; at Bletchley 23, 27; and Automatic Data Processing 353; and Berlin-Vienna tunnel operations 169–76; and bungled surveillance operations 140–2; buys missiles on the open market 407; closer relationship with MI5 and GCHQ 503–4; considered organisational basket-case 24; and Cyprus radio station 156; and end of influence over sigint 142; and Enigma decypherment 21; funding of 494; move to Broadway Buildings 17; move to Century House (1966) 195; move to Vauxhall Cross 496–7; no trade unions in 417; obsession with Russia 17, 31; origins of 14; and Penkovsky 322–3; and post-war re-absorption of GC&CS 67; recruitment to 61; Section V 37; Section VIII 57–9; Section Y (Carlton Gardens) 171, 178, 372; Technical Collection Service 132; and Yemen 164 Secret Service Committee 15 Security Commission 368–9, 381–2, 384 Security Service (MI5) 1, 57, 182, 361; and Automatic Data Processing 353; Britten affair 237; bugging operations 177; closer relationship with SIS and GCHQ 503–4; and CND 368; fears concerning Communists in trade unions 418–19; funding of 494; interception of telegrams and telexes 239; and John Cairncross 37; and Libyan Embassy affair 456; no trade unions in 417; origins of 14–17; and Peter Wright 223, 224; and polygraph 433–4; and possible sub-agents 382; problems with Orford Ness 287; and sigint security 70–1; ‘Squidgygate’ affair 482; and Venona 79, 84, 85, 86; ‘watchers’ operations 183–90; and West German defections 455–6; Whetstone radio monitoring station 190–1 Selby-Bennett, Harry 134, 139 Selwyn Lloyd, John 128, 158, 159 Semipalatinsk (Soviet Union) 107, 302 Services Liaison Department 70 Sexton, Jamie 305 Seychelles 334, 335 Shanghai 19 Sharq el-Adna (radio station) 156 Shayler, David 521 Shedden, Sir Frederick 86, 92 Sheldon, Robert 459 Shergold, Harry ‘Shergy’ 142 Short, Clare 523–4 Sidewinder air-to-air missiles 441 Sigaba (US cypher machine) 98–9 Sigdasys project (improved flow of sigint to front-line units) 451, 456 Sigint Conference (1946) 95 Sigint Electronic Warfare Operation Centre (Afghanistan) 535 sigint satellites 437–8 sigint ships 259–67 Sigint/EW Operations Centre in Regional Command (South) 534 SIGMod 539 signals intelligence (sigint); in Afghanistan 534–9; airborne 121–2, 125–33, 144, 202–7, 223, 251–2, 257, 267–74, 295, 377; and allied cut-offs 444–7; American 39–40, 143–4, 271–2, 273–4; benefits of 401–2; blue jackets (BJs) 17, 70–1; bugging 176–82; cost of 218–23; Croatian success 473–5; and Cuba 341–2; cypher security 57, 191–3; in Cyprus 320–39; D-Notice affair 238–41; defections from 228–9; dependence on SAS-type activities 250–1; and ‘Dodgies’ or Mystery Trips 6; and domestic/ international blurring 344; during Cold War 1, 5, 8, 108–24, 125–47, 257–8; during WWII 2, 5, 32–46, 57–8; expansion in India 30; failures of 253–5; and Far East 148–51, 164–8; GCHQ at heart of 5–10; German 35, 49–50, 62; global alliances 89–101; ground-based 117–20, 252–3; and Gulf War (1991) 468; in Hong Kong 151–5; importance of bases 151–5, 277–8; and Indonesia-Malaya confrontation 164–8; and invasion of Czechoslovakia 245–6; and invasion of Falklands 397–415; and Iraq War (2003) 525–6; and Kipling 13–14; legalities of 344; and making/influencing of foreign policy 321; in medieval times 4; and Middle East 155–64; modern formation of 58; and political leaders 2–4, 7; and private companies 17; problem of language 512–13; public disclosure of 356–64; rethinking of targets 345; relations between Western allies 444–7; release of wartime material on 355; and rescuing of enemy matériel 47–56; risks 203; and Russians 33–8, 280–1; seaborne 114–16, 133–47, 208, 259–67, 377–8; security disasters 228–38; size of 227–8; and speed of communication 4–5; support for front-line units 449; tactical units 250–2; and Third World 169; in Turkey 300–19, 330–1; value and importance of 60, 62–71 Signals Intelligence Centre 93 Signals Intelligence Service (USA) 74 Sillitoe, Sir Percy 86, 190 Silvey, Reg 412 Simakov, Alexander 478–9 Simkin, Anthony 187 Sinclair, Hugh ‘Quex’ 16, 22, 23 Sinclair, Sir John 142 Sinews (Sigint NEW Systems) 496 Singapore 19, 40, 96, 164, 166, 167 Singleton, Valerie 348 Sinkov Mission (1941) 39 Sinn Féin 500 Sinop (Turkey) 301 Six-Day War (1967) 253, 263–4, 271, 284 Skardon, William 87, 188 Skynet (communications satellite) 347–8, 403, 408, 438 Slessor, John 190–1 Slim, Field Marshal William 65 Sly, Ken 153–4, 375 Smallwood, Sir Denis 271 Smith, F.M. 302 Smith, Ian 3 Smith, Jack 151 Smith, Jacqui 543 Smith, Rupert 474–5 Smiths Industries 495 Snow, Leading Seaman ‘Snowy’ 135 social intelligence 178–9 Solidarity (Polish trade union) 432, 465 Somerville, John 121, 317–18, 419, 420, 428 Sony 480–1 Soothsayer (Army intercept equipment) 537 SOSUS (undersea microphones) 377 South Africa 209, 446, 454 South Georgia 390, 391, 393–6, 409 South-East Asia Command 110 Southern Thule 389, 390–1 Soviet Air Force 118, 119, 360; Air Defence Command 201; Strategic Air Command 256; Strategic Rocket Force 256 Soviet Army 175, 371; General Staff 255–6 Soviet Communist Party 19, 368 Soviet Navy 114–16, 133, 256, 301; Naval Intelligence 73; Northern Fleet 136 Soviet Union 33–8, 336, 349; apparent war preparations 255–7; and Berlin-Vienna tunnel operations 169–76; and biological weapons 611; capture of German cryptographic assets 50–1; Cold War espionage 8; elint on invasion of the Falklands 401; end of Cold War 493; enters WWII 28; ‘ferret’ programmes 112; Hitler’s invasion of 29, 32–3, 290; invasion of Czechoslovakia 244–7; Italian code-breaking concerning 52, 54; nuclear weapons 107–8, 112, 114, 116; and post-war confrontation 47; raid on Arcos building 18; Red Army 46, 78, 245–6, 249, 319; release of UK intelligence material 355; secret submarine missions against 6; and ‘strayed’ aircraft 126–33; successful sigint operations against 279–81, 344, 361; Turkish (Anglo-US) operations against 301–2; UK-US obsession with 17–19, 30–2, 45–6, 321–3; and Venona Project 72–88; and Yom Kippur War 291 Spain 44, 505, 533 Special Branch 141, 150, 151, 166, 187, 453, 456, 459, 509 Special Liaison Units 57–8 Special Radio Installation Flight (SRIF) 114 Spedding, David 494 spy planes 257, 267–74, 292–3, 296–7, 538 ‘Squidgygate’ affair (1990) 479–82 SR-71 Blackbird Mach 3 reconnaissance aircraft 284, 292–3, 296–7 Stalin, Joseph 31, 33, 46, 47, 173 Standard Cable & Wireless Ltd 17 Standard Telegraph & Cable Ltd 342 Stankovic, Milos 473 Stanmore (Middlesex) 62, 68 Stannard, Robert ‘Fred’ 210–11, 213, 218, 242 Starmer, Keir 543 State Research Association 361 stay-behind patrols 247–50 Stella Polaris (sale of Russian codebooks) 91 Stephanie operation (Canadian embassy intercepts in Moscow) 280–1 Stephenson, Sir Hugh 207, 221 Stevens, Geoffrey 40, 45, 50, 94 Stewart, Brian 353 Stewart, Michael 228 Stockholm 192 Straw, Jack 505, 519 Stripp, Alan 109 submarines 6, 114–17, 125, 133–9, 142–7, 197, 259, 264–7, 337, 377–8, 384, 397, 404 Suez Crisis (1956) 156–9, 160, 181, 182, 213 Sugar (Vienna tunnel code) 171 Sugar Grove (West Virginia) 262 Suharto, President 168 Sukarno, President 164, 165, 167–8 Sunay, Cevdet 311 Super Antelope programme (modernising/upgrading Polaris submarines) 337 Supreme Allied Commander Europe (SACEUR) 256 surveillance operations 4, 183–90, 197, 201–7, 208–9, 247–53, 292–3, 295, 296–7, 322, 368, 406, 421, 472, 480–1, 500, 506, 537–8, 540–50 SUSLO (Special United States Liaison Officer based in UK) 381 Sutton Common (Cheshire) 500 Sweden 31, 212, 214, 269 Switzerland 52, 212, 214, 215, 457, 492 Sykes, Richard 297, 298, 335, 593 Syria 156, 157, 271, 291, 300, 301–2, 304, 308, 336, 344 Tai Mo Shan (New Territories) 154 Taipei 195 Taiwan 152, 195, 323 Taliban 535, 537 Tanzania 511 Taper (Soviet cypher traffic) 54, 108 Tartus (Syria) 331 TASS News Agency 190 Taylor, Telford 43 Tebbit, Kevin 498, 504 Technical Committee of London Signals Intelligence Committee 267 Technical Radio Interception Committee 131 Tedder, Lord 5 Tel Aviv 157–8, 180 telephone tapping and intercepts 170–6, 180, 244–5, 299, 340, 341–6, 376, 377, 474–5, 479–83, 486, 499, 500–1, 523, 541–5 Tempest (radiation/emanation phenomenon) 209, 215–18 Templer, Sir Gerald 150, 219 terrorists, terrorism 9, 168, 277, 307, 320–1, 452, 531; and 9/11 509–15; domestic 539; and Heathrow plot (2003) 515–16; and IRA 498–503; Libyan 455–8; see also kidnapping and hostage-taking Teufelsberg (Germany) 478 Thatcher, Margaret 8; and Falklands conflict 298, 396, 400, 403–4; and mole-mania 363, 367; 1987 general election 433; obsession with secrecy 492; and polygraph 434; and removal of trade unions from GCHQ 415, 416–17, 423, 425, 426, 427, 428, 429, 430–1, 435; and tightening of the Official Secrets Act 363; and Zircon project 442, 460 Third World 203, 259, 334 Thistlethwaite, Dick 255 Thomas, Richard 544 Thomas, Teddy 234 Thompson, Julian 402, 410–12 Thompson, Ralph 94 Thompson-CSF (arms company) 489 Thomson, Mike 131 Thorneycroft, Peter 163 Thorpe, Peter 535 Tiananmen Square massacre (1989) 476 Tickell, Crispin 290 TICOM (Target Intelligence Committee) teams 48–56, 76, 78 Tiltman, John 19, 31–2, 42, 44, 67, 78, 79, 96, 213 Tirpitz (German battleship) 35 Titchner, Lambert 186 Tito, Josip 4 Tomlinson, Richard 521 Tonkin, Derek 330, 333 Tornado Multi-Role Combat Aircraft 345 Toumlin, George 432 Tovey, Brian 167, 414, 415, 421–2, 423, 424, 428, 433, 442, 448, 461, 490 trade unions 317, 368, 389, 416–36, 497–8 Trades Union Congress (TUC) 416, 417, 419, 426, 427, 509 Travis, Edward 27, 28, 36, 43, 48, 49, 53, 56, 60, 67–9, 69, 92, 94, 101, 121 Trawlerman (DIS computer scheme) 527–8 Trend, Burke 221, 240, 241, 242, 269–70, 288, 322–3, 354, 364 Trevor-Roper, Hugh 221 Tromsø 134 Truman, Harry S. 73, 85, 91, 101, 108, 109, 116 TRW Inc (telecoms company) 377 Tryst operation (British Embassy intercepts in Moscow) 280–1 ‘Tunny’ (German cypher machine) 28, 349 Turing, Alan 2, 25, 27, 349, 492 Turkey 52, 109, 131–2, 157, 268, 269, 299–319, 325–8, 330–1, 334, 338, 357, 423, 472; Air Force 312–13, 313; Army 315; Foreign Ministry 312–13 Turkish People’s Liberation Army (TPLA) 300, 303–19 Turkish People’s Liberation Front (TPLF) 300, 308–10, 312 Turnbull, Andrew 524 Turner, Charles 312–16, 318 TUSLOG (US Logistics Organisation in Turkey) 302 Tuxedo (British nuclear weapons stockpile in Cyprus) 163 25 Mayo (Argentine aircraft carrier) 404 Typex (UK cypher machine) 56, 98 Tyuratam (Soviet Union) 306 U-2 spy planes 142, 226, 292, 296, 332, 471 UK-USA Technical Conference (1946) 95 Ukraine 472, 533 UKUSA (UK-USA signals intelligence agreements) 7, 213, 241, 273, 376, 577; combined comsec/sigint agencies 242; and deterioration of Anglo-US relationship 285; development of 89–95; and elint 111; existence of 1948 aagreement 577; and GCHQ 95–9, 222; and Hong Kong 152; and Israel 471; and Korea 99–101; and satellite collection 437–8; second-party members 444, 447; standardisation of equipment in 424; third-party members 209, 447, 452; and tightening of security 381; value of British Empire to 149 Ultra (WWII decrypts) 1, 24, 26, 32–8, 41, 42, 43, 57, 59, 60, 62, 72–3, 113, 354, 362, 363 Underwater Development Establishment 145 Unit 8200 (Israel sigint agency) 470 United Nations 66, 295, 336, 445–6, 472, 523–4; Monitoring, Inspection and Verification Commission 520; Protection Force 472; Security Council 517, 522; Security Council Resolution 1373 511–12; Special Commission (UNSCOM) 470–1 United States; and Cold War espionage 8; Communications Intelligence Board 97, 152; cyphers worked on by GC&CS 17, 29; Department of Defense 295; Division of Scientific Intelligence 322; Europe Command 180; Foreign Intelligence Advisory Board 263; Information Service 305; National Photographic Intelligence Center 296; National Reconnaissance Office 449, 458; National Security Council 334; and proposed German-Mexican alliance 15; and UKUSA intelligence treaty 89 University of Pennsylvania 426 Unye (Turkey) 313, 316 US Air Force 96, 101, 129, 152; 47th Radio Squadron 118; Griffiss Air Force Base (New York) 292; Johnson Air Force Base (North Carolina) 293; Security Service 118, 120, 301; Strategic Air Command 272 US Army 39, 40, 42, 43, 44, 52, 74–5, 77, 91, 94, 99, 102, 336, 413, 474; Intelligence Support Activity 452; Marines 525; Security Agency 99, 152, 302, 345–6; Seventh Army Corps 467; Special Forces 249; US Army Air Force 110–11 US Navy 39, 40, 42, 43, 75, 91, 97, 99, 143, 207; US Naval Intelligence 157; Radio Research Station Program 262; Security Group 115, 301; Sixth Fleet 323–4; USS Belmont 260; USS Cochino 112–17, 135; USS Georgetown 260; USS Jamestown 260; USS Liberty 260, 263–4; USS Muller 260; USS Oxford 260; USS Pueblo 260, 264; USS Stickleback 143; USS Tusk 114–15; USS Valdez 260, 605; USS Vincennes 408 USM-49 (US sigint base in Turkey) 303 Vampire (UK intercept unit) 474, 534 ‘Vasiley’ (KGB officer) 232–4 Vass, Sir Douglas 423 Vassall, John 226, 238 Vatican 52 Venona project (intercepted Russian messages) 445–6; and Anglo-American collaboration 72, 78–80; and British Commonwealth 85–8; exploitation of reprinted pages 74–5; and exposure of agents 80–8; extreme secrecy of 73, 77–8; first code-breaks 75–6; and global sigint 90, 94, 98; and Manhattan Project 76; size and importance of 72–3; Soviets alerted to work on 73, 80–1 Vernon, Mike 422, 423 Viehoff (Germany) 50 Vienna 169, 170–1, 172, 372, 373, 375, 376 Vienna Summit (1961) 180 Vietnam 153, 167, 168, 178, 203, 376, 446 Vietnam War 123, 243, 252, 269, 271–2, 277, 279–80, 298, 356, 387 Virgin 545 Vladivostok 129 Voice of Egypt (radio station) 156 Wait, Dave 385–6 Wal Bin Chang 153 Waldegrave, William 429 Walker, John 264, 377, 384, 447 Walker, Walter 165, 166 War Office 22 War on Terror 533, 539–40 Warsaw Pact 114, 244, 245, 247, 248, 251, 253, 257–8, 319, 321, 369, 402, 465 Washington 7, 38, 39, 40, 41, 42, 43, 44, 45, 47, 57, 64, 74, 77, 82, 83, 90, 92, 96, 97, 99, 101, 102, 111, 119, 121, 143, 151, 157, 203, 205, 212, 243, 253, 292, 325, 329, 335, 355–6, 381, 413 Watchkeeper 450 drone 536 Watergate House (London) 15–16 Watergate scandal (1973) 4, 279, 288, 290, 293, 297, 298, 325–6, 356 Wavendon Manor (radio station, Buckinghamshire) 49 weapons of mass destruction (WMD) 516, 520, 523, 526, 528–31 Weatherill, Bernard 459 Weinberger, Casper 441 Weisband, William 80–1, 169 Welchman, Gordon 25, 26, 27, 43, 57, 61, 64, 65, 362–3, 364, 387 Wenger, Joseph 42, 53, 79, 95, 243 West Germany 345, 442, 447–9, 455–6 West Irian 167 West, Lord 544 West, Nigel 88 Western Union 240 Whaddon Hall (Buckinghamshire) 23, 57, 181 Wharfe, Ken 483 White, Ray 501 White, Sir Dick 142, 176, 181, 187, 225, 241–2, 243, 245, 246, 264, 285–6, 353, 354, 364, 453 Whitelaw, Willie 404, 426, 427 Wieck, George 214 Wiesbaden (Germany) 158 Wigg, George 227–8, 240 Wilkes, Detective Sergeant 378 Wilkinson, Peter 206 Williams, Sir Anthony 392–3 Wilson, Edmund 42, 54, 57, 68, 576 Wilson, Harold 225; and Anglo-US relations 356–7; antagonism towards the press 239, 242; and Chevaline project 440; and Cyprus problem 325, 329; and Diego Garcia 338; fascinated and terrified by intelligence and espionage 3, 168, 226–7, 357; and Radcliffe Committee 239–40; and Skynet 438; and U-2 flights from Cyprus 295–6; and Vietnam War 277 Wilson, Jim 292 Wilson, Richard 527 Winnifrith, John 418 Winterbotham, Frederick 35, 354 Wolfenden, Jack 383 Woodward, Admiral Sandy 402–3, 407, 408, 410–11 Wormwood Scrubs prison 238, 385–6 Wreford-Brown, Christopher 404–5 Wright, Georgina 317 Wright, Peter 216, 223, 224, 267, 363, 492, 538, 587–8 Wyllie, Sean 490 Y services (armed forces listening units) 26–7, 33–5, 63, 68, 103, 111, 117, 411 Yarallakos (Cyprus) 320, 328 Yardley, Herbert O. 38 Yemen 148, 163–4 Yom Kippur War (1973) 277, 290–5, 320, 337, 387 York, Duchess of 482 Young, Courtney 86 Yugoslavia 256, 257, 284, 471–5, 503, 512, 534 Yunnan (China) 151 ‘Yuri’ (KGB officer) 231–3 ‘Zhora’ (Weisband’s code name) 80–1 Zimmermann, Phil 490–1 Zimmermann Telegram (1917) 15 Zinnia (missile-detection system) 322–3 Zinser, Aguilar 519–20 Zionist movement 97, 109 Zircon project (GCHQ sigint satellite) 415, 438, 442–3, 449–50, 458–61 Acknowledgements On 9 December 1993 the Lord Chancellor, Lord Mackay of Clashfern, introduced the Intelligence Services Act in the House of Lords.

How We Got Here: A Slightly Irreverent History of Technology and Markets
by Andy Kessler
Published 13 Jun 2005

A Santa Barbara IMP went up in November and Utah in December 1969. One of the researchers at the Stanford site, Norm Abramson, was a surfer dude who spent a lot of time in Hawaii. The University of Hawaii had locations scattered across the Islands and was trying to figure out how to hook up a data network between them. It couldn’t afford to run undersea cable and modems were too slow, so they hit on the idea of using radio signals to transmit data. The problem was interference. Maui might transmit at the same time and step on the Big Island’s signal. It could use packet networks, but that didn’t solve the interference problem. In 1970, Abramson devised a system that checked for errors in the packets received.

The Longing for Less: Living With Minimalism
by Kyle Chayka
Published 21 Jan 2020

In one way, Apple follows the rule: The devices have only a few visual qualities. But it’s also an illusion. The company has to manufacture its own batteries to be as flat as possible and removes ports—see headphone jacks—any chance it gets. The iPhone’s function depends on an enormous, complex, ugly superstructure of satellites and undersea cables that certainly aren’t designed in pristine whiteness. Minimalist design encourages us to forget everything a product relies on and imagine, in this example, that the internet consists of carefully shaped glass and steel alone. The contrast between simple form and complex consequences brings to mind what the British writer Daisy Hildyard called “the second body”22 in her 2017 book of the same name.

Thank You for Being Late: An Optimist's Guide to Thriving in the Age of Accelerations
by Thomas L. Friedman
Published 22 Nov 2016

“In twenty years,” he added, “we went from maybe this is a good idea to there’s no turning back to hitting the physical limits … With ping I found out how close to the physical limits we were, and it was pretty startling. It is a way big revolution.” This revolution happened, Bucksbaum explained, thanks to a kind of Moore’s law that has been steadily quickening the transmission speeds of data and voice down fiber-optic cables. “The speed at which we can transmit data over undersea cables just keeps accelerating,” said Bucksbaum. The short version of the story, he explained, goes like this: We started out sending voice and data using a digital radio frequency over coaxial cable made primarily of copper wire. That is what your first cable/phone company sent into your house and into the box on your television set.

…

And it included wavelength division multiplexing, using different colors of light to carry different phone conversations at once—and then combinations of the two. They are not done accelerating. “The history of the last twenty years is that we just keep finding faster, better ways to divide the different properties of light to pack ever more information,” said Bucksbaum. “The rate of data transfer for an undersea cable today is now trillions of bits per second.” At some point, you end up “bumping up against the laws of physics,” he added, but we are not there yet. Companies are now experimenting not just with ways to change the pulse or the color of light to create more capacity, but also with new ways of shaping that light that can deliver more than one hundred trillion bits per second down their fiber lines.

The Rise and Fall of American Growth: The U.S. Standard of Living Since the Civil War (The Princeton Economic History of the Western World)
by Robert J. Gordon
Published 12 Jan 2016

The Civil War itself showcased these inventions when northern trains sped Yankee troops to the front and steamships blockaded supplies to the south from Britain, hastening southern defeat. And no longer was news delayed by days or weeks. Half a century earlier, the Battle of New Orleans had been fought on January 8, 1815, three weeks after the Treaty of Ghent was signed to end the War of 1812. Before development of the telegraph and undersea cable, news traveled very slowly. But during the Civil War, the daily newspapers carried dispatches announcing the outcomes of battles mere hours after they occurred. The flood of inventions that followed the Civil War utterly transformed life, transferring human attention and energy from the mundane to soaring skyscrapers and airplanes.

…

Within only two years, there were nine telegraph companies, whose 2,000 miles of wire stretched all the way from Portland, Maine, to Chicago and New Orleans.14 The telegraph network was ubiquitous in the east by 1855, the transcontinental telegraph debuted in late 1861, and after an abortive start in the late 1850s, a working undersea cable linking Britain and America was laid in 1866. Contemporary observers recognized the importance of the telegraph almost immediately. As early as 1847, the telegraph was seen as “facilitating Human Intercourse and producing Harmony among Men and Nations.” In the late 1860s, a writer anticipated “when the missing links shall have been completed of the great chain that will bring all civilized nations into instantaneous communication with each other … breaking down the barriers of evil prejudice and custom …” At the same time another optimistically suggested that “whenever science achieves a victory, a rivet is loosened from the chains of the oppressed.”15 Though the cost of sending a telegram was initially too high for the telegraph to gain widespread usage by private households, it was immediately viewed as essential by three industries—finance, railroads, and newspapers.

…

The telegraph could transmit prices of commodity and financial asset data, thus reducing or eliminating the role of arbitrage for traders who had particularly good or bad access to information. By squeezing margins, news provided by telegraph began to eliminate wholesalers and middlemen who had depended on differential access to information. A domestic financial transaction could be ordered and confirmed by 1890 in less than two minutes. Before the undersea cable, the six-week delay for a round-trip crossing from New York to London could lead to inefficient purchase and sale decisions for commodities and other goods, implying that the welfare benefits of the telegraph and cable included not just financial markets, but also the real sector of the economy.16 The telegraph became an essential tool to railroads in controlling the flow of passenger and freight trains.

The Dead Hand: The Untold Story of the Cold War Arms Race and Its Dangerous Legacy
by David Hoffman
Published 1 Jan 2009

The KGB, he recalled, harvested a rich crop of secrets from a walk-in to the Soviet Embassy in 1980, an employee of the National Security Agency, which ran American global electronic eavesdropping. Yurchenko said he only knew of this agent as "Mr. Long," and gave his debriefers some details. He said Mr. Long sold to the Soviets the details of the U.S. operation to tap the Soviet undersea cables in the Sea of Okhotsk. This was the monitoring operation known as Ivy Bells, which had been discovered and removed by the Soviets in 1981. (A second undersea cable-tapping operation in the Barents Sea had not been compromised.) The FBI launched a manhunt for Mr. Long, and four months later arrested Ronald Pelton, a communications specialist with the NSA who sold the classified data to the Soviets for $35,000.

Pax Technica: How the Internet of Things May Set Us Free or Lock Us Up
by Philip N. Howard
Published 27 Apr 2015

There is no consciousness-raising singularity here. But at the end of March 2013, more traffic flowed between China, Korea, and Japan than flowed between the United States, the United Kingdom, and Europe. A significant amount of digital traffic flows through the cables at the bottom of the Pacific Ocean. However, it’s not undersea cables that carry the most traffic, it’s overland cables. The fastest-growing region for internet traffic is within Asia—between China’s largest cities and between China’s cities and other cities in Asia. Indeed, one of the more pressing infrastructure problems in China is the need for faster internet connections from Shanghai, Hong Kong, and Beijing out to the country’s many large provincial capitals.

Who Gets What — and Why: The New Economics of Matchmaking and Market Design
by Alvin E. Roth
Published 1 Jun 2015

Thus responding to prices on the English market (which reflected supply and demand there) essentially took the better part of a month. After the cable was completed, price information could cross the Atlantic in a day. As a result, cotton shipments began to more closely match the fluctuations in the market, and prices on the cotton market became less volatile. Before the undersea cable, it was harder to match supply with demand because the news that arrived in New York was already more than a week old—slower than the pace of events even in that era. Speeding up the news let traders react to the market better. Speed had made for a better market in cotton. But as we’ll see next, speeding up the arrival of news so that decisions can be made with better information is different from speeding up decisions so that they have to be made before news arrives.

The Twittering Machine
by Richard Seymour
Published 20 Aug 2019

It supposedly represents a unique person, living her best life, from the best angle, in the best light. But it does so using a technology that, as Adam Greenfield puts it, distributes and smudges the self across ‘a global mesh of nodes and links’.60 This hard infrastructure, from sensors in the smartphone to cellular base stations, undersea cables, microwave relays and networks of users, organizes from end to end a person’s experience of the world, her selfhood. In addition to breaking up the self into digitalized components, the technologies of the selfie also have the alarming effect of making everyone look the same. Some of the repetitive banality of selfies can be blamed on the conventions of selfie-taking.

There's a War Going on but No One Can See It
by Huib Modderkolk
Published 1 Sep 2021

From these exchanges, data packets continue on their way to the data centre of the recipient’s ISP. That might be in the same country or it might be halfway around the world, in which case the packets speed across the oceans along thick, multi-layered, black cables on the seabed. Some of those undersea cables enter Europe through the Netherlands, at Beverwijk and Katwijk on the west coast and in the north at Eemshaven in the province of Groningen. The location of that last cable was enough to persuade Google to build a data centre in Groningen, conveniently close to the transatlantic superhighway.

The Grand Scuttle
by Dan Van der Vat

Heading out to sea, Müller sank the French destroyer Mousquet. The Penang raid behind her, the Emden slipped south-eastwards among the myriad islands of the Dutch East Indies, taking a two-day break for cleaning, running repairs and coaling. Müller then headed west to the British Cocos Islands to destroy the vital junction of undersea cables linking Australia, India and Africa at Direction Island. He anchored offshore on 9 November and put ashore a landing party, which demolished the relay station – but not before its operators were able to send the warning signal to the Allied searcher which was finally to put the Emden at bay. The Australian light cruiser, Sydney, larger, faster and more heavily armed than the German ship, and acting as advance escort of an Australian and New Zealand convoy, answered the call within hours.

Cyber War: The Next Threat to National Security and What to Do About It
by Richard A. Clarke and Robert Knake
Published 15 Dec 2010

The notion contained in the “arsonist principle” is one that can be applied to cyber war. While we talk about cyberspace as an abstract fifth dimension, it is made up of physical components. These physical components, from the high-speed fiber-optic trunks, to every router, server, and “telecom hotel,” are all in sovereign nations, except perhaps for the undersea cables and the space-based relays. Even they are owned by countries or companies that have real-world physical addresses. Some people like to contend that there is a “sovereignty problem” on the Internet, that because no one owns cyberspace in its entirety, no one has any responsiblility for its integrity or security.

The Girl in the Road
by Monica Byrne
Published 19 May 2014

Well, because of breakthroughs in materials science, the TALG is anchored to the seafloor by means of Gossamoor, synthetic silk modeled on the draglines of Darwin’s bark spider, native to Madagascar, which is not only the strongest substance known but weighs about twelve milligrams per thousand meters. And thanks to HydraCorp’s partnership with China Telecom, the anchors parallel the SEA-ME-WE 3 undersea cable that carries data between Mumbai and Djibouti before veering up the Red Sea. And how does the TALG survive the many intrusions of maritime traffic? Well, gentle viewer, it turns out that the segments are programmed to sense oncoming ships and take on seawater, sinking up to thirty meters to let the ship pass, and then pumping the water back out to regain buoyancy.

Daring Raids of World War Two: Heroic Land, Sea and Air Attacks
by Peter Jacobs
Published 29 Jun 2015

On arrival in Australia during April, the captain of the Bonaventure, Captain William Fell, actively sought opportunities for the midgets to be used. Initially, there were no possible targets identified, but the following month Fell was given the opportunity to use his midgets to cut some telegraph cables off Hong Kong. By cutting the undersea cables connecting Singapore, Saigon, Hong Kong and Tokyo, the Japanese would be forced to use radio communications and, therefore, open themselves to message interception. Missions to cut the Hong Kong to Saigon telephone cables were carried out in July 1945 by two midget subs under Operations Sabre and Foil.

Zeitgeist
by Bruce Sterling
Published 1 Nov 2000

“ECHELON is the legendary capitalist global surveillance system. It’s the worldwide signals intelligence directorate! ECHELON is the crown jewel of the antiprogressive Dark Forces!” “Uh, yeah. That would be the alleged phenomenon.” “ECHELON is run by the UK, USA, Australia, and New Zealand. It uses undersea-cable taps, and surveillance satellites like ‘Aquacade,’ ‘Rhyolite,’ and ‘Magnum.’ It taps the Internet through its major routing centers and does comprehensive word searches on e-mail traffic.” “Hey, shut up,” Tim protested over the satellite phone. “That’s all totally classified.” Starlitz put the phone on his shoulder and squinted in the sunlight.

Everything for Everyone: The Radical Tradition That Is Shaping the Next Economy
by Nathan Schneider
Published 10 Sep 2018

AP’s origins lie in unrecorded conspiracies among New York City newspapermen between 1846 and 1848. Their papers were fierce competitors, but they were stuck. If they all used the new telegraph lines separately, there wouldn’t be enough lines. So the competitors had no choice but to cooperate—to import news from Europe via undersea cables, to co-sponsor horse-borne dispatches from the Mexican front, to circulate one another’s reporting. The New York papers shared ownership and control of the enterprise, and they sold their dispatches to other organizations beyond the docks of the Hudson.30 The temptations to hoard this news-gathering resource were immense, and their defeat came only gradually, over the course of a century.

Ten Technologies to Save the Planet: Energy Options for a Low-Carbon Future
by Chris Goodall
Published 1 Jan 2010

One of the crucial reasons this location was chosen, apart from its tidal speeds, was the availability of a nearby connection to the electricity transmission system. One of the problems with tidal energy around the world is going to be connecting the turbines to the power grid. The best tidal locations tend to be far from high-voltage transmission lines and to use the full force of the Pentland Firth, the U.K. will need a new offshore undersea cable running down the east coast to London and on to the rest of Europe, using the same HVDC technology that will bring Saharan solar power to Germany and other countries, as described in Chapter 2. The third high-profile competitor for the wide-open market for tidal power generators is the Irish firm OpenHydro.

The Perfect Police State: An Undercover Odyssey Into China's Terrifying Surveillance Dystopia of the Future
by Geoffrey Cain
Published 28 Jun 2021

“We assess with high confidence that the increasing role of international companies and foreign individuals in US information technology supply chains and services will increase the potential for persistent, stealthy subversions,” the US National Security Agency (NSA), charged with intercepting communications around the world, wrote in a 2010 slide leaked by the whistleblower Edward Snowden.31 Edward Snowden’s leaked documents also revealed that the NSA was tracking twenty Chinese hacker teams attempting to break into US government networks and companies like Google. Huawei was also laying undersea cables that the NSA hoped to intercept, so it could read the communications of Huawei customers that happened to be “high-priority targets,” like Cuba, Iran, Afghanistan, and Pakistan.32 The NSA hacked Huawei’s headquarters, monitored the communications of its top executives, and ran an operation called Shotgiant aimed at finding a link between Huawei and the People’s Liberation Army.

Going Infinite: The Rise and Fall of a New Tycoon
by Michael Lewis
Published 2 Oct 2023

Oddly, the Bahamas hadn’t even made that initial list—­but then Ryan Salame visited to look for a vacation house to buy, and happened to learn that the Securities Commission of The Bahamas was putting the finishing touches on new crypto regulations.§ The Bahamas had great internet, piped in by undersea cable from Florida. It had a neutral tax system that gave you credit for taxes paid to other countries, empty office space, and a truly wild number of empty luxury condominiums waiting to be turned into worker housing. And it was sufficiently hungry for any kind of business that when Sam landed to check out the place he found himself meeting with the newly elected prime minister.

Nomad Century: How Climate Migration Will Reshape Our World
by Gaia Vince
Published 22 Aug 2022

High-voltage direct-current transmission lines will then relay this power to the cities in higher latitudes, as well as to regional cities. Models for such systems already exist – Australia, for instance, is currently building the world’s largest solar battery installation in its northern desert, which will send 24-hour renewable electricity to Singapore via a 4,500-kilometre undersea cable system, to be completed by 2027. Australia already produces far more solar power than it can use or store. North Africa is also sending high-voltage direct current to Europe from its solar arrays, and more regional transmission is promised from hydropower plants to cities. Africa’s deserts are prime for large-scale solar and wind power – Morocco already has the world’s biggest concentrated solar power project, in Ouarzazate, with others under construction.

Moonshot: The Inside Story of Mankind's Greatest Adventure
by Dan Parry
Published 22 Jun 2009

As the Earth spun on its axis, at least one of the three powerful antennas would be in sight of the Moon throughout the mission, with each station handing control to the next to maintain a continuous connection. Voice signals, tracking data and telemetry were passed to Houston via NASA's Communications Network, which linked the ground stations and other facilities through two million miles of landlines and undersea cables. Relying on six intermediate switching centres around the world, and two communications satellites, the network included redundant signal routes to increase reliability. Some remote switching points were triggered automatically whenever the CapCom began to speak. Each time he pressed his microphone button, it emitted a short beep known as a Quindar tone.

The New Harvest: Agricultural Innovation in Africa
by Calestous Juma
Published 27 May 2017

A decade ago it cost more than US$5,000 to install one kilometer of standard fiber-optic cable. The price has dropped to less than US$300. However, for Africa to take advantage of the infrastructure, the cost of bandwidth must decline. Already, Internet service providers are offering more bandwidth for the same cost. By 2011, the four undersea cables operating in Africa had resulted in a quadrupling of data transfer speeds and a 90% price reduction. Access to broadband is challenging Africa’s youth to demonstrate their creativity and African leaders to provide a vision of the role of infrastructure in economic transformation. The emergence of Safaricom’s M-PESA service—a revolutionary 52 THE NEW HARVEST way to transmit money by mobile phones—is an indicator of great prospects for using new technologies for economic improvement.26 In fact, these technologies are creating radically new industries, such as branchless banks, that are revolutionizing the service sector.27 Geographic Information Systems (GIS) The diffusion of geographic information systems is creating new opportunities for development in general and Africa in particular, with regard to agriculture.28 Several countries in Africa have invested in the development of space-based technologies through satellite programs in order to monitor land use and natural resources.

Heart of the Machine: Our Future in a World of Artificial Emotional Intelligence
by Richard Yonck
Published 7 Mar 2017

Shouldn’t it take great sophistication and understanding for one person to read and manipulate another person using such advanced technology? Well, yes and no. It certainly does take a highly sophisticated technology to accomplish all of this, just as it takes a very sophisticated technology to call a person instantaneously on the other side of the planet using an elaborate network of satellites, undersea cables, cell towers, and supercomputers, sometimes known as a smartphone. Most of us don’t give this latter process a moment’s thought because it has been removed from our awareness by layer upon layer of abstraction until we take no more notice of it than we do flicking a light switch. (Which by the way, is also abstracting numerous complex processes most of us never want to be bothered with.)

The God Species: Saving the Planet in the Age of Humans
by Mark Lynas
Published 3 Oct 2011

In the U.S., new wind installations fell by half in 2010 compared to the previous year, thanks to the recession and political wrangling over energy policy (or lack of it), even while the industry went from strength to strength in Europe and Asia.71 One of the big remaining hurdles in America is a straightforward lack of wires: New wind installations often lack grid connections to take their clean electricity to markets. In October 2010 Google Foundation became a partner in the Atlantic Wind Connection, a plan for a high-voltage undersea cable to bring large-scale offshore wind to the U.S. East Coast.72 In the U.K. onshore wind is increasingly being stymied by local opposition: According to one report, at least 230 local anti-wind farm campaigns now operate across the country, and approvals for new wind farms have fallen to an all-time low.73 From a climate change perspective, opposing wind is just as reprehensible as opposing nuclear, and as I discuss later in this book, I do not think any of the planetary boundaries offer compelling environmental reasons for completely ruling out wind—even though problems with bird and bat kills are real and severe in some cases.

The Myth of Capitalism: Monopolies and the Death of Competition
by Jonathan Tepper
Published 20 Nov 2018

Kindle Edition, locations 322–323. 63. https://www.seattletimes.com/business/tech-giants-put-the-squeeze-on-startups-squelching-their-chances-of-success/. 64. https://www.theguardian.com/technology/2017/oct/20/tech-startups-facebook-amazon-google-apple. 65. https://www.recode.net/2017/5/10/15602814/amazon-invested-startup-nucleus-cloned-alexa-echo-show-voice-control-touchscreen-video. 66. https://en.wikipedia.org/wiki/Google_data_centers. 67. https://peering.google.com/#/infrastructure. 68. http://www.dailymail.co.uk/sciencetech/article-5275893/Google-reveals-plan-build-THREE-new-undersea-cables.html. 69. http://www.wired.co.uk/article/google-facebook-plcn-internet-cable/. 70. https://www.ben-evans.com/benedictevans/2017/10/12/scale-wetxp. 71. https://www.theringer.com/tech/2018/5/18/17362452/microsoft-antitrust-lawsuit-netscape-internet-explorer-20-years. 72. https://www.sfgate.com/politics/article/Microsoft-Asked-Apple-to-Knife-the-Baby-Court-2980345.php. 73. https://promarket.org/google-facebooks-kill-zone-weve-taken-focus-off-rewarding-genius-innovation-rewarding-capital-scale/.

Rocket Billionaires: Elon Musk, Jeff Bezos, and the New Space Race
by Tim Fernholz
Published 20 Mar 2018

The internet was not as controversial as resource-extraction concessions or low-wage manufacturing, if only because its power to shape political events was only slowly becoming clear. His second key lesson was that putting wires underground was costly and time-consuming. It was no way to bring the information age to sparsely populated areas or those, like Rwanda, that are far from the undersea cables that link the continents together. Why not just use satellites to do the job of linking the local network to the outside world? In 2007, Wyler cofounded a company called O3b, which stood for the “other three billion”—a shorthand for the underserved chunk of the global population it targeted.

The Scientist as Rebel
by Freeman Dyson
Published 1 Jan 2006

To determine longitude at a remote place such as Dakar or Haiphong, it was necessary to compare the local time, obtained from local astronomical measurements, with Paris time, obtained by receiving an accurate time signal from Paris. The accuracy of maps therefore depended on the accuracy of long-distance transmission of time signals. The transmission of time signals, first by overland telegraph lines and undersea cables, and later by radio, was a difficult technical problem. Signals were attenuated by transmission losses and corrupted by ambient noise. The transmission introduced delays which had to be accurately calculated, so that Paris time could be correctly deduced from the observed time of reception of signals.

Augmented: Life in the Smart Lane
by Brett King
Published 5 May 2016

Today, we know Western Union as a money transmitter but, back in 1856, Western Union was the largest provider of telegraph services across the United States, and by 1890 its reach extended even across the Atlantic. Inflation adjusted, Western Union was capitalised at around US$850 million dollars (US$41 million actual) in 1876. Western Union was operating more than a million miles of telegraph lines and two international undersea cables at the time. When Alexander Graham Bell and his partners5 patented the telephone in 1876, initially referring to it as a “talking telegraph”, they offered Western Union the patent for the sum of US$100,000.6 The established communications monopoly declined their offer, deciding that higher dividends were more important than investment in new technology.

Sandworm: A New Era of Cyberwar and the Hunt for the Kremlin's Most Dangerous Hackers
by Andy Greenberg
Published 5 Nov 2019

Lee, of course, says little about the details of where that intelligence came from. But thanks in part to Edward Snowden, we know that it included a broad array of secret data-gathering tools, labeled broadly as “signals intelligence,” or “sigint,” that ranged from the ability to siphon vast quantities of raw internet data from undersea cables to hacking enemy systems administrators and looking over their shoulders at private networks. “When you’re given access to essentially the entirety of the U.S. sigint system and then surrounded with the smartest people doing this on the planet, you get spun up pretty quickly,” Lee says. For the next four years, he and a small team of around six analysts spent every working hour tracking the burgeoning, post-Stuxnet world of industrial control system hackers.

The Crux
by Richard Rumelt
Published 27 Apr 2022

For now, we think our reputation and customer relationships will carry us, but in the longer term … ? Our development efforts are currently concentrated in Ohio, where we have seventy-five people working on three different efforts. One is drone-based sensors. A second is working on sensors to be integrated into undersea cables, and the third is looking at integrating our sensors into Wi-Fi and Internet environments. Right now, they are wired rather than wireless. The owners brought new blood into the board last year, and … I would like to say it was a breath of fresh air. One of the new board members, ‘John Cherold,’ is from a finance background, and he says XRS is underperforming.

Age of Discovery: Navigating the Risks and Rewards of Our New Renaissance
by Ian Goldin and Chris Kutarna
Published 23 May 2016

From a user perspective, the Internet is an invisible field that connects us everywhere. But this user experience has a physical dimension—data centers and fiber optic cables that concentrate our connectivity in dangerous ways. The same strait and canal chokepoints through which the world’s ships must pass are also the best routes along which to lay undersea cables. Every year in Africa, the networks of whole countries go dark when one of these threads is cut by a passing ship’s anchor—or by an axe. In 2013, the Egyptian coast guard caught three men in a fishing boat who had pulled up, and were trying to hack through, a Suez cable deliberately. Their target, the SEA-ME-WE 4 cable, is one of the principal data links between Europe, Africa and Asia.

Wireless
by Charles Stross
Published 7 Jul 2009

Radio can’t punch through more than a few thousand miles of air directly, however powerful the transmitters; on Earth they used to bounce signals off the ionosphere or the moon, but that doesn’t work here—the other disks are too far away to use as relays. There’s a chain of transceiver buoys marching out across the ocean at two-thousand-kilometer intervals, but the equipment is a pig to maintain, very expensive to build, and nobody is even joking about stringing undersea cables across a million kilometers of seafloor. Misha’s problem is that the expedition, himself included, is effectively stranded back in the eighteenth century, without even the telegraph to tie civilization together—which is a pretty pickle to find yourself in when you’re the bearer of news that will make the Politburo shit a brick.

The Land Grabbers: The New Fight Over Who Owns the Earth
by Fred Pearce
Published 28 May 2012

At least the Purari Development Association had a formal existence when it was granted a special lease for 1.5 million acres. But the purpose raised eyebrows. It plans to construct a giant 1,800-megawatt hydroelectric plant that would flood much of the valley of the Purari River and send power by undersea cable 300 miles across the Coral Sea to Queensland. Some locals in the flood zone said they knew about the project, but others claimed they had “never been consulted or given permission” for the takeover. There seemed to be more local support for a scheme from a South Korean company known as Changhae Tapioka.

The Ultimate Engineer: The Remarkable Life of NASA's Visionary Leader George M. Low
by Richard Jurek
Published 2 Dec 2019

He named some of the recent practical applications in science and medicine that were currently providing major benefits, before he centered on the field of communications. “Breakthroughs are already being made. For example, fifteen years ago, no one would have predicted color television broadcasts from China would have been possible.” He discussed the use of satellites for voice communication, over the antiquated use of undersea cable systems. He discussed solar-energy applications, pointing out that NASA was now exploring “ways of collecting the sun’s energy” for practical use on Earth. Satellites, he said, while showing the audience a picture from space of sewage spills off Long Island, can also spot acid slides and sewage dumping and thus can provide officials with the tools to respond and combat these environmental problems.

The Hacker and the State: Cyber Attacks and the New Normal of Geopolitics
by Ben Buchanan
Published 25 Feb 2020

For espionage purposes, the Chinese have hacked military targets, valuable but obscure offices in the United States, key parts of the computing supply chain, major cloud service providers with gigantic amounts of data, and many other targets.8 The success of companies like Huawei in manufacturing, building, and repairing undersea cables and new 5G telecommunications infrastructure raises fears that the Chinese government may try to match the advantages in passive collection that the Five Eyes enjoy, according to statements of former government officials and leaked documents.9 Russian agencies have likewise continued to expand their cyber espionage repertoire.

Dangerous Ideas: A Brief History of Censorship in the West, From the Ancients to Fake News
by Eric Berkowitz
Published 3 May 2021

Add to that new, wide-ranging prohibitions on sedition—typical were New Zealand’s criminalization of words that “excite disloyalty” and the US law against speech that could bring the government into “disrepute”—and the foundation for what the conservative and later Nazi jurist Carl Schmitt called state intervention in “all spheres of human existence” was laid.39 Winston Churchill, then first lord of the Admiralty, expressed the British government’s view about war reportage when he told editors, “The best place for correspondence about this war will be in London,” and that it should be “fought in a fog.”40 As soon as the fighting began, in August, all the undersea cables surrounding Britain except those that were British-owned were cut, and all communications entering the country were intercepted. Britain’s secretary of state for war, Lord Horatio Kitchener, banned correspondents from the area surrounding the British Expeditionary Force. The hasty passage of the Defence of the Realm Act greatly magnified the state’s power to control information and punish its misuse.

The Ascent of Money: A Financial History of the World
by Niall Ferguson
Published 13 Nov 2007

In the three decades before 1914, trade in goods reached almost as large a proportion of global output as in the past thirty years.10 In a world of less regulated borders, international migration was almost certainly larger relative to world population; more than 14 per cent of the US population was foreign born in 1910 compared with less than 12 per cent in 2003.11 Although, in gross terms, stocks of international capital were larger in relation to global GDP during the 1990s than they were a century ago, in net terms the amounts invested abroad - particularly by rich countries in poor countries - were much larger in the earlier period.12 Over a century ago, enterprising businessmen in Europe and North America could see that there were enticing opportunities throughout Asia. By the middle of the nineteenth century, the key technologies of the industrial revolution could be transferred anywhere. Communication lags had been dramatically reduced thanks to the laying of an international undersea cable network. Capital was abundantly available and, as we shall see, British investors were more than ready to risk their money in remote countries. Equipment was affordable, energy available and labour so abundant that manufacturing textiles in China or India ought to have been a hugely profitable line of business.13 Yet, despite the investment of over a billion pounds of Western funds, the promise of Victorian globalization went largely unfulfilled in most of Asia, leaving a legacy of bitterness towards what is still remembered to this day as colonial exploitation.

Radical Technologies: The Design of Everyday Life
by Adam Greenfield
Published 29 May 2017

Beyond the satellites, camera cars and servers we’ve already identified, the moment-to-moment flow of our experience rests vitally on the smooth interfunctioning of all the many parts of this infrastructure—an extraordinarily heterogeneous and unstable meshwork, in which cellular base stations, undersea cables, and microwave relays are all invoked in what seems like the simplest and most straightforward tasks we perform with the device. The very first lesson of mapping on the smartphone, then, is that the handset is primarily a tangible way of engaging something much subtler and harder to discern, on which we have suddenly become reliant and over which we have virtually no meaningful control.

The Code Book: The Science of Secrecy From Ancient Egypt to Quantum Cryptography
by Simon Singh
Published 1 Jan 1999

Zimmermann Zimmermann had to encrypt his telegram because Germany was aware that the Allies were intercepting all its transatlantic communications, a consequence of Britain’s first offensive action of the war. Before dawn on the first day of the First World War, the British ship Telconia approached the German coast under cover of darkness, dropped anchor, and hauled up a clutch of undersea cables. These were Germany’s transatlantic cables—its communication links to the rest of the world. By the time the sun had risen, they had been severed. This act of sabotage was aimed at destroying Germany’s most secure means of communication, thereby forcing German messages to be sent via insecure radio links or via cables owned by other countries.

The Dawn of Innovation: The First American Industrial Revolution
by Charles R. Morris
Published 1 Jan 2012

The ensuing battle may be the only naval engagement of the war that turned almost entirely on a commander’s thoughtful battle preparation. The Plattsburgh harbor was on a bay sheltered from northerly winds. Macdonough anchored his vessels in a line broadsides out, in a narrow portion of the bay. The details of his positioning gave him two crucial advantages. The first is that he set kedges, or auxiliary anchors, with undersea cabling so crews could winch their ships around to change broadsides without losing anchorage. The second is that the narrowness of his anchorage site helped to neutralize the Confiance’s great advantage in heavy long guns. (Macdonough implicitly counted on the British penchant for attacking without regard to tactical details.)

This Machine Kills Secrets: Julian Assange, the Cypherpunks, and Their Fight to Empower Whistleblowers
by Andy Greenberg
Published 12 Sep 2012

Assange, then using the handle Proff, took note of Mudge’s cleverer hacks and often distributed L0pht’s advisories to the Best of Security group that Assange ran on Suburbia’s servers. But the two hackers had a mutual regard for the other’s alpha geek status that went beyond reading each other’s research, and they exchanged ideas over the undersea cables that connected Melbourne and Boston. More than once, they met in the physical world, including at least one meal together at the Chaos Communication Congress in Berlin. “Everyone knew everyone on IRC,” says one ex-member of the L0pht. “But when Mudge and Assange met at conferences or had dinner, that seemed like a different sort of connection.”

Empire: How Britain Made the Modern World
by Niall Ferguson
Published 1 Jan 2002

By the late 1840s it was clear that the telegraph would revolutionize overland communications; by the 1850s construction in India was sufficiently advanced for the telegraph to play a decisive part in suppressing the Mutiny.* However, the crucial development from the point of view of imperial rule was the construction of durable undersea cables. Significantly, it was an imperial product – a rubber-like substance from Malaya called gutta-percha – that solved the problem, allowing the first cross-Channel cable to be laid in 1851 and the first transatlantic cable to follow fifteen years later. When the Anglo-American Telegraph Company’s cable finally reached the American coast on 27 July 1866, having been successfully unrolled and dropped along the ocean floor by Isambard Kingdom Brunel’s mighty Great Eastern, it was plainly the dawn of a new era.

The Death of Money: The Coming Collapse of the International Monetary System
by James Rickards
Published 7 Apr 2014

In a surprising coda to the revelations of U.S. spying on allies emerging from defector Edward Snowden, Brazil announced plans in September 2013 to build a twenty-thousand-mile undersea fiber optic cable network from Fortaleza, Brazil, to Vladivostok, Russia, with links in Cape Town, South Africa, Chennai, India, and Shantou, China, to be completed by 2015. This system is tantamount to a BRICS Internet intended to be free from U.S. surveillance. The United States has long had excellent capability in tapping into undersea cables, so the actual security of the new system may be problematic. Nevertheless, the proprietary nature of this system could easily be adapted to include a BRICS interbank payments system, which would facilitate the use of any BRICS-sponsored alternatives to dollar payments. In addition to the regular meetings of BRICS leaders, a large number of ancillary and shadow institutions have sprung up around the BRICS, including the BRICS Think Tanks Council, the BRICS Business Council, and a BRICS virtual secretariat, among others.

Smart Cities: Big Data, Civic Hackers, and the Quest for a New Utopia
by Anthony M. Townsend
Published 29 Sep 2013

Founded in 1984 by husband and wife Len Bosack and Sandy Lerner, who built Stanford University’s campus network in the early 1980s, Cisco has grown into the world’s leading supplier for the sophisticated switches and routers that power the Internet. Cisco’s products not only push bits around offices, schools, and homes, but also sling them back and forth across undersea cables that link continents. It’s one of Silicon Valley’s largest and most-watched bellwethers. For a brief period in March 2000, at the height of the telecom bubble, it was the most valuable company in the world. But with size comes stagnation. Finding growth opportunities has become a constant struggle for Cisco, and to make a dent on the bottom line it needs to have billion-dollar payouts.

The Turing Option
by Harry Harrison and Marvin Minsky
Published 2 Jan 1992

He tapped the telephone on his belt. "I have my own number for this. About most of the time it rings wherever I am in the continental United States." "Is it a satellite link?" "No, satellite connections are too slow for most uses— particularly telepresence. Everything is fiber optics now— even the undersea cables. Cheap and fast. With plenty of room for communication with eight thousand megahertz band-width capacity available everywhere—and all of it two-way." Brian nodded. "I get your drift, Ben. What you're saying is that there is very little chance that I had a local mechanical backup. It was undoubtedly an electronic one.

Mapping Mars: Science, Imagination and the Birth of a World
by Oliver Morton
Published 15 Feb 2003

The once imperial cityscape is woven from threads that stretch throughout the world. Across the river to the east sits the squat black-glass bulk of Reuters, information from around the globe splashing into its rooftop dishes. Upstream and on the near side sit the long, low workshops where for more than a century men have made undersea cables to tie the continents together. New skyscrapers devoted to global businesses sit in the redeveloped heart of the docks that used to handle the lion’s share of the world’s sea trade. Within the park itself there are plants from every continent except Antarctica. At its foot sits the old naval college, where generations of Britannia’s officers, my late father included, learned to rule the waves.

Jihad vs. McWorld: Terrorism's Challenge to Democracy
by Benjamin Barber
Published 20 Apr 2010

Ignacio Ramonet argues that in the global economy neither capital nor work nor material is the determining factor, but rather the “optimal relationship between these three,” which pushes us into the world of information, communication, and administration where traditional nation-states can exert little control and are bound to feel more and more uncomfortable.13 Robert Kuttner reports that the state-of-the-art handle for the postindustrial company—which clearly is also the post—nation-state company—is “the virtual corporation” where “the company is no longer a physical entity with a stable mission or location, but a shifting set of temporary relationships connected by computer network, phone and fax.”14 McWorld is a kind of virtual reality, created by invisible but omnipotent high-tech information networks and fluid transnational economic markets, so the virtual corporation is not just a provocative turn of phrase.15 Without even trying, reporter Julie Edelson Halpert gives it concrete meaning in the portrait she draws of Ford Motor Company’s Mondeo project: Seeking to shave months and millions of dollars from car design, Ford has consolidated management of its European, North American and Asian design operations into a single international network using powerful work stations based on Silicon Graphics Inc. technology linked by Ethernet networking software. …The Ford system … was brought under a single “electronic roof” … based in Dearborn, Michigan. The other main sites on the network are in Dunton, England; Cologne, Germany; Turin, Italy; Valencia, California; Hiroshima, Japan; and Melbourne, Australia. The circuits—satellite links, undersea cables and land lines—are purchased from telecommunications carriers.16 The virtual corporation also exists in the labor market as an employer of “virtual” rather than actual laborers. The ideal virtual laborer is a robot: an interactive, “intelligent,” fully programmed worker capable of working twenty-four hours a day with no sustenance and minimal upkeep.

Ways of Being: Beyond Human Intelligence
by James Bridle
Published 6 Apr 2022

It was the same sensation I experienced when beginning to understand, and bring into view, the infrastructure of the internet: the vast, planet-spanning network of cables, wires, machines and electromagnetic signals which sustains and regulates humanity today. These microprocessors and data centres, undersea cables and wireless transmissions are our own mycorrhizal network, interpenetrating everyday life, managing our supplies and demands, disturbing the climate and touching upon our own skin – even, in the case of cochlear implants, pacemakers, and perhaps soon neural links, breaching it. It speaks volumes about our lives today that when we hear about networks and communication it is these artificial networks which we think of first.

Connectography: Mapping the Future of Global Civilization
by Parag Khanna
Published 18 Apr 2016

We are in only an early phase of reengineering the planet to facilitate surging flows of people, commodities, goods, data, and capital. Indeed, the next wave of transcontinental and intercontinental mega-infrastructures is even more ambitious: an interoceanic highway across the Amazon from São Paulo to Peru’s Pacific port of San Juan de Marcona, bridges connecting Arabia to Africa, a tunnel from Siberia to Alaska, polar submarine cables on the Arctic seabed from London to Tokyo, and electricity grids transferring Saharan solar power under the Mediterranean to Europe. Britain’s exclave of Gibraltar will be the mouth of a tunnel under the Mediterranean to Tangier in Morocco, from which a new high-speed rail extends down the coast to Casablanca.

…

Apple’s iOS 8 and the latest Android both feature encryption protocols that no longer allow any access to user data—preventing not only the U.S. government but also hackers (particularly from China) who have exploited previous versions’ back doors from accessing it.*5 The Internet’s earliest origins lie in efforts to create redundant communications in the event of enemy attack. Today it is becoming a network that can withstand any rupture, whether physically uprooted submarine cables or digitally disrupted services. The Internet now exists independently of the governments that created it; they function within cyberspace, not the reverse. The militarization of cyberspace thus does not prevent the Internet from remaining a universe of voluntary association, online commerce, and competition for mind share.

…

* * * *1 Web user data, long stored in digital silos, is now being harvested for sale through increasingly sophisticated services such as Facebook’s Atlas that allow advertisers to track users’ digital footprint across mobile, tablet, and other channels to target ads according to device. *2 There are already more mobile phone subscriptions in the world than people alive because many people in cities such as Dubai and Hong Kong have several accounts each. *3 Many submarine Internet cables have been laid along the same strategic sea lines of communication where the British navy installed telegraph cables in the late nineteenth century. The telegraph was the first communications system decoupled from transportation. The Economist’s technology editor Tom Standage cleverly termed the telegraph the “Victorian Internet

Debt of Honor
by Tom Clancy
Published 2 Jan 1994

I want the COs to drill their people up. You're never ready enough, Ron. Never." "True," the civilian conceded. He'd come over with SOSUS printouts to demonstrate that all known submarine contacts were off the screen. Two hydrophone arrays that were operated from the island of Guam were no longer available. Though connected by undersea cable to the rest of the network, they'd evidently been turned off by the monitoring facility on Guam, and nobody at Pearl had yet been able to trick them back on. The good news was that a backup array off Samar in the Philippines was still operating, but it could not detect the Japanese SSKs shown by satellite to be replenishing off Agana.

…

"I hope it's not about Wall Street," Jack replied in as unguarded a voice as he could muster. "No, it's about Guam. Why are the phone lines out?" "Bob, did you ask the phone company that?" Ryan tried. "Yeah. They say there was a storm that took a lot of lines down. Except for a couple of things. One, there wasn't any storm. Two, there's an undersea cable and a satellite link. Three, a week is a long time. What's going on?" the reporter asked. "How many people are asking?" "Right now, just me and a TV station in Little Rock that put a request up on the AP wire. Another thirty minutes and it's going to be a lot more. What gives? Some sort of—" "Bob, why don't you come on down here," Ryan suggested.

Industry 4.0: The Industrial Internet of Things
by Alasdair Gilchrist
Published 27 Jun 2016

For example, Microsoft Azure provides support for .NET applications and Google provides tools to support its own in-house applications such as Big Data tools and realtime stream processing. From a network perspective the major cloud providers,Amazon, Microsoft and Google provide potentially millions of concurrent connections, and Google run their own fiber optic network, including their own under-sea cables. The cloud is a huge enabler for the Industrial Internet as it provides the infrastructure and performance that industry requires but is at the same time financially compelling. However, there is one slight problem. Latency, which is the time it takes data to be transmitted from a device and then be processed in the cloud, is often unacceptable.

…

Some data, such as M2M sensors, will require in-flight or in-memory analysis; other data may be stored and later analyzed once in Hadoop. An example of a long-standing usage for high-velocity analysis is stock market and financial data. Financial institutions and banks have been analyzing this type of data at velocity, even going to the lengths of running a private submarine cable between exchanges in London and New York in order to shave a millisecond of the handling time of this valuable high-velocity Big Data. Data velocity in the IIoT context, or streaming data as it is known, requires real-time or close to real-time as possible handling and analysis. This constraint puts additional pressures on the data storage and handling systems.

Civilization: The West and the Rest
by Niall Ferguson
Published 28 Feb 2011

The opening of the Suez Canal (1869) and the Panama Canal (1914) shrank the world still further, the former reducing the distance of the London–Bombay route by more than two-fifths, the latter cutting the cost of shipping from the East to the West Coast of the United States by a third.48 By the late 1860s, thanks to the introduction of gutta-percha coating, undersea cables could be laid and telegrams sent from London to Bombay or to Halifax.49 News of the Indian Mutiny had taken forty-six days to reach London in 1857, travelling at an effective speed of 3.8 miles an hour. News of the huge Nobi earthquake in Japan in 1891 took a single day, travelling at 246 miles an hour, sixty-five times faster.50 Labour flowed across borders as never before.

Exploding the Phone: The Untold Story of the Teenagers and Outlaws Who Hacked Ma Bell
by Phil Lapsley
Published 5 Feb 2013

.: See Fettgather, Jim Whistle/whistling 2111 conference 163–164, 202, 206 Cap’n Crunch 154–155, 166, 202, 209, 326 Draper, John 153, 165–166, 209 Engressia, Joe 122–126, 130, 132, 173, 326, 335 Pyne, Charlie 77 Trips 154 Wozniak, Steve 218 See also: Davy Crockett Cat and Canary Bird Call Flute, Tonette Flute White House 800 number 229, 268, 280, 385–386, 393–394 Hotline 254 Secret telephone system 127–128 See also: Ford, Gerald; Nixon, Richard White Plains Tandem 2 175–176, 203, 207 WHRB (Harvard student radio station) 68–70, 72–73, 77 Wiretapping Ashley-Gravitt scandal 303–304, 306 Bookies 101 Engressia 127–128, 131 FBI 249–261, 272, 278 Legality 109–115, 288 Soviet undersea cables 182 Verification/blue box 249–261 White House 127–128, 229 See also: Greenstar; laws, 47 USC 605 Wink 54, 175, 371 Woolworth’s 29–30 Wozniak, Steve 218–225, 229, 243–244, 281, 309, 312–313, 333, 335 XBT: See crossbar tandem YIPL: See Youth International Party Line Yippies: See Youth International Party Young Engineering 66 Young, Shane 326 Youth International Party 185–186, 197, 214, 257 Youth International Party Line 185–186, 192–193, 197–200, 213–214, 230, 238, 240, 330–331 See also: Technological American Party Zzzzyzzerrific Funline 328 The Playground Table of Contents Cover Halftitle Title Copyright Dedication Contents Epigraph Foreword A Note on Names and Tenses Chapter One Chapter Two Chapter Three Chapter Four Chapter Five Chapter Six Chapter Seven Chapter Eight Chapter Nine Chapter Ten Chapter Eleven Photo Insert Chapter Twelve Chapter Thirteen Chapter Fourteen Chapter Fifteen Chapter Sixteen Chapter Seventeen Chapter Eighteen Chapter Nineteen Chapter Twenty Chapter Twenty-one Epilogue Sources and Notes Acknowledgments Index The Playground

Confessions of a Wall Street Analyst: A True Story of Inside Information and Corruption in the Stock Market
by Daniel Reingold and Jennifer Reingold
Published 1 Jan 2006

Gary alone cashed in over $700 million over the course of four years, second only to Phil Anschutz—who pocketed $1.9 billion on Qwest—as the industry’s champion roulette player.1 But the bankruptcy filing was particularly tragic in that the promise of this new world had been exposed as nothing more than hype. And even those of us who had questioned the hype had not questioned it enough. It was this century’s equivalent of the railroad industry boom and bust in the late 1880s. Customers might use all of those undersea cables one day. But not anytime soon. Global’s collapse came in the shadow of Enron’s sudden meltdown the previous December, which had taken the entire financial world by surprise—the auditors, the regulators, and the Street. Now everyone was scrambling to figure out who had left the barn door open and how.

Pacific: Silicon Chips and Surfboards, Coral Reefs and Atom Bombs, Brutal Dictators, Fading Empires, and the Coming Collision of the World's Superpowers
by Simon Winchester
Published 27 Oct 2015

The system has numberless branches and offshoots, and were its waters to be drained from the ocean and the planet dried out, its ridges would look like a web of fibers somehow helping to hold the earth together, like the stitches on a baseball or the sutures on a skull. Its existence was confirmed only recently, although there were suggestions as early as Victorian times that lines of unexpected shallows were to be found out in the mid-ocean deeps. HMS Challenger, surveying the Atlantic in 1872 to find the optimal route for an undersea cable, found that depths in the middle of the ocean were many thousands of feet less than expected, and a century later, German oceanographers noticed that this same upwelling continued around the African coast right up past Madagascar. Yet Albert Bumstead’s classic 1936 National Geographic map of the Pacific (the same one used, as mentioned, by Colonel Charles Bonesteel III to divide postwar Korea) still gives no clue that anything similarly significant had yet been noticed on the Pacific side of the world.

Countdown to Zero Day: Stuxnet and the Launch of the World's First Digital Weapon
by Kim Zetter
Published 11 Nov 2014

To track Hassan Ghul, an associate of Osama bin Laden who was killed in a drone strike in 2012, the NSA deployed “an arsenal of cyber-espionage tools” to seize control of laptops, siphon audio files, and track radio transmissions—all to determine where Ghul might “bed down” at night, according to Snowden documents obtained by the Washington Post.30 And since 2001, the NSA has also penetrated a vast array of systems used by al-Qaeda associates in Yemen, Africa, and elsewhere to collect intelligence it can’t otherwise obtain through bulk-data collection programs from internet companies like Google and Yahoo or from taps of undersea cables and internet nodes. Terrorism suspects aren’t the NSA’s only targets, however. Operations against nation-state adversaries have exploded in recent years as well. In 2011, the NSA mounted 231 offensive cyber operations against other countries, according to the documents, three-fourths of which focused on “top-priority” targets like Iran, Russia, China, and North Korea.

Chaos Monkeys: Obscene Fortune and Random Failure in Silicon Valley
by Antonio Garcia Martinez
Published 27 Jun 2016

Every time you load a new page in Facebook, not to mention most of the Internet, optical signals crisscross the globe to hundreds of waiting machines, all announcing, like some phalanx of royal trumpeters, your impending arrival. So what happens to those globe-spanning bid requests, assuming they don’t drown in their trans-Atlantic undersea cable? The people listening for your presence in billions of such requests per day are known as “demand-side platforms” (DSPs), and they’re the stockbrokers of this real-time media world, working in the employ of the advertiser or agency who wants to sell you something.* The DSP quickly unpacks the bid request, and queries its data for anything it knows about you: sites you’ve browsed, shopping carts you’ve abandoned, that airfare quote you got and never acted upon.

Dark Mirror: Edward Snowden and the Surveillance State
by Barton Gellman
Published 20 May 2020

Internet Companies in Broad Secret Program,” Washington Post, June 6, 2013, http://wapo.st/1LcAw6p, archived at https://archive.is/cYyFe. piece of the backbone: What is known as the backbone of the global communications network is a complex infrastructure with major components including trunk lines of fiber optic cable, high-capacity switches known as core routers, cable landing stations where undersea cables join terrestrial networks, and Internet Exchange Points. The majority of worldwide telephone and internet traffic passes to, from, or through U.S. infrastructure. $394 million: FY 2013 Congressional Budget Justification, Volume 1, National Intelligence Program Summary, on file with author.

The Deepest Map
by Laura Trethewey
Published 15 May 2023

In the best maps held by Seabed 2030, thin survey lines connect the continents, lighting up a patch of seafloor in an ocean of darkness. Those well-charted routes mostly follow shipping lanes used by international cargo fleets, which carry more than 95 percent of world trade, or they indicate the surveyed seafloor where submarine cables, which carry over 90 percent of the world’s internet traffic, lie.2 In certain places, such as the Gulf of Mexico off Louisiana and the Gulf of Sidra near Libya, the seafloor has been thoroughly sounded for fossil fuels buried beneath the primordial muck. But the majority of the international seafloor is shrouded in darkness: the world’s last great mystery.

…

It’s also a strategic advantage for a country to hint that it has more intelligence than it actually does. In the ocean-mapping world, where the military objectives of global powers often lurk behind expensive offshore operations, it’s best to question every story you hear. Like when Bell Laboratories built a telegraph cable across the North Atlantic—and a secret enemy submarine surveillance cable for the US Navy—back in Bruce Heezen and Marie Tharp’s day. Or the military funding of Bob Ballard’s search for the Titanic—as long as he found missing nuclear submarines first. Or Howard Hughes’s Glomar Explorer expedition to mine the Pacific seafloor, which was actually a covert operation to investigate Russian nuclear submarines.

…

See also Seabed 2030 Project autonomous mapping, 150 beauty of seafloor maps, 38–39, 86 confidentiality of, 56, 70–71, 83, 88, 103–4, 119, 123–27, 153 crowdsourcing and, 4, 127–29, 140–46, 149–50 difficulties of, 26–27, 39, 94 drones used for, 5, 150, 151–52, 159–62 history of, 5, 9, 10, 18–20, 25–26, 45, 55–61, 63, 65–66, 69–76, 103, 195 media coverage of, 62–63 multibeam sonars used for, 5, 15–16, 19, 20–21, 30, 35, 112 naming of seafloor features, 5, 106, 107–15 National Geographic map, 65–66 nautical charts distinguished from, 3n ocean conditions and, 4–5, 33–34, 35, 36, 38 perceptions of, 90, 238 political complications of, 5, 106 David Sandwell as pioneer of, 18 satellite predictions and, 3, 17, 18–19, 100, 140–41 seafloor terrain knowledge base and, 16 sediment sampling and, 1 submarine cables and, 9, 69–70 technology and tools for, 5–6, 15–16 territorial waters and, 9, 104 Marie Tharp’s expertise in, 69–71, 76, 77–81, 83–85, 87–88, 91, 112, 117, 124, 138, 161, 213–14 tragedies at sea and, 25–26 water depth and, 19, 26–27, 35, 112 women’s role in, 76–77 seafloor spreading theory, 81–82 seamounts deep-sea mining and, 201, 209 naming of, 5 satellite mapping of, 19 seafloor mapping and, 104, 105 water piling up on top of, 17 sea surface topography, 18 sediment plumes, 204–5 self-contained breathing apparatus (SCBA), 22 Shackleton, Ernest, 14, 92, 156, 169 Shell Ocean Discovery XPRIZE, 61 shipwrecks, on seafloor, 25, 32, 146, 169–71, 183–85, 189–90, 237–38 side-looking airborne radar (SLAR), 240 Smith, Craig, 214–15 Smith, Morgan, 171–76, 179–81, 183–89 Smith, Walter, 18–19, 124–25 Smith, William, 163 Snaith, Helen, 115–19, 123, 164 Solutrean hypothesis, 179 South China Sea, 86, 104, 114, 119–23, 146 Southern Ocean.

The Battery: How Portable Power Sparked a Technological Revolution
by Henry Schlesinger
Published 16 Mar 2010

Any reasonable man would have faded tactfully into obscurity. He published a self-serving book, wrote letters, and gave interviews placing the blame on everyone but himself. With Whitehouse out of the picture, the project was revitalized when Thomson’s theories were proven in other parts of the world with other submarine cables. New money was raised and a second attempt was in the planning stages even as the Civil War raged. The deathblow finally fell on what little remained of Whitehouse’s credibility when a new cable was stretched across the Atlantic. The redesigned cable, completed just months after the war’s end—with two sections turned in the same direction—differed significantly from the first.

…

At one point, telegraphing Grant, he urged, “Hold on with a bull-dog grip, and chew and choke, as much as possible.” IT IS DIFFICULT TO OVERESTIMATE the speed at which telegraph networks spread across the landscape. Within thirty years of Morse’s demonstration in 1844, there were some 650,000 miles of cable and 30,000 miles of submarine cable linking more than 20,000 towns and villages. By 1880 there were an estimated 100,000 miles of undersea wiring connecting continents. The world was becoming smaller. With the success of the telegraph, electricity was becoming a part of everyday life. Technology that was not readily understood by the “man in the street” was slowly integrating itself into his landscape.

Unfinished Empire: The Global Expansion of Britain
by John Darwin
Published 12 Feb 2013

Maintaining a wide correspondence was commercially vital: the business partner of John Gladstone (the prime minister’s father) thought nothing of sending off twenty letters or more after a long day in the saddle.15 John Gladstone himself covered 14,000 miles in 1791–2 speculating in American corn.16 At home and abroad, merchants clustered together to listen and gossip, count the ships in the harbour, tremble at rumours and watch each other’s activity. They formed chambers of commerce to pool the information that they were willing to share, and established ‘exchanges’ where produce prices were listed. From the 1850s the spreading network of telegraphs and submarine cables allowed vital price information to be gathered with revolutionary speed: a transatlantic cable was laid in 1866; by the 1870s India, Southeast Asia and even Australia were connected to London. Even so, in the City of London as everywhere else, much commercial intelligence spread by word of mouth, as it had always done.

…

London was the marketplace to which a great mass of produce was sent, partly because Britain imported so much food and raw materials for its industry, partly because of free trade, which attracted goods that would later be re-exported to other markets in Europe. A throng of specialist dealers and merchants examined, repackaged, auctioned off and sold on the goods that arrived. But many commercial transactions in London concerned goods that never reached it or any other British port. 10. British submarine cable system, 1929 This was because an enormous share of world trade used what was known as the ‘bill on London’ to reduce the inconvenience, cost and delay inseparable from purely bilateral trade. If a merchant in country A wished to buy goods from country B, he might find it difficult to strike an acceptable bargain.

…

Areas of white settlement in New Zealand by 1860 5. The Ashanti War, 1874: Wolseley’s march to Kumasi 6. Zululand in 1879: The line of British advance 7. The British invasion of Egypt, 1882 8. Atlantic commerce in the eighteenth century 9. Principal British trade routes and commodities, 1923 10. British submarine cable system, 1929 11. British foreign investment to 1914 12. India, 1858–1947 13. India in 1857 14. The Royal Navy and its stations, 1875 and 1898 15. Distribution of British troops, 1881 Notes ABBREVIATIONS CAB Cabinet Office Records in the National Archives CO Colonial Office Records in the National Archives PP British Parliamentary Papers PREM Prime Ministers’ Records in the National Archives TNA The U.K.

Platform Capitalism
by Nick Srnicek
Published 22 Dec 2016

In 1980 the level of annual investment in computers and peripheral equipment was $50.1 billion; by 1990 it had reached $154.6 billion; and at the height of the bubble, in 2000, it reached an unsurpassed peak of $412.8 billion.16 This was a global shift as well: in the low-income economies, telecommunications was the largest sector for foreign direct investment in the 1990s – with over $331 billion invested in it.17 Companies began spending extraordinary amounts to modernise their computing infrastructure and, in conjunction with a series of regulatory changes introduced by the US government,18 this laid the basis for the mainstreaming of the internet in the early years of the new millennium. Concretely, this investment meant that millions of miles of fibre-optic and submarine cables were laid out, major advances in software and network design were established, and large investments in databases and servers were made. This process also accelerated the outsourcing tendency initiated in the 1970s, when coordination costs were drastically cut as global communication and supply chains became easier to build and manage.19 Companies pushed more and more of their components outwards and Nike became an emblem of the lean firm: branding and design were managed in the high-income economies, while manufacturing and assembly were outsourced to sweatshops in the low-income economies.

Dark, Salt, Clear: Life in a Cornish Fishing Town
by Lamorna Ash
Published 1 Apr 2020

I stare at the variously whirring machines sending out readings that give a rough, brightly coloured sense of the world continuing beneath us. The Ground Definition Machine draws marks across the screen that resemble a red and green millipede slinking along the ground. If its body suddenly contorts upwards, suggesting the seabed rises dramatically, there is likely to be something large below us, such as an old anchor or submarine cables that we are in danger of pulling up in our nets, or, worse, the object could yank down one of the huge derricks and capsize us. On a trawler you are potentially only ever seconds away from catastrophe. Don guides the Filadelfia along the course he has set using the compass that he turns by minute degrees, following the pink line mapped out along the AIS, the Automatic Identification System – a tracking device that plots the voyage of every water-bound vessel around the world.

…

Instead, the ships that laid the cables along the ocean bed had to begin their work all over again. With the coming of telegraphy, Porthcurno, near Land’s End, suddenly became a thriving cosmopolitan area. Skilled men and women from cities around the UK were sent down there to work at the telegraph station, where many of the world’s submarine cables came ashore. The Cornish were somewhat suspicious of these telegraph stations that cropped up around the coast, attracting hordes of foreigners. In 1904, Cornish Evening Tidings reported a belief circulating amongst those living on the Lizard that the reason they had been suffering such awful weather recently was because of Marconi’s telegraph station in Mullion, declaring the high level of electricity coursing through their villages was ‘plainly an uncanny presence and capable of the worst, in the eyes of all good fishermen, who do not believe in upsetting the laws of nature’.

In the Plex: How Google Thinks, Works, and Shapes Our Lives
by Steven Levy
Published 12 Apr 2011

But Baidu’s biggest boost came from the Chinese government. The government would often slow or block Google’s service and at one point even redirected Google traffic to Baidu. An apparent whispering campaign attributed the problem to Google’s alleged ineptness in serving China. The most charitable fiction was that an undersea cable had broken, cutting off service from the United States. Google had hoped that its decision to create a search engine in the .cn domain—one that followed government rules of censorship—would lead to a level playing field. But even as Google had rolled out its .cn web address, there were indications that its compromise would not satisfy the Chinese government.

Lonely Planet Iceland (Travel Guide)
by Lonely Planet , Carolyn Bain and Alexis Averbuck
Published 31 Mar 2015

An Energetic Future Tourism now exceeds fishing as Iceland's dominant industry, and has helped Iceland bounce back from its banking crisis. But to ensure that prosperity continues, even if the tourism boom comes to a crashing halt, Iceland is shoring up its position as a green-energy superpower, looking to export its know-how (and quite possibly its actual energy, transmitted via undersea cables) to foreign shores. It's wooing more big-business energy users, to try to convince them to set up shop (large aluminium smelters are already here for the cheap, abundant power). Public and private enterprises are exploring the options for producing 'green fuels' such as biomethane and biodiesel.

The Long Game: China's Grand Strategy to Displace American Order
by Rush Doshi
Published 24 Jun 2021

•Provide Select Alternative Financing with Allies and Partners: The United States cannot and should not provide a counter to every project Beijing chooses to support. Working with allies and partners, the United States should fund alternatives to those projects that have the greatest strategic potential (e.g., dual-use port projects, undersea cables, airfields) or work to multilateralize Chinese funding to ensure the United States has a seat at the table. •Counter Chinese Technology Acquisition and Theft: China has sought to “catch up and surpass” the West in technology, and while some of this effort is driven by domestic investment and research, a significant portion also seeks to exploit the openness of US financial markets, universities, and companies to accelerate China’s technological programs.

This Is How They Tell Me the World Ends: The Cyberweapons Arms Race
by Nicole Perlroth
Published 9 Feb 2021

CHAPTER 7: THE GODFATHER Gosler’s favorite Price Pritchett quote can be found in his 1994 book, The Employee Handbook of New Work Habits for a Radically Changing World: 13 Ground Rules for Job Success in the Information Age. I relied on Matthew Carle’s 2013 retrospective on Operation Ivy Bells to describe the undersea cable tap: “40 Years Ago, The Navy’s ‘Operation Ivy Bells’ Ended With a 70s Version of Edward Snowden,” published by Business Insider. A history of Los Alamos is available via the U.S. Department of Energy’s Office of History and Heritage Resources in “The Manhattan Project: An Interactive History.”

After Tamerlane: The Global History of Empire Since 1405
by John Darwin
Published 5 Feb 2008

Mechanized transport on land and sea projected military force over far greater distances and (in pre-industrial terms) at almost lightning speed, allowing comparatively small numbers of soldiers to be deployed and redeployed in campaigns that might be thousands of miles apart. British troops could be shuttled between South Africa, India, China and even New Zealand. Steam-engined troopships and ‘strategic’ railways, like those built in India after 1860, economized on the size of the garrisons needed to safeguard foreign control. The electric telegraph and the submarine cable played a similar role, enabling instructions to be sent, warnings to be issued, help to be summoned in a matter of hours not weeks. Information became the invisible weapon in the European arsenal, worth thousands of troops and millions of pounds. The other great benefit that industrialism gave was also a product of speed.

…

By the 1870s they were colonizing vast new areas of the world, carving corridors of access into regions where travel had been difficult (and costly) and information scarce. The opening of the Suez Canal in 1869 meant the extension eastward of the steamship lines and their scheduled services, creating the great trunk road of merchant shipping all the way to Shanghai and Yokohama. Submarine cables and the overland telegraph could nowbring commercial and political news from East Asia to Europe within days, and then hours. But most of all it was the railway that revolutionized thinking about distance. The late nineteenth century was the greatest age of railway imperialism. The British and French built colonial railways in West, East and Southern Africa to bind restless hinterlands to their coastal bridgeheads.

…

The main driving force behind the newglobal economy was the great improvement in transport and its extension worldwide. Between 1869, when the Suez Canal opened, and 1914, when the Panama Canal was finished, much of the rest of the world was drawn into the nexus of communications that already existed between Europe and America. Steamships, railways and the electric telegraph, with its submarine cables, nowgirdled the earth. After 1900, the extension of this network to include every productive region seemed only a matter of time. The exchange of products on an ever-growing scale encouraged specialization and mutual economic dependence. As trade flows increased, they seemed to promote an inexhaustible appetite for more exotic products.

A Fiery Peace in a Cold War: Bernard Schriever and the Ultimate Weapon
by Neil Sheehan
Published 21 Sep 2009

Helena, where Napoleon was exiled to die. Ascension was to be the preliminary terminal point for Atlas and Titan. Twelve modified Second World War cargo ships, fitted out with appropriate gear, put to sea whenever there was to be a launch, filling in the gaps, particularly between Antigua and Ascension in the South Atlantic. Undersea cables tied many of the stations back to Cape Canaveral, providing instantaneous and secure communications. The radar at the stations, while sophisticated and precise, fulfilled the relatively simple task of tracking the missile. The radio monitors had a more complicated job. They had to pick up and record the telemetry from the sensors fitted into the missiles.

The New Gold Rush: The Riches of Space Beckon!
by Joseph N. Pelton
Published 5 Nov 2016

The builder assumed ownership of the “Africastar” satellite when it ran into financial difficulties, and the other satellites for the rest of the world were not deployed . Fixed Satellite Services (FSS) The first commercial satellite system was envisioned as a single integrated global system that would provide new international links across the oceans. As of the 1950s and 1960s there were coaxial submarine telephone cables that were able to provide international telephone connections. But the largest submarine cable had a capacity of 36 voice circuits. The technology then known as Time Assignment Speech Interpolation (TASI ) was able to double this capacity by assigning channels only when people were talking. This created a quite annoying clipping effect.

Uncommon Grounds: The History of Coffee and How It Transformed Our World
by Mark Pendergrast
Published 2 Jan 2000

With prices continually rising throughout the 1860s and 1870s, coffee monoculture seemed a sure way to riches.8 In 1867 the first Santos railway to a coffee-growing region was completed. In the 1870s the new coffee men, the Paulistas of São Paulo, pushed for more technological change and innovation—primarily to advance the sale of coffee. In 1874 the new submarine cable facilitated communication with Europe. By the following year 29 percent of the boats entering Brazilian harbors were powered by steam rather than sail. In 1874 there were only 800 miles of track; by 1889 there were 6,000 miles. The lines typically ran directly from coffee-growing regions to the ports of Santos or Rio.

…

At this point, however, Sielcken was the hero who saved the market, bidding the price back up to 17 cents. The Most Speculative Business in the World By century’s end, technology had made worldwide communication virtually instantaneous. Coffee exchanges in major European ports corresponded rapidly with New York. “Silently the submarine cable ticks off the news that supple fingers chalk and print of steamers leaving Rio and Santos on certain days, and with what cargoes,” wrote Richard Wheatley in 1891. Traders could ascertain the stocks of coffee at eight principal European ports in each month of the past two business years. “These facts and conditions, comparisons and symptoms, of the world’s commerce in coffee daily, weekly, and yearly, are under the eye of the broker,” Wheatley continued, “and guide his judgment in the contracts made so explosively on the floor of the Exchange.”

A Crack in the Edge of the World
by Simon Winchester
Published 9 Oct 2006

And the motion-picture industry was on the verge of being properly born by 1901, thanks to inventions being made at an almost feverish rate by the Edison Company and to the opening of scores of what were known as Film Exchanges, which led in turn to the opening of public cinemas (and which also allowed the San Francisco Earthquake to become the first natural disaster to be captured in moving pictures; hundreds of minutes of disaster film can still be found in archives). Marconi sent his first Morse Code signal by radio across the Atlantic that same year. The Atlantic Ocean had long before been crossed by telegraph wires; in 1903 it was to be the turn of the Pacific, which acquired its first submarine cable in the same year that Orville and Wilbur Wright launched their tiny biplane beside the sea at Kitty Hawk. A message sent from the White House took only twelve minutes to come back to the transmitting telegraphers, having circumnavigated the planet via a now fully connected skein of cables, and at what was then a barely imaginable speed.

…

Some, such as the huge quakes of central New Hampshire and Cape Ann, Massachusetts, are only dimly recorded and only in the geological record, since they happened long before there were Americans on hand to write accounts. More recently there are records from sudden single events that puzzle and perplex to this day: the Grand Banks Earthquake of 1929 (which broke submarine cables and spawned waves that killed dozens in the outports of Newfoundland); and three still notorious New York state events, Massena in 1944, Goodnow in 1983, Ardsley in 1985. They are sporadic, intermittent, unpredictable – and as a consequence they are much more dangerous to those living innocently by them than are the more generally expected earthquakes of California, where the cities make sure that buildings are made super-strong and the public is kept endlessly aware.

Free Speech: Ten Principles for a Connected World
by Timothy Garton Ash
Published 23 May 2016

For all the wonders of wireless, at some point our communications, broken down into individual ‘packets’ of digital data—that is, strings of the numbers 0 and 1, as commended to the Chinese emperor by Leibniz in the seventeenth century—are transmitted through physical, usually fibre-optic cables, laid under the oceans and snaking across continents. (Some of the main submarine cables can be seen in Map 8.) Map 8. The physical internet This map shows major submarine internet cable systems. Source: TeleGeography, 2015. In 2011, a 75-year-old granny called Hayastan Shakarian was scavenging for firewood and scrap metal near a railway line that runs past her village in rural Georgia. She cut into a cable—and the internet went down in most of neighbouring Armenia for about 12 hours.

…

See also Articles 18, 19, 20 (Covenant) international law, 27–31 International Telecommunication Union, 27, 354 internet, 16, 352f; access to, 1, 14f, 122; censorship, 360–63; code as law, 26; cybergovernance, 351–57 (352f); development of, 20–22; domain names, 22; ethical algorithms, 364–67; expiration date suggestion, 309; first message of, 11; first use of term, 11; and group polarisation, 133–34; growth of, 12–13; hourglass architecture, 355f; illusion of independence, 22; information privacy/sovereignty, 46, 289, 355–57; infrastructure iceberg, 348–51 (349f); as ‘Internet,’ 20–21; languages on, 175f; laws governing, 23, 54–56; mass migration and, 18; memory of as indelible, 305; ‘net neutrality,’ 21, 237, 353, 357–60; numbers on, 1, 12–13, 383n1; one-click-away principle, 126, 147–48; online as forever, 16; online distraction, 178–79; open and closed platforms, 51–52; opposing hate speech on, 220–21, 237–41 (240f); role of money, 367–69; smartphone and mobile internet, 11; as ‘splinternet,’ 54; submarine cable system, 349–50 (350f); surveillance as business model, 284 Internet Engineering Task Force, 21, 353 intrusion and exposure, 288 Iran, 32–33, 46, 48, 99, 157–58, 184–85, 239, 241, 271, 277, 328–29, 362 Iraq, 127, 139, 270, 326, 328–29, 340, 344–45 Ireland, 136, 188, 201f, 254, 291, 295, 310; Effin, 238 Irving, David, 158 isegoria, 76–77, 314 ISIL/Islamic State, 127, 133, 329 Islam.

America in the World: A History of U.S. Diplomacy and Foreign Policy
by Robert B. Zoellick
Published 3 Aug 2020

The agreements and accompanying declarations covered a diverse range of topics, including rules for submarines and a ban on poison gases, Pacific security consultations, China’s independence and territorial integrity (including the Open Door principles), Chinese control of its customs charges, undersea cable rights across the Pacific, and a Sino-Japanese accord to restore the Shantung Peninsula to China. The four-power accord among the United States, Britain, Japan, and France to consult on aggressive actions and controversies in the Pacific sparked opposition from senators fearful of commitments to “guarantee” security.

The Great Game: On Secret Service in High Asia
by Peter Hopkirk
Published 2 Jan 1991

The purchase of the Egyptian ruler’s entire holding, which effectively rescued him from bankruptcy, made Britain the largest shareholder in the Suez Canal Company. A second major improvement in communications with India was the opening, in 1870, of a direct submarine cable link with London. Five years earlier an overland telegraph line had been completed, but traffic went via Teheran and was thus vulnerable to interference or severance in time of war. The new submarine cable was far less vulnerable. ‘As long as England holds the empire of the sea the cables will be safe from enemies,’ declared The Times. ‘To grapple and raise them would require not only a knowledge of their exact position, and a ship specially fitted with proper apparatus and trained hands, but also more time than could be given to the task.

Billions & Billions: Thoughts on Life and Death at the Brink of the Millennium
by Carl Sagan
Published 11 May 1998

The real binding up and deprovincial-ization of the planet requires a technology that communicates much faster than horse or sailing ship, that conveys information all over the world, and that is cheap enough to be available, at least occasionally, to the average person. Such a technology began with the invention of the telegraph and the laying of submarine cables; was greatly expanded by the invention of the telephone, using the same cables; and then enormously proliferated with the invention of radio, television, and satellite communications technology. Today we communicate—routinely, casually, with hardly ever a second thought—at the speed of light.

No Place to Hide: Edward Snowden, the NSA, and the U.S. Surveillance State
by Glenn Greenwald
Published 12 May 2014

Some of these designated choke points are identified by cover names: According to the NSA, STORMBREW “is currently comprised of very sensitive relationships with two U.S. telecom providers (cover terms ARTIFICE and WOLFPOINT).” Beyond its access to US-based choke points, “the STORMBREW program also manages two submarine cable landing access sites; one on the USA west coast (cover term, BRECKENRIDGE), and the other on the USA east coast (cover term QUAIL-CREEK).” As the profusion of cover names attests, the identity of its corporate partners is one of the most closely guarded secrets in the NSA. The documents containing the key to those code names are vigilantly safeguarded by the agency and Snowden was unable to obtain many of them.

The Stack: On Software and Sovereignty
by Benjamin H. Bratton
Published 19 Feb 2016

“The rainfall from Typhoon Morakot caused rivers to flood in Taiwan flushing large volumes of sediment into the ocean. This led to several submarine landslides which broke at least nine communications cables 4000m down. It disrupted the Internet and telecommunications between Taiwan, China, Hong Kong and other parts of Southeast Asia”60 Flooded data centers and compromised undersea cables can knock out whole networks, which is especially concerning when you realize that “over 95% of global communications traffic is handled by just 1 million km of undersea fiber-optic cable. Rising sea levels increase the risk of flooding of coastal cable facilities and may also affect the stability of the seabed, making cables even more vulnerable.”

The Ransomware Hunting Team: A Band of Misfits' Improbable Crusade to Save the World From Cybercrime
by Renee Dudley and Daniel Golden
Published 24 Oct 2022

Beneath the glittering North Sea is a less well known attraction, one that has lured ransomware gangs from across the globe—and spurred Dutch law enforcement to be proactive and innovative in fighting them. Beverwijk is a landing station for one of the transatlantic fiber-optic cables connecting the United States to Europe. Due in part to the gradual slope of its shores, the Dutch coast is a popular area for landing the submarine fiber-optic cables that contribute to the country’s fast internet speeds. Hackers, frustrated by slow and unreliable connectivity where they live, have flocked—digitally—to the Netherlands to set up the servers they use to commit crimes. Legal conditions were also favorable. Internet providers weren’t liable for the contents of the servers they hosted.

…

two thousand vendors: “De Bazaar Beverwijk: Market in North Holland,” Lonely Planet, lonelyplanet.com/the-netherlands/north-holland/shopping/de-bazaar-beverwijk/a/poi-sho/1125240/1315672. widest sand beach: “Wijk aan Zee, the Best Kept Secret of the North Sea Coast,” WijkAanZee.net, wijkaanzee.net/en/wijk-aan-zee.php#:~:text=Wijk%20aan%20Zee%20is%20also,at%20the%20North%20Sea%20coast. fiber-optic cables: “Submarine Cable Map: Beverwijk, Netherlands, Atlantic Crossing-1 (AC-1),” TeleGeography, submarinecablemap.com/landing-point/beverwijk-netherlands. top marks: Author interview with Marijn Schuurbiers, team leader, Dutch National Police High Tech Crime Unit, November 5, 2021. a security competition: The passage about the HTCU’s Capture the Flag competition is drawn from author interviews with Pim Takkenberg, former HTCU team leader, September 7, 2021, and February 8, 2022.

1989 The Berlin Wall: My Part in Its Downfall
by Peter Millar
Published 1 Oct 2009

Reuters in those final days before the whirlwind advent of electronic information and financial services that would transform it, was still a trust owned by a conglomerate of British and former imperial media interests. The business founded by Paul Julius Reuter, a German Jew, using carrier pigeons to link fledgling French and German telegraph lines had moved to London in 1851 after the laying of the Dover-Calais submarine cable, and subsequently spread a network of correspondents across the world. Reuters got a famous scoop in 1865 when a correspondent arriving on a ship from New York passed a message to a waiting boat off the coast of Ireland enabling the news of President Abraham Lincoln’s assassination to be telegraphed to London and the continent nearly a full day before the mail ship docked in Southampton.

The Green New Deal: Why the Fossil Fuel Civilization Will Collapse by 2028, and the Bold Economic Plan to Save Life on Earth
by Jeremy Rifkin
Published 9 Sep 2019

A 2019 feasibility study on a proposed Pan-American interregional power grid that will stretch from Alaska to Chile and potentially be in place by 2030 has sparked a conversation across the Americas on how this intercontinental technological alignment will likely affect the economy, social life, and governance of nation-states in this region of the world.16 Another report published in 2019 details plans to lay a submarine power cable between Europe and North America to trade solar- and wind-generated green electricity across the Atlantic Ocean.17 Similar feasibility and deployment plans to establish an Africa-wide power grid and a European–African power grid are afoot. We are in the early stages of creating a global interconnected electricity power grid—a digital Pangaea—that is likely to come online in bits and pieces between now and the late 2030s, connecting the human race for the first time in history.

…

Long Yongtu, Digital Silk Road: The Opportunities and Challenges to Develop a Digital Economy Along the Belt and Road (Beijing: Post & Telecom Press, 2017), 1–8; Morgan Stanley, “Inside China’s Plan to Create a Modern Silk Road,” March 14, 2018, https://www.morganstanley.com/ideas/china-belt-and-road (accessed March 1, 2019). 16.  Arman Aghahosseini et al., “Analysing the Feasibility of Powering the Americas with Renewable Energy and Inter-Regional Grid Interconnections by 2030,” Renewable and Sustainable Energy Reviews 105 (2019): 187–204, doi:10.1016/j.rser.2019.01.046. 17.  Arturs Purvin et al., “Submarine Power Cable Between Europe and North America: A Techno-economic Analysis,” Journal of Cleaner Production 186 (2018): 131–45, doi:10.1016/j.jclepro.2018.03.095. 18.  Kerstine Appun, Felix Bieler, and Julian Wettengel, “Germany’s Energy Consumption and Power Mix in Charts,” Clean Energy Wire, February 6, 2019; Rob Smith, “This Is How People in Europe Are Helping Lead the Energy Charge,” World Economic Forum, April 25, 2018, https://www.weforum.org/agenda/2018/04/how-europe-s-energy-citizens-are-leading-the-way-to-100-renewable-power/ (accessed March 5, 2019). 19.  

Trading at the Speed of Light: How Ultrafast Algorithms Are Transforming Financial Markets
by Donald MacKenzie
Published 24 May 2021

The antennas on them link the trading going on in the datacenter of the CME, the Chicago Mercantile Exchange, to that taking place in other, similar datacenters elsewhere. In the first instance, these are the New Jersey datacenters in which shares, sovereign bonds, and currencies are bought and sold, but signals radiating from the towers also cross the oceans, by submarine cable or shortwave radio, to datacenters in Greater London, Frankfurt, Mumbai, Singapore, Hong Kong, Tokyo, and beyond—to everywhere that automated trading takes place. The microwave towers are the clearest material manifestation of a feature of the financial system that was long in the making: the central role of futures trading—in particular, of trading on the CME—as a driver of what happens on many other exchanges.

…

On McKay’s calculations, this greater capacity (together with an LMDS link being slightly faster than E-band, because LMDS needs fewer hops and it is possible for an LMDS link to hug the geodesic particularly closely) is sufficient to remove any structural advantage of private links over quasi-public data transmitted via LMDS.18 McKay’s LMDS network (which became fully operational in the summer of 2019) is, however, not the final act in the material political economy of data transmission in New Jersey, as we shall see in chapter 7. The Materiality of the Small HFT’s “materiality of the large” (the mechanisms of data transmission among datacenters) has an even larger-scale transoceanic aspect: the submarine fiber-optic cables, shortwave radio links (Van Valzah 2018), and—perhaps eventually—swarms of low-earth-orbit satellites that carry financial data from continent to continent. Let me, though, set these aside here (for discussion of them, see MacKenzie 2019c), and end this chapter on HFT’s physicality by returning to “the materiality of the small”: to what goes on inside datacenters.

Globish: How the English Language Became the World's Language
by Robert McCrum
Published 24 May 2010

In 1881 the Cambridge historian J. R. Seeley published The Expansion of England, which argued, in a surprisingly modern way, that the exploitation of new technology might save Britain from inexorable decline. More than steamships and railways, wrote Seeley, it would be the electric telegraph and submarine cable that would interweave the fabric of empire. Seeley was a man of his time: the mid-nineteenth century was supremely an age of new technology. In America, Samuel Morse had pioneered his code in 1844. Four years later, the Associated Press began to use the telegraph for its news reporting. In the Indian Mutiny of 1857 the telegraph played an important role in frustrating the sepoy rebellion.

Sextant: A Young Man's Daring Sea Voyage and the Men Who ...
by David Barrie
Published 12 May 2014

A chronometer, on the other hand, was always available (so long as it worked), and it enabled the navigator to keep track of the time during the intervals between lunar observations. In practice, the two methods were entirely complementary, and they were to be used in tandem for at least seventy years. It was only with the invention of the electric telegraph and the laying of submarine telegraph cables that lunars ceased to be essential. Chapter 8 Captain Cook Charts the Pacific Day 9: Wind W by S force 4–5. Not a very good day. Various mishaps: failure of engine circulating water when trying to charge batteries; the binnacle light wouldn’t go on; and the washing up liquid drained away—so we now have to wash up in plain seawater.

…

See also Polaris Sterne, Laurence, 191 Stevenson, Robert Louis, 238 Stokes, Pringle, 195–201 Stonehenge, 23–24 storm warning systems, 216 Strait of Juan de Fuca, 147 Straits of Magellan Admiralty chart, 234–35 and Bougainville’s explorations, 115 discovery of, 194–95 and FitzRoy’s travels, 216 and Port Famine, 199–200 and Slocum’s circumnavigation, 229 and voyage of the Beagle, 201–2, 204–5 Stromness, South Georgia, 260 submarine telegraph cables, 84, 226 summer solstice, 23–24 Summer Triangle, 16 Sumner, Thomas, 219–22 Sumner Line, 222 sun, 20, 20–21, 26, 69, 233, 274, 300n6 sun compasses, 23 Supplement to the Voyage of Bougainville (Diderot), 114 supplies for sea voyages. See provisions for sea journeys surveys, 11 Surville, Jean de, xvi Swallow, 115, 121 Swift, Jonathan, 168–69n Sydney, Australia, 163 Tahiti accuracy of location for, 62n and Bligh’s explorations, 43, 158 and Bougainville’s explorations, 115–18 and Cook’s explorations, 88, 90–91 and Flinders’s explorations, 188 and mutiny of Bounty crew, 37 and Polynesian navigational skills, 263, 264 and Vancouver’s explorations, 144 and voyage of the Beagle, 211 Tail of the Bank, 22, 36 Taiwan, 126 Tamar Bay, 196 Tamar River, 162 Tanamo, 219, 219n, 269 tangent screw, 75 Tasmania, 135, 161–63, 169n, 171 telegraph, 84, 215–16, 226 theodolites, 146, 169, 207 Thetis, 200, 212–14 Thistle, John, 170 Thomson, James, 199 tides and tidal currents Agulhas Current, 182 and astronomy, 24–25 Bass Strait, 161 and Batavia wreck, 51 Cape Horn, 53–54 and Cook’s explorations, 97 and dead reckoning, 34 and departures, 272 and early humans, 23 Elephant Island, 249 English Channel, 5, 167 Great Barrier Reef, 100–101 Gulf Stream, 18 and La Pérouse’s explorations, 127, 132, 143 Labrador Current, 18 and lunar motion, 25–26 Mont St.

Cataloging the World: Paul Otlet and the Birth of the Information Age
by Alex Wright
Published 6 Jun 2014

The Union would also create a series of comparative exhibits to survey “the chief facts of universal history and the various phases of civilization,” as well as surveying a litany of other topics, from the grand—philosophy, science, medicine, and the law—to the highly particular—the history of the postal service, Preliminary plan for the World Palace of International Associations (Avant-­ projet de Palais Mondial des Associations Internationales). From Publication no. 25, Office Central des Associations Internationales (1912). Reproduced with permission from the Mundaneum. 117 C ATA L O G I N G T H E WO R L D interoceanic canals, submarine cables, and wireless telegraphy. Taken together, it would all add up to “a vast geographical and ethnographical museum.”14 In hopes of accelerating their efforts and building a wider base of international support, Otlet and La Fontaine solicited funding for the Union of International Associations from the Carnegie Foundation in America, which had already demonstrated support for similar initiatives designed to promote the public good.

Empire Lost: Britain, the Dominions and the Second World War
by Andrew Stewart
Published 1 Nov 2008

The recipient did not, however, object as he thought the admonishment was deserving and told the same to his former boss Berendsen, who was just settling in as the New Zealand minister in Washington. As he told him, if Britain had published something similar without prior consultation, he was pretty certain that Australia, at the very least, 'would have up and left the Empire, and the scream would have been heard all the way to Whitehall without benefit of submarine cable or wireless aerial'. He thought the DO faced an 'impossible job' and that New Zealand was not entirely without blame in terms of what had happened.66 The quarrel was ultimately resolved, but the message coming out of the discussions could not be so easily ignored, containing as it did an inexorable movement away from London and the pre-war Imperial system.67 One view, promoted by a renowned post-war Australian historian, saw it as 'not an aberration in Australian foreign policy but its logical outcome'.

Hidden Figures
by Margot Lee Shetterly
Published 11 Aug 2016

As the craft flew overhead, it telemetered a torrent of data to the closest tracking station, everything from its speed and altitude to its fuel level and the astronaut’s heart rate. The tracking stations captured the signals with their sixty-four-foot receiving dishes, then relayed this data plus voice communications through a jumble of submarine cables, landlines, and radio waves to the computer center at Goddard. The IBM machines used the inputs they received to make calculations based on the orbit determination programs. Via high-speed data lines—a blazing 1 kilobyte per second—Goddard sent Mission Control real-time information on the spaceship’s current position.

Outposts: Journeys to the Surviving Relics of the British Empire
by Simon Winchester
Published 31 Dec 1985

We passed a small forest of radio aerials: this was where the Marconi Wireless Telegraph Company had set up its first transmitter in 1912, and had sent a message in morse to King George. The Admiralty built a second set of mighty aerials just before the Great War, in the mountains to the west of the capital; the first time they were tested some hapless matelot turned on too much power and miles of wiring burned out in a flash and a drizzle of sparks. A submarine cable laid to Montevideo snapped, twice. Then a set of rhombus aerials was built for a space-research team that came to the Falklands in the mid-Sixties, and a private circuit was established between the unlikely twin towns of Stanley and Darmstadt, in West Germany. But after seven years it closed, too—leaving behind, as seems the custom here, a cluster of odd-shaped radio masts.

Cybersecurity: What Everyone Needs to Know
by P. W. Singer and Allan Friedman
Published 3 Jan 2014

Second, as international relations expert Robert Axelrod writes, while there may be no actual physical territory or borders in cyberspace, these governments “have woken up to an interesting fact: every node of the network, every router, every switch is within the sovereign borders of a nation-state and therefore subject to its laws or travels on a submarine cable or satellite connection owned by a company that is incorporated in a sovereign nation-state and therefore subject to its laws. In other words, there is no non-sovereign, ‘free’ part of cyberspace.” The outcome of this is an important shift. Countries have long sought to control the Internet within their own borders, both legally and operationally.

The House of Morgan: An American Banking Dynasty and the Rise of Modern Finance
by Ron Chernow
Published 1 Jan 1990

It specialized in deals of strategic import, requiring sensitive contacts abroad or covert government support. Many of its deals were directed against British interests; first and foremost, 23 Wall operated as an arm of Washington. Consider telecommunications. After the war, the United States feared the British military monopoly of undersea cable communications, which had yielded invaluable wartime intelligence. The U.S. Navy favored the use of a new private corporation, supported by Washington, to battle Britain in the emerging field of radio technology. Privately, President Wilson notified General Electric that he wanted to counter Britain’s cable monopoly with an American radio monopoly.

Slouching Towards Utopia: An Economic History of the Twentieth Century
by J. Bradford Delong
Published 6 Apr 2020

The telegraph enlarged the relevant group from the village or township or guild to, potentially, the entire world. Spanning the globe with telegraphs was difficult. Particularly difficult to set up were the submarine telegraph cables. The year 1870 saw the English engineer Isambard Kingdom Brunel’s SS Great Eastern—then the largest ship ever built (nothing larger was to be built until 1901)—lay the submarine telegraph cable from Yemen to Mumbai, completing the undersea line from London. Future dukes of Wellington, and millions besides, no longer took months conveying news and commands from London to Bombay and back.

…

In the 1980s there was another huge technological shift gathering force that would have powerful influences on world trade and far beyond: information technology. There came a true revolution in the cost of transporting not goods but bits, not material objects but information. The communication and data global internet—and the massive fiberoptic submarine and subsurface cables, plus the narrowcast and broadcast transmitters, receivers, and satellites, on which it rides—transformed the world again starting in the 1990s. I HAVE NOT WRITTEN much in this book about precisely how new technologies have advanced human collective powers over nature, about how they allowed us to organize ourselves in new ways, and what they were and did.

To the Ends of the Earth: Scotland's Global Diaspora, 1750-2010
by T M Devine
Published 25 Aug 2011

In consequence, the number of letters flowing between Britain and the USA, which had reached 2 million by 1854, had tripled to 6 million twenty years later. Mail times from the UK to Australia halved between the 1830s and 1850s and halved again in the 1870s as a result of the expansion in steam services and the opening of the Suez Canal.40 By 1851 telegraph wires traversed the USA and five years later a transatlantic submarine cable successfully linked Britain and America. A striking feature of the investment saga was how agents and managers from Scottish companies were able to travel abroad regularly to check on the performance of their shareholdings to an extent that would have been impossible in the days of the sailing ship and the horse and carriage.41 Information on financial markets also became abundant and more detailed in the national and regional press, the periodical literature, such as The Statist and Blackwood’s Edinburgh Magazine, and in specialist pamphlets.

AI 2041: Ten Visions for Our Future
by Kai-Fu Lee and Qiufan Chen
Published 13 Sep 2021

The secretary-general of the United Nations listened to the outlaw hacker’s theory. And then, once the plan’s feasibility had been confirmed by the experts of the Special Advisory Group, he gave Robin the green light. 176 seconds. Robin’s plan was for humans to cut off the power grid and submarine cable connections themselves. They would have to shut down root servers, signal transfer facilities, and all electronic equipment to minimize the impact of high-altitude EMPs and shorten the subsequent recovery time. It would be shock therapy for the global Internet. It would realize the anarchist ideal countless hackers had dreamed of since the dawn of the Internet age.

Great North Road
by Peter F. Hamilton
Published 26 Sep 2012

Libra day, and outside the camp personnel were gathering for the Sunday-night barbecue, which was fast becoming a tradition for the expedition camps. He told his e-i to establish a secure link to Vermekia. A secure connection through a six-thousand-kilometer e-Ray relay above the jungle, then an undersea cable, followed by another four-thousand-kilometer landline with dozens of civilian relays and cells was something of a joke, but the call was audio only and AIA encryption was still the best. “Two deaths?” Vermekia asked. “One death, one missing,” Vance said, wishing he didn’t sound so defensive.

Adventures in the Anthropocene: A Journey to the Heart of the Planet We Made
by Gaia Vince
Published 19 Oct 2014

Importantly, the project won’t only power Europe – African countries will also benefit, including the host solar generators in Morocco, Tunisia and Egypt. HVDC transmission is only one option – with the increasing advances made in high-temperature superconductors, efficient transmission of electricity can also be made over larger distances using submarine cables. Geothermal energy produced in Iceland, and hydroelectricity produced in Norway, can be transmitted in this way to Britain, for example. Another option is to store and transport the energy in hydrogen – the solar energy is used to split water into oxygen and hydrogen, and the reaction reversed using the hydrogen in a fuel cell to produce electricity where it is needed.

Turing's Cathedral
by George Dyson
Published 6 Mar 2012

“If only specific pairs of bases can be formed, it follows that if the sequence of bases on one chain is given, then the sequence on the other chain is automatically determined.”14 The mechanism of translation between sequence and structure in biology and the mechanism of translation between sequence and structure in technology were set on a collision course. Biological organisms had learned to survive in a noisy, analog environment by repeating themselves, once a generation, through a digital, error-correcting phase, the same way repeater stations are used to convey intelligible messages over submarine cables where noise is being introduced. The transition from digital once a generation to digital all the time began in 1953. The race was on to begin decoding living processes from the top down. And with the seeding of an empty digital universe with self-modifying instructions, we took the first steps toward the encoding of living processes from the bottom up.

The Pentagon's Brain: An Uncensored History of DARPA, America's Top-Secret Military Research Agency
by Annie Jacobsen
Published 14 Sep 2015

A three-hundred-foot-tall radio tower built nearby made it possible for the men in the bunker to communicate directly with U.S. defense officials and scientists running this secret operation from aboard the Task Force Command ship USS Estes, sixty miles out at sea. The men inside the bunker were members of the bomb’s firing party, a team of six engineers, three Army technicians, and one nuclear scientist. Miles of waterproof submarine cable connected the racks of electronic equipment inside the bunker to the Castle Bravo bomb, which was located on a separate island, nineteen miles across Bikini’s lagoon. “In the bunker we felt secure,” recalled Bernard O’Keefe, one of the nuclear weapons engineers who had advocated for this test.

The Information: A History, a Theory, a Flood
by James Gleick
Published 1 Mar 2011

When the first telegraph office opened in New York City on Wall Street, its biggest problem was the Hudson River. The Morse system ran a line sixty miles up the eastern side until it reached a point narrow enough to stretch a wire across. Within a few years, though, an insulated cable was laid under the harbor. Across the English Channel, a submarine cable twenty-five miles long made the connection between Dover and Calais in 1851. Soon after, a knowledgeable authority warned: “All idea of connecting Europe with America, by lines extending directly across the Atlantic, is utterly impracticable and absurd.”♦ That was in 1852; the impossible was accomplished by 1858, at which point Queen Victoria and President Buchanan exchanged pleasantries and The New York Times announced “a result so practical, yet so inconceivable … so full of hopeful prognostics for the future of mankind … one of the grand way-marks in the onward and upward march of the human intellect.”♦ What was the essence of the achievement?

Likewar: The Weaponization of Social Media
by Peter Warren Singer and Emerson T. Brooking
Published 15 Mar 2018

LIKEWAR 181 “the first non-linear war”: Surkov is famed less for his writing than as a principal architect of the modern Russian state. Natan Dubovitsky [Vladislav Surkov], “Without Sky,” quoted in Peter Pomerantsev, Nothing Is True and Everything Is Possible: The Surreal Heart of the New Russia (PublicAffairs, 2014), 2. 181 crew of the CS Alert: “British Cable Steamer Cuts Germany’s Trans-Atlantic Submarine Cables,” Palestine: Information with Provenance Database, accessed March 20, 2018, http://cosmos.ucc.ie/cs1064/jabowen/IPSC/php/event.php?eid=4289. 181 all seven of Germany’s: The exact order by which the lines were cut remains a matter of contention. See Chad R. Fuldwider, German Propaganda and U.S.

Growth: From Microorganisms to Megacities
by Vaclav Smil
Published 23 Sep 2019

This exponential growth, averaging about 63%/year (doubling every 13 months), is much faster than the rate predicted by Moore’s law, and the growth curve (including the anticipated 2021 total of 105.8 TB/s) fits best a five-parameter logistic curve indicating roughly a sevenfold increase during the 2020s (figure 4.28). Concurrently, maximum speed for modems was rising by 40%/year and capacity of submarine fiber-optic cables was growing by 33%/year: the record holder in 2018 was the 13,000-km long submarine cable between Los Angeles and Hong Kong whose six fiber-optic pairs will be able to transmit 144 terabits in both directions (Hecht 2018). Per capita Internet traffic rose from about 10 MB/month in 2000 to 10 GB/month in 2016 and it is expected to reach 30 GB/month by 2021 (CISCO 2017).

Heaven's Command (Pax Britannica)
by Jan Morris
Published 22 Dec 2010

It was a flamboyant admission of failure—for possession of the hill without artillery could really achieve nothing—by a commander without the magic gift of victory. ‘Think lovingly and sadly,’ Colley had written to his wife the evening before, ‘but not too sadly or hopelessly of your affectionate husband.’1 10 Majuba proved an abdication in a wider sense, too. The news reached London the following day, by a new submarine cable from the Cape, and the War Office at once resolved to send further reinforcements to South Africa from India, Ceylon, Bermuda and Britain, and to appoint Sir Frederick Roberts, one of the Empire’s most successful generals, to succeed Colley. But Gladstone thought otherwise, and in another unpredictable volte-face, he immediately opened peace negotiations with the Boers, to the contempt of the Conservatives and the dismay of the Queen.

How to Hide an Empire: A History of the Greater United States
by Daniel Immerwahr
Published 19 Feb 2019

The British, champions of the cable game, had by the early twentieth century gained control of more than half the world’s cables. They also, through Malaya, possessed the world’s sole supply of the natural latex gutta-percha—the only material until plastic that could effectively insulate deep-sea submarine cables. Yet mere preponderance wasn’t enough. The British obsessed over acquiring an “all red” network, red being the color of the British Empire on the map. Such a network, passing only through British territory, would offer protection from foreign powers that might cut or tap Britain’s cables. Britain achieved its all-red network and, with it, invulnerability.

Chinese Spies: From Chairman Mao to Xi Jinping
by Roger Faligot
Published 30 Jun 2019

This was comparable to the Japan External Trade Organization’s system, and capable of transmitting any economic data both to a central database and to the companies involved, in order to facilitate their interventions abroad.9 Headquartered in Beijing, the national branch then included 200 stations that used Asian civilian satellites. The international branch had 100 stations serving South-East Asia via satellite, North America, Japan and Oceania by submarine cable, and Europe, Africa and Asia from satellites in the Indian Ocean.10 It is possible to trace the birth of Chinese economic surveillance back to this birth of MOFERT/MOFTEC. It was supported by services directly linked to the upper echelons of the CCP, such as the United Front Work Department, which, as we have seen, intervened with the Chinese diaspora abroad.

The Best of 2600: A Hacker Odyssey
by Emmanuel Goldstein
Published 28 Jul 2008

With the exception of bilateral agreements between neighboring countries to make each other’s national systems compatible, especially in border regions, all international systems in use are: C5, C6, C7, and R2. R2 is limited to a single numbering region by policy and must use one of the three remaining systems for overseas working. There are few technical limitations to prevent R2 from working with satellites, TASI, or other analog/digital underseas cables. The spec is flexible enough to allow overseas working, but is not done at the present time. R2 is likely to displace C5 on the remaining analog trunks in the near future. National Signaling Systems CCITT 1, 2, and 3 are early international standards for signaling the distant end. C1 is just a 500 Hz line signaling tone, and was used to alert the operator at a distant switchboard that there was traffic and no DC path, due to amplifiers or repeaters on a relatively long circuit.

A Peace to End All Peace: The Fall of the Ottoman Empire and the Creation of the Modern Middle East
by David Fromkin
Published 2 Jan 1989

On 27 November 1884, Clemenceau told the French Chamber of Deputies that "Bismarck is a dangerous enemy, but even more dangerous perhaps as a friend; he showed us Tunis, placing us in conflict with England."18 In Parliament and in his journal, La Justice, Clemenceau denounced the acquisition of colonies as a financial and military burden, a distraction from the problem of the German frontier, and a clever German-inspired move that Berlin hoped would drive France into quarrels with Britain. In opposing the policy, he exposed the financial corruption that accompanied French colonial politics. La Justice's suggestion of sinister manipulations in the Tunisia affair were not far off the mark: there were speculations in real estate, railway con-cessions, and submarine cable telegraph concessions, whatever their relation might have been to the formulation of government policy. The financial corruption surrounding the adventure in Indochina was even more lurid. Clemenceau's accusations and exposures de-stroyed reputations and brought down governments. He became known as "the wrecker" even before he became known as "the tiger."

The Pentagon: A History
by Steve Vogel
Published 26 May 2008

This is the War Department The Pentagon had to be fully functioning before the high command moved in, and, as much as anything else, that meant working telephones. To meet the War Department’s fantastic need for telephone service—likened to providing a system from scratch for a city the size of Trenton, New Jersey—an ambitious plan had been devised to lay a dozen submarine cables one-third of a mile across the bottom of the Potomac River. These would connect the switchboard with the Chesapeake and Potomac Telephone Company’s central office in Washington. Frank E. Watts, a twenty-six-year-old telephone company employee from Alexandria, reported with other workers to the Arlington bank of the Potomac near the Pentagon on a rainy and chilly morning.

The Rise of the Network Society
by Manuel Castells
Published 31 Aug 1996

There are unconfirmed reports that a single-wire telegraph, using Salva’s scheme, was actually constructed between Madrid and Aranjuez (26 miles) in 1798. However, it was only in the 1830s (William Cooke in England, Samuel Morse in America) that the electric telegraph was established, and in 1851 the first submarine cable laid out between Dover and Calais (Garratt 1958); see also Sharlin (1967); Mokyr (1990). 38 Forbes (1958: 148). 39 A good history of the origins of the information technology revolution, naturally superseded by developments since the 1980s, is Braun and Macdonald (1982). The most systematic effort at summarizing the developments of the early information technology revolution was conducted by Tom Forester in a series of books (1980, 1985, 1987, 1989, 1993).

Andrew Carnegie
by David Nasaw
Published 15 Nov 2007

AC, RTW, 1; see also original copy of travel diary in Dunfermline. 12. J. W. Vandevort, notebooks, “Around the World,” 1878–79, n. p., Vandevort. 13. AC to GFMc, October 23, 1878, box 6, folder 2, CS. 14. John Steele Gordon, A Thread Across the Ocean (New York: Perennial, 2003), 211–12; Bill Glover, FTL Design: History of the Atlantic Cable and Submarine Telegraphy, Cable Timeline: 1850–1900: http://www.atlanticcable.com/Cables/CableTimeLine/index1850.htm. 15. Vandevort, notebooks, “Around the World,” Vandevort. 16. AC, Autobiography, 197, 204; William W. Stowe, Going Abroad: European Travel in Nineteenth-Century American Culture (Princeton: Princeton University Press, 1994), 1, 13–15. 17.

…

City of Eros: New York City, Prostitution, and the Commercialization of Sex, 1790–1920. New York: W. W. Norton, 1992. GLADSTONE, W.E. “Mr. Carnegie’s ‘Gospel of Wealth’: A Review and a Recommendation,” Nineteenth Century XXVII (1890). GLOVER, BILL. “FTL Design: History of Atlantic Cable & Submarine Telegraphy, Cable Timeline: 1850–1900,” http://www.atlanticcable.com/Cables/CableTimeLine/index1850.htm. GOODENOUGH, SIMON.The Greatest Good Fortune. Edinburgh: MacDonald, 1985. GORDON, JOHN STEELE.A Thread Across the Ocean. New York: Perennial, 2003. GREEN, CONSTANCE MCLAUGHLIN.Washington: Village and Capital, 1800–1878.

Paris 1919: Six Months That Changed the World
by Margaret Macmillan; Richard Holbrooke; Casey Hampton
Published 1 Jan 2001

The Russian question, the League of Nations and the new borders in central Europe had all come up, even if they had not been completely settled. Progress had been made, too, on some of the crucial details of the German treaty by special committees: on war damages and on Germany’s capacity to make reparation; on Germany’s borders, its colonies and its armed forces; on the punishment of German war criminals; even on the fate of German submarine cables. The big question, though—how to punish Germany and how to keep it under control in the future—had barely been touched on by Clemenceau, Lloyd George and Wilson, the only men who could really settle it. Also emerging was what a Swiss diplomat called the “great surprise at the conference”: a close partnership between the British and the Americans.

The Rise and Fall of the Great Powers: Economic Change and Military Conflict From 1500 to 2000
by Paul Kennedy
Published 15 Jan 1989

The Political Divisions of Europe in the Sixteenth Century 3. Charles V’s Inheritance, 15194 4. The Collapse of Spanish Power in Europe 5. Europe in 1721 6. European Colonial Empires, c. 1750 7. Europe at the Height of Napoleon’s Power, 1810 8. The Chief Possessions, Naval Bases, and Submarine Cables of the British Empire, c. 1900 9. The European Powers and Their War Plans in 1914 10. Europe After the First World War 11. Europe at the Height of Hitler’s Power, 1942 12. Worldwide U.S. Force Deployments, 1987 Tables & Charts TABLES 1. Increase in Military Manpower, 1470–1660 2.

The Way of the Gun: A Bloody Journey Into the World of Firearms
by Iain Overton
Published 15 Apr 2015

THE MANUFACTURERS Guns as the first product in history – revelations on manufactured death – the workings of a pistol factory on the Black Sea coast in Turkey – massive gun profits and political influence in New York, USA – gaining access to the hell-dog boardrooms of the world’s largest gun manufacturer foiled November 1851 was, by any stretch of the imagination, a remarkable month. It saw the publication of Moby-Dick by Herman Melville; a book seen by some, with its monumental ambition, as the first modern novel.1 It was the month that celebrated the laying-down of the first protected submarine telegraph cable, on the bed of the English Channel, foretelling an era of global communications.2 And, on a London day when the sky at dawn had been the colour of the Thames but was now brilliant and blue, it saw Samuel Colt standing before the bewhiskered and besuited members of the Institution of Civil Engineers in London, waiting for them to fall quiet.

The Pirates of Somalia: Inside Their Hidden World
by Jay Bahadur
Published 18 Jul 2011

Whenever I glanced out to sea, the Victoria’s distance from shore appeared to fluctuate, at times so close that I could make out the colour of the deck cranes, other times so far to sea that she was almost lost to the horizon. Whether the ship was adrift and being swept in and out with the tide, or whether it was an optical illusion, I was unable to discern. I was told that the pirates on board had ordered the Victoria’s crew to keep the vessel in constant motion, in order to prevent US frogmen or submarines from latching cables onto its hull. The pirates erroneously believed US naval forces had employed these tactics during the Maersk Alabama incident to tow the ship’s doomed hijackers into sniper range. In town, Omar, the Colonel, and I ate breakfast in a small lean-to, reclining in bare feet on woven mats.

The Pursuit of Power: Europe, 1815-1914
by Richard J. Evans
Published 31 Aug 2016

Accepting the primacy of the Greenwich meridian, and advocating a national standard time based on the 15th meridian, just east of Berlin, Moltke went on to declare that what was needed was ‘a unity of time reckoning for the whole of Germany’. It came eventually in 1893. The need to standardize time had already become urgent with the spread of telegraphing systems across not only Europe but also the world. The first submarine cables were laid across the English Channel in the early 1850s, and already in 1852, Edward Highton (1817–59), a pioneer of the electromagnetic telegraph, commented: ‘Time and space are all but annihilated. Years are converted into days, days into seconds, and miles have become mere fractions of an inch.’

Chief Engineer
by Erica Wagner

But when he and Emily and Edmund—still under their charge, for the time being, though he would shortly skip off on his own to Vienna—landed in England, rest and recuperation were surprisingly far from his mind. The first four pages of the letter remarking on the proposed trip to Auteuil are a characteristically detailed account of a visit to the Hooper’s Telegraph Works in London’s Millwall Docks, where submarine telegraph cable was manufactured. He was especially interested in the method of splicing cable used, and was evidently thinking hard about how the method could be adapted for the thousands of miles of wire that would shortly be crossing the East River. “The splices are rather new—in splicing the No. 13 Bessemer, they file it off like a bridge splice and wrap it around with pure tin wire,” he wrote, and drew a neat, delicate image.

Seapower States: Maritime Culture, Continental Empires and the Conflict That Made the Modern World
by Andrew Lambert
Published 1 Oct 2018

Instead it compelled Algiers to end the enslavement of European sailors, and crushed the Atlantic slave trade. Britain exploited technology, money and power to create the first globalised economy. It knocked down trade barriers, by force or by finance, pioneered new forms of capital movement, invented and laid the first global communications network, the submarine telegraph cable, and used it to build new markets. Britain created a world economy to sustain the seapower fleet that made it a great power. After 1815 the only threat to Britain’s global dominance came from an alliance between France, the only European great power with an oceanic navy, and either Russia or the United States, continental states with significant fleets.

The Vast Unknown: America's First Ascent of Everest
by Broughton Coburn
Published 29 Apr 2013

The scramble for Africa, 1876-1912
by Thomas Pakenham
Published 19 Nov 1991

There were few signs of British rule, apart from the coconut matting where some people played cricket in the square, the Union Jack flapping on the pole beside the tin-roofed government building, and the single battalion of redcoats in the thatched-roof barracks beside the convent. But there was one vital innovation, a small telegraph office, the end of a line of posts marching beside the cart track across the rolling veld to the blue hills of Natal, and so (by way of Durban and the submarine cable route via Aden) connecting Britain’s newest Crown Colony with London. This telegraph line had been a rushed job, and so had cost a fortune, £30,000, about a fifth of the territory’s budget for the year. But it would come in useful, the imperial authorities agreed, in case there was a ‘row’, meaning a war, with the Boers.

Project Azorian: The CIA and the Raising of the K-129
by Norman Polmar and Michael White
Published 1 Dec 2010

The Outlaw Ocean: Journeys Across the Last Untamed Frontier
by Ian Urbina
Published 19 Aug 2019

Multilateral regional fishery management organizations regulate commercial fisheries harvests; the International Maritime Organization manages shipping; and the International Seabed Authority regulates mining of the international seabed. But the management of other activities like dumping, laying submarine internet cables, bio-prospecting for new medicinal discoveries, and military weaponry testing that affect the ocean column or floor are regulated by a single-sector approach or virtually not at all. Rarely do the different categories of regulators work together to get a holistic view of the combined pressures on an area.

Energy and Civilization: A History
by Vaclav Smil
Published 11 May 2017

New industries energized by coal, coke, and steam were set up to supply national and international markets with unprecedented speed. The making of high-pressure boilers and pipes started after 1810. The production of rails and railway locomotives and wagons rose rapidly after 1830, as did the making of water turbines and screw propellers after 1840. Iron hulls and submarine telegraph cables found large new markets after 1850, and commercial ways of making inexpensive steel—first in Bessemer converters after 1856, then in open-hearth (Siemens-Martin) furnaces during the 1860s (Bessemer 1905; Smil 2016)— found new large markets for finished products ranging from cutlery to rails and from plows to construction beams.

The Boundless Sea: A Human History of the Oceans
by David Abulafia
Published 2 Oct 2019

D. 174 Gokstad ship 369 gold 57 , 76 , 106 , 244 , 505 , 540 , 547 , 588 , 640 , 717 African 471–4 , 475 , 481 , 484 , 496 , 500–501 , 506 ; Guinea/Mina coast 507 , 508 , 509–11 ; ‘River of Gold’ 471 , 473 , 474 , 484 , 496 , 500 , 502 , 511 artefacts found in Broighter 331 Aztec 635 bars 509 from Bia-Punt 79 boat model 331 bracelets 509 bribes 207 , 782–3 bullion famine 500–501 Californian 852–3 Caribbean 531 , 553 , 555 , 582 , 588–9 and China 216 of Chrysé and Argyré 134 coins 164 , 172 , 352 , 478 , 687 , 730 , 803 and Columbus 524 , 528 , 529 , 531 , 588 , 589 and crusades 481 dust 207 , 216 , 475 , 510 , 531 , 840 and economic warfare against Islam 501 , 537 , 546 and El Dorado 553 , 590 and encomienda system 531 English ambitions for 663–4 ‘fool’s gold’ 664–5 of Funan 140 and Hispaniola 531 , 582 Inca 635 ingots 156 and Japan 194 , 216 , 221 , 232 , 524 , 847 leaf 216 mining 174 and the Moluccas 611 and Oc-èo 141 of Ophir 80–81 , 82 in Philippines 574 and the Portuguese 471–4 , 501 , 502 , 506 , 507 , 508 , 509–11 , 546 , 548 , 560 , 635 , 652 ; funding imperial expansion 511 ; gold funding imperial expansion 511 of Ptolemy’s Syria 97 in Red Sea 176 Roman 108 and Russia 801 , 802 , 803 and Singapore/Temasek 286 and slavery 582 and Sofala 548 and Solomon 81 and the Spanish 531 , 553 , 555 , 582 , 588–9 , 729 and Śri Vijaya 153 tribute 546 Viking raids on 362 Welsh 302 Golden Hind 665 Gomes, Diogo 488 Gomes, Fernão 504 , 506 Gonneville, Binot Paulmier de 557–60 Gonville Hall, Cambridge 669 gooseberries 114 Gothenburg 717 , 722 , 723 , 724 , 725 archipelago 726 ‘Little London’ and the British 723 Reformed Church 723 Gotland 353 , 357 , 366 , 367 , 368 , 418 , 419 , 423 , 424 , 425 , 427 , 437 , 448 Bronze Age 368 Gotlanders’ Saga 368 Gouvea, Francisco de 653 Graf Spee 888 grain trade xix , 57 , 104 , 106 , 238 , 276 , 352 , 432 , 442 , 444 , 470 , 860 see also barley ; maize ; millet ; rice ; rye ; wheat Ålander 874–6 Black Sea 872 British imports 881 and Ceuta 475 , 478 and the Dutch 428 , 675 , 676–7 English 389 , 391 , 444 , 458 grain surpluses 50 and Hansards 428 , 429 and Iceland 390–91 Santo Domingo imports from Spain 587–8 Spanish 587–8 Gran Canaria see Grand Canary Gran Colombia 852 Granada 475 , 476 , 477 , 485 Grand Canary (Gran Canaria) 472–3 , 501 , 505 grapes 114 graves see burial ; tombs Gray, Robert 793 , 794 Great Britain see Britain Great Britain , SS 864 Great Chronicle of London 533 Great Collection of Buddhist Sutras 234 Great Depression 883–4 , 886 Great Eastern , SS 864 Great Gift 773 Great Northern War 802 Greece/Greeks Alexandrian Greeks 99 and Bereniké 111 Buddhism and Greek culture 90 coins 326 Greco-Roman merchants see merchants: Greco-Roman/‘Roman’ Greek pirates 130 Greek towns in northern Italy 334–5 and Hellenization 46 , 93 Ionia see Ionia maritime trade from 19th century 872–3 Massalia and Greek travellers 331–2 Mycenaeans see Mycenaeans and Persia 85 pottery 94 , 323 settlers in Africa 841 ships 109 , 872–3 and Socotra 129–30 trade routes of Bronze Age Greece 314 Greek language 111 of Socrotans 130 Greene, Daniel 795 Greenland xix , xxi , 362 , 380 , 393–409 , 721 Black Death in 405 Christianity 397 , 400 Corte Real’s rediscovery 534 Eastern Settlement 397 , 399 , 400 , 404 , 406 Eskimos: Dorset 398–9 ; Inuit 398 , 399–400 , 404 , 405 , 406–7 falcons 398 hunting 397 and Iceland 393–8 , 400–402 named Labrador 534–5 Norse discovery and settlement 393–402 , 403–7 , 655 and Norway 712 paganism 400 Southern Settlement 397 trade 398 , 402–4 , 407 , 408 Western Settlement 397 , 404 , 408 women 406 Greenlanders’ Saga 407 , 409 , 410–11 , 413 Greifswald 418 , 426 Gresham, Sir Thomas 669 Gresham College 669 Grijalva, Juan de 614 Grim Kamban 380 Grimaldo, Juan Francisco de 588 Grotius, Hugo 679–82 , 773 , 822 Guadeloupe 768 Guam 573–4 , 856 Guanches 472 , 488 Guangdong 159 Guangzhou/Canton xxix–xxx , 136 , 138 , 147 , 149 , 163 , 164 , 172 , 199 , 240 , 243–50 , 636–7 , 640 , 642 , 723 , 725 , 814 and the British 709 , 814 , 825 ‘Canton System’ 789 and Empress of China 785 , 786–8 and the French 827 and furs 806 Hoppos 789 merchants in 824 , 825 and New Julfans 706 North American trade with 786–91 and opium 825–6 and West Indies 711–12 Guatemala 581 guavas 644 Guðrið, wife of Þorfinn Karlsefni 411 , 412 , 413 Guðroð Crovan 378 Guðrum 361 Guerra, Luis 553 Guerra brothers 553 guilds, merchant 183 , 184 , 213 , 249 , 355 , 669 , 789 Guinea 504–8 and the Brandenburgers 716 coast 489 , 490 , 504 , 505–12 , 545 , 669 , 702 , 712 , 716 , 718 ; Carlsborg fort 717 , 718 and the Danes 712 , 716 , 721 and England 716 and France 561 and the Portuguese 504–8 , 754 scramble in 17th century for 716–19 and the Spanish 754 and the Swedes 716 , 718 trade see trade: Guinea trading stations 716 Guinea Bissau 504 Gujarat/Gujaratis 51 , 602 , 604–5 Diu see Diu, Gujarat Gulden Zeepaert , ’t 741 Gulf Stream xv gum arabic 839 Gunnbjorn Ulf-Krakuson 396 Gustavus Adolphus of Sweden 716 , 717 Gwadar, Oman/Pakistan 829 Gypsies 46 Habsburgs 439 , 604 , 703 Angra’s Spanish Habsburg fortifications 752 and France 672 Magnus Intercursus treaty between England and Habsburg rulers of Flanders 669 and New Christians 702 , 704–5 Portuguese 1640 rebellion 705 and São Felipe, Cape Verde 753 southern Netherlands transfer from Spanish to Austrian Habsburg rule 722 in Spain 614 , 764–5 Hacho, Mount 474 , 475 , 478 Hadhramawt, south Arabia 128 hairstyles 170 Haithabu 365–7 , 371 , 421 Haiti 527 , 685 see also Hispaniola Hakata (Fukoaka) 196 , 212 , 224 , 652 Hakata Bay 196 , 197 , 215 , 216–17 , 223 , 224 , 225 Hakluyt, Richard 658 Hakusuki, battle of 192 Halley, Edmund 750 Hallstatt culture 326 , 330 , 331–2 Hamburg 424 , 437 , 445 , 449 , 673 , 676 , 702 , 712 , 880 Chilehaus 880 and Portuguese Jewish merchants 701 and Second World War 891 , 892 Sephardim 701 trading links with South America 880 Hamburg–Amerika Line 869 Hamilton, Alexander 818–19 Hamwih (Southampton) 353 Hangzhou (Quinsay) 221 , 224 , 226 , 238 , 239 , 240 , 245 , 248 , 250 Hannibal 845 Hanoi 138 Hans, king of Denmark 464 Hansa, German xxii , 403 , 415–31 , 432–46 , 668 and Antwerp 673 Baltic members’ importance 433–6 and Bergen 418 , 419 , 443–6 and Bruges 433 , 438–9 , 440 , 441–2 cogs 426 , 428 , 451 and Cologne 428 , 433 , 438 , 441 , 442 , 444 , 447 , 448 , 452 Diets (Hansetage ) 420 , 432–6 , 437 , 453 and the Dutch 425 , 428 , 668 , 674–5 , 676 , 683 ; conflict with Spain 685 and England 419 , 447–53 , 456 , 458 , 463 ‘Hanseatic League’ 419 , 432–3 , 452 and herring 674 and Iceland 461–3 Kontor 440 , 444 , 445 , 446 , 447–8 and Norwegians 442–6 and Portugal 431 , 477 post-plague ‘Hansa of the Towns’ 432–6 Prussian members 452 and Teutonic Order 420–21 , 428 , 436 , 437 , 441 , 452 trade routes 424–5 , 438 HAPAG 881 , 885 , 886 , 891 , 904 Harald Fairhair 362–3 , 373 , 380 , 384 , 385–6 , 389 , 401 , 402 Harald Hardraða 361–2 , 377 , 378 Harali 57 Harappā 63 Harland, Edward 870 Harland and Wolff shipyard, Belfast 870 , 899 Harold Godwinsson 362 harpoons 28 , 304 , 399 Harris, William xxiii Harrison, John, chronometer 812 Hartog, Dirk 736 Harun ar-Rashid 170 , 365 Hasekura Tsunenaga 632 Hastings, Francis Rawdon-Hastings, 1st Marquess of 821 Haswell, Robert 793 Hathab/Astakapra 129 Hatshepsut/Hashepsowe, Queen 75–6 , 77 , 78 , 82 Havana xviii , 591–3 , 617 , 701 , 754 Hawai’ian islands xxiii , 3 , 20–21 , 25–7 , 810 , 813–17 , 859 agriculture 26–7 first settlers 24 Kaua’i 25 , 814 , 815 and Māori language 132 and Marquesas 24 , 30 O’ahu 24 , 25 , 814 , 816 pineapples 859 sweet potatoes 29 and the USA 816 , 817 , 856 , 859 Hawaiian Citizen 903 Hawaiki 5 , 33 , 34 , 35–6 , 37–8 Hayam Wuruk of Majapahit 280 Hayes’ Island 739–40 hazelnuts 114 Hazor 80 Heaney, Seamus 351 Hebrides 328 , 377 , 378 Viking raids 372 Hedeby see Haithabu Heian see Kyoto/Heian Hekataios 88 Hellenization 46 , 93 Helluland 409 Helsingborg 425 Helsingør (Elsinore) castle 713 Helsingør narrows 685 hemp 194 Henriques, Moses Josua 719 Henry I of Schwerin 425 Henry II of England 458 Henry III, Holy Roman Emperor 402 Henry III of England 450 , 469 Henry IV of England 452 Henry V of England 476 , 477 Henry VI, Holy Roman Emperor 449 Henry VII of England 526 , 532–3 , 534 , 557 Henry VIII of England 557 , 656 Henry the Impotent, king of Castile 504–5 Henry the Lion 424 , 425 , 427 Henry of Navarre 676 Henry the Navigator, Prince Henry 431 , 465 , 469 , 476 , 478 , 479–80 , 481 , 484 , 485 , 499 , 501 , 502 , 503 , 514 and the Azores 487 , 488 Belém statue 480 and Madeira 484 , 485 hens see chicken Herjólfsnes, Greenland 404 , 405 , 406 , 410 Hermapollon 112–13 Herodotos 88–9 , 90 , 334 herring 306 , 420 , 426 , 428 , 429–30 , 436 , 451 , 562 Baltic 430 , 436 , 451 , 674 and the Dutch 668 , 674 , 683 , 685 North Sea 430 , 436 , 674 Heyerdahl, Thor 27–30 , 299 , 727 Heyn, Piet 686 Heysham, Robert 749 Heysham, William 749 hides see skins/hides Hideyoshi Toyotomi 628 , 629 , 630 , 646–7 , 649 , 651–3 Dutch merchants in 650–51 Himilco 333 Himyar 122 , 123 , 166–7 Hindenburg 898 Hindostan , SS 865 , 867 Hinduism/Hindus 109 , 183 , 548 Brahmins see Brahmins Buddhism interwoven with Hinduism 293 and Madagascar 759 and Majapahit 280 and Melaka 293 , 295 Oman and Hindu merchants 834 and the Portuguese 547 , 548 , 606 ; Portuguese Christianizing mistakes over images 541 and Socotra 128 and Vietnam 144 Vishnu statues 159 Hine-te-aparangi 32 , 34 Hippalos 102 , 109 Hirado, Japan 692 , 693–4 Hiram, king of Tyre 73 , 80 Hispaniola 491 , 555 , 582–6 , 656 , 768–9 , 770 cattle 590–91 and Columbus 520 , 521 , 527–8 , 529 , 530 , 531 , 582 , 585 , 769 and the Dutch 683 gold 531 , 582 and Havana 592 hides 590 , 591 salt 588 Santo Domingo see Santo Domingo, Hispaniola sugar 582 , 589–90 , 768–9 Taínos 520 , 521 , 527–8 , 529 , 530 , 531 , 582 , 583 Hitler, Adolf 887 , 888 Hjǫrleif 384–5 Hoëdic 306 , 308 Hojeda, Alonso de 534 , 550 , 551 , 552 , 554 Hokianga nui a Kupe 33 Hokkaido 191 Hokule’a 20 Holocene period 303 , 304 Holt, Alfred 864–7 , 868 shipping company see Blue Funnel Line hominids, early, in Flores 7 Honduras 591 , 592 honey 194 , 389 , 391 , 444 , 470 Hong-wu 256 , 260 Hong-xi 267 Hong Kong 637 , 639 and Blue Funnel Line 867 , 891 and Britain 639 , 826–8 , 893 Chinese bazaar 827 Chinese settlers 827 Club 827 as a container port 904 , 907 and the Depression 886 merchants 789–90 , 808 , 824 and Mizrahi Jews 841 opium trade 827 , 828 P&O office 892 and People’s Republic of China 639 , 892–3 , 907 and pirate raids 867 and Second World War 887 , 892 ; and post-war period 892–3 and the USA 893 Victoria Island 826 , 867 Wallem shipping company 890 Hong Kong and Whampoa Dock Company 905 Honolulu 903 Hopewell 706 Hoppos, Canton 789 Hoq cave, Socotra 128 Horace 170 Hormuz 8 , 254 , 264–5 , 267 , 269 , 598–9 , 604 , 606 , 608 , 610 , 634 Strait of 52 , 62 , 166 , 598 , 634 Horrea Piperataria 114 horses 220 , 404 , 489 , 581 , 652 , 730 horse-breeding on Cape Verde 489 , 502 and yellow fever 855 Horta, Azores 750–51 horticulture, Lapita 12–13 Hotu Matu’a 30 houses circular 321 , 324 , 329 in circular villages 321 with dry-stone foundations 321 Inuit 399 Madeiran 484 merchant 425–6 , 443 , 445–6 , 838 oblong 321 of timber and wattle 366 Howqua (Hong merchant) 790 Huang Chao 172 Hudson, Henry 665 Hudson Bay 408 , 557 , 658 , 664 , 665 , 666–7 Huelva 319 Huguenots 676 , 696 , 697 Hull 447 , 451 , 462 , 871 human sacrifice 367 , 368 , 730 , 809 , 810 Humboldt, Alexander von 852 Hundred Years War 469 Hung Wu-ti 219 , 220 hunting 302 , 304 , 306–7 , 397 , 399 in Greenland 397 grounds 404 , 406 , 794 by Ming emperors 256 seals 399 turtles 105 whales see whales/whaling Hussein, sultan of Johor 821–2 Hutchison Ports 905 Hygelac 348 Hyksos 75 Hyogo 218 Iberia xxii , 305 see also Castile/Castilians ; Catalans ; Galicia ; Portugal/Portuguese ; Spain/Spaniards Atlantic culture of 324–5 Bronze Age 316 , 322 Christian vs Muslim struggle in 12th century 467–8 circular villages 321 Iron Age 330 Neolithic 308–9 , 313 pin-brooches 326 roundhouses 329 ships in Mediterranean from Atlantic Iberia 470 and Spanish Reconquista 468 , 477 swords 318–19 visitors to London 455 Ibn al-Muqaddam 182 Ibn Battuta 248 Ibn Hawqal 171 Ibn Iyas 595 Ibn Sab’in 475 Ibrahim Pasha, Pargalı 603 , 607 ice 303–4 Arctic 339 , 655 , 657 , 660 , 661 , 664 glaciers 304 receding 908 Ice Age 303–4 ‘Little Ice Age’ of 17th century 767 Iceland xix , 362–3 , 383–8 Black Death in 403 , 405 and Bristol 461–3 and Columbus 410 , 524 and England 461–3 and Greenland 393–8 , 400–402 and Hansards 461–3 and Norway 712 paganism 387–8 , 400 , 407 sagas 370 , 380 , 384 , 386 , 387–8 , 396 , 444 ; Egil’s Saga 389 , 391 ; Orkneyinga Saga 311 , 363 , 364 , 373 , 375 , 376 tallow 721 trade 390–92 , 461–3 vaðmal cloth 390–1 , 403 Ictis island 336 Ideal-X 902–3 Idrisi, Muhammad al- 130 , 157–8 , 471–2 , 475 Ieyasu Tokugawa 629 , 630–31 , 632 , 653–4 , 692 , 693–4 iguanas 12 Iki island 224 Imari wares 650 Imhof family and banking house 672 Imma 353 immigration see migration Imperator 885 Imperial Airways 898 Incas 29 , 618 , 635 , 727 incense 71–3 , 122 , 243 see also frankincense ; myrrh Arabian 94 , 115 Assyrian 80 Chinese 286 in Jewish Temple 80 and the ‘Muscateers’ 831 stands/vessels 80 Sumerian 80 Inchcape, James Lyle Mackay, 1st Earl of 884 India Ajanta caves 129 and Armenians 706 , 707 , 708 and the British/English 690–91 , 714 , 851 see also East India Company, English ; and Armenians 707 Buddhist monasteries 118 , 128 Cabral in 542 carpets 548 and China 125 , 137 , 138–9 , 144–6 , 147 , 152 , 161 , 282–3 see also Indo-China cloth see cloth: Indian coastal ethnic/religious diversity in 9th century 183 coins introduced to 102 da Gama in 540–41 and the Danes 712 , 714 , 715 diamonds 703 and Egypt 182–3 , 187–8 da Gama in 543 Goa see Goa Gujarat see Gujarat/Gujaratis as interchange point between trade networks 187 and Japan 152 luxury of elite 188 mail sent with P&O 867 and Malaya 282 and Mediterranean 108 merchant guilds 183 , 184 merchants from see merchants: Indian Mughal invasions 604–5 Muslims in 183 , 261 de Nova in 542 and Oman 831 , 832 and Ottomans 603 , 605 pepper see pepper: Indian and the Portuguese 540–43 , 604–5 , 606 ‘Roman’ merchants and southern India 52 Romans in 118 and silver 188 and Śri Vijaya 161 steamship link with Persian Gulf 868 Tamils see Tamils trade see trade: Indian and Vietnam see Vietnam: and India and West Indies 711–12 wine imports 116 Zheng He in 261 Indian Ocean 43–69 , 103 and Abbasid rise 170 and Alexandria 96 , 103 , 187 and American ‘Red Sea men’ 781–2 and British India Steam Navigation Company 868 and Buddhism 43–6 , 118 control struggle over 594–610 Dias’s discovery of link with Atlantic 514 , 536 , 537 ‘extensions’ xxii ; Red Sea see Red Sea ; South China Sea see South China Sea and the Genoese 703–4 Greco-Roman xviii and Gujarat 602 historic exit points 46–7 and the ‘Indies’ 46 , 99 , 517 and ‘Lascars’ 841 as link between Mediterranean and South China Sea 338 Madagascar as link with Atlantic 758 , 780 Magellan’s experience of 568 maritime network 43 , 90 , 93 , 124 , 125 , 147 , 167–70 , 279 Mediterranean interaction 98 , 116 , 117 , 124 , 167–70 , 338 , 517 Melaka as link with South China Sea 272 , 295 , 569 , 636 monsoons 50 , 51 , 52 , 96 , 102 , 171 , 182 , 546 navigation 117 and Ottomans see Turks/Ottomans: and Indian Ocean Pacific Ocean contrasted with 43 of the Periplous 103–10 , 116 , 119 , 124 , 125–8 , 134 and Persian Gulf see Persian/Arabian Gulf piracy widespread through 707 and Portugal see Portugal/Portuguese: and Indian Ocean Ptolemy’s view of 89 , 105 , 496 , 727 and Red Sea see Red Sea ‘Silk Route of the Sea’ xx , 265 ; and Singapore 281–9 Singapore as link with South China Sea 272 Spice Islands see Moluccas/Spice Islands spices in Indian cuisine 714–15 and Strait of Malacca see Melaka (Malacca): Strait of Malacca tides 103 , 107 trade see trade: Indian Ocean trade routes see trade routes: Indian Ocean and the Turks see Turks/Ottomans: and Indian Ocean and the USA 781–2 , 795 winds 51 , 110 see also monsoons and Zheng He’s voyages 254–5 , 256–69 , 280–81 , 293 ‘Indies’ areas covered by term in Middle Ages 46 , 517 , 523 Columbus and the term 46 , 517 East Indies see East Indies ; Indonesia/Indonesians spice trade see spices/spice trade transatlantic attempts to reach 517 , 522–35 , 537 , 550–64 , 566–8 , 571–2 West Indies see West Indies indigo 107 , 717 , 754 Indiké 120 Indo-China 137 , 147 , 249–50 see also Cambodia ; Vietnam and Fan-man’s conquests 144 Faxien’s voyages 144–6 and Sanskrit 144 Indonesia/Indonesians 6 , 43 , 147 see also East Indies ; Malaya/Malays and Buddhism 166 Bugi pirates 819 , 823 Chola and Indonesian trade 273 and the Dutch see Netherlands/Holland and the Dutch: and the East Indies/Indonesia Federation of Malaya and Indonesian Republic 893–4 and Madagascar 132–3 merchants 10 , 132 , 165 Moluccas see Moluccas/Spice Islands and the Portuguese see Portugal/Portuguese: and East Indies/Indonesia and Ryukyu Islands 232 sailings between India and China 125 spice trade with Madagascar 132–3 Sumatra see Sumatra and Vietnam 143 Zabaj 155–8 see also Śri Vijaya, Sumatra Indus River 50 , 53 , 68 , 88 , 106–7 Indus Valley and civilization 53 , 62–9 and Dilmun 66–7 see also Dilmun pottery 62 , 63 seals 63–4 and Sumer 62 trade with Mesopotamia 66–7 , 69 industrialization 861 , 865 , 869–70 , 896 , 907–8 infant mortality 307 Ingolf Arnarson 384–5 ingots bronze 320 copper 60 , 242 gold 156 silver 165 tin 164 Ingstad, Anne Stine and Helge 413 Innocent VIII 513 Inquisition 491 , 592 , 656–7 , 673 , 685 , 699 , 700 , 703 , 705 , 707 , 708 , 730 , 749 , 754 , 769 , 770 , 775 Intan wreck 164–5 Inuit 398 , 399–400 , 404 , 405 , 406–7 , 408 , 411 , 663 Ionia 82 , 85 Ionian Islands 682 , 872 , 873 Ipswich 353 Iran see Persia/Iran Iraq 68 , 85 , 164 , 598 Chinese trade with 185 Ottoman government of 831 Sumer see Sumer/Sumerians Ubaid 53–4 Ireland Bronze Age 316 , 320 circular villages 321 currachs 331 , 382 and the English 767 Irish Free State: and Second World War 888–9 ; and the USA 885 Irish monks 381–3 , 386–7 linen industry 457 migrants fleeing famine 860 monuments 301 and Orkney 311–12 and the potato 752 promontory forts 328 trade 457 and US immigration policy 885 Viking raids 361 , 372 , 379–80 , 381 Irian 280 Irish Channel 312 Irish Sea 313 iron 97 , 104 , 133 , 142 , 158 , 185 , 408 , 807 , 888 aboard SS London 861 early ironmaking 316 and Greenland 398 , 408 , 413 and Iceland 391–2 magnetized 243 and Marianas 619–20 nails see nails and Norsemen 408 ore 242 pyrites 664–5 Spanish 455 steamships 826 , 864 , 868–9 , 882–3 Swedish 724 technology 314–16 irrigation 50 on Hawai’i 25 , 26 Irvine, Charles 723 Isaac the Jew 186–7 Isabella of Castile 464 , 489 , 504–5 , 522 , 527 , 529 , 534 , 582 death 530 , 555 Isadora, Aelia 111 Isbister, Orkney 311 Isfahan 705–8 Isidore of Seville 135 , 409 Isis 122 Iskandar Shah 282 , 285 , 290 Islam/Muslims xx , 94 , 546 Abbasid dynasty 164 , 170 and Africa 500 , 502 Almohads 175 , 468 in Bombay 789 caliphate rivalries in 8th century 170 in China 244 , 258 Christians vs Muslims in Spain 423 , 477 coins 364 and the crusades see crusades dirhams 366 European struggle against Muslim power 46 see also crusades forcible conversion of Jews 181 Indian 183 ; in Calicut 261 and interest payments 837 and Ma Huan 269–70 and Madagascar 759 and Malaya 282 Mamluks see Mamluks and Melaka 291–2 , 293 , 295 merchants see merchants: Muslim ‘Moors’ 292 , 465–7 , 537 , 540 , 541 , 569 , 617 , 707 Muslim fleets 465–7 Muslim pilgrims 866 Muslims of Persian origin in 9th century 184 and Pasai 276 and Portugal see Portugal/Portuguese: and Islam ‘purity of blood’ doctrine and hiding of Muslim ancestry 696 in Quanzhou 244 and Red Sea closure to non-Muslims 187 relationship between Muslim governments and Jews/Christians 836–8 Shi‘ite 173 , 599 in Spain see Spain/Spaniards: Muslim (al-Andalus) spread along trade routes 146 , 166 and Śri Vijaya 147 Sunnis see Sunni Muslims and Tengah/Muhammad 291–2 , 293 and Zheng He 257–8 Íslendingabók 398 Ismail Pasha 849–50 Israel, Jonathan 674 , 686 Israel, Six-Day War (1967) 896 Israelites and incense 80 and Jewish Temple 80 , 81 Italy 115 , 240 see also Sicily ; Tuscany bankers 450 , 454 , 532 , 590 , 703 ; in England 454 French invasion 526 and Germany: landward connections 441 , 442 ; and Second World War 889 Greek towns in northern Italy 334–5 and London 454–5 merchants see merchants: Italian migrants fleeing poverty 860 and Ottomans 601 and Second World War 889 wars with Charles V 611 Zheng He in 254–5 Ivan III of Russia 677 Ivan IV, the Terrible 659–60 , 677 Ívar Bárdarson 403–4 ivory 505 , 547 , 717 , 840 craftsmen 503 elephant 47 , 59 , 61 , 62 , 63 , 76 , 105 , 107 , 118 , 121 , 122 , 153 , 233 , 265 , 278 , 323 , 324 , 398 , 405 , 833 and the Omanis 829 , 831 , 833 and the Portuguese 501 , 504 , 635 and the USA 833 walrus xix , 385 , 391 , 397 , 404 , 408 Izu islands 230 Jackson, Andrew 852 Jacob 782–3 Jacobites 723 Jacobsz, Ariaen 738–9 Jakarta see Batavia/Batavians Jamaica 520 , 530 , 749 , 761 , 769–79 and the English 765 , 768 , 769–79 Jews 773 Miami ferry service 900 and Panama Canal workforce 853–4 , 855 Port Royal 765 , 769–79 silver 777 , 778 and slave trade 777 and the Spanish 771 , 773 sugar trade 777 , 779 Jambi 160 , 273 , 279 , 287 James I of England 690 James II of England 777 James III of Majorca 473 James of Bruges 488 church in Azores 491 Jamestown, Virginia 557 , 766 Jan Mayen Island 396 Jang Bogo see Chang Pogo Jansz/Janszoon, Willem 736 Janus Imperiale 454 Japan/Japanese 6 , 190–209 , 212–29 Ainus 797 archers 211 , 230 armour 211 artists 200 Ashikaga shoguns 219 , 220 , 231 bakufu 212 , 217 , 222 , 223 , 225 , 229 and ‘barbarians’ 847 on Behaim’s globe 525 bonzes (religious teachers) 647 Buddhism see Buddhism: in Japan and Cabot 532 and Carletti 621 , 647 , 648–9 cash 214 Ceremony of National Purification 199 and China 135 , 152 , 188 , 189–90 , 192 , 197 , 198–209 , 212–21 , 228–9 , 235 , 236 , 251 , 287 ; and Hideyoshi’s assault on Korea 650–51 ; Hideyoshi’s conquest plans 646 ; Japanese mercenaries in Manila garrison vs Chinese 626 ; and Philip II’s plan to invade China 627 ; trade 201–2 , 212–20 , 228 , 638 , 647 , 649 and Christianity see Christianity: and Japan coal 846 and Columbus 524–5 , 537 and copper 692 , 847 daimyos (warlords) 644 , 646 , 647 , 650 , 654 and the Dutch see Netherlands/Holland and the Dutch: and Japan early ethnic variety 191 and European technology 847 and European weapons 644 and First World War 883 and Germany, Second World War 887 , 888 and gold 194 , 216 , 221 , 232 , 524 , 847 guilds (za ) 213 Iki 224 imperial court 191 , 193 , 194 , 196 , 197 , 200 , 203 , 205 , 209 , 213 , 214 , 215 , 217 , 222 , 223 , 228–9 and India 152 inflation 214 Inland Sea 199 , 204 , 212 , 218 , 232 , 652 inter-clan battles 219 , 230 and Java 220 junks 628 , 637 kami-kaze pilots 229 kaolin 650 and Korea 190 , 191–9 , 200 , 204 , 223 , 225 , 646 , 649–51 Kyushu see Kyushu island lacquered boxes xix , 216 and Macau 635 , 638 , 639 , 642 , 644 , 654 and Manila 622 , 628–30 , 632 , 633 Meiji regime 847 , 883 , 888 mercenaries 626 , 628–9 merchant fleets 883 , 896 merchants 189 , 221 , 616 , 629 , 630 , 649 and Mexico 630–31 Ministry of Commerce 847 Mongol attacks on 221 , 222–8 paintings 198 paper 200 , 201 , 213 , 743 pearls 216 , 221 peasants 213 ‘People of the Sea’ 213 and Philippines 617 , 622 , 628–30 , 632 , 633 , 648 , 652–3 Polo’s account of 221 and the Portuguese 606–7 , 628 , 630 , 632 , 635–8 , 639 , 642 , 647 , 653 , 654 , 692–3 ; and access to Chinese silk 652 ; and Dutch 692–3 ; and Hideyoshi 646–7 ; and Jesuits 644 , 646–7 ; translation of Portuguese sailing manuals into Japanese 644 pottery see pottery/ceramics: Japanese power fragmentation in 14th century 230 , 231 and rice 808 , 847 and the Russians see Russia: and the Japanese and Ryukyu Islands 229–30 , 232 , 614 sake 212 samurai 223–4 , 225 , 644 , 649 on Schöner’s globe 572 seafaring 199 and Second World War 887–8 , 890 , 891 Shikoku island 218 shipbuilding 212 , 896 ships see boats/ships: Japanese shogunate abolition (1868) 847 Shōsō-in imperial treasury, Nara 194 , 207 , 217 and Siam 220 and silk 194 , 200 , 201 , 628 , 630 , 633 , 847 silver 638 , 641 , 642 , 647 , 801 , 847 and the Spanish 627 , 629–30 , 632 , 633 , 652–3 ; and Dutch 692 Takashima 227 and tea 215–16 , 230 , 649 , 847 trade see trade: Japanese Tsushima see Tsushima island and the USA 846–7 ; and Second World War 888 and Vietnam 632 wakō pirates 218 , 219 , 220 , 222 , 637 , 638 and Waldseemüller 556 and Wallem 872 Japan current xv Japan Sea 6 , 43 , 198 , 200 Jardine, William 824 , 825 , 827 Jarlshof, Shetland 372 Java/Javans 7 , 113 , 154 , 162 , 272 , 279–81 , 287 , 689 , 736 Batavia see Batavia/Batavians and Britain 820 and China 163–4 , 188 , 244 , 260 , 262 , 268–9 , 280–81 , 286–7 and the Dutch 688 and Japan 220 Majapahit kingdom 279–81 , 284–5 , 287 , 290 merchants 292 and Mongols 228 , 272 , 279 and Norway 871 Parameśvara and Javan rule over Sumatra 287 , 289–90 Polo and ‘Great Java’ 272 , 280 , 734 and the Portuguese 578 and Ryukyu Islands 233 ships 285 ; junks 548 , 549 , 606 and Singapore/Temasek 284–5 , 286 , 288 , 289 spice merchants in 250 and Śri Vijaya 159 , 160 , 272–3 , 279–80 , 287 trade 163–4 Jefferson, Thomas 792 Jeffreys, George, 1st Baron 777 Jehosaphat 82 Jelling, Jutland 353 Jenkinson, Anthony 659–60 , 706 Jerónimo de Jesús de Castro 653 Jeronymites 589 Jerusalem 460 , 523 , 531 , 537 Temple 80 , 81 , 728 and Third Crusade 449 Jesuits 628 , 640 , 642 , 643 , 644–7 , 648 , 653 and Franciscans 653 jet 141 jewellery 141 , 265 see also specific gemstones American silver coins melted down for 754 bracelets see bracelets brooches 326 , 353 Crown Jewels 450 Lapita 15 necklaces 56 , 102 Jews 52 , 96 Adeni 181 of Alexandria 99 anti-Semitism 697 , 700 , 855 , 884 in Azores 840 bankers 450 , 855 of Cairo 124 , 536 in China 238 and citron 178 and England 700 , 769–70 , 839 and flax 176 , 178 forced conversion: to Christianity 537 ; to Islam 181 French anti-Semitism 855 of Fustat 176 Genizah documents 174 , 175–6 , 182 , 184 German 422 , 886 Gibraltar settlement 656 impurity laws and eastern drinking vessels 174 , 187 and the Inquisition 699 , 700 in Jamaica 773 Khazars 171 London’s Spanish and Portuguese Synagogue 700 , 838 Majorcan 480 in Manchester 839 in Mangalore 186 merchants 175–6 , 181 , 183 , 184 , 238 , 354 , 697–702 , 834 , 841 Mizrahi 841 Moroccan 697 , 836–8 , 840–41 Muslim governments’ relationship with 836–8 of Persian origin 184 and the Portuguese 536 , 537 , 697 , 699–700 , 701 , 702 , 717–18 , 772–3 , 838 , 840 see also Christianity: Portuguese New Christians (of Jewish origin) ; children from Portugal sent to São Tomé 493 , 494 ‘purity of blood’ doctrine and hiding of Jewish ancestry 696 Red Sea and Jewish–Christian confrontation 130 , 166–7 Sephardic see Sephardim and silk 176 in Singapore 823 in Spain 698 suspected secret Jews: in the Canaries 697 ; in Havana 592 Tabernacles Feast 178 Yemenite 181 , 182 Jian-wen 251 Jiddah 595 , 598 , 602 Jikaku Daishi (Ennin) 202–9 , 210–11 Jingdezhen 174 João I of Portugal 476 , 477 , 484 João II of Portugal 494 , 496 , 508 , 512 , 513 , 514 , 527 , 537 John, king of England 449 , 469 John I of Portugal 469 John, Prester 46 , 512 , 523 , 536 , 540 Johnson, Lyndon B. 903 Johor 239 , 679 , 818 , 821 Jon Grønlænder 405–6 Jordan 83 Josephus 135 Jourdain, John 689 , 690 Jtj (wife of Parekhou) 77 Juana la Beltraneja 505 Juana the Mad, queen of Castile 555 Julfa 706 see also New Julfan merchants Jungdezhen 649 junks xx , 134 , 137 , 233 , 234 , 575 , 637 Chinese 135 , 217 , 243 , 246 , 247 , 259 , 270 , 278 , 621 , 622 , 623 , 624 , 625 , 628 , 636 , 638 , 781 , 867 Japanese 628 , 637 Javanese 548 , 549 , 606 Melakan 643 and the Portuguese 636 , 637 , 642 , 643 Jutes 344 , 347 , 348 Jutland 685 battle of 882 Ka‘aba, Mecca 163 , 171 Kahiki 25 Kaiana, Hawai’ian prince 814–15 Kaifeng 238 Kalevala 366 Kalinga 288 Kalpe see Gibraltar Kamakura 212 , 214 , 215 , 217 , 223 , 224 , 229 Kamara (Puhar) 119 Kamaran Island 601 Kamchatka Peninsula 800 , 802 , 803 , 807 Kamehameha I of Hawai’i 814 , 815 , 816 , 817 Kamehameha II of Hawai’i 816–17 Kané 123 Kane (god) 26 Kangp’a see Chang Pogo Kanton, near Stockholm 725 kaolin 650 Karachi 63 , 91 Karelians 422 Karl Hundason, king of the Scots 376 Katharine Sturmy 461 Kathāsaritsāgara 143 Kattigara 121 , 143 Kaua’i 25 , 814 , 815 Kaundinya 140 , 141 Kaupang, Norway 353 , 355 , 365 , 390 Kawano Michiari 224–5 kayaks 399 Keats, John 565 Kedah 152 , 161 kelp 196 Kemal Reis 607 , 610 Kempe, Margery 452 Kendrick, John 792 , 793 , 794 Ken’in (Buddhist monk) 202 Kennedy, John F. 903 Kent 360 Ketill the Fool 386 Kettler, Jacob 719 , 720 Khazars 171 Khmer kingdom/kings 140–41 , 158 and Champa 249–50 and Song China 244 Kholmogory 660 Khubilai Khan 221 , 222 , 223 , 224 , 227 , 228 , 247 , 248 , 263 , 272 Ki no Misu 204 Kidd, William 761 Kiev/Rus 365 , 367 , 423 Kilwa 131 , 176 , 543 , 546 , 547 , 568 Kim Yang 211 kingfisher feathers 135 , 139 King’s Mirror 399 , 400 Kingston Bay, Jamaica 771 , 778 , 855 Kiribati Islands 14 , 31 , 614 Kiritimati 31 Kish/Qays 173–4 , 180 Klysma 102 , 117 knives 194 knǫrrs 371 , 385 , 390–91 Koch, John 330 Kodiak Island 793 , 805 Kōfuko-ji monastery, Nara 235 Koguryō, Korea 192 , 194 Kola Peninsula 658–9 Kolkata see Calcutta Kollam 183–4 Komr 158 Kon-Tiki 27 , 28 Kongo slaves 493 , 503 Konstantin of Novgorod 427 Koptos 78 , 111 , 112 , 113 , 116 Korea and Buddhism 166 , 194 , 198 celadon 232 and China 189 , 191 , 192 , 198 , 210 , 236 , 239 , 244 , 245 , 287 , 650 and Christianity 695 House of Li 219 and Japan 190 , 191–9 , 200 , 204 , 223 , 225 , 646 , 649–51 Kim dynasty 193 Koguryō kingdom 192 , 194 Koryō 219 , 222 , 223 merchants 189 , 207 ; merchant communities from Quanzhou 245 and Mongols 222 , 249 navy 650–51 North 193 Paekche kingdom 192 , 194 Parhae kingdom 194 , 197–8 and Red Sea 117 and Ryukyu Islands 232 , 234 ships 206 , 650–51 silk 244 , 247 Silla kingdom 192–3 , 194 , 195 , 197 , 198 , 199 , 204 , 211 slaves 210 tea bowls 649 trade 194 , 198 , 216 , 630 Kōrokan, Japan 196–7 Koryō, Korea 219 , 222 , 223 Kos, Greek island 116 Kosh, Socotra 129 Kosmas Indikopleustes 130 Kowloon 828 Kra Isthmus 137 , 144 , 273 Krakatoa 119 Kronstadt, Baltic 807 , 809 Kruzenshtern, Johann-Anton von 806–9 Kulami 182 Kumiai Indians 28 Kunemura 232 Kupe, Polynesian navigator 32–6 Kuril Islands 797 , 803 , 804 Kurland 715–16 , 719 Kuroshio current xv Kuwait 54 , 67 Kuy (war-god) 26 Kwallŭk 194 Kylsant, Owen Philipps, 1st Baron 883–4 Kylsant Royal Mail 883–4 Kyoto/Heian 212 , 216–17 , 218 , 223 , 228 , 644 Minamoto assault on 230 Tōfuku-ji monastery 218 Kyushu island 191 , 195 , 196 , 197 , 204 , 208 , 215 , 216–17 , 218 , 220 , 652 see also Hakata (Fukoaka) and Hideyoshi 646 Japanese converts to Christianity 654 pirates (wakō ) 218 La Gomera 589 La Isabela 528 , 585–6 , 588 La Navidad 585 La Pérouse, Jean-François de Galaup, comte de 809 , 812 La Tène culture 326 , 330 , 332 , 334 Labrador 464 , 533 , 557 Greenland given name of 534–5 lacquer 220 boxes xix , 216 , 703 Lady Hughes 788 Lady Washington 792 , 793 Lagash 64 , 65 Lagos, Algarve 497–9 , 502 Lancaster, James 687–9 Langdon Bay, Kent 319–20 , 321 Lanzarote 471 , 472 , 473 Laodicea 108 , 122 lapis lazuli 57 , 59 , 62 , 107 , 194 , 202 , 238 Lapita people/culture 11–16 , 19 agronomy 12 boats 15–16 horticulture 12–13 pottery 12 , 14 trade 15 Lapps/Sami 373 , 655 larch 408 Las Casas, Bartolomé de 554 , 582 , 584 , 585 , 591 Las Coles 585 Lasa 265 Latin 111–12 Latvia 420 , 422 peasants 719 Lavenham 453 Lavrador, João Fernandes 534 Lazarus, Emma 860 Le Havre 562 , 564 lead 333 , 640 Leander 807 leather 455 , 839 Lebanon 81 , 91 Ledyard, John 785 Lee Kuan Yew 894 Legázpi, Miguel López de 615 , 616 , 618 Leif Eiríksson (Leif the Lucky) 400 , 407 , 410 , 411 , 413 Leisler, Jacob 782–3 lemons 573 , 752 juice 688 lemurs, giant 133 leopards 99 , 266 skins 323 Lesseps, Ferdinand de 849 , 854 lettuce 643 Leuké Komé 115 Levant Company 691 Levant trade 460–61 Lewis, Bernard 836 Lewis, David 13 Lewis chessmen 397 Liang shu 137 Libelle of Englyshe Polycye 455 Libya/Libyans 73 , 79 , 90 Liefde , De 631 , 691 life expectancy 307 Lima 697 Limahon 625 , 627 limes 573 , 752 limewood 339 Limyriké 109 Lin Feng/Limahon 625 , 627 Lin Zexu 825–6 Lindisfarme 360 , 364 , 372 linen 373 , 444 , 457 , 540 , 548 , 624 Linnaeus, Carl 725 lions 99 , 259 , 266 , 269 , 276 skins 323 liquorice 194 Lisbon xxii , 370 , 468 , 469 , 470 , 476 , 477 , 487 , 510 , 545 , 600 , 606 , 617 , 648 , 676 , 701 , 840 archives 594 , 634 businessmen funding transatlantic voyages 525 Castilian siege of 476 and the Dutch 677 , 682–3 investors 561 , 703 and Madeiran sugar 485 shipbuilding 484 and spices 682–3 , 684 unloading of cargoes in transit to avoid customs in 745 , 751 Litany of Oengus 381 Lithuania 437 , 452 Liubice 423 Liujiagang 268 , 270 Liverpool 864 , 865 , 866 , 868 , 871 , 873 , 874 , 875 , 880 , 885 , 887 air raids 891 post-war recovery of shipping companies 897 and Second World War 891 , 892 Livonia/Livs 420–21 , 441 Livorno 619 , 676 , 682 , 700 , 772 , 838 , 872 Lixus 323 Lloyds 861 Lo Yueh 135 Loaísa, Garcia Jofre de 579 lodestones 243 , 370 Lomellini family 470 London 419 , 438 , 780 , 885 air raids and damage 891 , 892 , 896 and Bensaúde 840 and Bristol cloth 458 cosmopolitanism 455 Genoese in 455 and Greek merchants 872–3 and Hansards 447–8 , 449 , 450 , 453 and Italians 454–5 and Navigation Act (1651) 780 New Christians of 700 , 708 and New Julfans 706 Port of 454 , 839 , 880 , 891 , 892 , 896 , 897 ; docks 904 post-war recovery 897 Royal African Company 760 Royal Exchange 669 Royal Society 792 , 820 and under-sea cables 864 Sephardim 701 Spaniards in 455 Spanish and Portuguese Synagogue 700 , 838 Steelyard and the Hansard Kontor 447–8 , 453 tax exemptions 455 Viking raid on 360 and wool 454 London , SS 860–61 London Missionary Society 813 Long Melford 453 Longe, William 456 longitude, fixing 16 , 731 , 812 Loos, Wouter 740 Lopes Pereira, Manuel 704 Lopez, Fernando 755–6 Lopius, Marcus 793 Lothal 63 , 64 Lotus Sutra 199–200 , 215 Louis the Pious 337 Louis XI 459 Louis XII 559–60 Louis XIV 720 Louisiana , USS 859 Love Boat 900 Low Countries Flanders see Flanders Netherlands see Netherlands/Holland and the Dutch and Portuguese merchants avoiding customs duties 745 Portuguese sugar imports 669 and Spain 673 , 685 see also Netherlands/Holland and the Dutch: and the Spanish Low German language 426 Middle Low German 423 Lu-Enlilla 58–9 , 65 Lu-Mešlamtaë 59 Lu Xun 138 Luanda, Angola 593 , 701 Lübeck 418 , 419 , 422 , 423 , 424–6 , 427 , 429 , 440–41 , 444 , 445 , 447 , 449 , 450 , 452 , 470 Alt-Lübeck 423 Bone-Cutter Rebellions 440 and Hansa Diets 432 , 433 , 436 , 437 herring 430 Lübeckers in England 449–50 marzipan emporium 429 New Council 440–41 salt 430–31 Society of the Circle 440 , 442 Lucca 454 Lucena, Vasco Fernandes de 513 Lucian of Samosata 102 Lucqua (Hong merchant) 808 Luftwaffe 889 Lüneburg Heath 431 Luo Mao-deng: The Grand Director of the Three Treasures Goes Down to the Western Ocean 254 , 259 , 265 Lusitania , RMS 882 luxury goods 58 , 75 , 82 , 139 , 192 , 199 , 215 , 217 , 232 , 349 , 420 , 427 see also specific goods by name Arctic 412 , 414 , 415 Chinese customs rates on 241–2 , 273 European demand from new colonies in mid-16th century 588 exotic luxuries see exoticism: exotic goods, luxuries and foods and German Hansa 420 , 427 and Hawai’i 816 Japan and Portuguese luxuries 647 Mediterranean 672 and Russia 802 Luzon island 616 , 618 , 622 ceramics 652 Lvov 708 lychees 246–7 Lydia 85 Lynn, England 447 , 449 , 451–2 , 462 Steelyard 453 lynx 306 Lyons 562 Ma Huan 256–7 , 261 , 264–5 , 269–70 , 280 Maasvlakte 897 Mabuchi Kamo 847 Macaronesia 481–96 , 745–55 see also Azores ; Canary Islands ; Cape Verde Islands ; Madeira Macassar 10 Macau (Macao) xviii , xxix , 592 , 617 , 627–8 , 634 , 638–42 , 648 , 709 , 725 , 787 , 808 , 826 ‘black ships’ of Macau/Great Ship of Amacon 641–2 , 644 , 645–6 , 647 and the British 806 and gambling 907 and Japan 635 , 638 , 639 , 642 , 644 , 654 Jesuits 645 and New Julfans 706 opium trade 827 and the Portuguese 638–42 , 643–4 , 787 ; New Christians 703 silk 640 and silver 763 and tax 639 and the USA 806 Macaulay, Thomas Babington, 1st Baron 722 mace 611 , 684 , 703 , 781 Macnin, Meir 839 Macnin family 836 Macpherson, W.