Humble Pi: A Comedy of Maths Errors
by Matt Parker · 7 Mar 2019
programmers feel as they try to finish writing their code. Deadly code The most dangerous 256 error I have found so far occurred in the Therac-25 medical radiation machine. This was designed to treat cancer patients with bursts of either an electron beam or intense X-rays. It was able to
…
the X-ray beam) had been placed in between the electron beam and the patient. For this, and a host of other safety reasons, the Therac-25 looped through a piece of set-up code, and only if all the systems are verified as being in the correct settings could the beam
…
had a number stored with the catchy name of Class3 (that’s just how creative programmers can be when naming their variables). Only after the Therac-25 machine had verified that everything was safe would it set Class3 = 0. To make sure that it was checked every time, the set-up loop
…
but merely because the value had rolled over from 255 back to zero. This means that roughly 0.4 per cent of the time a Therac-25 machine would skip running Chkcol because Class3 was already set to zero, as if the collimator had already been checked and verified as being in
…
1987 in Yakima Valley Memorial Hospital in Washington State, US (now Virginia Mason Memorial), a patient was due to receive eighty-six rads from a Therac-25 machine (rads is an antiquated unit of radiation absorption). Before the patient was to receive their dose of X-rays, however, the metal target and
…
5, but it lived on as a piece of vestigial code. In general, reusing code without retesting can cause all sorts of problems. Remember the Therac-25 radiation therapy machine, which had a 256-roll-over problem and accidentally overdosed people? During the course of the resulting investigation it was found that
…
, had the same issues in its software, but it had physical safety locks to stop overdoses, so no one ever noticed the programming error. The Therac-25 reused code but did not have those physical checks, so the roll-over error was able to manifest itself in disaster. If there is any
…
some serious problems can result when non-technical users are faced with an overly technical error message. This was one of the problems with the Therac-25 radiation machine with roll-over issues. The machine would produce around forty error messages a day, with unhelpful names, and as many of them were
…
died from the resulting radiation overexposure. When it comes to medical equipment, bad error messages can cost lives. One of the recommended modifications before the Therac-25 machines could go back into service was ‘Cryptic malfunction messages will be replaced with meaningful messages.’ In 2009 a collection of UK universities and hospitals
Engineering Security
by Peter Gutmann
how dangerous it can be to assign arbitrary probabilities to events, in this instance for a fault tree, was illustrated in the design of the Therac-25 medical electron accelerator. This led to what has been described as the worst series of radiation accidents in the 35-year history of medical accelerators
…
,000 rads (a normal dose from the machine was under 200 rads, with 500 rads being the generally accepted lethal dose for fullbody radiation, the Therac-25 only affected one small area which is often less radiosensitive than the body as a whole). The analysis had assigned probabilities of 110-11
…
for common mishaps that occur in the course of human activities range from about 10-2 to 10-10 incidents per hour [133]. In the Therac-25 case it was exactly these (supposedly) extraordinarily unlikely events, with a probability of one in a billion and one in a hundred billion, that caused
…
Security and Privacy, Vol.3, No.5 (September/October 2005), p.66. 292 Threats [131] “Report on the Therac-25”, J.Rawlinson, OCTRF/OCI Physicists Meeting, 7 May 1987. [132] “An Investigation of the Therac-25 Accidents” Nancy Leveson and Clark Turner, IEEE Computer, Vol.26, No.7 (July 1993), p.18. [133] “Designing
…
correct value from a selection of several actually forced them to think about the problem. A particularly notorious instance of user satisficing occurred with the Therac-25 medical electron accelerator, whose control software was modified to allow operators to click their way through the configuration process (or at least hit Enter repeatedly
…
a host of other design problems) led to situations where patients could be given huge radiation overdoses, resulting in severe injuries and even deaths (the Therac-25 case has gone down in control-system failure history, and is covered in more detail in “Other Threat Analysis Techniques” on page 259). Even in
…
better protect them”, Min Wu, Proceedings of the First Workshop on Trustworthy Interfaces for Passwords and Personal Information, June 2005. [142] “An Investigation of the Therac-25 Accidents” Nancy Leveson and Clark Turner, IEEE Computer, Vol.26, No.7 (Jul 1993), p.18. [143] “Fighting Phishing at the User Interface”, Robert Miller
…
end result really does have the properties that it’s supposed to have. Another example of the need for post-release testing occurred with the Therac-25 medical electron accelerator that’s already been mentioned in “Other Threat Analysis Techniques” on page 259 and “Safe Defaults” on page 462, which had a
…
Security (3rd edition)”, Rudolf van Renesse (ed), Artech House, 2005. “Handbook of Paper and Board”, Herbert Holik (ed), Wiley – VCH, 2006. “An Investigation of the Therac-25 Accidents” Nancy Leveson and Clark Turner, IEEE Computer, Vol.26, No.7 (Jul 1993), p.18. “An Improved Experience for New Users of Firefox”, Ken
Robot Futures
by Illah Reza Nourbakhsh · 1 Mar 2013
directly or solely responsible for the behavior of a complex robotic system. Brainspotting 101 Technology ethics and design courses frequently study the tragedy of the Therac-25 to understand how much can go wrong when poor design, incorrect training, and simple errors are compounded (Leveson and Turner 1993). The
…
Therac-25 was a radiation therapy machine that provided focused radiation to cancer victims to destroy malignant tumors by rapidly moving a high-energy radiation beam. The
…
one hundred times the intended dose of radiation, inducing massive pain in the patient and, eventually, killing patients through radiation sickness. Many aspects of the Therac-25 therapy process are partially to blame for this. The interface was poorly designed, making incorrect data entry easy. Training for the operators was lightweight, and
…
Street science, 115 Structure, 27–31, 46 Synapse, 97–99, 123, 124 Telepresence, 37, 65–73, 102, 104, 107, 117, 124 Terrill, Rufus, 24, 25 Therac-25, 101 Traffic calming, 113, 114 Turkle, Sherry, 62 Urban search and rescue (USAR), 74–78, 124 Vagabond, 56, 57, Vigilante robot, 110 Water quality, 114
Overcomplicated: Technology at the Limits of Comprehension
by Samuel Arbesman · 18 Jul 2016 · 222pp · 53,317 words
undergo radiation treatment for cancer of the cervix. The patient was prepared for treatment, and the operator of the large radiation machine known as the Therac-25 proceeded with radiation therapy. The machine responded with an error message, as well as noting that “no dose” had been administered. The operator tried again
…
’s having indicated that no dose of radiation was delivered. This was not the only instance of this radiation machine malfunctioning. In the 1980s, the Therac-25 failed for six patients, irradiating them with many times the dose they should have received. Damage from the massive radiation overdoses killed some of these
…
a fact of life, and yet the safety analysis almost completely ignored the risks they present. The people responsible for ensuring the safety of the Therac-25 misunderstood technological complexity, with lethal consequences. In hindsight it’s almost easy to see where they went wrong: they downplayed the importance of whole portions
…
: LOSING THE BUBBLE In 1985, a patient entered a clinic: Story and analysis from Nancy G. Leveson and Clark S. Turner, “An Investigation of the Therac-25 Accidents,” Computer 26, no. 7 (1993), 18–41. “software does not degrade”: Quoted in Leveson and Turner, “An Investigation.” the way machines count: Machines—or
…
, 3–4 “natural history” of, 103–4 philosophy of, 79–81 self-contained ecosystems in, 4 Teece, David, 144 Thales, 139 Theory of Everything, 113 Therac-25, overdose failures of, 67–69 Three Mile Island nuclear disaster, 12, 126 time zones, 2, 51–52 tinkering, 118, 125–26, 127, 132, 191 Torvalds
Dreaming in Code: Two Dozen Programmers, Three Years, 4,732 Bugs, and One Quest for Transcendent Software
by Scott Rosenberg · 2 Jan 2006 · 394pp · 118,929 words
-bit variable, but the number was too high, a buffer overflowed, and the system froze.) From 1985 to 1987 a radiation therapy machine named the Therac-25 delivered massive X-ray overdoses to a half-dozen patients because of software flaws. During the 1991 Gulf War, a battery of American Patriot missiles
…
.gsfc.nasa.gov/nmc/tmp/MARIN1.htm. James Gleick tells the story of the Ariane 5 bug at http://www.around.com/ariane.htm. The Therac-25 bug is detailed in a paper by Nancy Leveson and Clark S. Turner in IEEE Computer, July 1993, at http://courses.cs.vt.edu/~cs3604
Concepts, Techniques, and Models of Computer Programming
by Peter Van-Roy and Seif Haridi · 15 Feb 2004 · 931pp · 79,142 words
the interleavings. In the history of computer technology, many famous and dangerous bugs were due to designers not realizing how difficult this really is. The Therac-25 radiation therapy machine is an infamous example. Because of concurrent programming errors, it sometimes gave its patients radiation doses that were thousands of times greater
…
-Wesley, 1997. [127] [128] Doug Lea. Concurrent Programming in Java, 2nd edition. Addison-Wesley, 2000. Nancy Leveson and Clark S. Turner. An investigation of the Therac-25 accidents. IEEE Computer, 26(7):18–41, July 1993. [129] Henry M. Levy. Capability-Based Computer Systems. Digital Press, Bedford, MA, 1984. Available for download
…
, 382 theorem binomial, 4 Church-Rosser, 331 Gödel’s completeness, 634 Gödel’s incompleteness, 634 halting problem, 681 theorem prover, 117, 634, 662 Therac-25 scandal, 21 thinking machine, 621 third-party independence, 335 32-bit address, 78 32-bit word, 74, 174 this, see self Thompson, D’Arcy Wentworth
Beautiful Testing: Leading Professionals Reveal How They Improve Software (Theory in Practice)
by Adam Goucher and Tim Riley · 13 Oct 2009 · 351pp · 123,876 words
. Kidwell, P. A. 1998. “Stalking the Elusive Computer Bug.” Annals of the History of Computing, 20: 5–9. McPhee, N. “Therac-25 accidents,” http://www.morris.umn.edu/~mcphee/Courses/Readings/Therac _25_accidents.html. Smithsonian National Museum of American History. “Log Book With Computer Bug,” http:// americanhistory.si.edu/collections/object.cfm?key
New Laws of Robotics: Defending Human Expertise in the Age of AI
by Frank Pasquale · 14 May 2020 · 1,172pp · 114,305 words
Moreover, lawyers have grappled with the problem of malfunctioning computers for decades, dating back at least to the autopilot crashes of the 1950s and the Therac-25 debacle of the 1980s (when a software malfunction caused tragic overdoses of radiation).29 Nevertheless, some proposals would severely diminish the role of courts in
…
; and facial recognition, 128; and non-state actors, 162; and online media, 98; and “terror capitalism,” 166–167. See also bioterrorism; 9 / 11 terrorist attacks Therac-25, 40 Thomas, Raymond, 241n69 Thrall, James, 42 Three Body Problem (Liu), 209 Tokui, Nao, 219 Tokyo University, 68–69 Toyama, Kentaro, 82 Toyota, 6 transportation
Coders at Work
by Peter Seibel · 22 Jun 2009 · 1,201pp · 233,519 words
engineering; part of engineering is working out various safety properties, which matter. Doing a browser they matter. They matter more if you're doing the Therac-25. Though that was more a thread-scheduling problem, as I recall. But even then, you talk about better languages for writing concurrent programs or exploiting
Geek Sublime: The Beauty of Code, the Code of Beauty
by Vikram Chandra · 7 Nov 2013 · 239pp · 64,812 words
rocket that went off course and self-destructed forty seconds after lift-off because of an error in converting between representations of number values; the Therac-25 radiation therapy machine that reacted to a combination of operator input and a “counter overflow” by delivering doses of radiation a hundred times more intense