load shedding Concepts

Private Equity: A Memoir

by Carrie Sun · 13 Feb 2024 · 267pp · 90,353 words

ago about being exhausted yet unable to bring herself to say anything. I looked at Val, who had told me, after Boone agreed to my load-shedding, that she planned to send an email to Ari saying “ok now let’s cut back the work.” Before she sent it she spoke to

Lonely Planet Cape Town & the Garden Route (Travel Guide)

by Lucy Corne · 1 Sep 2015 · 1,203pp · 124,556 words

its past. Cape Town Today The legacy of the city’s stint as World Design Capital 2014 (WDC2014); the threat of electricity blackouts (known as ‘load shedding’) due to a struggling national grid; political tussles between the Democratic Alliance (DA), who run the city and the Western Cape, and national governing party

Building Secure and Reliable Systems: Best Practices for Designing, Implementing, and Maintaining Systems

by Heather Adkins, Betsy Beyer, Paul Blankinship, Ana Oprea, Piotr Lewandowski and Adam Stubblefield · 29 Mar 2020 · 1,380pp · 190,710 words

interplay between reliability and security that can cause unexpected outcomes. The password manager’s failure was triggered by a reliability problem—poor load-balancing and load-shedding strategies—and its recovery was later complicated by multiple measures designed to increase the security of the system. The Intersection of Security and Privacy Security

…

underlying platform What mechanisms components use to communicate (such as RPCs, message queues, or event buses), how requests are routed, and how load balancing and load shedding are implemented and configured How unit testing, end-to-end functional testing, production readiness reviews (PRRs), load testing, and similar validation activities are integrated in

…

guarantee a safe response. Instead, servers should adjust how they respond to load based upon current conditions. You can use two specific automation strategies here: Load shedding is done by returning errors rather than serving requests. Throttling of clients is done by delaying responses until closer to the request deadline. Figure 8

…

-3 illustrates a traffic spike that exceeds the capacity. Figure 8-4 illustrates the effects of load shedding and throttling to manage the load spike. Note the following: The curve represents requests per second, and the area under it represents total requests. Whitespace

…

-3 also distinguishes the uncontrolled nature of degraded traffic (the backward-slashed area) prior to system crash. Figure 8-4 shows that the system with load shedding rejects significantly less traffic than in Figure 8-3 (the crosshatched area), with the rest of the traffic either processed without failure (whitespace area) or

…

area). Figure 8-3. Complete outage and a possible cascading failure from a load spike Figure 8-4. Using load shedding and throttling to manage a load spike Load shedding The primary resilience objective of load shedding (described in Chapter 22 of the SRE book) is to stabilize components at maximum load, which can be especially

…

capacity unavailable—not just the capacity for the excess requests. When this capacity is gone, the load just shifts elsewhere, possibly causing a cascading failure. Load shedding allows you to free server resources even before a server’s load reaches capacity, and to make those resources available for more valuable work. To

…

requests the server receives from clients (if clients send requests sequentially), which means that you can redirect the resources saved during wait times. Similar to load shedding, you could define policies to apply throttling to specific offending clients, or more generally to all clients. Request priority and cost play a role in

…

selecting which requests to throttle. Automated response Server utilization statistics can help determine when to consider applying controls like load shedding and throttling. The more heavily a server is loaded, the less traffic or load it can handle. If controls take too long to activate, higher

…

servers with lower loads Providing DoS protections that can assist in response to malicious clients if throttling is ineffective or damaging Using reports of heavy load shedding for critical services to trigger preparation for failover to alternative components (a strategy that we discuss later in this chapter) You can also use automation

…

self-reliant failure detection: a server that determines that it can’t serve some or all classes of requests can degrade itself to a full load-shedding mode. Self-contained or self-hosted detection is desirable because you don’t want to rely on external signals (possibly simulated by an attacker) to

…

to determine and record levels of system degradation, regardless of what triggered the problem. This information is useful for diagnosing and debugging. Reporting the actual load shedding or throttling (whether self-imposed or directed) can help you evaluate global health and capacity and detect bugs or attacks. You also need this information

…

should address the differences in available capacity by using the means covered in “Controlling Degradation”. After failover, such a system switches to using throttling and load-shedding policies tuned for the alternative component. If you want the system to fail back after the failed component recovers, provide a way to disable that

…

been used at Google to demonstrate the wide spectrum of approaches to continuous validation. Inject anticipated changes of behavior You can validate system response to load shedding and throttling by injecting a change of behavior into the server, and then observing whether all affected clients and backends respond appropriately. For example, Google

…

eliciting resilient responses requires less load. You can quantify the impact of your experiment by comparing the monitoring signals from other failure domains. By adding load shedding and throttling, you further increase the quality of output from the experiment. Oversubscribe but prevent complacency Quota assigned to customers but not consumed is a

…

of their relatively static nature, yet offer significant improvements. High-availability services are the next most cost-effective solution. Consider these options next: Consider deploying load-shedding and throttling capabilities if your organization’s scale or risk aversion justifies investing in active automation for resilience. Evaluate the effectiveness of your defenses against

…

provides for a cheap way to examine how much you might be able to improve the service’s availability. It’s also cheap to abandon. Load-shedding and throttling capabilities, along with the other approaches covered in “Controlling Degradation”, reduce the cost of the resources the company needs to maintain. The resulting

…

, Delete it! Liberia, Criminal Actors libFuzzer, How Fuzz Engines Work linters, Automated Code Inspection Tools LLVM Clang, How Fuzz Engines Work load balancing, Defendable Architecture load shedding, Load shedding location separation, Location Separation-Isolation of confidentialityaligning physical and logical architecture, Aligning physical and logical architecture isolation of confidentiality, Isolation of confidentiality isolation of trust

…

Begin-Practical Advice: Where to Begin resistance to change, Changing Culture Through Good Practice response mechanism deployment, Deploy Response Mechanisms-Automated responseautomated response, Automated response load shedding, Load shedding reliability/security tradeoffs, Failing safe versus failing secure throttling, Throttling response plansauditing automated systems, Auditing Automated Systems communicating when email/instant messaging system is compromised

Site Reliability Engineering: How Google Runs Production Systems

by Betsy Beyer, Chris Jones, Jennifer Petoff and Niall Richard Murphy · 15 Apr 2016 · 719pp · 181,090 words

“Testing for Cascading Failures”. Serve degraded results Serve lower-quality, cheaper-to-compute results to the user. Your strategy here will be service-specific. See “Load Shedding and Graceful Degradation”. Instrument the server to reject requests when overloaded Servers should protect themselves from becoming overloaded and crashing. When overloaded at either the

…

frontend or backend layers, fail early and cheaply. For details, see “Load Shedding and Graceful Degradation”. Instrument higher-level systems to reject requests, rather than overloading servers Note that because rate limiting often doesn’t take overall service

…

queue size based on the current number of threads in use, processing time for each request, and the size and frequency of bursts. Load Shedding and Graceful Degradation Load shedding drops some proportion of load by dropping traffic as the server approaches overload conditions. The goal is to keep the server from running out

…

picking requests that are more important and prioritizing. Such strategies are more likely to be needed for shared services. Graceful degradation takes the concept of load shedding one step further by reducing the amount of work that needs to be performed. In some applications, it’s possible to significantly decrease the amount

…

on-disk database or use a less-accurate (but faster) ranking algorithm when overloaded. When evaluating load shedding or graceful degradation options for your service, consider the following: Which metrics should you use to determine when load shedding or graceful degradation should kick in (e.g,. CPU usage, latency, queue length, number of threads

…

degraded mode automatically or if manual intervention is necessary)? What actions should be taken when the server is in degraded mode? At what layer should load shedding and graceful degradation be implemented? Does it make sense to implement these strategies at every layer in the stack, or is it sufficient to have

…

a small subset of servers near overload in order to exercise this code path. Monitor and alert when too many servers enter these modes. Complex load shedding and graceful degradation can cause problems themselves—excessive complexity may cause the server to trip into a degraded mode when it is not desired, or

…

? How do you mitigate unavailability of your dependencies? Example action items Implement request deadlines to avoid running out of resources for long-running requests. Implement load shedding to reject new requests early in overload situations. Client Behavior On a traditional website, there is rarely a need to take abusive behavior from legitimate

…

that can be referenced using standard terms Standard dimensions for monitoring instrumentation A standard format for request debugging logs A standard configuration format for managing load shedding Capacity of a single server and determination of “overload” that can both use a semantically consistent measure for feedback to various control systems Frameworks provide

…

application code. Under this model, SREs assume responsibility for the development and maintenance of large parts of service software infrastructure, particularly control systems such as load shedding, overload, automation, traffic management, logging, and monitoring. This model represents a significant departure from the way service management was originally conceived in two major ways

…

perform acceptably when overloaded with traffic. For times when load is high enough that even degraded responses are too expensive for all queries, practice graceful load shedding, using well-behaved queuing and dynamic timeouts; see Chapter 21. Other techniques include answering requests after a significant delay (“tarpitting”) and choosing a consistent subset

…

Investigate running index MR/fusion continuously prevent jennifer Bug 5554824 TODO Plug file descriptor leak in search ranking subsystem prevent agoogler Bug 5554825 DONE Add load shedding capabilities to Shakespeare search prevent agoogler Bug 5554826 TODO Build regression tests to ensure servers respond sanely to queries of death prevent clarac Bug 5554827

…

traffic volume File descriptor leak bug fixed (bug 5554825) and deployed to prod Looking into using flux capacitor for load balancing (bug 5554823) and using load shedding (bug 5554826) to prevent recurrence Annihilated availability error budget; pushes to prod frozen for 1 month unless docbrown can obtain exception on grounds that event

…

, Ensuring Business Continuity correctness guarantees, Workflow Correctness Guarantees development of, Introduction to Google Workflow stages of execution in, Stages of Execution in Workflow graceful degradation, Load Shedding and Graceful Degradation GTape, Gmail—February, 2011: Restore from GTape H Hadoop Distributed File System (HDFS), Storage handoffs, Clear, Live Handoff “hanging chunk” problem, Trouble

…

Balancing at the Virtual IP Address policyLeast-Loaded Round Robin, Least-Loaded Round Robin Round Robin, Simple Round Robin Weighted Round Robin, Weighted Round Robin load shedding, Load Shedding and Graceful Degradation load tests, Overload Behavior and Load Tests lock services, Lock Service, Distributed Coordination and Locking Services logging, Examine Lustre, Storage M machinesdefined

…

Q “queries per second” model, The Pitfalls of “Queries per Second” Query of Death, Process Death queuingcontrolled delay, Load Shedding and Graceful Degradation first-in, first-out, Load Shedding and Graceful Degradation last-in, first-out, Load Shedding and Graceful Degradation management of, Queue Management, Reliable Distributed Queuing and Messaging queuing-as-work-distribution pattern, Reliable

…

tests reliable replicated datastores, Reliable Replicated Datastores and Configuration Stores Remote Procedure Call (RPC), Our Software Infrastructure, Examine, Criticalitybimodal, Bimodal latency deadlinesmissing, Missing deadlines propagating, Load Shedding and Graceful Degradation, Deadline propagation queue management, Queue Management, Reliable Distributed Queuing and Messaging selecting, Latency and Deadlines retries, Retries-Retries RPC criticality, Criticality(see

The Practice of Cloud System Administration: DevOps and SRE Practices for Web Services, Volume 2

by Thomas A. Limoncelli, Strata R. Chalup and Christina J. Hogan · 27 Aug 2014 · 757pp · 193,541 words

to pay for unused capacity. Shared resource pools are not just appropriate for machines, but may also be used for storage and other resources. Load Shedding Another strategy is load shedding. With this strategy the service turns away some users so that other users can have a good experience. To make an analogy, an

…

immediate response, such as a simple “come back later” web page, rather than requiring them to time out after minutes of waiting. A variation of load shedding is stopping certain tasks that can be put off until later. For example, low-priority database updates could be queued up for processing later; a

…

might cause problems if they are put off forever. There is, after all, a reason they exist. For any activity that is delayed due to load shedding, there must be a plan on how such a delay is handled. Establish a service level agreement (SLA) to determine how long something can be

…

extend the deadlines. Low-priority updates might become a high priority after a certain amount of time. If many systems are turned off due to load shedding, it might be possible to enable them, one at a time, to let each catch up. To be able to manage such situations one must

…

web era, 456 with multiple backend replicas, 12–13 with shared state, 75 three-tier web service, 72–74 Load sharing vs. hot spares, 126 Load shedding, 139 Load testing, 215 Local business listings in Google Maps, 42 Local labor laws, 43 Logarithmic scaling, 476 Logistics in disaster preparedness, 318–320 Logistics

…

code review systems, 269 defined, 120 disasters. See Disaster preparedness Overflow capacity factor in service platform selection, 67 Overload failures DoS and DDoS attacks, 139 load shedding, 139 scraping attacks, 140–141 traffic surges, 138–139 Oversubscribed systems defined, 53 spare capacity, 125 PaaS (Platform as a Service), 51, 54–55 Packages

…

phase flow, 196 Service Deployment and Decommissioning (SDD), 404, 437–438 Service latency in cloud computing era, 471 Service level agreements (SLAs) Error Budgets, 152 load shedding, 139 monitoring, 334 oncall, 286–287 Service Level Indicators (SLIs), 334 Service Level Objectives (SLOs), 334 Service Level Targets (SLTs), 334 Service life cycle, 155

…

vs. traditional enterprise IT, 148–149 Site reliability practices, 151–152 Size batches, 178–179 caches, 108–110 SLAs (service level agreements) Error Budgets, 152 load shedding, 139 monitoring, 334 oncall, 286–287 SLIs (Service Level Indicators), 334 SLOs (Service Level Objectives), 334 Sloss, Benjamin Treynor Google Error Budget, 396 site reliability

Nomad Century: How Climate Migration Will Reshape Our World

by Gaia Vince · 22 Aug 2022 · 302pp · 92,206 words

heatwave in the spring of 2022 across India and Pakistan meant hundreds of thousands of people were unable to work after 10 a.m., with load-shedding power outages leaving people without access to cooling or refrigeration. Cooling is not just going to be a problem in the tropics, where there is

The Key Man: The True Story of How the Global Elite Was Duped by a Capitalist Fairy Tale

by Simon Clark and Will Louch · 14 Jul 2021 · 403pp · 105,550 words

Electric was controlled by Abraaj and required billions of dollars of loans and investment to improve its operations and reduce power cuts, which he called “load shedding.” After the meeting, Ambassador Patterson sent a cable to State Department colleagues asking for help to encourage Abraaj to raise the required funds for Karachi

…

Electric. “Black outs and load shedding are a serious impediment to economic productivity. Poor power delivery has also led to large public demonstrations,” the American diplomatic cable said. “Embassy Islamabad requests

The Elements of Choice: Why the Way We Decide Matters

by Eric J. Johnson · 12 Oct 2021 · 362pp · 103,087 words

a $62 million airliner in the river and they call you a hero. Is this a great country, or what?”2 Sullenberger borrowed the term load shedding from electrical utilities: when demand exceeds their capacity, a utility may take parts of its network off-line, shutting down, say, power to a factory

…

they are overwhelmed, and they decide to ignore what they hope are nonessential parts of the problem. Choosing a plausible path is a form of load shedding: it involves making decisions about what is crucial information to consider for meeting goals and what information is nonessential and can be taken off-line

…

life expectancy, 76–81, 136 annuities, 79–81, 336n Social Security benefits, 88–98 life-extension care, 310–13 light bulbs, 136–38, 254–55 load shedding, 23–24, 27, 28, 44, 61 local warming, 67–68 Loewenstein, George, 174–75 London Zoo, 57 loss aversion, 346n Lotz, Sebastian, 134–35, 143

…

–44 “stupid human tricks,” 4 subliminal perception, 300–1 subscription services, 126–27, 267, 314–15 Sullenberger, Chesley “Sully,” 22–29, 44, 102, 332–33n load shedding, 23–24, 27, 28, 44, 61 Sunstein, Cass, 11, 95, 118, 121, 135, 308 supermarket shelves, 209–11 sustainability and targets, 248–50 Sydnor, Justin

The Nature of Software Development: Keep It Simple, Make It Valuable, Build It Piece by Piece

by Ron Jeffries · 14 Aug 2015 · 444pp · 118,393 words

will frustrate a user or provoke a retry loop. As such, back pressure works best within a system boundary. At the edges, you also need load shedding and asynchronous calls. In our example, the API server should accept calls on one thread pool and then issue the outbound call to storage on

…

. Consumers will experience slowdowns. The only alternative is to let them crash the provider. Apply Back Pressure within a system boundary Across boundaries, look at load shedding instead. This is especially true when the Internet at large is your user base. Queues must be finite for response times to be finite. You

…

thing to do under high load is turn away work we can’t complete in time. This is called “load shedding,” and it’s the most important way to control incoming demand. Load shedding happens very quickly when a socket’s listen queue is full, and a quick rejection is better than a slow

…

safety limits on retries). Wrapping Up We looked at the interconnect layer in this chapter, where instances come together to form systems. Load balancing, routing, load shedding, and service discovery are some of the key issues to consider when building this layer. Depending on your organization, you may have existing solutions in

Ghost Train to the Eastern Star: On the Tracks of the Great Railway Bazaar

by Paul Theroux · 9 Sep 2008 · 651pp · 190,224 words

in the city. ‘Suppose there’s a power cut?’ I asked. Such things were common, and barely concealed under the euphemisms ‘brownout’, ‘rolling blackout’ or ‘load shedding’. ‘What happens then?’ ‘Last July we had power cuts. Ninety-three centimetres of rain in sixteen hours.’ That was more than three feet of rain