load shedding

Site Reliability Engineering: How Google Runs Production Systems
by Betsy Beyer , Chris Jones , Jennifer Petoff and Niall Richard Murphy
Published 15 Apr 2016

time requirements, Balance in Quality training for, Learning Paths That Are Cumulative and Orderly-A Hunger for Failure: Reading and Sharing Postmortems training materials, Creating Stellar Reverse Engineers and Improvisational Thinkers typical activities, Life of an On-Call Engineer one-phase pipelines, Initial Effect of Big Data on the Simple Pipeline Pattern open commenting/annotation system, Collaborate and Share Knowledge operational loadcross-industry lessons, Automating Away Repetitive Work and Operational Overhead managing, Managing Operational Load ongoing responsibilities, Managing Operational Load types of, Dealing with Interrupts operational overload, Operational Overload operational underload, A Treacherous Enemy: Operational Underload operational work (see toil) out-of-band checks and balances, Choosing a Strategy for Superior Data Integrity, Out-of-band data validation out-of-band communications systems, What went well outage trackingbaselines and progress tracking, Tracking Outages benefits of, Unexpected Benefits Escalator, Escalator Outalator, Outalator-Reporting and communication Outalatoraggregation in, Aggregation benefits of, Outalator building your own, Outalator incident analysis, Analysis notification process, Outalator reporting and communication, Reporting and communication tagging in, Tagging overhead, Toil Defined overload handlingapproaches to, Handling Overload best practices for, Overloads and Failure client-side throttling, Client-Side Throttling load from connections, Load from Connections overload errors, Handling Overload Errors overview of, Conclusions per-client retry budget, Deciding to Retry per-customer limits, Per-Customer Limits per-request retry budget, Deciding to Retry product launches and, Overload Behavior and Load Tests request criticality, Criticality retrying requests, Deciding to Retry(see also retries, RPC) utilization signals, Utilization Signals(see also cascading failures) P package managers, Packaging packet encapsulation, Load Balancing at the Virtual IP Address Paxos consensus algorithmClassic Paxos algorithm, Reasoning About Performance: Fast Paxos disk access and, Disk Access Egalitarian Paxos consensus algorithm, Stable Leaders Fast Paxos consensus algorithm, Reasoning About Performance: Fast Paxos, The Use of Paxos Lamport’s Paxos protocol, How Distributed Consensus Works(see also consensus algorithms) performanceefficiency and, Efficiency and Performance monitoring, Worrying About Your Tail (or, Instrumentation and Performance) performance tests, System tests periodic pipelines, Challenges with the Periodic Pipeline Pattern periodic scheduling (see cron) persistent storage, Disk Access Photon, Number of Replicas pipelining, Batching planned changes, Planned Changes, Drains, or Turndowns policies and procedures, enforcing, Enforcement of Policies and Procedures post hoc analysis, Setting Reasonable Expectations for Monitoring postmortemsbenefits of, Postmortem Culture: Learning from Failure best practices for, Google’s Postmortem Philosophy-Introducing a Postmortem Culture, Postmortems collaboration and sharing in, Collaborate and Share Knowledge concept of, Postmortem Culture: Learning from Failure cross-industry lessons, Postmortem Culture-Postmortem Culture example postmortem, Example Postmortem-Timeline formal review and publication of, Collaborate and Share Knowledge Google's philosophy for, Google’s Postmortem Philosophy guidelines for, Ensuring a Durable Focus on Engineering introducing postmortem cultures, Introducing a Postmortem Culture on-call engineering and, A Hunger for Failure: Reading and Sharing Postmortems ongoing improvements to, Conclusion and Ongoing Improvements rewarding participation in, Introducing a Postmortem Culture triggers for, Google’s Postmortem Philosophy privacy, Choosing a Strategy for Superior Data Integrity proactive testing, Encourage Proactive Testing problem reports, Problem Report process death, Process Death process health checks, Stop Health Check Failures/Deaths process updates, Process Updates process-induced emergencies, Process-Induced Emergency Prodtest (Production Test), Detecting Inconsistencies with Prodtest product launchesbest practices for, Progressive Rollouts defined, Reliable Product Launches at Scale development of Launch Coordination Engineering (LCE), Development of LCE-Infrastructure churn driving convergence and simplification, Driving Convergence and Simplification launch coordination checklists, The Launch Checklist-Example action items, Launch Coordination Checklist launch coordination engineering, Launch Coordination Engineering NORAD Tracks Santa example, Reliable Product Launches at Scale overview of, Conclusion processes for, Setting Up a Launch Process rate of, Reliable Product Launches at Scale techniques for reliable, Selected Techniques for Reliable Launches-Overload Behavior and Load Tests production environment (see Google production environment) production inconsistenciesdetecting with Prodtest, Detecting Inconsistencies with Prodtest resolving idempotently, Resolving Inconsistencies Idempotently production meetings, Communications: Production Meetings-Attendanceagenda example, Example Production Meeting Minutes production probes, Production Probes Production Readiness Review process (see SRE engagement model) production tests, Production Tests protocol buffers (protobufs), Our Software Infrastructure, Integration Protocol Data Units, Load Balancing at the Virtual IP Address provisioning, guidelines for, Provisioning PRR (Production Readiness Review) model, The PRR Model, Production Readiness Reviews: Simple PRR Model-Continuous Improvement push frequency, Motivation for Error Budgets push managers, Ongoing responsibilities Python’s safe_load, Integration Q “queries per second” model, The Pitfalls of “Queries per Second” Query of Death, Process Death queuingcontrolled delay, Load Shedding and Graceful Degradation first-in, first-out, Load Shedding and Graceful Degradation last-in, first-out, Load Shedding and Graceful Degradation management of, Queue Management, Reliable Distributed Queuing and Messaging queuing-as-work-distribution pattern, Reliable Distributed Queuing and Messaging quorum (see distributed consensus systems) R Raft consensus protocol, Multi-Paxos: Detailed Message Flow, Stable Leaders(see also consensus algorithms) RAID, Overarching Layer: Replication Rapid automated release system, Continuous Build and Deployment, Rapid read workload, scaling, Scaling Read-Heavy Workloads real backups, Backups Versus Archives real-time collaboration, Collaborate and Share Knowledge recoverability, Challenges of Maintaining Data Integrity Deep and Wide recovery, Knowing That Data Recovery Will Work recovery systems, Delivering a Recovery System, Rather Than a Backup System recursion (see recursion) recursive DNS servers, Load Balancing Using DNS recursive separation of responsibilities, Recursive Separation of Responsibilities redundancy, Challenges of Maintaining Data Integrity Deep and Wide, Overarching Layer: Replication Reed-Solomon erasure codes, Overarching Layer: Replication regression tests, System tests release engineeringchallenges of, Release Engineering continuous build and deployment, Continuous Build and Deployment-Configuration Management defined, Release Engineering instituting, Start Release Engineering at the Beginning philosophy of, Philosophy-Enforcement of Policies and Procedures the role of release engineers, The Role of a Release Engineer wider application of, Conclusions reliability testingamount required, Testing for Reliability benefits of, Conclusion break-glass mechanisms, Expect Testing Fail canary tests, Canary test configuration tests, Configuration test coordination of, The Need for Speed creating test and build environments, Creating a Test and Build Environment error budgets, Pursuing Maximum Change Velocity Without Violating a Service’s SLO, Motivation for Error Budgets-Forming Your Error Budget, Error Budgets expecting test failure, Expect Testing Fail-Expect Testing Fail fake backend versions, Production Probes goals of, Testing for Reliability importance of, Preface integration tests, Integration tests, Integration MTTR and, Testing for Reliability performance tests, System tests proactive, Encourage Proactive Testing production probes, Production Probes production tests, Production Tests regression tests, System tests reliability goals, Embracing Risk sanity testing, System tests segregated environments and, Pushing to Production smoke tests, System tests speed of, The Need for Speed statistical tests, Testing Disaster stress tests, Stress test system tests, System tests testing at scale, Testing at Scale-Production Probes timing of, Production Tests unit tests, Unit tests reliable replicated datastores, Reliable Replicated Datastores and Configuration Stores Remote Procedure Call (RPC), Our Software Infrastructure, Examine, Criticalitybimodal, Bimodal latency deadlinesmissing, Missing deadlines propagating, Load Shedding and Graceful Degradation, Deadline propagation queue management, Queue Management, Reliable Distributed Queuing and Messaging selecting, Latency and Deadlines retries, Retries-Retries RPC criticality, Criticality(see also overload handling) replicasadding, Capacity and Load Balancing drawbacks of leader replicas, Capacity and Load Balancing location of, Location of Replicas, Quorum composition number deployed, Number of Replicas replicated logs, Number of Replicas replicated state machine (RSM), Reliable Replicated State Machines replication, Challenges of Maintaining Data Integrity Deep and Wide, Overarching Layer: Replication request latency, Indicators, The Four Golden Signals request profile changes, Request profile changes request success rate, Measuring Service Risk resilience testing, Practices resourcesallocation of, Hardware, Managing Machines exhaustion, Resource Exhaustion limits, Resource limits(see also capacity planning) restores, 1T Versus 1E: Not “Just” a Bigger Backup retention, Retention retries, RPCavoiding, Deciding to Retry cascading failures due to, Retries considerations for automatic, Retries diagnosing outages due to, Retries handling overload errors and, Handling Overload Errors per-client retry budgets, Deciding to Retry per-request retry budgets, Deciding to Retry reverse engineering, Reverse Engineers: Figuring Out How Things Work reverse proxies, What went well revision history, First Layer: Soft Deletion risk managementbalancing risk and innovation, Embracing Risk costs of, Managing Risk error budgets, Motivation for Error Budgets-Benefits, Error Budgets key insights, Benefits measuring service risk, Measuring Service Risk risk tolerance of services, Risk Tolerance of Services-Example: Frontend infrastructure rollback procedures, What we learned rollouts, New Rollouts, Rollout Planning, Progressive Rollouts root causeanalysis of, Practices, Google’s Postmortem Philosophy(see also postmortems) defined, Definitions Round Robin policy, Simple Round Robin round-trip-time (RTT), Distributed Consensus Performance and Network Latency rows, Hardware rule evaluation, in monitoring systems, Rule Evaluation-Rule Evaluation S Safari® Books Online, Safari® Books Online sanity testing, System tests saturation, The Four Golden Signals scaledefined, Choosing a Strategy for Superior Data Integrity issues in, Scaling issues: Fulls, incrementals, and the competing forces of backups and restores securityin release engineering, Enforcement of Policies and Procedures new approach to, Practices self-service model, Self-Service Model separation of responsibilities, Recursive Separation of Responsibilities serversvs. clients, Our Software Infrastructure defined, Hardware overload scenario, Server Overload preventing overload, Preventing Server Overload-Always Go Downward in the Stack service availabilityavailability table, Availability Table cost factors, Cost, Cost defined, Indicators target for consumer services, Target level of availability target for infrastructure service, Target level of availability time-based equation, Measuring Service Risk types of consumer service failures, Types of failures types of infrastructure services failures, Types of failures service health checks, Stop Health Check Failures/Deaths service latencylooser approach to, Other service metrics monitoring for, The Four Golden Signals service level agreements (SLAs), Agreements service level indicators (SLIs)aggregating raw measurements, Aggregation collecting indicators, Collecting Indicators defined, Indicators standardizing indicators, Standardize Indicators service level objectives (SLOs)agreements in practice, Agreements in Practice best practices for, Define SLOs Like a User choosing, Service Level Objectives-Objectives control measures, Control Measures defined, Objectives defining objectives, Objectives in Practice selecting relevant indicators, What Do You and Your Users Care About?

…

For details, see “Testing for Cascading Failures”. Serve degraded results Serve lower-quality, cheaper-to-compute results to the user. Your strategy here will be service-specific. See “Load Shedding and Graceful Degradation”. Instrument the server to reject requests when overloaded Servers should protect themselves from becoming overloaded and crashing. When overloaded at either the frontend or backend layers, fail early and cheaply. For details, see “Load Shedding and Graceful Degradation”. Instrument higher-level systems to reject requests, rather than overloading servers Note that because rate limiting often doesn’t take overall service health into account, it may not be able to stop a failure that has already begun.

…

For example, Gmail often uses queueless servers, relying instead on failover to other server tasks when the threads are full. On the other end of the spectrum, systems with “bursty” load for which traffic patterns fluctuate drastically may do better with a queue size based on the current number of threads in use, processing time for each request, and the size and frequency of bursts. Load Shedding and Graceful Degradation Load shedding drops some proportion of load by dropping traffic as the server approaches overload conditions. The goal is to keep the server from running out of RAM, failing health checks, serving with extremely high latency, or any of the other symptoms associated with overload, while still doing as much useful work as it can.

Building Secure and Reliable Systems: Best Practices for Designing, Implementing, and Maintaining Systems
by Heather Adkins , Betsy Beyer , Paul Blankinship , Ana Oprea , Piotr Lewandowski and Adam Stubblefield
Published 29 Mar 2020

Figure 8-3 also distinguishes the uncontrolled nature of degraded traffic (the backward-slashed area) prior to system crash. Figure 8-4 shows that the system with load shedding rejects significantly less traffic than in Figure 8-3 (the crosshatched area), with the rest of the traffic either processed without failure (whitespace area) or rejected if lower priority (forward-slashed area). Figure 8-3. Complete outage and a possible cascading failure from a load spike Figure 8-4. Using load shedding and throttling to manage a load spike Load shedding The primary resilience objective of load shedding (described in Chapter 22 of the SRE book) is to stabilize components at maximum load, which can be especially beneficial for preserving security-critical functions.

…

Creating a safeguard by permanently overprovisioning servers wastes money and doesn’t guarantee a safe response. Instead, servers should adjust how they respond to load based upon current conditions. You can use two specific automation strategies here: Load shedding is done by returning errors rather than serving requests. Throttling of clients is done by delaying responses until closer to the request deadline. Figure 8-3 illustrates a traffic spike that exceeds the capacity. Figure 8-4 illustrates the effects of load shedding and throttling to manage the load spike. Note the following: The curve represents requests per second, and the area under it represents total requests. Whitespace represents traffic processed without failure.

…

This approach reduces the rate of requests the server receives from clients (if clients send requests sequentially), which means that you can redirect the resources saved during wait times. Similar to load shedding, you could define policies to apply throttling to specific offending clients, or more generally to all clients. Request priority and cost play a role in selecting which requests to throttle. Automated response Server utilization statistics can help determine when to consider applying controls like load shedding and throttling. The more heavily a server is loaded, the less traffic or load it can handle. If controls take too long to activate, higher-priority requests may end up being dropped or throttled.

The Elements of Choice: Why the Way We Decide Matters
by Eric J. Johnson
Published 12 Oct 2021

Sullenberger, in discussing his decision, revealed that he immediately established what was important: So I quickly set priorities, I load shed—pared down this problem to its essential elements—did the few things that had to be done, did them very well and I was willing to goal sacrifice. I knew that the highest priority was to save lives, and I was more than willing to give up trying to save the airplane very early on in order to do that. And that was an easy choice for me to make, although as [First Officer] Jeff Skiles would tell me later, “You put a $62 million airliner in the river and they call you a hero. Is this a great country, or what?”2 Sullenberger borrowed the term load shedding from electrical utilities: when demand exceeds their capacity, a utility may take parts of its network off-line, shutting down, say, power to a factory, to provide resources to another more crucial location, like a hospital.

…

What did the gauge enable Sullenberger to think about, and what did it enable him to ignore? How did it help him load-shed? He says that in the midst of everything, he considered the location for a water landing. The Hudson is cold in mid-January. The water temperature was 41 degrees Fahrenheit, and the air a chilly 19 degrees. Hypothermia was a real risk. Even if they successfully landed, the plane might float for only a short time. Sullenberger knew that there were many boats along Manhattan’s west side, and a ferry terminal nearby. Because he was able to load-shed, he considered not only where he could safely land the plane but rescue possibilities as well, thinking about what would happen next.

…

2 Sullenberger borrowed the term load shedding from electrical utilities: when demand exceeds their capacity, a utility may take parts of its network off-line, shutting down, say, power to a factory, to provide resources to another more crucial location, like a hospital. Pilots use this phrase when they are overwhelmed, and they decide to ignore what they hope are nonessential parts of the problem. Choosing a plausible path is a form of load shedding: it involves making decisions about what is crucial information to consider for meeting goals and what information is nonessential and can be taken off-line. The way information is displayed to the chooser influences their choice of a plausible path. Lost in the discussion of the bravery of Sullenberger, First Officer Skiles, and the three flight attendants was the role of the cockpit displays.

Shorting the Grid: The Hidden Fragility of Our Electric Grid
by Meredith. Angwin
Published 18 Oct 2020

In some states, RECs can be purchased from distant states. Restructured states or areas: States (or areas within states) that have joined an RTO market. RMR: Reliability Must Run. Rolling blackouts: A form of load shedding, in which a utility schedules power outages for one area and then returns the power to that area and schedules a power outage for another area. The blackout (load shedding) “rolls” from place to place. See Load shedding. RTO: Regional Transmission Organization. Southwest Power Pool (SPP): The Regional Transmission Organization whose territory covers much of the Midwest and some areas of the South. SPP territories lie mostly to the west of MISO territories.

…

Nobody is responsible IN CALIFORNIA, AS ELSEWHERE, the distribution utilities were responsible for keeping their lines and substations running. The lines and substations ran fine. The RTO (CAISO) was responsible for dispatching the plants that were available for dispatch. If too many of them were offline for “maintenance” and there had to be “load shedding” (rolling blackouts), well, CAISO is not responsible for plant maintenance, just for dispatch. Every individual plant makes money when it runs. If a plant needs maintenance or whatever at an awkward time, it’s not making money at that time. That individual plant is not responsible for keeping the grid going.

…

In early April, Chicago-based energy provider Exelon Corp. said it would close two large natural-gas units at Mystic Station, Massachusetts. In its report about possibilities for the winter of 2024–25, ISO-NE had included the loss of these two plants in one of its scenarios. The ISO-NE report concluded that Mystic’s possible closure would cause 20 to 50 hours of “load shedding” (meaning rolling blackouts) and hundreds of hours of grid operation under emergency protocols. When Exelon made its closure announcement, ISO-NE realized that the danger of rolling blackouts was suddenly more immediate than 2024. It now hopes to provide “out-of-market-cost recovery” — subsidies — to persuade Exelon to keep the Mystic plants operating.

The Practice of Cloud System Administration: DevOps and SRE Practices for Web Services, Volume 2
by Thomas A. Limoncelli , Strata R. Chalup and Christina J. Hogan
Published 27 Aug 2014

Additional capacity can be found at other service providers as well. A public cloud computing provider can be used as the shared pool. Usually you will not have to pay for unused capacity. Shared resource pools are not just appropriate for machines, but may also be used for storage and other resources. Load Shedding Another strategy is load shedding. With this strategy the service turns away some users so that other users can have a good experience. To make an analogy, an overloaded phone system doesn’t suddenly disconnect all existing calls. Instead, it responds to any new attempts to make a call with a “fast busy” tone so that the person will try to make the call later.

…

A variation of load shedding is stopping certain tasks that can be put off until later. For example, low-priority database updates could be queued up for processing later; a social network that stores reputation points for users might store the fact that points have been awarded rather than processing them; nightly bulk file transfers might be delayed if the network is overloaded. That said, tasks that can be put off for a couple of hours might cause problems if they are put off forever. There is, after all, a reason they exist. For any activity that is delayed due to load shedding, there must be a plan on how such a delay is handled.

…

Establish a service level agreement (SLA) to determine how long something can be delayed and to identify a timeline of actions that should be undertaken to mitigate problems or extend the deadlines. Low-priority updates might become a high priority after a certain amount of time. If many systems are turned off due to load shedding, it might be possible to enable them, one at a time, to let each catch up. To be able to manage such situations one must have visibility into the system so that prioritization decisions can be made. For example, knowing the age of a task (how long it has been delayed), predicting how long it will take to process, and indicating how close it is to a deadline will permit operations personnel to gauge when delayed items should be continued

Ten Technologies to Save the Planet: Energy Options for a Low-Carbon Future
by Chris Goodall
Published 1 Jan 2010

To the electricity provider, these reductions may be equivalent to having an extra power station available; however, a spare power station that sits idle 90 percent of the time would be a much more expensive and polluting solution. The major load-shedding programs around the world typically cover about 2 percent of peak power use, enough to cope with temporary energy deficiencies as long as wind is not too great a percentage of total electricity supply. For Spain and Denmark, countries where wind occasionally provides a very large fraction of total electricity demand for several hours at a time, this scheme would not be enough on its own. However, many people in the electricity industry think that load-shedding programs could cover 10 percent of peak power use or even more.

…

Otherwise, the voltage or frequency of the alternating current would move outside the tolerances of home appliances and business equipment, possibly causing damage. Pumped storage is a way of quickly adjusting supply, but there are also methods to almost instantaneously reduce demand. In the jargon of the electricity industry, this activity is known as “load shedding.” Some manufacturing companies, for example, have agreements that allow their electricity supplier to disconnect them at a few moments’ notice. In return, they pay lower prices for their electricity. This system works well. Although it is designed primarily to shave a little off the sharp daily peaks of electricity demand, there is no reason why the same approach couldn’t be adapted to deal with temporary shortfalls in wind power at all times of day.

…

A signal carried over the mobile phone network might trigger an electronic on/off switch at the wall socket of those electric appliances that use large amounts of electricity. A program involving all three of these techniques for balancing supply and demand—better electricity grids, greater energy storage, and load shedding—allows electricity operators to deal with the occasional unexpected drop in wind power. There will be some costs, and some inconvenience, such as washing machines unexpectedly switching off for an hour or so, but it can be done. In many parts of the world, regional electricity grids are already finding ways to reduce peak demand.

The Nature of Software Development: Keep It Simple, Make It Valuable, Build It Piece by Piece
by Ron Jeffries
Published 14 Aug 2015

The caller will stop waiting for a response on the original request and probably fire a retry at us (exactly when it hurts the worst!). Preventing Disaster With that perspective, we can see that the best thing to do under high load is turn away work we can’t complete in time. This is called “load shedding,” and it’s the most important way to control incoming demand. Load shedding happens very quickly when a socket’s listen queue is full, and a quick rejection is better than a slow timeout. More generally, we want to shed load as early as possible so we can avoid tying up resources at several tiers before rejecting the request. Load balancers near the network edge are the ideal place.

…

At some point, though, the API server still has a thread waiting on a call. As we saw in ​Blocked Threads​, blocked threads are a quick path to downtime. At the edge of your system boundary, blocked threads will frustrate a user or provoke a retry loop. As such, back pressure works best within a system boundary. At the edges, you also need load shedding and asynchronous calls. In our example, the API server should accept calls on one thread pool and then issue the outbound call to storage on another set of threads. That way, when the outbound call blocks, the request-handling thread can time out, unblock, and respond with an HTTP 503. Alternatively, it could drop a “create tag” command in a queue for later indexing.

…

That way you can tell whether it’s a random fluctuation or a trend. Remember This Back Pressure creates safety by slowing down consumers. Consumers will experience slowdowns. The only alternative is to let them crash the provider. Apply Back Pressure within a system boundary Across boundaries, look at load shedding instead. This is especially true when the Internet at large is your user base. Queues must be finite for response times to be finite. You only have a few options when a queue is full. All of them are unpleasant: drop data, refuse work, or block. Consumers must be careful not to block forever.

Smart Cities: Big Data, Civic Hackers, and the Quest for a New Utopia
by Anthony M. Townsend
Published 29 Sep 2013

Smart meters that communicate directly with smart appliances might automatically reschedule a load of wash for later in the day when demand and prices are likely to fall. Even the most sophisticated load-shifting scheme will one day meet its limit. That’s when utilities wield their trump card—load shedding—a kind of targeted blackout. Traditionally, load shedding was a manual process. Utilities would cut deals with large users of electricity like factories and universities to shut down power during peaking crises in return for a discount on their regular rates. Smart meters will allow these miniblackouts to be replaced by sophisticated surgical drawdowns on sacrificial facilities and equipment.

…

While peaking plants can also be highly efficient—most are natural-gas–powered turbines—they are far more costly per unit of power to build and run. If only the peaks could be evened out, fewer peaking plants would be needed and utilities could focus more on ruthlessly fine-tuning base load plants to be as lean and clean as possible.48 Smart grids offer two tricks to even out the peaks: load shifting and load shedding. Load shifting, the gentler of the two, tries to spread demand for power away from peak periods of demand through price incentives. In their simplest form, smart meters allow businesses and consumers to see the true cost of generating electricity during periods of high demand. As they fire up those costly peaking plants, utilities simply pass the higher generating cost along to consumers.

…

The wiggle room that once existed in the form of reserve generating capacity is fast disappearing, raising the possibility of regular blackouts in the future. During the 1990s, demand for electricity grew by 35 percent in the United States, but generating capacity increased by only 18 percent.49 According to Siemens, smart grids will help utility engineers sleep at night, since load shedding and load shifting could reduce national electricity needs by up to 10 percent. 50 Environmentalists will cheer because improved demand management removes a key obstacle to greater reliance on renewable generating sources, which are notoriously unreliable base capacity—the sun doesn’t always shine and the wind doesn’t always blow.

Building Microservices
by Sam Newman
Published 25 Dec 2014

Timeouts and circuit breakers help you free up resources when they are becoming constrained, but bulkheads can ensure they don’t become constrained in the first place. Hystrix allows you, for example, to implement bulkheads that actually reject requests in certain conditions to ensure that resources don’t become even more saturated; this is known as load shedding. Sometimes rejecting a request is the best way to stop an important system from becoming overwhelmed and being a bottleneck for multiple upstream services. Isolation The more one service depends on another being up, the more the health of one impacts the ability of the other to do its job.

…

Index A acceptance testing, Types of Tests access by reference, Access by Reference accountability, People adaptability, Summary Aegisthus project, Backup Data Pump aggregated logs, Logs, Logs, and Yet More Logs… antifragile systems, Microservices, The Antifragile Organization-Isolationbulkheads, Bulkheads circuit breakers, Circuit Breakers examples of, The Antifragile Organization increased use of, Microservices isolation, Isolation load shedding, Bulkheads timeouts, Timeouts AP systemdefinition of term, Sacrificing Consistency vs. CP system, AP or CP? API key-based authentication, API Keys, It’s All About the Keys application containers, Application Containers architects (see systems architects) architectural principlesdevelopment of, Principles Heroku's 12 factors, Principles key microservices principles, Bringing It All Together real-world example, A Real-World Example architectural safety, Architectural Safety, Architectural Safety Measures artifactsimages, Images as Artifacts operating system, Operating System Artifacts platform-specific, Platform-Specific Artifacts asynchronous collaborationcomplexities of, Complexities of Asynchronous Architectures implementing, Implementing Asynchronous Event-Based Collaboration vs. synchronous, Synchronous Versus Asynchronous ATOM specification, Technology Choices authentication/authorization, Authentication and Authorization-The Deputy Problemdefinition of terms, Authentication and Authorization fine-grained, Fine-Grained Authorization service-to-service, Service-to-Service Authentication and Authorization single sign-on (SSO), Common Single Sign-On Implementations single sign-on gateway, Single Sign-On Gateway terminology, Common Single Sign-On Implementations automationbenefits for deployment, Automation case studies on, Two Case Studies on the Power of Automation autonomymicroservices and, Autonomous role of systems architect in, Summary autoscaling, Autoscaling availabilityin CAP theorem, CAP Theorem key microservices principle of, How Much Is Too Much?

…

JSON web tokens (JWT), HMAC Over HTTP K Karyon, Tailored Service Template key-based authentication, API Keys Kibana, Logs, Logs, and Yet More Logs… L latency, How Much Is Too Much? Latency Monkey, The Antifragile Organization layered architectures, Microservices librariesclient, Client Libraries service metrics, Service Metrics shared, Shared Libraries Linux containers, Linux Containers load balancing, Load Balancing load shedding, Bulkheads local calls, Local Calls Are Not Like Remote Calls logsaggregated, Logs, Logs, and Yet More Logs…(see also monitoring) security issues, Logging standardization of, Standardization logstash, Logs, Logs, and Yet More Logs… loose coupling, Loose Coupling, Orchestration Versus Choreography, Loose and Tightly Coupled Organizations M man-in-the-middle attacks, Allow Everything Inside the Perimeter Marick's quadrant, Types of Tests maturity, Maturity mean time between failures (MTBF), Mean Time to Repair Over Mean Time Between Failures?

Nepal Travel Guide
by Lonely Planet

Avoid marches, demonstrations and disturbances, as they can quickly turn violent. The website www.nepalbandh.com warns of any upcoming strikes. When roads are closed the government generally runs buses with armed police from the airport to major hotels, returning to the airport from Tridevi Marg at the east end of Thamel. Load Shedding Electricity cuts (‘load shedding’) are a fact of life in Kathmandu and Pokhara, especially in winter, when water and thus hydro power levels are at their lowest. Electricity is currently rationed across Kathmandu, shifting from district to district every eight hours or so. Most hotels post a schedule of planned electricity cuts, which can last up to 16 hours a day in both Kathmandu and Pokhara.

…

Information Emergency Ambulance service ( 4521048) Provided by Patan Hospital. Fire Brigade ( 101, 4221177) Police ( 100, 4223011; www.nepalpolice.gov.np; Durbar Sq) Red Cross Ambulance ( 4228094) Tourist Police Bhrikuti Mandap ( 4247041); Thamel ( 4700750) Dangers & Annoyances Electricity cuts (‘load shedding’) are a fact of life in Kathmandu; they last for up to 16 hours a day in winter when hydro power levels are at their lowest. Electricity is currently rationed across the city, shifting from district to district every eight hours or so. Most hotels post a schedule of planned electricity cuts. Try to choose a hotel with a generator and make sure your room is far away from it.

…

Electricity Electricity is 220V/50 cycles; 120V appliances from the USA will need a transformer. Sockets usually take plugs with three round pins, sometimes the small variety, sometimes the large. Some sockets take plugs with two round pins. Local electrical shops sell cheap adapters. Blackouts (‘load shedding’) are a fact of life across Nepal, especially in Kathmandu; these peak in February with up to 16 hours a day of cuts. Power surges are also likely, so bring a voltage guard with spike suppressor (automatic cut-off switch) for your laptop. Embassies & Consulates Travellers continuing beyond Nepal may need visas for Bangladesh, China, India, Myanmar (Burma) and Thailand.

The Key Man: The True Story of How the Global Elite Was Duped by a Capitalist Fairy Tale
by Simon Clark and Will Louch
Published 14 Jul 2021

Her officials called Karachi Electric’s chief executive, Naveed Ismail, and listened carefully to what he had to say. Naveed said that Karachi Electric was controlled by Abraaj and required billions of dollars of loans and investment to improve its operations and reduce power cuts, which he called “load shedding.” After the meeting, Ambassador Patterson sent a cable to State Department colleagues asking for help to encourage Abraaj to raise the required funds for Karachi Electric. “Black outs and load shedding are a serious impediment to economic productivity. Poor power delivery has also led to large public demonstrations,” the American diplomatic cable said. “Embassy Islamabad requests the Department seek Consulate Dubai’s assistance.”

Imagining India
by Nandan Nilekani
Published 25 Nov 2008

There is no escaping our infrastructure problems, even here in India’s capital city—the Delhi newspapers during my visit have been full of headlines on the city’s power outages. A single day that week, as one outraged journalist wrote, had seen “10 periods of ‘load-shedding.’” In fact this particular bit of bureaucratspeak we use, “load-shedding,” reveals how a growing economy has found it difficult to look its crisis in the face.bt Our bad roads and power cuts are a reminder of our prereform years—it is here that we can most clearly see the evidence of India’s old structures, the tattered vestiges of socialism in an emerging free-market economy.

…

br It is a different story that this rule has often been violated by state governments. bs Bangalore can lay claim to pioneering bond issues by cities in India—India’s first such bond issue was by the Bangalore Mahanagarapalika in 1997 of Rs 1 billion, with a coupon rate of 13 percent. bt As a technical term, “load-shedding” means power cuts to tackle spikes in excess demand. But what India faces is consistent and severe power shortage. bu This view of roads and railways as an investment toward safety—to move people and goods in and out quickly, and avoid being cornered by enemies—has plenty of precedent. The Romans built Britain’s major road systems when they had occupied the restive island, and many of these still exist.

Nerds on Wall Street: Math, Machines and Wired Markets
by David J. Leinweber
Published 31 Dec 2008

Smart meters with Internet communication capabilities were the enabling technology for the next wave of technologydriven savings, designed to bring the suppliers into the process. This was 336 Nerds on Wall Str eet done using simple command-and-control load-shedding measures that allowed utilities, with prior agreement of larger customers, to shed loads during periods of peak demand or to remotely adjust airconditioning thermostats upward to reduce demand when needed. The next step, still in its infancy, is to introduce real-time electricity pricing. This will allow consumers to make their own economically motivated load-shedding decisions, and to program their meters to implement those decisions for them. Day Trading for Electrons In the best seller Hot, Flat, and Crowded: Why We Need a Green Revolution—and How It Can Renew America (Farrar, Straus & Giroux, 2008), New York Times columnist Thomas Friedman calls the use of smart meters to create a market-based system of energy technology (ET), “when IT meets ET—day trading for electrons.”

Adventures in the Anthropocene: A Journey to the Heart of the Planet We Made
by Gaia Vince
Published 19 Oct 2014

It is hard to see how that will happen. Drought conditions are already causing blackouts as the nation’s hydropower struggles with lack of water and poor energy infrastructure. As throughout the developing world, most of the blackouts are planned load-shedding intervals, where the government tries to manage the poor supply by rationing electricity among different regions in turn. The load-shedding is seriously impacting Nepal’s fledgling electric car industry, its 700 Safa Tempos (three-wheeled electric passenger vans), which cleanly ply Kathmandu’s streets, charging at thirty-two stations and transporting around 100,000 people a day.

Countdown: Our Last, Best Hope for a Future on Earth?
by Alan Weisman
Published 23 Sep 2013

Everyone is back: men in white kurtis and prayer caps; hijab’d women brilliant as tropical birdlife, wrapped in multihued loose pants and tunics called salwar kameez; other women in black chadors, even transvestites in chadors, all threading their way through the stalled traffic, buying provisions and tea. Since 10:00 a.m., that traffic has gone from mere paralysis to pandemonium, as Lyari’s stoplights are out for the next three hours. All but the most privileged parts of Karachi are subject to load-shedding—daily rolling blackouts—because the city can’t possibly keep up with demand. There were fewer than a half-million people here in 1947. Today’s 21 million is a forty-two-fold increase. No one could have prepared for this. Three days earlier, when the grenade attacks began, everyone stayed hidden until long after the explosions ended.

…

He has six children with his first wife, and three with her. She wants to keep pleasing him so he’ll be attentive. Alma takes her hand. “But in your condition, more pregnancies are a risk.” The overhead fan quits and the deadened air is immediately stifling. The women exchange worried glances, because load-shedding was over for the day, and unexpected blackouts often signal yet a new civil disturbance. Asma pulls a flashlight from her desk drawer and motions for the next client. An entire bench stands and approaches: five women in white burqas bordered with lace, their dark eyes peering through woven lattice grills in their face hoods.

The Reluctant Carer: Dispatches From the Edge of Life
by The Reluctant Carer
Published 22 Jun 2022

‘I couldn’t stop looking at it!’ says Mum of one’s nose-piercing. It’s as though characters from their television have walked directly into the room. 25 January 2018 I get an encouraging email from my friend Ron who lives abroad and knows a lot about energy: In electricity there is a term: load-shedding. Required to keep the power supply constant, in some way. A technical thing. I hope for some equivalent for you. He is right. I go to London, shed that load a little, write and reflect and confess. The Mighty Fall 28 January 2018 In my brief absence my benign, or perhaps lacklustre, regime has been replaced by a surveillance-gathering operation run by the unlikely but effective partnership of my deaf mum and suspicious sister.

Engineering Security
by Peter Gutmann

A novice driver has to manually and consciously perform actions such as changing gears and checking the rear-view mirror, while for an experienced driver these actions occur automatically without any conscious effort. To cope with the inability to handle the driving process via automatic actions, novice drivers load-shed by ignoring one of the two main aspects of driving (speed control and steering), with the result that they crawl down the road at an irritatingly slow speed while they concentrate on steering. It’s not until both aspects of vehicle control have become automatic processes that the novices progress to the level of more experienced drivers [108]. 138 Psychology This load-shedding phenomenon is particularly pernicious in situations like aircraft control, where a multitude of events such as an unexpected occurrence during an already complex operation like takeoff or landing can overload a pilot’s ability to maintain full awareness of the situation [109][110][111].

…

It’s not until both aspects of vehicle control have become automatic processes that the novices progress to the level of more experienced drivers [108]. 138 Psychology This load-shedding phenomenon is particularly pernicious in situations like aircraft control, where a multitude of events such as an unexpected occurrence during an already complex operation like takeoff or landing can overload a pilot’s ability to maintain full awareness of the situation [109][110][111]. In order to cope with the situational awareness problem the pilot load-sheds, and the result of this process is written up by the board of inquiry as “pilot error”. Mind you, you can sometimes use the load-shedding phenomenon to your advantage. If you suspect that someone’s lying to you, you can increase their cognitive load to the point where they can no longer both maintain the lie and deal with the extra load. One way of doing this is to get them to narrate their story in reverse, which is awkward because it runs counter to the natural sequence of events and increases the chances that the liar will trip themselves up [112].

…

You can see the conflict between conscious and unconscious processing yourself if you write something simple like your name or the weekday repeatedly across a piece of paper and at some point start counting backwards from 100 while you write. Look at what happens to either your writing speed or writing quality when you do this, depending on which load-shedding strategy you choose to adopt. Now try it again but this time sign your name (an automatic process for most people) and see what happens. This simple experiment in fact mirrors some of the early investigations into the phenomenon of attention that were carried out in the 1950s, which involved seeing whether and how much performing one task interfered with another [115][116].

The Return of Marco Polo's World: War, Strategy, and American Interests in the Twenty-First Century
by Robert D. Kaplan
Published 6 Mar 2018

While it is true that Pakistan’s government is increasingly being held accountable by a burgeoning media and nongovernmental organizations—thereby expanding civil society at the top end of the spectrum in Islamabad and Lahore—and it is also true that interparty warfare in Islamabad has lessened somewhat, the country in fundamental ways continues to deteriorate. Electricity blackouts (“load shedding,” as they are called) are more persistent now than ever, and water shortages are worsening. The situation is fluid. Nuclear power and coal imports may soon alleviate the power blackouts, even as the army has reportedly moved away somewhat from encouraging Islamic radicalism. But Pakistan’s population growth is still above 2 percent annually, meaning its population doubles every thirty-five years.

Wiseguy: The 25th Anniversary Edition
by Nicholas Pileggi

Judy, who was a friend of the family, was already at my house when my brother and I got there. She looked like a Kansas preacher’s daughter. That, of course, was what made her such a good courier. Skinny, dirty-blond hair, dumb pink-and-blue hat and crummy Dacron clothes out of the Sears catalogue. Sometimes, with heavy loads, she’d borrow a baby for the trip. She looked so pathetic that the only people who ever stopped her were Travelers Aid social workers looking to stir up business. Judy was going to hang around the house until I got back with the stuff. Then, after we had all had dinner, I was going to drive her to the airport for her flight to Pittsburgh.

Nomad Century: How Climate Migration Will Reshape Our World
by Gaia Vince
Published 22 Aug 2022

Demand for cooling will soar this century, becoming a key social justice issue, especially during heatwaves, when lack of access will prove deadly. Cooling already uses 20 per cent of global energy production, and this is expected to triple by 2050. Months of heatwave in the spring of 2022 across India and Pakistan meant hundreds of thousands of people were unable to work after 10 a.m., with load-shedding power outages leaving people without access to cooling or refrigeration. Cooling is not just going to be a problem in the tropics, where there is already fast-rising demand, but in today’s temperate zones where vast populations will be headed. Insulation will help manage this burden, and strategic use of water – used for cooling by architects and planners for centuries – will also play a role.

Private Equity: A Memoir
by Carrie Sun
Published 13 Feb 2024

We arrived in the conference room and sat equidistantly around the large table. I looked at Jay, whose hair was uncombed. I looked at Lena, who had been complaining mere hours ago about being exhausted yet unable to bring herself to say anything. I looked at Val, who had told me, after Boone agreed to my load-shedding, that she planned to send an email to Ari saying “ok now let’s cut back the work.” Before she sent it she spoke to one of the traders who’d been with Carbon since the aughts. “Why stop now and get a lower bonus,” he had said to her. He advised her to care less, do less work, not make it known that she was dialing it back.

Lonely Planet Cape Town & the Garden Route (Travel Guide)
by Lucy Corne
Published 1 Sep 2015

The Natural Environment Shaped over millions of years, the Cape’s dramatic features are carpeted by the richest floral kingdom in the world. Wine The forces that created this 350-year-old industry, and how it is tackling the working practices that blighted its past. Cape Town Today The legacy of the city’s stint as World Design Capital 2014 (WDC2014); the threat of electricity blackouts (known as ‘load shedding’) due to a struggling national grid; political tussles between the Democratic Alliance (DA), who run the city and the Western Cape, and national governing party the African National Congress (ANC); and is Cape Town the most racist city in South Africa? These are among the ongoing conversation topics across the Mother City.

When McKinsey Comes to Town: The Hidden Influence of the World's Most Powerful Consulting Firm
by Walt Bogdanich and Michael Forsythe
Published 3 Oct 2022

GO TO NOTE REFERENCE IN TEXT “For almost 95 years”: “McKinsey’s Partners Suffer from Collective Self-Delusion,” Economist, March 4, 2021. GO TO NOTE REFERENCE IN TEXT “When you have people”: Testimony of Mieszala to Commission of Inquiry into State Capture, Dec. 10, 2020, 157. GO TO NOTE REFERENCE IN TEXT In January 2020, the company was forced: “Eskom Misled Ramaphosa About Extent of Load-Shedding: David Mabuza,” TimesLIVE, Jan. 9, 2020. GO TO NOTE REFERENCE IN TEXT South Africa’s president had cut short: “President Cuts Short Egyptian Visit to Attend to Electricity Crisis,” South African Government News Agency, Dec. 11, 2019. GO TO NOTE REFERENCE IN TEXT The media reported that 2021 could: Jackie Cameron, “Loadshedding: 2021 to Be Worst Year Yet for SA Electricity Crisis—Chris Yelland, Energy Expert,” BizNews, Jan. 20, 2021.

Extreme Money: Masters of the Universe and the Cult of Risk
by Satyajit Das
Published 14 Oct 2011

A minister extols the virtues of India, citing statistics on growth, resource availability, and opportunities. There is no mention of the fact that the vast majority of the Indian population has no access to sanitation, clean water, education, or healthcare. There is no mention of the aging colonial era infrastructure where inadequate electricity supply results in daily load shedding or brownouts interrupting power supplies for several hours most days. An American banker finds India fascinating and full of opportunity. “A billion people, a billion consumers, wow!” He is fascinated that I, an Indian, do not speak Indian but Bengali, one of the hundreds of languages spoken in India.

The Architecture of Open Source Applications
by Amy Brown and Greg Wilson
Published 24 May 2011

Unlike the consistent hashing approach, two keys that are next to each other in the key's sort order are likely to appear in the same partition. This reduces the size of the routing metadata, as large ranges are compressed to [start, end] markers. In adding active record-keeping of the range-to-server mapping, the range partitioning approach allows for more fine-grained control of load-shedding from heavily loaded servers. If a specific key range sees higher traffic than other ranges, a load manager can reduce the size of the range on that server, or reduce the number of shards that this server serves. The added freedom to actively manage load comes at the expense of extra architectural components which monitor and route shards.

Ghost Train to the Eastern Star: On the Tracks of the Great Railway Bazaar
by Paul Theroux
Published 9 Sep 2008

Because of that, TCS provided a gym, a cafeteria and a resident doctor. And all employees commuted to work by the company shuttle service, which stopped at various hubs in the city. ‘Suppose there’s a power cut?’ I asked. Such things were common, and barely concealed under the euphemisms ‘brownout’, ‘rolling blackout’ or ‘load shedding’. ‘What happens then?’ ‘Last July we had power cuts. Ninety-three centimetres of rain in sixteen hours.’ That was more than three feet of rain in a little more than half a day! But Mr Randeria was smiling. ‘We had two hundred per cent redundancy backup. I’ll show you.’ He took me to a towering building at the rear of the complex.