Building slack into systems

What the covid-19 pandemic has revealed, at least as far as the US is concerned, is how delicately balanced the supply and distribution systems are. As long as things are normal, everything appears to run smoothly. But given a large enough disruption, the system can not only not cope, it cannot reconfigure itself quickly enough to meet the challenge. In this case, we have discovered that the supply of goods and services is highly dependent on a just-in-time supply chains for each item that are finely tuned for maximum efficiency and eliminate the need for costly stockpiling of supplies. But the sudden change in the way people live and work has resulted in shortages in some areas along with gluts in others, with no means for quickly redistributing the resources to reach a new equilibrium.

In an article in the May 4, 2020 issue of the New Yorker magazine titled What the Coronavirus Crisis Reveals About American Medicine, Siddhartha Mukherjee looked at how the US was unable to not only cope with the sudden and immediate large demand for masks, gloves, gowns, ventilators, and other personal protective equipment (PPEs), it was not able to do so for months after the initial deficiencies were exposed. Because so much of production of these items was in China, when that country found itself needing all its production output to deal with the domestic problem, there was little left for the rest of the world.

He argues that the tight but efficient supply chain networks need to have a certain amount of built-in slack to cope with disruptions. Slack does not necessarily consist of stockpiles of a lot of important stuff that can be drawn upon in an emergency, though some stockpiling is necessary. It also means having a plan to repurpose sources of supply in other areas to meet the needs in one area that suddenly is in need. US automakers switching to making ventilators is an example of what slack could be. But because there had been no prior planning for such a contingency, the effort has been stumbling and slow. This kind of slack should be planned for in advance and kept in reserve.

He gives an example of what happened in 1997 when a fire broke out in a company that was the sole supplier of a small but key brake system component for Toyota automobiles. The sudden loss of that item brought Toyota car production to an immediate halt. Panicked executives immediately went into emergency brainstorming mode and what happened next was extraordinary. Because Toyota’s success is a source of national pride in Japan, other companies quickly rallied round to support it and within just three days they had repurposed their factories to provide Toyota with the necessary part. Within a month, the company had rebounded and only suffered the loss of production of an astonishingly small number of 70,000 vehicles.

The point of the story is that this experience taught Toyota management a lesson, that they needed to build slack into their production supply chains so that if there was a similar disruption anywhere else for any other part, the company had a plan to deal with it. And they have done so.

Toyota’s recovery from the Aisin factory fire in 1997 can sound like a story of triumph, as, in many respects, it was. But the company’s executives realized that it was also a story of failure. The company shouldn’t have been so vulnerable to such an event. The fire, along with a later disaster—the 2011 earthquake, which cut off its supply of a crucial microchip—taught Toyota the value of redundancy and risk assessment. It modified its just-in-time system to allow for at least a month’s worth of specialized components, building strategic slack into its operation. It created a database, called rescue, with dozens of companies organized into tiers, their risks regularly evaluated under conditions of adversity, and information on sixty-eight hundred parts continually updated. The company maintains constant communication with its suppliers under “ordinary operating conditions.” But it also trains employees to operate during disasters, and evaluates the risk to the entire company if nodes in the network should falter. No enterprise is truly disaster-proof, but in cultivating networks of mutual loyalties the company has engineered resilience.

Mukherjee points to entire areas in the medical system in the US that are fraught with weak links and roadblocks that are unable to respond quickly to crises. What the US needs is a system that is structured similarly to what Toyota did, that will enable the country to respond nimbly to future pandemics that will inevitably hit the system again. The necessary resources, both material and human expertise, exist for doing so. What is needed is a commitment to long-term planning.

Slack can be costly. As Greninger put it, “Right now, I have machines and reagents to test tens of thousands of patients for sars-CoV-2. That’s basically all the clinical virology lab is doing. What will happen when the epidemic is over?” Once the incidence of covid-19 subsides, so will the sense of urgency when it comes to building infrastructure, or stockpiling equipment—masks, ventilators, reagents—that might sit unused in warehouses for a decade or more. We need purchasing procedures that control costs without creating conditions in which critical supplies vanish during a crisis. We need a Strategic National Stockpile that has sufficient inventory to ease temporary shortages. But, most of all, we need an identified capacity—a network that can be activated on demand, repurposing manufacturing lines, recalibrating agency protocols.

The question is whether the experience of this crisis will lead to such a radical reconceptualization or, as has happened so often before, once this crisis passes, all the lessons that have been bitterly learned will be forgotten or ignored. Given past experience, I am not hopeful.


  1. Pierce R. Butler says

    What is needed is a commitment to long-term planning.

    You not fum around here, are ya?

    We call that socialism in these parts!

  2. sonofrojblake says

    “What is needed is a commitment to long-term planning.”

    If you asked me to characterise the single biggest difference between American and Japanese business practices (and I have first hand experience of both) I would talk about that exact thing. US companies are buggers for considering only the bottom line, today. The Japanese see the big picture.

  3. says

    There were anti-terrorism experts who used to say off-the-wall things like “if someone disrupted the trains on the east coast of the US for a month, there would be starvation in New York.”
    I always thought that kind of thing was ridiculous, since there is a certain amount (quite a bit, really!) of redundancy built into the system. But it’s thought-provoking.

    I used to imagine that effective governments have teams of people, somewhere, who analyze things like a 3 month-long trucker’s strike and ask “does this indicate that we have a potential problem?” It wasn’t until I spent some time working on disaster planning and incident response for a couple large companies that I realized that governments are typically nowhere near that competent. For one thing, they are too susceptible to fads and economic trends that invalidate the plans in ways that are hard to predict. I can say that Exxon-Mobil did have pandemic plans, and hurricane plans, and a few other types of fairly predictable disasters.

    After 9/11, when I was researching my book on homeland security, I talked to a few of the folks who were in the cybersecurity disaster planning group under Richard Clarke. There was considerable disenchantment because some of the concerns they identified correctly were ignored and a lot of the really bad problems never occurred to them, either. For example, the overload/collapse of NYC’s cell phone grid on 9/11, which made it impossible for some first responders and crisis managers to talk. Prior to 9/11 different groups used wildly incompatible radio systems and communication protocols, and if the police needed to talk to a particular fire station, fairly often someone would just phone the duty officer’s personal cell-phone. And that all suddenly stopped working. One of the problems with gigantic systems is that they already contain myriads of work-arounds as part of their normal functioning, so someone trying to understand the system is really looking at a broken architectural diagram that represents and ideal but highly inaccurate state. I have since found out from construction people and architects that the same often applies to real buildings (and now I assume all complex systems) -- in order to make it work, it’s often importantly different from the plan, in undocumented ways. So, when it fails, it fails hard and mysterious -- but there are work-arounds and often it’s more resilient than anyone could have expected.

    If you’re interested in how complex systems fail, you can start with Charles Perrow’s Normal Accidents and compare and contrast that with Scott Sagan’s The Limits of Safety. They both make good arguments that it is more or less impossible to plan for the unplanned, but (and this is what I really did not like) the unplanned is often incomprehensible for quite a while. So disaster planning entails building systems that are robust against incomprehensible failures.

  4. flex says

    Over the past nine years the automotive industry has been seriously concerned about ensuring their supply of parts can continue to arrive at the plants all along the supply chain in case of a disaster. This was brought clearly to their attention after the 2011 earthquake in Japan. Toyota wasn’t the only automaker using microprocessors from the fabrication site that went down. I well remember the frantic need to first buy as many parts as possible from parts already in distribution, and then test and qualify a new microprocessor. It took well over a year to complete all the open tasks, even though most companies had identified a new microprocessor to use within a few days.

    So there is a push to qualify a single part from two (or more) suppliers, and at the same time the electrical component suppliers have been qualifying different plants around the world to make the same parts. For example, a capacitor used as, say, and ESD suppression capacitor may be sourced to two different suppliers, and one of the those suppliers may have two plants making the same part. It’s not precisely slack, although there is some emphasis on ensuring some safety-stock exists (which is the term used for parts which are held to be used in case of a supply-chain disruption).

    Related to the switchover to ventilator production at the Ford plant. I was not at all involved in this. I used to work closely with Ford products, but for the past ten years I’ve been working with FCA and now GM. But I know a lot of how the plants work and I’ve been in the Rawsonville plant many times. The workers in that plant are well trained in assembly. I am certain that given parts and instructions they could assemble most products.

    So the difficulty in changing over to building ventilators was largely a problem of supply. I.e. what are the parts needed? Who makes the parts? Can the manufacturers which make the parts increase their supply? And how does Ford get the parts to the plant? Setting up assembly lines is a paltry task (albeit necessary) compared to getting a supply chain established.

    The scuttlebut is that Ford started looking at doing this in late-March, early-April. But Ford is not primarily a medical supplier, and did not have a supply chain set up to make this switch. If the federal government had invoked the WPA, and got the logistics experts from the Pentagon involved, Ford would probably have had plans in hand, and suppliers arranged within a couple weeks. As it was Ford had to figure all this out on their own. Frankly, I think they did a pretty good job of it. But there are people in the Pentagon who not only plan for this sort of thing, but also perform these sort of tasks on a regular basis. They have lists of the suppliers who provide the medical equipment to the military, and all their sub-suppliers, and probably the assembly and repair instructions. The WPA could have been invoked to not just help Ford, but help coordinate the entire supplier chain.

Leave a Reply

Your email address will not be published. Required fields are marked *