In our last episode, we touched on nuclear safety, and so we’ve decided to dig a little deeper into that topic on today’s podcast. We’ll be discussing the paper entitled, “Disowning Fukushima: Managing the credibility of nuclear reliability assessment in the wake of disaster.” by John Downer (2014), published in the journal Regulation & Governance.
The paper’s abstract reads:
This paper reflects on the credibility of nuclear risk assessment in the wake of the 2011 Fukushima meltdown. In democratic states, policymaking around nuclear energy has long been premised on an understanding that experts can objectively and accurately calculate the probability of catastrophic accidents. Yet the Fukushima disaster lends credence to the substantial body of social science research that suggests such calculations are fundamentally unworkable. Nevertheless, the credibility of these assessments appears to have survived the disaster, just as it has resisted the evidence of previous nuclear accidents. This paper looks at why. It argues that public narratives of the Fukushima disaster invariably frame it in ways that allow risk-assessment experts to “disown” it. It concludes that although these narratives are both rhetorically compelling and highly consequential to the governance of nuclear power, they are not entirely credible.
Discussion Points:
Quotes:
“It’s a little bit surprising we don’t scrutinize the ‘control’ every time it fails.” - Drew
“In the case of nuclear power, we’re in this awkward situation where, in order to prepare emergency plans, we have to contradict ourselves.” - Drew
“If systems have got billions of potential ’billion to one’ accidents then it’s only expected that we’re going to see accidents from time to time.” - David
“As the world gets more and more complex, then our parameters for these assessments need to become equally as complex.” - David
“The mistakes that people make in these [risk assessments] are really quite consistent.” - Drew
Resources:
Disowning Fukushima Paper by John Downer
The Safety of Work on LinkedIn
David: You're listening to The Safety of Work Podcast episode 101. Today we're asking the question, when should incidents cause us to question risk assessments? Let's get started.
Hey, everybody. My name is David Provan. I'm here with Drew Rae. We're from the Safety Science Innovation Lab at Griffith University in Australia. Welcome to The Safety of Work Podcast.
In each episode, we ask an important question in relation to the safety of work or the work of safety, and we examine the evidence surrounding it. Drew, following on from our touch on nuclear power in episode 100 when we reviewed Perrow's book on normal accident theory, you had a few reflections on pros, anti nuclear sentiment in that paper. So we decided to follow it up with this episode and paper today.
I guess the experts who do risk assessments into nuclear technology have the view that it's safe. I guess this tends to be the view that, I guess, the broader society adopts as sort of belief, faith, trust in these experts in their assessments. I guess what we're going to explore today is, what happens when they get it wrong and the incident occurs? Or if an incident does occur, does that mean they've got it wrong?
A while ago, I mentioned in one or two of these episodes that you seem to have this hit list of about 10 or so papers that you definitely threw across to me when I started my PhD. I guess, designed to make a new student think a bit more critically about the world. This was one of those papers. What is it about this paper that you like? What is it about this paper that would make it useful for anyone in safety to be thinking about?
Drew: David, I guess I should start off by saying that I don't actually have a hit list of the same papers I send to every student. It's a question I often get. Can you give me a bunch of papers that will make me think interesting thoughts about safety? I actually do curate the student for student from what I think each student will respond to and where they need to sort of be thinking more deeply about things.
This particular paper, I really like, because there's this thing that happens that—I know you've experienced this, too—when you're just starting off doing a literature review, and you're really interested in a topic, and you're trying to find papers that answer that topic, then all of a sudden, you find this other person who has thought the same things you've thought. It's like they've gone down this path slightly ahead of you.
You immediately have this really positive reaction, oh, this is someone who's really guiding me. Then you have this moment of real doubt. Hold on, is my research even worthwhile because someone's already done it? Then you gradually, you sort of come to terms with it and realize, no, just that your knowledge has been expanded by this person who's gone before, and that's going to carry you even further yourself down the path.
I was doing a literature review about criticisms of quantitative risk assessments, because I was trying to sort of formulate my own critical analysis. I was looking for people who had previously done analysis to find out what both the main arguments were. But also, I want to better understand what the defenses were of quantitative risk assessments, because most of the literature is critical, so why do people do it at all?
I ran into this paper, which is simultaneously a critique of quantitative risk assessment, but it's also an explanation of how people are able to keep believing that quantitative risk assessment is a good idea in the face of all the criticism. This paper is geared at the ultimate criticism of risk assessment, which is when the risk assessment turns out to be horrifically wrong. How do you then recover and keep believing that risk assessment is a good idea?
I actually interviewed John for my old podcast, DisasterCast, about this paper. He was sort of swimming in this environment where Fukushima had happened. He had all of these friends who were engineers who do risk assessment for a living. He was just really interested in how they come to terms with—this is what they do—explaining to people why nuclear accidents won't happen and nuclear accidents happen. What do they then say next?
It's almost like the way when people predict the opposite. When cults predict the end of the world, and they predict the end of the world on a particular date, and the date comes about and the world doesn't end, how do they recover? Spoiler alert, they don't recover by immediately saying, oh, my theory was bunk. The world's not going to end. Let's abandon the cult and all go in our separate directions.
Likewise, people who do safety don't see a major accident and say, oh, gosh, we're not very good at this, are we? Let's pick a different profession. I think that's a really interesting question. How do you recover your beliefs in the face of such a big challenge to those beliefs?
David: Yeah, great summary, Drew. Maybe let's introduce the article. You mentioned the author there briefly by first name. The author is John Downer. The title of this paper is Disowning Fukushima: Managing the credibility of nuclear reliability assessment in the wake of disaster.
It was published in 2013, so two years after the Fukushima incident. It was published in a journal titled Regulation and Governance. I think this is the first time we've drawn from this journal. I've never heard of it before. Have you ever heard of this journal?
Drew: This is the only article, I think, I have ever actually cited coming out of that journal. I don't want to say anything positive or negative. It's just a journal I don't know much about. Yeah, I think I'm just going to leave it there, because that is a space we often [...].
We don't actually know the credibility of the journal. We can look online and see what other people say. But ultimately, knowing most of the stuff they publish doesn't seem to be particularly of interest to me. I don't know either the pros or against the quality. I'm going to judge each individual article on its merits.
David: Yeah. I think we know how many journals are out there. I guess we try to stick to the peer reviewed journals in the podcast. I guess you can form your opinions of this paper. There is no real detailed data collection and analysis in this paper. We can look at the strength of the arguments and the theory being proposed.
Drew, just to frame up the paper and to take some points out of the abstract, the paper reflects on the credibility of nuclear risk assessments, particularly after Fukushima. The argument goes that our policymaking around nuclear and even the broader community acceptance of nuclear energy has long been premised on this understanding that we've got these experts. These experts, mostly engineers, can objectively and accurately calculate the probability of having these catastrophic events.
I guess the Fukushima disaster or any disaster that goes in the face of these assessments lends itself to the substantial body of social science research, which we speak a bit about on the podcast, that sort of suggests that these calculations are fundamentally unworkable for a whole bunch of reasons we'll talk about today. Nevertheless, though, the credibility of these assessments seemed to continually survive the disasters in their industries, not just in nuclear, but in other high risk industries as well.
We're going to look at, why our faith, trust, belief in these risk assessments can endure beyond the disaster? There's a quote after the abstract. I quite like that idea of dropping a quote after the abstract. We've used it ourselves a few times.
That's from Terry Pratchett who says, "After eight years involved with the nuclear industry have taught me that when nothing can possibly go wrong, and every avenue has been covered, then it's time to buy a house on the next continent." When everything is 100% safe, then it's probably a good indicator that it's not safe at all.
Drew: Yeah. David, before we sort of get too much into the meat of the article, I think it's worth reflecting a little bit on how different industries have these different narratives associated with them. When we think about airline travel, the constant argument is our people are genuinely assumed to be a little bit afraid of flying, but we reassure them by just saying, flying is totally safe. When people do comparisons, like the chance of dying in an aircraft is less than the chance of dying on the way to fly in that aircraft, you're going to have a car accident on the way to the airport, not an accident in the plane.
Then we have rail, which is just sort of assumed to be generally safe. Then we've got the narrative that like cars are unsafe, but they're commonplace, and we use them anyway. The narrative around nuclear is really interesting, because no one thinks that nuclear power is safe. It's always portrayed even by the experts as this inherently dangerous thing. But the argument is, it's this dangerous thing, but don't worry about it, because the experts are doing their best to look after it. It's a dangerous thing that's been made safe by all of the safety precautions.
That's kind of a fragile belief, because you've got to trust that those safety precautions are adequate. Then every time we have a hint that they're not, we've got this boiling possibility underneath. I always find that interesting, because my personal view is that the reality is that nuclear power is not fundamentally dangerous. It's dangerous in the way that coal fired power or chemical plants are dangerous in that there are bad things that can happen. Those bad things are bad, but they're not especially bad for nuclear.
The half life of radiation might be in the hundreds of thousands of years, but heavy metals don't have a half life. They're there permanently. We have heavy metals in all sorts of different processing. Once we sort of set it up as the demon that's under control, then the control immediately becomes this really suspect thing. It's a little bit surprising that we don't scrutinize the control every time it fails.
David: Yeah, and I guess the paper sort of draws out some of those examples of aviation and even infrastructure like bridges and tunnels. It goes that people understand that planes occasionally crash. They don't expect the individual plane that they're flying on to crash, but understand that planes crash. But when it comes to nuclear power, there's almost this total absolute belief that people just don't expect disasters to happen.
I guess as we'll talk about in this paper, even after the last disaster does happen, I guess we bounced back into that world that we're never going to have another one again. I guess that's what we'll talk about in this paper today. We would never believe that for aviation no matter what the airline's told us. If they said, no plane is ever going to crash again, we wouldn't believe that. When the nuclear industry comes out and says, we're never going to have a nuclear incident again, we seem to go, oh, okay, well, that's good.
Drew, I guess one of the things that was called out early on in this paper, because what we're going to do is we'll talk a little bit about how we think about some of the general things about probabilistic risk assessment. Then we're going to talk about these four narratives that, I guess, experts and others tell themselves as to why the risk assessments, we should still trust in them after incidents occur.
I just wanted to throw out something about the nuclear industry, because maybe the nuclear industry has convinced itself that it is very, very safe. So much activity has gone on to the prevention side of the risk management exercise that this article just calls out just some of the limitations in things like emergency response planning, particularly in the wake of Fukushima and some really specific scenarios that had never been considered some response actions that had never been considered, even to the point of evacuations, availability of medication, and these things.
I guess, Drew, you've done a bit of work with Ben Hutchinson on fantasy plans. I guess this idea that if we think something's really safe, then we're probably not going to invest significantly or usefully in what we actually need to do if something does happen.
Drew: David, I think that in the case of nuclear power, we're in this awkward situation where in order to prepare emergency plans, we have to contradict ourselves. Those contradictions are difficult from a public relations point of view, but they're also just difficult from a psychological point of view.
Logically, what you really want around a nuclear power plant is you want radiation monitoring. But why on earth would you need radiation monitoring around a nuclear power plant that you have said is never going to have radiation release? Surely this putting up of dosimeters in the nearest towns, that's telling people that there's a possibility of radiation. At the same time, you'd rather tell them, no, it's impossible, there'll never be radiation.
How do you have plans for a controlled release of radioactive gas, which is something that people always end up needing to do in nuclear crisis situations? When you're promising people, we're never going to release gas, how do you have defense in depth and talk about, how are we going to handle it if we've ever got radiation in the outbuildings, when the whole point of the design is that you never have radiation in the outbuildings? You've got these contradictions between the way it's supposed to work, and the way it might end up being, and admitting that it might end up being in these terrible states.
For example, one of the things at Fukushima was they just had no plans for, what happens if we have to flush seawater through this thing? Because admitting that you might have to flush out the plant with sea water is admitting that you could be in a situation where all of your other stuff doesn't work. And you promised yourself that all the other stuff was going to work.
David: Yeah, Drew. This systemic failing on the response side, where systemic, from a sense of if the assumptions about the credible scenarios that you carry into your normal risk assessment for normal operations becomes those same assumptions and scenarios that you bring into your emergency situation. Like you say, it's never considered that we don't have access to the cooling systems, even though we're talking about the on-site dosimeters, they were designed for normal operations with background radiation, as opposed to anything, like any kind of doses that will be seen during an incident.
Operators had to borrow flashlights from nearby houses, because they didn't have sources of emergency lighting and things like that. Just this idea, and we'll get back to practical examples at the end. But carrying the limitations of your risk assessments into limitations of your emergency plans might just give yourself a bit of a double whammy.
Drew: Yeah. It's almost like you need two separate groups of people. One group that's told, make it never happen and the other group that's told, start with the assumption that it happens, and have that same group be the same set of people. It just requires total cognitive dissonance.
David: Yeah, that's a good point. Let's talk briefly about, I guess, probabilistic risk assessment. I guess you could talk far more than briefly on that. But I guess this idea of probabilistic risk assessment is industries have created and adopted a whole series of calculative tools that gives operators, designers, regulators, a means of "objectively" establishing that nuclear accidents were sort of too improbable to merit serious discussions.
Drew, I guess a lot of this was earlier in the 60s, and that was based on a set of studies called the WASH studies. Do you want to talk just a little bit about the background into the WASH studies in the nuclear industry?
Drew: Sure. One thing that I think has been fairly lost to history is that most of these very early quantitative risk assessments, and the language that you use there, David, is really telling, they were not told, give us a realistic understanding of the likelihood of a nuclear accident. That was never the question. The assignment was, demonstrate that the risk associated with this accident is sufficiently low.
A lot of these methods are not designed with a spirit of scientific inquiry. They're methods designed with an ability to provide a rationalization for why we should believe a particular number that has in fact been determined in advance. We've already been told what is acceptable, you're now tasked with producing an analysis that matches acceptable.
Unlike with some of the things that came before WASH to do with nuclear weapons, which were much, much more explicitly just following straight down that path, you've been told, you've told us that it's less than one in a million, okay, prove that it's less than one in a million. Fantastic, here's our report that says it's less than one in a million.
With wash, they did actually put these early reports under peer review. The initial calculations showed that nuclear power was acceptable. They were also incredibly convoluted, incredibly unreadable, incredibly unrealistic.
I used to actually give my poor students that assignment to read the original WASH-1400 report, and it's just impossible. Even if you are an engineer and understand math, this is not an exercise in transparency. It's basically, we were expecting you just to read the abstract and believe us. We were never intending that anyone else would actually check our calculations. If we did, we might have actually explained what those calculations were.
The original risk assessment was peer reviewed and was very heavily criticized. One of the things that it was criticized for apart from the total lack of transparency about how it was done, was the lack of any sort of consideration of uncertainty. People were plucking numbers out of the air basing their calculations on those numbers. They're never considering, okay, what if these numbers happen to be wrong?
Oddly, the next step was not to reject the analysis. The next step was just do it again, but this time do it again with uncertainty calculations. We basically went from this science of calculating risk to the sub-science of calculating uncertainty. Risk assessments became a calculation of risk plus calculation of uncertainty. Neither of those being particularly scientific.
We could actually just make it to continue the Terry Pratchett since it's already in the episode. It turtles all the way down, that if we're going to do the calculations of uncertainty, then we're going to need to do a calculation of the uncertainty with the uncertainty, because we use the same methods we used to calculate the risk.
David: Yeah. Drew, in this paper, we'll talk about one of the prominent manufacturers reporting to a UK regulator before Fukushima, that the risk of a core damage incident to be in the order of one incident per reactor for every 1.6 million years. And the probability of a core meltdown is infinitesimal. We're not even prepared to put a number on the probability of a meltdown, because we've got this one in every 1.6 million years just for a core kind of incident.
Even reactors of the age of Fukushima were sort of considered to be by those assessments, probabilistic risk assessment to be sort of one in 100,000 years type of incident, but still not, I guess, resulting in a meltdown.
Drew: David, I'll just pull you there for a moment, because these figures, they just sound ridiculously low, but I think we need to just point out just how ridiculously low they are. These French figures about the reactor do not include in them any consideration of where that reactor happens to be. The assumption is that that core damage can only come from the reactor itself.
Japanese history has not been well recorded for the necessary 100,000 years. But just think back to Japan the last 100 years. Just think of those things that have got nothing to do with a nuclear weapon factor that are capable of flattening a city that have happened to Japan in the last 100 or 150 years. We have had multiple city destroying earthquakes. We have had multiple cities destroying floods or tornadoes.
We've had a war that did a pretty good job of flattening multiple Japanese cities in that time. The likelihood of a core damage incident being 1.6 million years, given that there have been multiple, multiple, multiple types of events that are capable of flattening the entire city, the reactors in, over the timespan of just 100 years, just shows how ridiculously those figures don't take into account all of the possible causes of the accident happening.
David: Yeah. Drew, I guess the next few sections before we get into sort of the detail talks about objectivity versus a sociological perspective. There's this probabilistic risk assessment engineering perspective that's all about transparency, objectivity around decision making, and that we actually can calculate these numbers. We can understand what might happen and how often it might happen.
Then you've got the sociological perspective where John Downer kind of ties back into normal accident theory and Charles Perrow's work, which we reviewed last week and said, accidents are caused by very improbable confluences of events where no risk calculation could ever anticipate. If systems have billions of potential billion to one accidents, then it's only expected that we're going to see accidents from time to time, because if you've got that many possibilities even at low probabilities, you're going to see incidents.
Drew: David, I'm sure not all of our listeners are familiar with what these risk assessments actually look like. There's something I should throw in here. Downer is aware of this. He's not hiding it, but he's just sort of assuming that you know the answer in the background. The way these risk assessments are conducted, is typically with a thing called fault tree analysis. There are a couple of alternate methods like Markov analysis, but in nuclear power, a lot of these assessments are presented as fault trees.
The idea of the fault tree is that it does, in fact, calculate the confluence of events. It takes the probability of each individual event, and then it creates these complicated logic structures that show what if this event happens at the same time as this event, or this one, but not this one, but then this one as well. It actually calculates all of those probabilities and combines them together.
It's not that those confluences are probable. Each individual combination of them is, in fact, quite improbable. And the possible combinations have been considered and taken into account. But what's typically missing from those analysis is it assumes that we know the probability of each individual event. Therefore, we can just treat them as individual things and combine them.
It never takes into account all of the things that make these apparently improbable combinations, things that are likely to happen all at once. The chance of diode number 337 failing at the same time as transistor number four failing may be one in 100,000 multiplied by one in 100,000. Chance of them both happening at the same time is going to be one in 10 million. But if the entire buildings are underwater, then they're both guaranteed to fail at the same time.
If the maintenance regime is failing, then they're both going to be under maintained, both likely to fail. The risk assessments don't properly take into account these reasons why apparently independent events might actually be quite likely to happen at the same time.
David: Drew, let's talk about how we rationalize these disasters or incidents in the context of these risk assessments. We've got this probabilistic risk assessment. It's as much detail as you've just described there. We have this number, and that number is, I guess, acceptable to the operators, the regulators, and can throw these out to the communities with a one in 1.6 million year chance of a core incident. We're not even going to calculate the chance of a meltdown because it won't happen.
I guess what John Downer goes on to propose and talk through is that there's sort of four overlapping narratives. The way that we can talk about the incident and the risk assessments, we can talk about them in a certain way so that we can continue to have faith and trust in the system and in these assessments. I might just introduce these four, and then we'll go through each one if that's okay.
The first one is that the risk assessments themselves did not actually fail, and he calls this an interpretive defense. How we interpret what the risk assessment was going to do and how we interpret the incident can show that, yes, the incident occurred, but the assessment itself didn't fail.
The second one was that the failure of one assessment is not relevant to the credibility of other risk assessments. This is like the relevance defense. Okay, we might have missed something at Fukushima, but we haven't missed anything anywhere else.
The third one is that the assessments were sound, but people didn't behave in the way that they were supposed to behave, or they didn't obey the rules. This is what Downer calls the compliance defense. You can't really hold the engineers and the risk assessors responsible when people inside the plant or the operations company acts or does things in a certain way that they weren't supposed to do.
The fourth is probably the most, I guess, least defensive one, which is that, okay, the assessments were flawed, but now we know how those assessments are flawed, and we fixed it. This is what Downer calls the redemption defense. Okay, there might have been a problem with our assessments, we've had this incident, we've learned from it, we've gone to fix the other assessments, it's not going to happen again. Drew, do you want to talk about these four one by one, and I guess the argument people making, and the counter argument that Downer makes?
Drew: Yeah. Let's try to give a fair version of what each of these defenses actually sounds like before we point out the criticisms of them. I want to sort of make sure that we do in fact acknowledge why these defenses do sound reasonable. Shall I have a go at the interpretive defense to start with?
David: Yeah, do that.
Drew: Okay. Your interpretive defense is ultimately, yes, we had a risk assessment. Yes, the accident happened. But that doesn't mean that the risk assessment was bad. Now, there's a few different versions of this. One of them is that there is a genuine argument that Fukushima was not actually an accident. It got lots of media attention, it got lots of press, it got lots of scaremongering. But ultimately, the different layers of defenses ultimately worked.
What we saw was the first couple of layers of defenses failing, but then defenses are supposed to do, the emergency plans were put into practice, the emergency plans did in fact work. That Fukushima, basically, wasn't a real accident. The risk assessments which said you're never going to have that accident, what you were scared off was a meltdown, so why are you picking us up for successfully preventing a meltdown?
There's a slightly different version of this, which says, okay, Fukushima did happen, but we said the chance was one in a million. It happened once. You can't say, oh, we were wrong about it happening one in a million when it only happened once. And then there's a third version, which is the one that most commonly gets used for Fukushima, which is that in order to do a risk assessment, you've got to make certain assumptions.
We've got to say, okay, a reactor is never going to be perfect, we've got to design it to meet certain conditions. We design it to meet the risk of an airplane crashing into it, but it's unrealistic to expect us to be successful protecting it from a nuclear bomb exploding next door. We call this the design basis. We design nuclear power plants to withstand reasonable conditions, those conditions have limits. We designed Fukushima to survive certain conditions that were suitable for a nuclear power plant in Japan.
We did our risk assessment on that basis and an event, which rocked the world, which rocked the region happened. That was outside the design bases for Fukushima. The fact that Fukushima survived that event is miraculous. It went above and beyond its risk assessment. The risk assessment wasn't wrong.
You don't blame Fukushima for being partially damaged by an event which flattened a region. That's sort of the interpretive events. We don't need to worry about the risk assessment, because the risk assessment was right. It gave us an accurate perception of the risk within what it was expected to do.
David: Drew, I guess that last one's a little bit like force majeure, which is that if the risk assessment was based on, say, the plant being 10 meters underwater, because it was 20 meters underwater, the risk assessment was never designed to provide probability of that particular scenario. The assessors would say, well, we could have if we had wanted to include that particular scenario, and then your other points, which is yes, it only happened once. If it happens twice, our risk assessment is wrong, but it only happened once.
The first one, it wasn't actually a meltdown. Yes, there were some fatalities. The numbers of those fatalities vary, but it wasn't a meltdown. That's the interpretive defense. Quite rational narratives that regulators, operators, designers, even the community can hold and listen to. Do we want to talk about, I guess, reexamining that from maybe how Downer would see the more irrationality of that particular narrative?
Drew: Yeah. Downer's answer is really simple and easy and I have 100% sympathy for. You can't take a risk assessment and say, oh, the risk assessment was fine, we just got the scope wrong, because the scope is supposed to be part of the risk assessment. Part of the risk assessment is supposed to be considering what are the reasonable external events that the risk assessment is supposed to take into account.
This defense might exculpate individuals. If a third tier engineer asked to write the fault tree, you are given a set of assumptions and told, write a fault tree in accordance with these assumptions. Okay, that's fine. That gets you off the hook, but it doesn't get the person who gave you those assumptions off the hook for making the wrong assumptions. Those assumptions are part of the risk assessment process.
That absolutely applies once we take this up to the level of regulating an industry. If the regulator is letting you make bad assumptions in order to make your risk assessments, then we cannot say that the risk assessment based regulation regime is working at all. There's one specific thing that happened with Fukushima that can easily be called out for this. This is just the assumption of how high a tide comes in as a result of the tsunami. The reactor was supposed to be able to withstand.
This directly involves things like constructing the seawall. If you're told, assume that the highest tide that can come in is 10 meters, and you build a seawall that can manage that 10 meters, it's hard to criticize you. But the person who told you, build it so that it can withstand 10 meters, or to have done reasonable checks about what is a reasonable tide that might come in, now you can say, well, okay, this was a spectacular 20 meters, that was far more than they expected, but their expectation was dumb.
You look back at the history of floods in the Fukushima area. You cannot say that more than 10 meters was something that they should not have anticipated. It's unrealistic to have anticipated 20 meters, but this would have failed at 19, it would have failed at 18, it would have failed at 17, it would have failed at 10.5. All of those, 10.5 particularly, is something that happens every 20 years or so.
It was an unreasonable thing. To just exclude that from the risk assessment is to exclude one of the key failings of risk assessments. Risk assessments are really bad at anticipating what are the reasonable parameters for external events that might happen.
David: Yeah, Drew. Downer sort of says that the real art of risk assessment or risk calculation isn't just about applying a formula correctly, but it's about framing that formula correctly. Like you said, what are all the different operating conditions that this plant might encounter during its lifetime? If you turn around and say that, well, this event was beyond the design basis, this event was outside our parameters, it's basically essentially saying that an integral element of a risk assessment process, which is actually framing all of the credible and possible scenarios correctly, was wrong.
The calculations may have all been right, but the framing of the whole assessment process was not right. I guess in terms of this interpretive defensive goes, it may not be reasonable for anyone involved in this industry to go, well, the risk assessment didn't fail because the incident happened. Maybe you can do that if a satellite falls out of the sky, or something happens. I guess as the world gets more and more complex, then our parameters for these assessments need to become equally as complex.
Drew: If you were still a fan of the design basis argument, then I'd urge you to consider what the residents of Fukushima were told. They were not told, we're building a set of nuclear power plants. These nuclear power plants will be safe, so long as we don't have a particularly high tide. They weren't told these things will be safe, so long as earthquakes are no worse than they've been in the last decade or so.
If we get a tsunami or if we get a bad earthquake, all bets are off, you're at risk. They were told this is safe. This has been designed appropriately, had a risk assessment done, and the risk assessment says it is safe. If you're going to claim nuclear power plants are safe within the design basis, then you should be advertising those design bases to the population along with credible evidence of how often that design basis might be broken.
If the risk is one in a million within the design basis, but the chances that design basis has been wrong, or that's going to happen every decade or so. then it's that second number that you should be sharing with the population.
David: Yeah, that's a scary realization. Drew, three more narratives. The second one is what Downer calls the relevance defense. Do you want to talk a little bit about that?
Drew: Okay. The relevant defense is, I think, my favorite, because this is basically saying, and this is what we do with many accidents, is we call out the people who suffered the accident as a bad actor. Risk assessment generally is good, but Fukushima did it wrong. That one, I think, is fascinating in the case of Fukushima, because you've only got to look at exactly what was said after Three Mile Island and after Chernobyl.
After Three Mile Island, the Soviet said, this could never happen in Soviet Russia, because it's the type of accident that only happens under American individualized commercialized regimes. And then Chernobyl happens and the Americans say, oh, this is the thing that can only happen under Soviet regimes with lack of individual accountability.
What everyone agreed after both of them is that the ideal sort of culture where this would not happen, is a culture that combined individual accountability with social responsibility, i.e. Japan. Until the moment the accident happens in Japan and everyone says, oh, those awful Japanese with their subservience to authority culture. I think that history makes obvious just how this relevance argument only works if you only have exactly one example of something happening.
The moment starts to happen in more than one place and time, then you've got to accept that there are things that are more than just one place and time, exceptional things causing it. In particular, in order to properly make the relevance defense, you've got to be able to predict in advance which risk assessments are bad, not just afterwards, which risk assessments were bad. Because if you can't tell till afterwards, then that's useless as an explanation for why your risk assessments don't have those exact same problems. I'm sorry, that's actually me talking. Shall we shift to what Downer says about the relevance defense?
David: No, and it's very close. It's a good interpretation of it. He talks in his relevance that I guess we know in safety, this idea of differencing by distancing, which if we can find a way to think that the aspects of the incident aren't the same as aspects of another operation, then we can discount the plausibility of that scenario.
In this case, he talks about, well, this is an old 1960s Mark 1 style plant, then this problem doesn't exist with more recent plants and assessments, or we can turn around and say, well, the culture, like you said, the national culture and the operators are different, or even in this case, very much the regulatory regime was called out specifically to say that the assessment and regulatory environment in Japan wouldn't happen in Europe or the US.
I guess his relevance defense is really what we might know broadly in safety as I interpreted as this differencing while distancing situation. We see this in aviation incidents as well. Particularly after the Boeing incidents with their MCAS system, there was a lot of discussion about the African and the Asian pilots involved in those two fatal aircraft incidents and sort of saying that this incident wouldn't happen with Western pilots.
It was sort of how Europe, the US, and even Australia, were able to turn around and say, well, no, no, no, we're not worried about these planes, because we have Western pilots in them, and they would behave differently to some of those other cultures. I guess that's this relevant thing, Drew. Rather than seeking ways for how we might be similar, we sort of seek out ways to find a difference and justify an existing position.
Drew: David, I'm glad you pulled out the MCAS example, because I also want to point out exactly what happened once we eventually realized, no, it's not just the African and Asian pilots, it's actually the software, everyone immediately turned on Boeing and their quantitative risk assessment. It was, oh, look at how badly Boeing did these quantitative risk assessments singling out Boeing and distancing themselves from Boeing, again not recognizing the fundamental limitations of the quantitative risk assessment process. Instead, it was just, oh, Boeing did it wrong, they should have done it better.
Even once we stopped distancing from the pilots, we still distance from the particular example of the risk assessments. The big question that needs to be answered if you want to mark this defense is, how do you know that your risk assessments don't have the same problems? If we didn't realize the problems with Fukushima risk assessments until Fukushima happened, if we didn't realize the problem with Boeing's risk assessments until the MCAS accidents happened, then we're not going to realize the problems with your risk assessments either until an accident happened.
If you want to argue against that, publish the damn risk assessments. Put them online or find the fault in them, because they're the same faults that are in every risk assessment. The mistakes that people make in these things are really quite consistent. If you believe the risk assessments are free of fault, then publish, otherwise you don't get to make the relevance defense.
David: I think the further challenge that defense as Downer mentions is this idea that, particularly the US manufacturer involved, the idea is that they would use the same assessments like you said earlier no matter where that plant was in the world. To think that the seismic variables in Japan are particularly different, maybe, from somewhere like San Francisco or some of the other volcanic areas in Europe, it doesn't hold up.
He also says that, at the time of the accident, we should remember that Japan had a first class reputation for managing complex engineering infrastructure. There was actually a Washington Post article after Fukushima that said, "If the competent and technologically brilliant Japanese can't build a completely safe reactor, then who can?" That was kind of in the face of this whole relevance defense, which is where even, I guess, that was one occasion that the media really tried to actually go, well, should we be asking more questions here?
Drew: I dare our listeners to find an article written about the Japanese approach to safety before Fukushima that is not, why can America and Europe not be more like Japan? Why are our trains not as safe? Why are our airplanes not as safe? I dare to find one that says, before the accident, look at how dangerous Japan's regulatory regime is.
Also David, I don't think we made this really explicit. The type of reactor here, this was not a Japanese reactor. This was an American reactor of a type that is still in existence in America. Yes, there are more recently designed reactors, but it's not like everyone shut down all the old ones.
If it's really that the new ones are safe, and the old ones are really, really dangerous, then why is our regulatory regime not said, oh, look at how dangerous all these old ones are, we need to get rid of them immediately. Why have we not done that before Fukushima if that's what we really believe?
David: Drew, the third defense here overlaps a little bit with the relevant one, but it's really about the compliance defense or Downer calls it the compliance defense. It's more in relation to how the company that operates the plant operates it and more specifically how the workers who control the plant, how they act and behave in terms of what it means for the risk assessment itself. Do you want to talk a little bit about that?
Drew: This is like the inverse of the design basis. The design basis says, we've got to design the plant to cope with certain external events. We can't be responsible if the external events are beyond what we're expected to design to. The compliance defense says that when we design a plant, we've got to make reasonable assumptions that people will look after it, and they will manage it appropriately and they won't stuff up in the management of it.
It's basically like our risk assessment can't possibly be supposed to take into account fill stuffing up. Our risk assessment was fine, but no one could have anticipated the fill. Of course, the criticism of this is very similar to the criticism of the design basis. You're supposed to anticipate the filling. You're supposed to anticipate reasonable human behavior.
If you want to make the argument that there is a lack of compliance, you've got to show that this is more than just normal human behavior, because you're really supposed to take into account normal humans, not super humans. If Fukushima was an act of deliberate sabotage, then absolutely, I think it would be quite reasonable to say, this was our safety risk assessment, not our security risk assessment. This wasn't an accident, this was a security failing. Then of course, you'd have to explain the security failing, but at least safety would be redeemed.
You can't say that these were humans behaving humanly. That's terrible. How dare these well-trained, well-educated Japanese operators behave like normal well-trained, well educated Japanese operators. Instead of the super humans we imagine the operators in the US, Germany, and France to be.
David: Yeah, this idea that human error or the actions of an operator are not a reflection on the quality of a risk assessment. It's that narrative. I guess how Downer reexamines this is he said, look, we've heard all this every single time. We've seen the reports after Three Mile Island that pointed to operator error. We've seen it after Chernobyl as human error is the primary cause, even the plant director and five other operators at Chernobyl being sentenced to long, long prison terms. And we get it after Fukushima as well.
What Downer says is that, even if we feel like we can blame a nuclear accident on errors or misbehavior of people, it doesn't redeem these assessments that claim the accident was not credible, because I guess such an argument would say that we've got these perfect assessments, we've got perfect control over the risks, we've got a perfect set of rules. If everyone follows those rules, then our risk assessments are correct.
I guess in here, Downer sort of drew on a lot of the stuff that we've covered a few times in the podcast here about complex systems, performance variability, and local rationality. Our listeners will be familiar with just how different situations are that operators face every day and just how incomplete their information of the overall functioning of the system is. If you don't have an error tolerant system, then your risk assessments are really not that good to you.
Drew: Yeah. Imagine if we made that same argument. Driving to work is perfectly safe, so long as every other driver always obeys every road rule. Flying is perfectly safe, so long as pilots are never tired. If we assume human perfection, that's just not a rational assumption to make, particularly given that you've got previous evidence of people making these exact same mistakes in previous nuclear power plants.
David: Drew, the fourth argument here is the redemption defense. Do you want to talk about that as it's one of the most straightforward ones?
Drew: We have learned from Fukushima. Yes, Fukushima was bad, but don't worry, because we learn from our mistakes, and we're not going to make those same mistakes again. One thing I don't know if Downer was actually aware of in this paper was there was a whole massive spate of conferences around the world in the subsequent couple of years. Basically, everyone retitled their risk assessment or safety conference learning from Fukushima, or how do we avoid repeating the mistakes of Fukushima?
I think this is a very sincere defense. All of these are not intended to be cynical. They're all intended to be sincere. Okay, we understand now that Fukushima, the problem was that they didn't properly consider the design basis. Okay, all of our risk assessments now I've got rules that we have properly considered our design bases.
Fukushima, they didn't properly consider the fact that the emergency generators might go down at the same time as the power plant. All risk assessments now have to include a proper assessment of what the likelihood that the generators will function at the same time is whatever caused it to fall down. It's like, yes, there was a hole in the dam, but our fingers are there now. Don't worry, there's no leak because we've spotted the leak, and we've fixed it.
David: I guess, Drew, Saudi talks about the EU came out and said, we're going to reassess all of our nuclear power plants. As the NRC in the US sort of said, well, we don't think that our assessments to date have adequately weighed the risks of emergency Generac generators or flooding, and we're going to do that. The IEA had a five-point plan to strengthen reactor oversight, because like you said, Drew, we've now had a good look at flooding risk, we've now had a good look at emergency generators going down when the plant goes down.
I guess, Drew, it's kind of like two things. One, is it down? It says, well, there's actually no clear evidence that that much actually changed as a result of those review processes, particularly for established plants. The second thing is, you've corrected a couple more things that you know about now, but how does that give you any confidence that the things that didn't actually happen at Fukushima that could still be systemic weaknesses with your risk assessments?
Drew: David, I want to quote from a bit earlier in the paper, both to illustrate this point and also just as a continuing explanation for why I love this paper. He quotes Terry Pratchett, but he also quote quotes, I'm Charles Schulz in Peanuts. We've got an international audience. Maybe not everyone's as big a fan of Snoopy as I am personally, so I'll basically explain.
There's this one comic strip that all throughout Peanuts runs for decades. This almost exact same comic strip gets published every now and then. It consists of the main character of peanuts, Charlie Brown, and his friend Lucy is holding a football ready for him to kick it. Charlie runs up to kick the football, Lucy whips it away, Charlie Brown folds over. And then every subsequent comic, the first panel is Lucy explaining why this time, she's not going to take away the football. And every single time, she takes away the football.
The lesson we're supposed to get from that is not listening to Lucy's carefully reasoned explanation for why it will be different this time. It examined the larger lesson. Lucy can't be trusted with these explanations. The trouble with the redemption defense is it works exactly once. Once it happened twice, you've got two things to explain now, not just that you fixed the mistake, but you've got to explain why your fix last time didn't work. And then the next time, you've got to explain why your fix the previous time didn't work.
We're now at the point where you have to explain why this time is so different from all of the previous times, because the history has been repeated, risk assessments fail in all sorts of ways. Fixing up the process doesn't stop. Warning people about the particular mistake doesn't stop it. Building up general competency and risk assessment doesn't stop it.
You're patching the holes in something that is constantly full of holes. The lesson that needs to be learned is not how to redeem it, but learn that it's not to be trusted.
David: Drew, this is actually a fun paper. I'm not sure if it's open access and people can access it. But maybe if you give us a hoy, we'll think it through. At the risk of this being a long episode, do you want to say anything before we go on to some practical takeaways?
Drew: The final thing I just want to say is that this paper isn't really about what is wrong with quantitative risk assessments. There's a whole other body of work about that. We've addressed some of it in previous episodes. I'm sure we'll address it in future episodes.
What this episode is interesting about is the way in which people are able to rationalize why they keep doing safety in the same way in the face of criticism. Not just in the face of academic criticism, but in the face of the universe telling you that you're wrong, but you still find reasons to still believe that your safety practices work. That extends well beyond quantitative risk assessment, it extends to anything we do, where we keep repeating the same things, even though we keep having safety accidents. It's a lesson not for quantitative risk assessment of nuclear power. It's a lesson for us all to really think about our own ways of defending things that we do
David: Great, Drew. I had a bit of a go at some practical takeaways. Do you want me to start us off and see what you think?
Drew: Yeah, go for it.
David: There's this idea that I think we've explained today, but uncertainty is significantly present in all risk assessments. What have we included in the basis of design? What have we included as credible scenarios? What have we considered in terms of the actions of people? Our assessments of likelihood and probability, what are the assumptions underlying all of those?
There is so much uncertainty and everything that we do in a risk assessment that when you hear someone saying that something is definitively safe, I guess the message will be worried, because maybe they actually genuinely believe that it is definitively safe.
Drew: Yup, and I think that applies in the other direction as well. When you are being a responsible conveyor of safety information, it is so tempting to want to reassure other people, but the ethical thing to do is to be transparent about the limitations of your own claims and analysis. The right thing to do at Fukushima was to put up the dosimeters in the nearby town, and explain to the residents that even though we're doing everything possible, this is also a necessary layer of defense, not to say, oh, we'll worry them if we admit the possibility, so we're not going to do it.
David: Drew, this redemption defense has been found and fixed. you might find and fix one problem in relation to any incident, but that does nothing to correct any systemic weaknesses that you might have with your processes inside your organization. To think that we've had this problem, we've only had this problem, we've fixed this problem, and we don't need to look any further than that or worry any further than that, is a mistake.
Drew: David, I'm going to throw in a quick add here with your permission for a free product. Nothing that we're getting any money out of. It's a thing called the White Rose intelligent customer handbook that we prepared while I was working at York, which is a free guide to reviewing safety documents. I'm happy to send that out to any of our listeners who want it, totally free to use.
One of the principles in that document is, particularly for regulators and customers when you're reviewing things, is don't find mistakes and send those back to the original producer of the document saying, here is the mistake, please fix it. Find a mistake, and assume that that is systematic of a category of problems, and look for that category being fixed.
Your networks are even as simple as, when you're reviewing someone else's essay, you don't correct every billing mistake. You correct the first two instances, then you tell them to do a spellcheck. You go back next time, you don't check those particular mistakes, you check for the general pattern of spelling. The same thing with risk assessment. Don't fix the particular problems, recognize that they're symptoms of a systematic problem, and look for a way to fix the systematic problem.
David: Great, Drew. We might be able to post that publication on LinkedIn or something as well during the course of the week. The third is that your risk assessments of complex systems, particularly involving people and high risk technology, can never identify all potential failure modes. Even these really, really detailed quantitative failure modes effects analysis and things that fault trees that Drew has mentioned. It means that a risk assessment process can never be set and forget.
We've done it at the time that we've designed the plant, and now we don't have to worry for the life of the plant. We need to have ways of constantly getting feedback on modes of operation, what's happening in industry, and properly scrutinizing where the world is showing us that our representation of the world in our risk assessments may not be as true as we'd like it to be.
Drew: Three things that are inevitably missing for every risk assessment of a complex system are anticipating all the mistakes that people will make, anticipating the way the technology will change and evolve over the next few years, and anticipating all of the little plastic connectors that can break causing your complex technology to fail. No one ever successfully remembers all of those things. The lesson there is not to remember all of those things, it's that no one ever successfully remembers them.
David: Yeah, it's interesting. I was talking to someone who was an engineer involved in a reliability assessment of an LNG facility and said, what's the failure mode you're worried about? He said, the seals on a particular type of pump.
Of everything that could go wrong in that entire plant, it was the pump seals that he was worried about after three or six months, where they wouldn't start leaking or not. I guess that's like the O rings in the Challenger launch. Of all of the complex technologies, it was a set of O rings on the rocket boosters in cold temperatures. That was enough to crash the system, literally.
Drew: The second shuttle was basically the equivalent of Styrofoam foam. It's the little plastic and rubber bits that get you.
David: Drew, the fourth one here is that there are going to be assumptions and bases of design for every risk assessment. We've never considered all of the potential scenarios that could be involved, even not all the credible scenarios. I guess if you're reviewing or involved in risk assessments, be wary of those, doing a bit of scenario analysis, asking a few what if scenarios. This idea that, ah, the basis of design is designed for a 10 meter wave height. Well, what if we get something that's 30 meters, what would happen then?
I was involved in an offshore platform that we were doing some work on manning it, putting more people at it, I guess, putting people offshore. Similarly, not too long after Fukushima, actually, the team had done a pushover analysis about maximum wave height before this thing would push over. Remember having a really deep conversation at the time in the wake of Fukushima about, how much bigger was the wave height they considered than the biggest wave height ever recorded in that part of the world in the last 100 years?
The initial starting point was only marginally more. It's interesting to see different industries can repeat these mistakes, but asking what if. What if there's a 40-meter wave instead of a 25-meter wave? At least, then you'll start to understand where you've got catastrophic failure potential if something out of the ordinary happens.
Drew: Yup. One thing I say in all of my system safety classes is that every assumption you make is an obligation. An assumption isn't something that restricts your analysis. An assumption is something that immediately goes onto your to do list of things that you need to check and ensure. The moment you write down my assumption is that the largest wave is 10 meters. That's an obligation to go out and find good evidence that, in fact, that is right. And if not, replace it with the correct number and redesign accordingly.
David: I guess to the next point, the next practical takeaway here about your assumptions of the operators in these systems. Being aware of assessments that are not tolerant or dismissive of the variability of operator capability, their understanding of the system, and their task performance.
When you see a risk assessment that is almost technology only and just makes an explicit assumption that operators will be trained, they will understand what to do, and they will act in accordance with the designers intentions all the time, that's unlikely to be a very resilient type of system. Being aware of those assessments that don't, I guess, I don't know what the ratio should be, but worrying 50% about the engineering design and worrying 50% about how operators are actually going to work with that design once we switch it on.
Drew: There are really basic statistical techniques that can guide us here. If you want to make a claim that's as simple as, oh, the operators are going to make this mistake one in 1000 times, then I want to see that you've observed operators do that in realistic conditions 2000 times. If it's the sort of thing that is too dangerous to actually observe operators doing it, then don't make the assumption, because you could never validate that that's, in fact, what's going to happen.
David: Drew, finally, my last point there, I guess, in the wake of this sort of incident and this industry is just because something's approved as safe by a regulator doesn't actually make it safe. I guess in all of these, whether it's aviation, oil and gas, mining, or nuclear, you know, all of these incidents that have happened to facilities or operations have, in some way, been endorsed or approved in some way by a regulator, but it doesn't make them safe.
Drew: I'm going to agree with you, David, but I'm not quite sure we do that as a takeaway except to be scared.
David: Wow. I guess we hear that argument as a justification in the industry as well. The safety case has been approved. People won't necessarily want to enter into any debate about the limitations of that assessment, because it's been approved by the regulator.
If that's the defense that's being used in your organization that has been approved by a regulator, then maybe that's not a great time to just stop based on that solely.
Drew: Yeah. I think something we need to be honest about as people working with these sorts of systems is that regulators, for reasons that are no fault of the regulator, lack the capacity to provide adequate peer review of design. We can talk maybe about that in another episode about some of the reasons why that's the case. That means then that we can't use the fact that the regulator has approved and accepted something as evidence of its adequacy, because that's not what regulators are capable of doing.
David: Drew, the question we asked this week was, when should incidents cause us to question our risk assessments?
Drew: David, I love your answer here, which is exactly what I was going to say too, which is that we should be questioning risk assessments constantly. Incidents should just be a reminder that this is something we should be doing constantly.
David: All right, that's it for this week. We hope you found this episode thought provoking and ultimately useful in shaping the safety of work in your own organization. Join us on LinkedIn or send any comments, questions, or ideas for future episodes to feedback@safetyofwork.com.