On today’s episode of Safety of Work, Drew is out, so we have a special guest, Russell McMullan!
Russell is a safety practitioner based in New Zealand. He joins us to tell us about his work and together we wonder if safety and design processes change the design itself.
Topics:
Quotes:
“We don’t always have opportunities in safety science to get some objective artifacts…”
“I’ve never heard of a designer saying they’ve got plenty of time and a ton of budget, so let’s, you know, get down and optimize for safety.”
“And if we start by listing the operations before we list the hazards or risks, then we’re going to be in a much better place.”
Resources:
David: You're listening to the Safety of Work Podcast, episode 69. Today, we're asking the question, do safety in design processes change the design? Let’s get started.
Hi everybody, my name's David Provan and today I'm joined by Russell McMullan. Russell’s a research practitioner based in New Zealand. In each episode of the Safety of Work Podcast, we ask an important question in relation to safety of work or the work of safety and we examine the evidence surrounding it.
Today, Russell is going to talk us through his research where he explored specifically safety in design processes. We’re going to explain what those are. What Russell wanted to understand was to what extent do those safety in design processes actually change the design and make the design safer.
Russell, welcome to the Safety of Work Podcast. How about you tell us a little bit about yourself and your background and how you came to be researching safety in design?
Russell: Thanks very much, David. It certainly is fantastic to be on the podcast. How did I come to research safety in design? Well, it actually goes back to the DisasterCast Podcast when I was listening to Drew on those. He talked about the Master of Safety Leadership, which I thought sounded like a pretty good idea. My background is 18 years in the Royal New Zealand Air Force as an aeronautical engineer. The later parts of that are in systems integration and systems safety.
I left the Air Force in 2011, joined a consultancy, worked across multiple sectors in safety critical systems, and more recently since 2015, I've been employed on a large rail infrastructure project with a heavy focus on operational safety in the railway licensing and those aspects.
I really wanted to align my masters’ research with my role and really get to the bottom of this whole idea of safety in design. If we’re going to commit a lot of resources and a lot of time to it, I wanted to understand how we might make it more effective, how we might get better use from our designers and in the work that they do, and avoid any waste the way possible.
David: Great. I've been in and around large infrastructure projects, predominantly in the oil and gas industry but a little time in rail and general construction. We do increasingly spend a lot of time on these safety in design processes with a lot of hope that we use these processes to create much safer designs from a construction point of view, from an operational point of view. I suppose having read ahead of your research findings, I think we're going to have an interesting discussion today just on how those processes work or maybe don't work as we think they might be an organization.
Why was this interesting? I know it aligns with your background but going into this, what are some of the things that you really wanted to understand? What was your specific research question?
Russell: I really wanted to understand, is safety in design effective, just fundamentally? We talk about it a lot. Every organization seems to have a guideline that they suggest people should follow. Almost every safety risk practitioner I encounter has written their own guideline that they want to show me. But I really want to understand its effectiveness. Are we seeing the outcomes of safety in design? Is it very well-defined even as a concept?
Fundamentally, it's just revisiting that whole idea of what we’re doing effectively and I guess that aligns to that idea of reality-based safety. We really just want to make sure that we're building on a solid foundation.
David: I think we would ask a few questions, particularly over the last 12 or 18 months, with some of the issues that some of the Boeing aircraft—the 737 Max 8—had around software changes and designs, and the risk assessment processes around those. There are a few things that happen in safety critical industries that raise questions from time to time about these processes, but there’s also this big expectation that we have that there’s a lot of specific safety activity that goes on during the design phase of these systems and projects.
Maybe, why don’t you tell us a little bit about how you approach studying something like this? Like you want to understand whether safety in design is effective. What was your research design? How did you do that?
Russell: That’s a great question, David. In terms of research method, I had a range of options available to me. I could have undertaken interviews, I could’ve sat in on as many safety in design sessions as possible. But I thought the most practical way to really explore how safety in design is being undertaken is to try and collect up all of those outputs from a safety in design activity. Whether they're risk registers, altered designs, or some other thing that I can use to really objectively explore and someone could arguably pick up those same artifacts and come to the same set of conclusions if you like.
I then undertook content analysis and thematic analysis on that information that I was provided to try and reach that point of saturation we had there was really nothing more to be gained from that information. I went into it with four main questions. The first question was what are the artifacts that are created in practice? What are provided to me? How are practitioners using safety in design to implement safety outcomes? And what are the justifications used within safety in design?
I was also looking to see if there was an idea of a thing called risk shuffling, which was suggested in the literature where people make decisions that moves a risk from one phase in the life cycle of, say, construction into operations where for instance that minimize the safety risk in construction but inadvertently created an operational risk that exist for the life of the asset or vice-versa.
David: I think that's a really good approach to do that. We don't always have opportunities in safety science to get some objective artifacts and safety in design is one of those safety work practices, if you like, that does generate output documents all the way through to the safety cases like we spoke about in episode 68 of the Safety of Work Podcast. I think that then you can actually go okay, how to interrogate these documents? How do I look to see what's actually being recorded as the outcome of the safety in design processes?
You do these thematic and content analysis processes. What do you find? What are the key findings that you came to out of that analysis?
Russell: The first thing that I had to think about was the limitations around the information that I’d received. I approached basically as many people as I could. I've got some pretty good contacts into the broader design industry. I really ended up with a much smaller set of outputs and artifacts than what I was expecting. I was provided ultimately 31 unique safety in design outputs from about 20 different organizations. But surprisingly, almost all of the people that provided me the information, you and me personally and I guess there’s this idea of trust that underpins sending me information.
Some people couldn't send me their information because of internal concerns that they were sharing things that they shouldn't be sharing or some perception of liability should that information get out of their control. That significantly limited the study. But in terms of what I found, I found firstly, everybody seems to call these outputs and artifacts different names. There's no consistent naming. I received mostly hazard register type outputs with a couple reports, one presentation, a couple of meeting records.
I was really expecting some things like an altered design or maybe someone sending me some marked up drawings or something like that. I didn't receive any of those things. People tend to think of safety in design as a safety in design risk register and that is the main output.
David: I can understand the practical challenges of getting some of that information from organizations having being inside businesses their reluctance to share. Like I mentioned before when we did the episode last week, Drew talked at length about some of the challenges in getting safety case documents publicly available so that they can be researched and they can be interrogated by interested stakeholders.
It's just something that we seem to hold on closely, too, which is a bit of a shame because I think you’ve made a call at the end of your research for someone to go and actually try to test your conclusions more broadly. I think it would be a fantastic opportunity if you could get a very large set of outputs to either confirm or challenge some of your findings because some of the findings that you did come to with those 31 artifacts were both surprising and maybe a little bit concerning to me as well in terms of those processes.
Russell: Yes, I guess one of the first things I was really looking for is how well described are the changes in the design given that the whole idea of safety in design is to minimize or eliminate the safety risk by modifying the design. That's the primary activity of safety in design process or even the core idea of safety in design is to modify the design. There are two secondary outputs. One is to communicate risks downstream, the residual risks that you are unable to deal with in the design that you might have unified.
Thirdly, as an assurance activity to record the fact that you've done something. Maybe even who was involved and to what degree you've done it. Just to give people in the future some idea of what you've done. But when I went looking for changes that were claimed in these outputs and these artifacts of 4000 lines of safety controls that were listed, I found only four changes to the design that were claimed.
David: I recall those numbers jumped off the page at me. I think it was something like you said 4000 controls across 3000 individual hazards or risks and four design changes. Maybe that's the rate, maybe 4 in 4000 is actually the rate and that's what it is. Maybe those four are really great impactful design changes, but intuitively, for me that felt like a low number.
Russell: Yeah, I agree and when I drew out and actually put on my safety practitioner hat, looked at some of the risks that have been identified, and thought in my own mind what might have been undertaken. I came up with some pretty basic ideas like one was a gate that would swing open, they didn't modify it by putting an automatic spring gate closer on it. They suggested that an administrative control of teaching people, reminding them to close the gate was more appropriate, little things like that, so I tend to agree.
I found people claiming controls outside of the hierarchy of controls. The hierarchy of controls is the general idea that you eliminate the control by modifying the design before applying administrative controls such as training or PPE. But what I found is people would use things completely different that we're just not even part of that hierarchy. They would talk about awarding the job to a competent contractor as a claim for having minimized a specific hazard.
There was no modification to the design. There's no discussion on the controls that might be applied either administratively or otherwise. It was some claim around some completely tangential matter such as the design is subject to peer review, somehow claiming that is making the design safer. Now, that might do that, but it's just not within the hierarchy of controls which is in theory what safety in design is specifically looking for.
David: I see that in risk registers, not necessarily in safety in design risk register outputs but just occupational safety risk register where there’ll be specific hazards. The controls will include things like a prestart risk assessment toolbox or inspection processes of these general safety practices that maybe they’re there to provide some oversight of all the safety risks, but they don't specifically address any individual safety risk. This finding surprised me a little bit because I would have thought safety in design practitioners are really experiencing concepts of layers of protection, independent protection layers, and what actual effective control is in terms of its functionality, its reliability, its availability and all of these things that go into risk engineering. It surprised me that we were seeing this type of information on safety in design output documents.
Russell: I totally agree. I was really expecting a lot more positive actions in terms of modifying the design and making those claims within these outputs. To be fair, I think it could just be in the infrastructure sector that we’re seeing this and that's why someone really needs to broaden out the study and look at some other domains because that's not my experience coming from aviation.
David: Yeah, so maybe this adopting of these processes from industries into other industries and the process gets done, like you said earlier, safety in design is done maybe as a compliance activity or maybe is an insurance activity. That draws in that whole thing that we talk about every single episode on the podcast about safety work versus the safety of work. What's my mindset going into this particular practice in the organization? Am I going in with the mindset of producing a risk register or am I going in with the mindset of changing the design to make it safer?
Russell: That’s exactly right. I guess the main thing I found is this idea of messages to the future. A lot of the artifacts I got were very complete looking tables and spreadsheets, often with lots of nice colors. They look complete. There's a lot of text and a lot of green on the right hand side; people using a common red and the green type status. When you pick them up they look pretty good. I guess that falls into that whole assurance argument of if I got a big stack of paper and it looks pretty good then it must be all right.
When you actually dig a little bit deeper into the content. I found things like what we coined as messages to the future. A lot of these messages to the future were either a message for someone in the future design stage, some message or instruction to a contractor or to users or maintenance. Sometimes it was a reminder to follow the law or follow the rules. Sometimes they were just messages to the unknown, like someone would write a statement and you couldn't determine who's the same there or anything like that.
Then there are also hidden messages where regularly in safety risks registers, you have a column called existing controls. There’ll be some little action that they're expecting someone to take in the future that's written in that column that no one will probably ever find. No one will ever go looking for it because they have no reason to. That really gave me this idea that maybe the designers don't have a great understanding of the work that's being undertaken because some of the messages are really precise, that's quite good. But some of the messages were very vague. Yet that vague message was being used as a claim to reduce a safety hazard or risk in some way. Yet, when you explore it, it's very difficult to unpack and really confine your mind to the effect that that has on that risk.
David: I think that that third thing there, messages to the future, there's a lot that can be wrapped up in that, like you said, there are different sub themes. Also, in my mind there's a lot of, you mentioned, reshuffling earlier. There’s a lot of, well actually we can resolve this in operations with a procedure and then you started raising some questions in the papers.
What organizational pressure is the designer under? Does the designer actually feel that they've got the autonomy to propose or recommend a change? Is there a norm or a climate in the infrastructure project that we want to prioritize? The cost of the construction of the capital project over these extra nonessential but potentially important for safety type of design issues. That would be one thing which we would have got, I suppose. You could’ve done interviews as well or something like that, you might have been able to test some of these assumptions that you've actually alluded to in your papers well.
Russell: Definitely. I certainly recognize that designers have enormous pressure. They are constrained by codes and standards, by budget, by time. I've never heard of a designer saying they've got plenty of time and a ton of budget, so let's get down and optimize for safety. They pulled 20 different ways by as many people. Yet, when I read through all of the artifacts that I had, I really just got a sense that there was a defending of the current state and very little ability to influence the design. I really came away with a little bit of a feeling that we’re putting in a lot of effort into this process and it's just really not working for us as we might expect it to.
I've had people over time saying, well people just aren’t doing safety in design properly. I question what do they mean by properly? There's plenty of different guidelines out there that all suggest many different things. This is hard. We expect a fire engineer to have their work peer-reviewed and have a certain level of expertise and be signed off by various people. In theory, they’re mitigating one small set of risks and yet we try to undertake safety in design which is looking at every risk in theory that the system might encounter in the future and its entire operational life and minimize all of them optimally in the design without affecting project cost or time.
David: I think that's a great description of the reality of a practitioner responsible for safety in design in terms of their role. The cynical part of me and I think you’ll allude to this in part of your paper where you say if we assume that we've got a set of standards that we’re complying with, we've got a functional specification of what we want our system or what we're designing to actually deliver, we've got a schedule, we've got a cost.
A lot of design decisions are made that are balancing all of those things all the time. When we come along and do the safety in design process, you talked then about defending the position. We really are doing the safety in design process boxed into a corner with all of these decisions that have already been made, the pressures that are already there, the tradeoffs that have already been made, and the designer maybe it's only in 4 out of 4000 opportunities to make a subtle tweak to what's going on.
Or maybe the messages of the future is really, really important. Maybe the safety in design role is not actually to change the design—I'm just hypothesizing now. Maybe the safety in design process is to actually look at all those decisions and tradeoffs that have already been made and broadcast safety information down the line so that people have it somewhat early. Maybe that's the role.
Russell: Well, I did ask myself that very question and I wasn't seeing what I thought I'd see. Once I've done the research, I actually had to circle back around through literature and go, have I assumed that? Have I just assumed that safety in design is to modify the design? I went back through all of the literature and guidelines. It's pretty clear that the purpose of safety in design as written by all the major guidelines and books and things is to modify the design to communicate the risks that you haven't been able to optimize the design downstream and record for assurance purposes. We’re missing at least some part of that. At least, the recording of what we have done.
David: That reminds me of you saying as written there through literature, this might be safety in design as imagined in the literature and safety in design has done in organizations.
Russell: Very much so.
David: But I think it's fascinating. You’ve got these 31 artifacts, lots of different companies, different domains, and you found these three big broad themes about safety in design at least in the sample of this study really didn't do much to change the design. Lots of the risk controls that were produced out of safety in design process is to claim risk reduction were some general broad statements outside of the hierarchy of controls. There was a lot of let's say broadcast messaging to the future for other people at some point—a designer down the track, an operator down the track, or contractor down the track.
I think they're really useful and interesting themes for people to reflect on with their own safety in design processes. Stepping on from that, what would be some advice? What have you taken out of this in your practitioner role, but also what advice would you give to people in organizations, people who do safety in design processes in real companies on real systems?
Russell: That's a great question. I think that's probably the most important one, is moving forward. How do we make the most of this effort that we’re putting in to ensure that we're not creating waste or wasting people's time? Also getting the benefit that we’re seeking.
The first one I think is to start from that basis that you spoke about where designers are already boxed in the corner. We already know that something has been built to code for instance, if they're building to a code. It's already going to be signed off and peer-reviewed. Let’s not write that into the register at all. Let's just maybe make an assumptions list that those things are going to be true. That we’re going to have a complete contract to build it, we’re going to train our operators. Just write one of those little things down once. Don’t stick it against each hazard or risk.
I think the next most important thing is that we are really clear around the operations, situations, and things that are going to occur and whether or not we understand them. If we are designing or undertaking a safety in design activity on a piece of instruction, we don't look at the design and critique the design.
We try to understand the operations that this thing is going to be used for and then we try to understand the risks that arise from those operations, either historically through work that's been done or using this idea of foresight of what might occur in the future. If we start by listing the operations before we list the hazards or risks then we’re going to be in a much better place. Then we ask ourselves what have we done in the design already to help us minimize or eliminate these risks? Where we haven't done anything we say, look we can't think of anything that we've already done to help minimize or eliminate that.
Just put it in [...] or we’ve considered it, just so that it's really clear an artifact that nothing's been done. Then we ask ourselves what more might we do? How could we modify the design to make it safer against this hazard or risk? How might we further control it with the specific aim of modifying the design? How might we modify the design to make it better? If we can't, we can't. If we can, we put that idea up and work it through the process. See if the designer can accommodate it, given whatever time and other constraints that they have.
Ultimately, if we want to pass down those hazards or risks that we've been unable to manage downstream, I think we’re better off to start articulating what the hazard or risk that we've identified and let those downstream make their own judgment around its probability and its potential consequences.
I think we are better off just articulating that we've seen it, articulating in whatever language we can use to describe it as best we can, but those further downstream are probably a better place to understand the nature of the risk and how it might manifest itself and do harm or something like that.
David: I think that's a really practical and comprehensive process for people to follow and I liked the way that you purposefully used language when you were describing that to say how might we change the design. I’d almost see that as like a column header in a safety in design output register. Some of these things even have specific columns for a design column and, like you say, a later phase operational risk.
I also really like the way you describe just identifying the risk and not necessarily prescribing the magnitude of that risk or the best way of controlling it because other people at other project phases are going to be a better place to make those decisions. I think Drew said on one of their podcasts about risk assessment, one of the good ways to test whether risk assessment is effective is have the conversation about what can we do right now.
Even that example you gave about the gate, just letting people know that this gate should stay closed and the design won’t automatically keep the gate close. Whether it's signage or training or something else, then you leave that to someone else, but you at least point out the fact that this gate to be closed and the design doesn't ensure that it is always closed. That's almost giving just a list of hazards to handover to the next phase of the project and let them figure it out.
Russell: Yeah and practical experience, those downstream and those service operations or those undertaking construction really do seem to have very different ideas of what's risky and what isn't compared to the designers and those involved in the safety in design process. Even when you've got the operators in the room and the construction people in the room, you try to do as much as you can by modifying the design to make it safer, of course, but anything that you do downstream, their perception of risk is going to be quite different.
David: Russell, you got your masters’ in safety leadership. This was your thesis paper. You've now written it into paper that you are just about to submit to a journal. You're about to go through that publication process yourself. Our listeners probably know it could be 6 or 12 months before people see it, but you might be able to make a preprint available on a ResearchGate or something at some point in time. You're also just about to embark on a PhD. I'm interested in your decision for going off and doing that and also what you plan on researching.
Russell: That's a really good set of questions, David. Firstly, yeah of course, I’ll make the paper available, no problems there at all. I think what I've found is quite interesting and I'd like other people to explore in their own organizations to see if they're seeing the same things in their safety in design outputs.
The masters’ was great. I really enjoyed the research phase immensely. I found it really valuable. Strangely, I could have written a lot more and I think there's a lot more to explore. I started studying a little bit later in life. I'm in my late 40s now and the family are at a certain point where I could probably take on a little bit more time. I think there's a lot more work to be done in this area and I've embarked on the next stage of the journey.
What I really want to look at in the next phases of study are how designers are making these decisions. This idea, this notion all designers writing these things down, telling us these things, and writing these messages to the future. I really want to see behind the curtain and understand the rationale for that.
You spoke about institutional logics in a previous podcast, and the research that you've undertaken. I basically apply that framework to designers and design organizations, and look at how they make decisions around safety. If they are indeed boxed in a corner and safety is really the last thing they think about, I think we really need to understand that. But if it isn't, that's good too.
What are the constraints? What is the world that they live in that is resulting in these artifacts and outputs that have these things that we’re seeing? If we can understand that, we can maybe influence that and then get better outcomes.
David: I think it's a fascinating research area. It's one that I hadn't had too much of a look at until you started researching this area, but like you say, I'm thinking the opening line of your paper, or the opening line of your executive summary, you make these comments about how safety in design processes give us this opportunity to fix problems easily and cheaply before we construct and operate these systems.
If we're not taking every advantage we possibly can take to make these things for the life cycle of the system or the asset, then we're probably missing an opportunity in safety. It surprised me that probably there wasn't more research in this area but after 69 episodes of the podcast, the lack of research in an area of safety doesn't really surprise me as much anymore.
Russell: I’ll agree with that.
David: Thanks so much for your time today Russell. I am going to ask you the question, so the question for this episode was do safety in design processes change the design? Your answer would be?
Russell: Generally, the answer for that is no. In some very minute number of cases, yes. I think this is an area that we need to apply a lot more attention to.
David: Thanks for joining the Safety of Work Podcast. We hope you enjoyed this episode and found it useful for changing the safety of work in your own organization. Send any comments, questions, or feedback directly to us at feedback@safetyofwork.com.