In this episode, we’ll be discussing another listener-suggested paper, from the journal of Safety Science, May 2022 - “Rewarding safety performance: Improving safety or maintaining beliefs?” by Bitara et al.
This was very in-depth research within a single organization, and the survey questions it used were well-structured. With 48 interviews to pull from, it definitely generated enough solid data to inform the paper’s results and make it a valuable study.We’ll be discussing the pros and cons of linking safety performance to monetary bonuses, which can often lead to misreporting, recategorizing, or other “perverse” behaviors regarding safety reporting and metrics, in order to capture that year-end dollar amount, especially among mid-level and senior management.
Discussion Points:
Quotes:
“I’m really mixed, because I sort of agree on principle, but I disagree on any practical form.” - Drew
“I think there’s a challenge between the ideals here and the practicalities.” - David
“I think sometimes we can really put pretty high stakes on pretty poorly thought out things, we oversimplify what we’re going to measure and reward.” - Drew
“If you look at the general literature on performance bonuses, you see that they cause trouble across the board…they don’t achieve their purposes…they cause senior executives to do behaviors that are quite perverse.” - Drew
“I don’t like the way they’ve written up the analysis I think that there’s some lost opportunity due to a misguided desire to be too statistically methodical about something that doesn’t lend itself to the statistical analysis.” - Drew
“If you are rewarding anything, then my view is that you’ve got to have safety alongside that if you want to signal an importance there.” - David
Resources:
The Safety of Work on LinkedIn
Drew: You're listening to The Safety of Work podcast episode 97. Today we're asking the question, should we link bonus pay to safety performance? Let's get started.
Hey, everybody. My name is Drew Rae. I'm here with David Provan. We're from the Safety Science Innovation Lab at Griffith University in Australia. Welcome back to The Safety of Work podcast.
In each episode, we ask an important question in relation to the safety of work or the work of safety, and we examine the evidence surrounding it. Today, we're going to tackle an issue which I guess falls into the work of safety, looking at the issue of safety performance, and bonus pay for individuals. David, over to you to introduce the topic.
David: Yeah. Great, Drew. I guess many organizations have incentive schemes for safety and targets, I guess, KPIs around safety performance. This can range from shopping vouchers through to other types of rewards. I guess, specifically today, we're talking about formal payment schemes that provide financial cash bonuses if certain safety performance targets are met. I guess it will be a lot of debate and discussion, and some of which will hopefully get through today.
A lot of work, Drew. I'm not sure how for me with the work that Professor Andrew Hopkins and Sarah Maslen at ANU, Australian National University, have done in relation to this. They published a book called Risky Rewards and the subtitle is How Company Bonuses Affect Safety. I guess it contained a lot of case studies, but also some results of 11 interviews that were done with different company leaders around 2015.
I guess what they framed up in that piece of work were three questions. (1) Do our incentive schemes work as they're intended to work? (2) Do financial rewards motivate the right behaviors, or do they have unintended consequences? (3) Are the right indicators being used? Drew, what are your sort of general thoughts on this topic and some of this body of work?
Drew: Just as a matter of social justice, I guess I'm pretty much in favor of profit sharing as a general idea, where if companies make extra money, then a large amount of that money should be returned to the employees. That should be more true for businesses that really profit off the back of the work of their employees as opposed to capital investment. But I don't see any particular reason why that should be targeted at senior executives.
I think that tends to immediately start distorting the incentives. In fact, a lot of ways that senior executives get rewarded are at the expense of the well-being of their employees. Even in the financial realm, we get the idea of people actually getting bonuses for cutting employee numbers and cutting employee salaries, which seems like directly perverse to the social justice of profit sharing.
When it comes to safety, I have really mixed feelings if we're going to have bonuses in the first place. Particularly, if we're going to have these bonuses that are rewarding executives, then we shouldn't be incentivizing people to ignore safety in return for other things. Logically, safety should be part of the mix.
On the other hand, that's like adding in one extra perverse incentive to a system that already has heaps of perverse incentives. It doesn't necessarily fix it, it just skews it to yet another way that executives are encouraged to skew the performance of the company to match their own personal interests, rather than the interests of the company and the employees. I'm really mixed because I sort of agree on principle, but I disagree in any practical form.
David: Yeah, I think there's a challenge between the ideals here and the practicalities. I have many experiences with this, but I had the opportunity. When I used to have a real job to set company targets and to actually define how they would link into bonus schemes in over about five years, I sort of changed them every year just experimenting with what went on in the organization with those targets.
I wanted to get away from recordable injury rates. I remember one year, I changed the bonus pay to fatality potential incidents, the number of fatality potential incidents, because the year before we'd had 20 across the organization, so I thought setting a 20% reduction down to 16% would focus people on the fatality risks and we would have maybe just a more focused approach in the year.
At the end of that year, we went from 20 the year before down to five. And I thought worse. That's a 75% reduction. Looking through, I kind of noticed that I'm not sure that the overall severity of incidents changed. But a lot of things were classified in a way that didn't represent them as a significant incident.
I guess the ideal of the situation and then the practicality of what it is that we can actually use and there are things like listing rules, which are very particular about what you can and can't do with financial incentives and things like that. Anyway, that was my experience, Drew. I guess it mirrors some of the things we'll talk about in the paper today.
Drew: The only time I've worked in an organization where I felt that there was very positive linking of safety performance to other targets was where it was an issue directly in the control of the person getting the bonus. I really don't think outcome measures are, but there are lots of things under the control of an executive that we want to incentivize.
For example, if we're going to give bonuses for early project completion to a project manager, then we want to say, okay, you've also got to complete all the safety activities before you can get that bonus. That's a nice, neat link. Or if we've got a senior executive, why not tie it to your promise to implement certain systems. Have you actually implemented those systems?
You've promised to attend certain meetings, you promised to do a certain number of safety walk arounds, whatever you think of safety walk arounds, tie people to the promises that they've made for things that they have personal control over. At least, then you're encouraging them to put some time and thought into safety.
David: Yeah. To be devil's advocate there, I also was responsible for setting a target once to improve the completion of safety actions on time to take that up from, I think at the time, we had about 50% of the safety actions completed on time. We set a target already, we got pretty close, and then went and did a bit of an assurance review on the actions in the sample that we chose.
We found two thirds of them closed out not done. I've kind of got a bit skeptical of anything that we count the more pressure we apply to that. Whether it's through financial incentives or management pressure, we'll get the answer that we want, but it just might not be the actual answer.
Drew: Yeah. The obvious rejoinder to that is if you're stalking tens of thousands of dollars of bonuses on a metric, spend more than five minutes trying to think of what the metric should be. Often, we put in these really simple things. We incentivize closeouts. Okay, so people focus on close outs.
Maybe we should have thought about whether what we really wanted was close out or whether we wanted them to actually do the actions. I think sometimes we can really put pretty high stakes on pretty poorly thought out things. We oversimplify what we're going to measure and reward.
David: I think that might actually be the punchline at the end of the episode, Drew. We might be done. Let's continue anyway. This paper was suggested by a listener. It's always a little bit easier, because I guess we at least know that there's one listener that will be interested in our discussion today. But if you do have those ideas, send them through.
We've talked about safety performance measurement a few times. We've talked about a report by Matt Hallowell and his team on the statistical invalidity of recordable injury rates. It's a work earlier, I think around podcast 35 by Helen Lingard and her team on leading indicators in the construction industry. This is an interesting extension of that.
The indicator matters. But also regardless of the indicator, what's the impact of having targets and reward schemes for safety. Drew, I also recall, we did an episode relatively early on on rewarding behavior itself and the implications of that. Drew, I might introduce the paper if that's okay and we'll go from there.
Drew: Go for it, David.
David: All right. The title of this paper is called Rewarding safety performance: Improving safety or maintaining beliefs? The authors are Fawaz Bitar, Diane Chadwick-Jones, Robert John de Boer, and Marcin Nazaruk. I know all of these authors, Drew. I'm sure you do as well.
Diane is no longer at BP. This study was done inside the BP organization. I guess under Fuzzy's leadership of safety at BP, there's been quite a reasonable number of contributions to the safety science literature based on case study work inside the BP organization. There was some work into just culture programs and a whole range of human performance type topics.
I guess it's just great to see organizations that are trying to bridge the gap between doing good quality research and I guess understanding the real life practices inside organizations. Robert John de Boer is a professor at SDO University of Applied Sciences in the Netherlands, and Marcin Nazaruk is at Baker Hughes.
He also does a lot of work with the International Association of Oil & Gas Producers, IOGP. I'm actually doing some work with him on resilience at the International Atomic Energy Authority at the moment, Drew. A fairly good combination of academics and practitioners in this paper.
Drew: Yeah, it's an interesting mix of authors. I didn't realize that the first author was a senior management within BP. My reading of the paper, I'd almost assumed that they were a PhD student, and that this was in sort of early work as part of a PhD project from the style of it.
David: No, I don't know how that first authorship popped up, but I guess that's the way that that happens. Fuzzy's a great leader. He's done quite a lot of good work there. It's a recent paper, Drew. Only published in May 2022, a couple of months ago in the Journal of Safety Science.
Drew, I thought we wouldn't go into a long background on organizational reward systems, because we've just done a little bit of that in the introduction. By design, organization reward systems, I guess, hold the assumption that what we're trying to do is add external or extrinsic sources of motivation to, I guess, an employee's intrinsic motivation to be safe and manages motivation to run a safe company by saying, let's just apply a bit more motivation for people in the organization to do the things that we want them to do.
Financial rewards for safety, they've been recommended a number of times. The Baker Report in the Texas City accident talked a lot about linking financial rewards to process safety indicators. Even the Institute of Occupational Safety and Health as recently as 2017 in the UK, sort of, really strongly encourages organizations linking financial rewards to safety.
However, Drew, the literature in this topic is fairly vague and inconclusive, I guess, as to whether financial rewards drive different behaviors than they would otherwise. Based on this paper, I guess, some studies in the literature review said that there are other processes like training and collaborative goal setting that are more impactful. Any sort of thoughts about the literature on this topic?
Drew: I find it rather interesting, the way the safety literature sometimes takes other aspects of organizational life and assumes that they're immutable. It makes sense if you're giving rewards for all other aspects of performance. It makes total sense that, why ignore safety? Surely, it rewards people and encourages them to reward safety.
If you look at the general literature on performance bonuses, you see that they cause trouble across the board, that performance bonuses don't achieve their purposes for financial or other performance either. They cause senior executives to do behaviors, which are quite perverse. It's the exact opposite. You're assuming that people are not aligned with the goals of the company to start with, which is usually wrong, then you deliberately put in place a system that is designed to further align them.
What that system actually does is cause them to take behaviors which are directly contrary to the interests of the company. That happens routinely with any sort of performance bonus. The question is, do we try to fix the whole system? Or do we throw in safety so at least safety is just as bad as everything else?
David: Yeah. I think as we get to the end of this paper, I guess that's the challenge of to do or not to do it. That's the question here and what's the least worst decision to make. I guess that's the major concern in the literature, Drew. Does this actually create these perverse actions, the management of the measure, as opposed to behaviors consistent with the achievement of the outcome that that measure is, I guess, meant to be related to?
Drew: I do like the way they finish off their literature review, though.
David: Oh, quoting a paper of ours?
Drew: Yes.
David: Yeah. Let's talk about that in a moment. It's always nice to see someone read a work that you've written. They're in relation to our safety work versus the safety of work paper. We talked about the fact that there are some things that are done in the interest of safety that create value based on the structures and beliefs that they maintain, and reinforcing the organization as opposed to the direct risks that they manage, and had difficulties to remove anything that you're doing for safety.
In that sense, I kind of likened the performance bonuses as a performative activity as opposed to an instrumental activity. Performance bonuses are not actually designed to be instrumental in driving safety risk reduction in our business. They're designed to create a feeling, an emotion, and a belief in the organization about the importance of the topic like a performance. What are your thoughts, I guess, around that?
Drew: First thing, I really liked the very nuanced reading of our paper. I don't think we've had that many people cite our work who've really sort of got the idea that the work of safety as institutional work can have a value in its own right, that it's not just this distinction between things that are good for safety and things that are not, that we can have these things that are not immediately instrumental, but that are still fundamental to how we value and manage safety in our organization. I really sort of liked that reading of it.
I think that this is the right way to look at cash bonus schemes, because in the short-term, they're definitely going to be dysfunctional in many ways. It's unlikely that you're offering someone an extra $1000 this year and suddenly going to prevent an accident. The question is, in the long run, what sorts of structures, cultures, and ways of thinking do offering these bonuses support?
Do they support structures, which are generally waiting for safety more than they would otherwise? Or do they support structures, which are systematically misreporting, misrecording, twisting safety? That's really the question rather than in the short-term, when they immediately work or immediately don't work.
David: Yeah. I think we've got a reasonable answer in the literature that in the short-term, they don't really do much instrumental work. This research is going to reinforce that. We might have an answer to that.
I don't know about you, Drew, but I guess when you talk about bonuses and KPIs to people inside organizations, they tend to hold one of two, I don't know if it's work beliefs, but thoughts about the issue. It's either like what gets measured gets managed. If you talk to senior people, they're very much about that if you're measuring it, it'll get managed. Then another school of thought of people in the organization will say, what gets measured, gets manipulated.
I guess they're the two sides of the coin. There are some interesting findings in this paper about depending on what level of the organization you're talking to, you're going to get one of those two responses.
Drew: David, I just want to do a quick look ahead to a single finding in the paper that I think is interesting and worth talking about now.
David: All right.
Drew: We can be really cynical about bonuses and say, okay, bonuses cause people to miss report things, to distort things, in order to earn the financial reward. One of the things they found in this study is that most of the people in the company didn't understand exactly how the bonus was calculated in the first place.
I don't think we should be too cynical that people are sitting down with their spreadsheets and working out how to manipulate figures, given that they didn't even know how the bonus was determined. It would be a prerequisite. Before you could manipulate it, you'd need to understand it at a bit of a mathematical level.
David: There's also an individual finding in the paper just on that linking performance bonuses to incident rates significantly increased the time that senior leaders spent involved in the activity of classification of incidents.
Drew: We'll have to talk about whether that's a good thing or a bad thing.
David: There's a lot of things to do in learning from the incident's process. I'm not sure classification is the part of the process that should get the majority of the attention. All right, Drew, method. Okay, we're a single company, I guess.
This one part designed one part, I guess, a convenient sample. I guess, if you're the head of safety inside an organization, then you've got an opportunity to get your organization to support some research as opposed to trying to do something like this across an industry. Also quite specifically, with a single company research, Drew, what they wanted to do was explore this topic without the results being confounded by variations in the design of the incentive scheme across different organizations.
They could look at role differences at different levels of an organization. They could look at geographic differences all within the same reward framework. I thought that was kind of like a good opportunity to go narrow and deep as opposed to broad and shallow on this topic.
Forty-eight employees participated. This is sort of a targeted sample. They identified about 70 people at all levels from senior leader to frontline workforce. I guess they're really trying to understand a couple of things. One was, what people thought the impact of having this bonus scheme was to the safety of work and then what people thought could be improved. Before they even got into that, they asked people whether they understood what the bonus scheme was and how it worked. Any thoughts on the research design, Drew, at that point, the sample?
Drew: Do we know if any of these guys are likely to be listening to the podcast, David?
David: Yeah, they will. Pass it on.
Drew: Okay. Let's say the good things first. One really good thing is the nice alignment between the questions they were asking and the method. You remember that if we're evaluating bonus schemes, we're not going to learn about how it directly distorts behavior just by doing interviews. They were careful that their research questions are not about directly, do bonus schemes work or not work? They deliberately ask questions about how people perceive, and think about the bonus scheme, and how they think it might affect their own behavior and other people's behavior.
They've been really careful about the questions that they're asking, both in terms of research questions and their actual interview questions. Forty-eight interviews is a heck of a lot as you and I both know from trying to do this sort of work, David. This is a real deep dive. And putting it within a single company has a lot of advantages, because we're all talking about the same scheme. We're getting a lot of data about how different people might interpret the exact same scheme that's going on in the background, so I really love that.
As we get into the paper, you start to see lots of statistics that they're using, which, frankly, is methodologically incoherent. You can't do qualitative research like this and do things like count the number of participants who mentioned a theme, or count the number of times the theme occurs, or try to do statistical comparisons between one geographic region and another. That's just all the guff that gets in the road of otherwise good research.
I don't like the way they've written up the analysis. I think there's some lost opportunity due to a sort of misguided desire to be too statistically methodical about something that doesn't lend itself to statistical analysis. I'd rather they put that thought and attention into what good qualitative research looks like instead of trying to make their qualitative research look like it's quantitative research.
Knowing the journal it's published, that may be entirely the fault of the peer reviewers that made them do with it. If they put in all those statistics because the peer review said so, then I apologize to the authors, and I sympathize with them. If they chose to do that themselves, then that wasn't time well spent. But overall, this is a good method for the questions that they're asking.
David: Yeah. I think Drew, I wasn't really going to talk too much. I was sort of going to make that comment, which was that the statistical significance is kind of irrelevant based on the sample. But I don't know why those statistical tests were being used, because there was some really good quality descriptive work in the analysis.
It felt like what emerged from this paper, Drew, is a whole bunch of really interesting questions that would actually now suit a very large scale survey sample that you could actually design based on the findings in here, and then you could be as quantitative as you want it to be.
Drew: Exactly.
David: Before we end the results, the analysis method, basically, they might need to do some analysis. You start by reading the transcripts, and then you're trying to extract what are the meaningful statements or pieces of information inside or snippets, and then you give them a label. Oh, this person said that bonus schemes encourage people to down classify incidents or something.
What you're trying to do is just extract snippets of meaning out of that theme and then do that, obviously, transcript by transcript. This is why 48 interviews is a lot of data. They obviously shared it around. The way that they did that is they all coded the same data independently, and then made sure that they were coding it in a similar sort of way, and then they're able to divide up the analysis research.
Drew, you and I have done that personally in the past, and it's not a bad process. We ended up with, I guess, over 1000 of these coded snippets of data. I guess it was those 1000 things that they started using to do some of their statistical analysis on. Drew, coming from semi structured interviews, I don't think that makes any statistical sense to do that.
For example, they said that the majority of this data relates to suggestions for improvement, and that's really just because that's the way that the questions we get. If you ask someone what's the link between this and that, it's going to be a short answer. If you ask someone what are the things that you think could be done to improve this, it's going to be a longer answer. I guess that's obviously why the majority of the information is where it is, Drew. That's at least my thoughts on the design.
Drew: Yes, exactly. How about we just ignore the statistics and get into the really rich, qualitative data that they've produced?
David: All right. First, when do you see across three areas? Understanding the link between the bonus scheme payments and safety performance. This is sort of a general question of what's the link between the two.
I guess, Drew, 19 out of the 48 people could say how they were linked to bonus payments. If you do the math, 31 people couldn't accurately describe the way that the system worked. It was predominantly the supervisors and the frontline staff that really didn't understand how it worked. The middle and senior managers knew how it worked, but the rest of the organization didn't really.
Drew: David, should we say a little bit about how the bonuses were calculated in this company? Do you think that's relevant at all?
David: Sure. Drew, my understanding from the paper is that the performance bonuses were tied to a combination of lagging indicators based on personal safety and process safety. The company also measured globally two leading indicators, but those two leading indicators weren't connected to the actual bonus payment scheme itself. Did that match your understanding?
Drew: I think there are sort of three factors that went in. Each employee has a maximum potential bonus that then gets multiplied by their performance against these lagging indicators. They have one of these stacked performance ranking systems, where in their performance evaluations, they are forced to rank their people. If you're higher up the rankings, you get a full score. If you're lower down the rankings, you get a lower score.
Automatically, not everyone can get the same bonus, because there's this forced choice that if you lower down the ranking, you'll be forced to get a lower bonus than someone else. There's a bit of individual performance, as well as performance against the metrics. But the individual performance isn't based on safety, it's just based on overall ranking.
David: I couldn't work out because at the end of the paper, it's said that that individual ranking thing has been removed, or didn't work out if that was done after the study, or it was actually removed, but people didn't know about it.
Drew: There were certainly people in the study complaining about the forced ranking system.
David: Yeah. but I didn't know whether it was actually a historical thing or not, but yes. I guess in the scheme, they're fairly general. These are the incident rates. Based on that, there's a bonus of a certain amount.
Drew, the impact on safety, I guess, is where people were asking, how does the performance bonus system impact the safety of work? And 87% of people in the survey, so all but a handful of people suggested that it had a limited or a negative impact on safety, which is kind of a lot of people out of 48. Let's say 42, 43 people who said either this doesn't help much or it does the opposite of help. Okay, it hinders things.
Drew: Yeah. I thought the mix of reasons, though, were really interesting. Some people said they didn't think it mattered because how they behaved as an individual, they didn't really understand how that fitted into the overall target. It's too distant, too lagging, to affect what I'm going to do right now. Other people said, actually, it's just too small, like implying that actually the problem isn't the bonus, the problem is that there's not enough of a bonus.
If it was a bigger bonus, it would have a bigger effect. Other people actually sort of were mixing up this should and does, saying that, because it's an ethical obligation, it shouldn't be incentivized for money, which really doesn't say whether it does or doesn't, just that they're uncomfortable with the idea that it is.
Even though they all seem to agree, they actually agree in ways that seem to be almost contradictory. There's a real mix of feelings about how bonuses work and when they should be there in the first place. If they are there, I bet they're bigger, smaller. Do they make a difference? Wide range of views?
David: Yeah, and all those examples that you gave. I wasn't aware of this, but the way that the performance bonus is linked to the incident rates, it's employee incidents only. Some people said, well, it's actually contractors who do most of the work at this company, so they're excluded from the calculation for the bonus scheme. I thought that was obviously a way of trying to get the things that were linked to the bonuses more into the control of the people who are impacted by them.
Drew: That's a great perverse incentive to make the contractors do all the dangerous stuff. Because if they get hurt, it doesn't affect the bonus.
David: It has a few unintended consequences there. Almost half of the respondents are unprompted when asked about how this impacts on safety. Half of the respondents mentioned that it actually showed that safety was a priority for the company. They weren't talking about, what does it directly do for operational decision making? I thought that it was quite interesting that 50% of people unprompted when they're asked about the link actually talked about this performative link that it contributes because it shows that it's a priority, it doesn't contribute directly.
People are saying that it shows safety is a priority. But there are a few other negative things as well, like people thought it influenced. There are a few other ways that people thought it negatively impacted the safety of work that influence reporting. To not report something or to reclassify something, people didn't feel it was based on the actual performance of the individual.
Drew, this last piece I've mentioned at the start, none of the negative comments that we've mentioned now about the impact on safety were raised or said by any of the senior managers. When asked how this impacts on safety of work, nothing negative was said by senior managers. I guess that's an interesting one. I don't know. What do you make of that?
Drew: I was going to ask you how you interpreted that. My immediate thought is that this is consistent with attitudes towards behavioral safety schemes. Senior people seem to have a very naive behavior list view of how frontline workers are incentivized. They sincerely believe that handing out lollipops makes people like them and sincerely believe that rewarding people with bonuses causes them to act safer.
It just seems to be something in the way managers are trained or socialized to sort of view the world as having these big effects from quite small deliberate actions within the control of the managers. I don't see that as self-interest, I see that as probably a genuine belief that they have, that this is a tool in their hands, and they want to believe that the tool works rather than anything cynical about the fact that they get the highest bonuses. I think they would see how it operates for other people quite differently from how they would see how it operates in their own lives.
David: Yeah. I guess you can only do what you can do. I guess, as senior people in organizations, you've got limited tools at your disposal to impact some of these things. It is one of those blanket tools that you can run across the organization and go, okay, this is going to shift the dial a little bit. Everyone's going to get $1000, then everyone's going to take safety a little bit more seriously. But I think that is unhelpful oversimplification of organizational behavior at the frontline.
Drew: Yeah. I think there really is a dangerous equivocation between the idea that safety is a priority and the idea that we are doing good things for safety. I don't see it as a good sign that the impact of this scheme is that it shows that safety is a priority. I think that sort of performative belief in safety as a value is directly associated with things like underreporting, because it incentivizes people to sincerely believe that they're doing the right thing by driving down the number of reported incidents. I don't think that it's unquestionably a good thing that we rate safety as a really high value. That can pervert our behavior even worse than cynical behaviors sometimes.
David: Yeah. We'll get to the conclusion shortly after we discuss one more topic. I might disagree with the authors a little bit there too about what they've suggested around making safety a value for the company. We'll get to that in a moment. But before we do, the last part of the interviews were about what suggestions people had for the link between performance bonus schemes and safety, so what to do.
Some people suggested things like making them both leading and lagging indicators. Lots of suggestions around safety, work type leading indicators, risk control, verification, safety leadership, visits, safety observations, lots of things that we'd seen in safety work leading indicators. I guess that's just to make things more into the control of the people as opposed to the outcomes.
People saying you're avoiding numbers games altogether just measuring quality, not quantity, making things specific for a site, like having different sort of performance bonus schemes for production sites, drilling sites, project sites, office sites, and things like that, having the individual contribution count more than the aggregate result, and then this idea of maybe a strong support for different scheme for leaders in the workforce, people who are on site, what is their scheme look like, and people who are maybe senior leadership and off site something a bit different. Drew, your thoughts on, I guess, some of those suggestions. What else did you take out of that?
Drew: I always find people's suggestions more interesting for what problems they reveal and for whether the solutions themselves would solve those problems. I think people are very good at identifying problems and not so good at identifying what solution would actually fix it.
We look at leading and lagging indicators. That clearly indicates a problem that these lagging indicators are too distantly removed from the actual activities they want to incentivize. But I think you and I, David, both probably agree that rewarding people for leading indicators is just going to encourage people to do more make-work in the name of safety.
David: Yeah, and I think this is the challenge. Dekker quotes that, "You're just counting what you can count rather than what counts." There was actually a specific quote from one of the interviewees in this paper that, "We're just chasing measures that we can measure rather than those that actually make a difference to safety." I guess that's why we're still debating performance metrics in relation to safety now 30 years since we've known the limitations of our existing performance measures. I guess the alternative hasn't been widely adopted yet.
Drew: And then we've got another one here, this suggestion of measuring quality rather than quantity, avoiding numbers games altogether, which points to a real problem that anything that we try to quantify is necessarily going to be abstract and distorted. But we know that when we switch to subjective measures for performance evaluation that it immediately has a gender and racial bias to it.
Again, we can clearly see the problem, but the proposed solution actually would make things worse. We'd end up just rewarding people for being male and white rather than for their safety performance. Why we go to quantified measures in the first place is to try to get rid of some of this discrimination that happens in performance evaluation.
David: Yeah. Again, Drew, I think it's the least worst choice. I suggest I mention your thoughts on this, because I put it in the practical takeaways, but I might just mention it now. Thinking about what outcome you're trying to create and then trying to find a way to try to measure that. If you think about safety leadership visits, when you asked the question, well, what do I want my leadership visits to do?
Do I measure my leaders doing one management walk around every month? Then yes, I'll get one management walk around every month. Or what am I trying to create as a result of that? Let's say you're trying to create three things. I want to demonstrate to workers that leaders care about them, I want leaders to understand work has done and what's really going on on the site, and I want leaders to understand and support the management of fatal risks in the business.
Am I better off counting a leadership visit being done once a month, or am I better off asking all of the workers three questions? Do I think my leader cares about me? Do I think my leader understands my work? Do I see my leader spending most of their time on the really important safety issues? I don't know, Drew, your thoughts about it. If people do leadership visits, then they can do leadership visits, but they've actually got to try and create that relationship with their workforce.
Drew: My thought is that if you were successful at doing those things that you're asking for, then you wouldn't need to incentivize them in the first place. If you had managers who genuinely cared and understood about safety, then they would be doing the right things as part of intrinsic value, they wouldn't need to be incentivized to do them.
Really, what we're trying to achieve, and this gets back to Hopkins' criticisms of incentive schemes and what the people in things like BP Texas were worried about, we're not trying to do these schemes to incentivize safety in the first place. We're trying to do these schemes to ensure that we can distribute some profits back to the company and to the workers through our bonus schemes without creating perverse incentives against safety.
Really, if the best we can do is avoid our bonus scheme, hurting safety, I would count that as a win. That would suggest even more simple things like giving the safety manager a veto over the bonuses. Your bonus gets calculated according to other things that the safety manager has to sign off on.
If you've done enough things to annoy the safety manager over the past year, then you don't get your bonus. All we're really doing is putting a check to say, don't downplay safety in your pursuit of other things or we won't reward you with your bonus.
David: Nice, Drew. I had kind of like this visceral reaction to the relationship between the safety organization and the rest of the organization of applying that veto.
Drew: Positive or negative visceral reaction there, David.
David: Quite negative. I could see the rubber stamping of a whole lot of things. The least worst choice. The authors, I guess, suggest that the annual cash bonus in their belief practice should be discontinued in the long-term because of its limited influence on safety in the first place and its potential adverse effect on safety outcomes, as we've talked about. But at the same time, they were very clear that there needs to be alternative means and mechanisms for ensuring that the company's commitment to safety is communicated and believed across the organization.
Drew, let's tackle these in a few parts and then we'll do the practical takeaway. Your thoughts about abolishing a cash bonus for safety in favor of other ways of creating a culture in which safety is a value.
Drew: I'm not certain that that's realistic, certainly, for a lot of positions in the company. If the logic of the way in which we reward people, incentivize people, and talk about performance, is in terms of targets, incentives, and bonuses, then we can't run safety under a totally different logic. We would have to actually remove that entire structure and logic of thinking, which I think is outside of our power when we're just trying to influence safety as a single thing. I don't think we can do anything that is separate from the way we generally think about bonuses and remuneration in a company.
David: Yeah, and that's practically what the conclusion that I hold, Drew, which is that if you are rewarding anything in your business, production, finance, employee engagement, if you are rewarding anything, then my view is that you kind of have to have safety alongside that if you want to try to signal that importance there.
If you're talking about removing that whole structure, then yeah, great, do that. But I think taking safety out of a performance structure or not putting it into a performance structure, probably, is a worse choice than having it in there.
Drew: Yup. I'll go a step further and say I think that is most important the more senior you are in the company. I wouldn't see that as being universally true if we had a whole workforce bonus scheme. I think it's really irrelevant for frontline employees whether we include or don't include safety in their bonus.
Just give them the bonus, because it's outside their control whether safety is achieved. Whereas it's more for senior executives and investment in safety that we need to be careful about, making sure that we've got safety as part of that package.
David: I had an interesting debate with the senior leader of an organization recently, Drew. I'm interested in your take on this. I guess they're using the Hudson maturity model. That's a whole nother conversation, but this idea that our listeners will understand from calculative to proactive and generally enforce all that. I'm trying to understand what they do with their safety KPIs and linking to bonus schemes in each kind of level.
I had this debate. If you are going to link them to be generative, then I guess if you look at that descriptor, you should have more than 50% of your entire bonus pool devoted to your safety metrics, and you probably should devote that 51% or more of your entire bonus to getting a generative outcome on a safety culture score.
The company kind of struggled, and I guess that's why some of these descriptions of these cultures are a bit impossible, because you could probably never see an organization. Most of the organizations have like 5% of the overall bonus on safety. It's entirely immaterial.
Drew: The idea of linking your bonus to a score on a safety maturity metric seems absolutely bizarre. I would have thought, for example, let's say we're talking about a senior management team actually taking action on safety. A generative culture would be about transforming the way the company manages safety. In other words, literally, what they should be doing is changing their metrics. That's what they should be rewarded for.
You're measuring them based on a metric when you're supposed to be coming up with new metrics. It's just an absolutely perverse incentive. The actual act of rewarding people based on a standardized metric, that is a calculative culture.
David: It is. I guess, institutions don't have radically different structures and processes as well. Maybe what you're saying there under those things is that those things are never actually possibly achievable, because an organization can never possibly change enough to do what some of those cultural descriptions say.
Drew: Yeah. The idea that you could have multiple organizations that are as far apart in their fundamental ways of thinking that some of those scales imply just makes very little sense when you've been inside a real organization. Where I think that sort of thinking might have value is that it does point towards localizing what the performance metrics are. I think that's one way to break up performance.
Break up perverse incentives is to target the incentive towards what people are able to control. That's part of why I said don't apply broad scale metrics to your frontline workforce, because they've got very little in their control. If you're going to reward them for anything, reward them for day-to-day behaviors on a day-to-day basis. Don't reward them at the bonus at the end of the year. You'll see maybe actually outcome measures aren't that bad as a metric. The least bad thing to measure them on is just the overall safety outcome.
David: I'll try another one on, Drew. Yeah, you go with this one. I did some work with a company recently, a refinery, and their metric for the year. Again, about localizing it for one of the refinery sites was to implement an additional 50 engineering layers of protections for process hazards.
One a week for the next year, so it's all about going back into their hazard analysis processes and looking at, where can we put instrumented systems? Where can we put additional barriers and protections? And then they have to come back with evidence that they've got an additional 50 independent protection layers, engineering, independent protection layers for process safety risks within 12 months.
Drew: My immediate thought is for the poor assurance engineer on day 364 of that year, sitting down trying to invent 50 things that have been done in the past year and reclassify things to count them as layers of protection. But actually, overall, I don't mind that scheme. It's got everyone thinking that the way to get our bonus is to find new ways to add protection.
Sure, there'll be some attempts to reclassify. Sure, anything they do is an improvement. They'll be trying to claim it as a layer of protection, but you are at least incentivizing people mostly in the right direction. If we're looking at least worst, that's not too bad.
David: Drew, that's good. I did make some jokes, because people have challenged me quietly off the record to find some things where you'll say, actually, that might work. My goal for some of these was to keep throwing out examples of things that I'm involved in and trying, at one point, to say, that might work.
All right. Some takeaways, Drew, I guess. I'm interested in your thoughts on this idea of measuring quality, not quantity. What are the outcomes, behaviors we actually want? Is what we're measuring actually giving us insight into that or maybe directing anything useful towards that if that's even possible? Thoughts?
Drew: I think that's absolutely the right mindset to have when you're trying to implement these systems. You shot down in flames, my idea of having the senior safety manager determined quality. We're going to run into that same problem that we ended up trying to create a metric for something that fundamentally is a subjective judgment.
David: Yeah. Some of these things are really hard with quality, not quantity, because most of the organizations that our listeners might work for will be listed organizations. They'll be subject to security investment rules, stock exchange listing rules, remuneration policies within their organizations, their shareholder reports, and you can't just pay people stuff for no reason.
You can't pay people stuff without a transparent and audible process for determining how much you pay people, which is why quantitative numbers based systems that can be audited, verified, and accounted for are unnecessary in lots of jurisdictions. Drew, there are a lot of constraints around what you can do with these things.
Drew: But we do, in other areas, use things like 360 degree performance evaluations. I don't see a fundamental problem in at least making your bonus conditional or prorated to a particular evaluation based on your individual goals for the year and your individual achievement against those goals as perceived by your peers, the people above you, and the people below you.
As with any performance evaluation, there's a little bit of perverse incentive there. There's a little bit of horse trading, a little bit of back and forth. But at least it sends that same clear signal that your bonus is conditional on you having safety objectives and you meeting those safety objectives.
David: Drew, you're fond of faith in human ethics. My last story of this podcast is going to be as involved. I know a person who shared it with me. Employee engagement was tied to everyone's remuneration. That team had gone from an engagement score in the 20s to in the 90s in 12 months.
I spoke with some people in that team, because I wanted to understand how he did it. They said, last year, we were the worst performing team. And every single week for the last 12 months, we've had engagement improvement sessions and workshops. So we all agreed that there was no way we're going to go through another year of all this extra work to do, so we all agreed how we're going to respond to this survey this time around.
None of which was known to the leadership group of that particular team who had suddenly thought that they were now the best leaders in the world. I don't know, Drew.
Drew: On the other hand, their goal was to improve the employee engagement score. They absolutely met that goal. Why should they not be rewarded for it?
David: Yeah. I think 360 degrees of it, and I guess the examples that we're sharing. It won't be any surprise to our listeners. This is all about the least worst outcome. Hopefully, most of what you've taken out of this is the least worst outcome.
I guess the last practical thing that I wanted to say, Drew, was, understand what is going on in your organization with your current system. Nothing stops any of you doing what Fuzzy and Diane did with BP here and speak to 50 people at all different levels of your organization.
Ask them these questions, do you know how this works? Do you think it has an impact on safety? Do you think we could do it differently? You want to know the unintended consequences that this system might be having and there might be ways to make it better.
Drew: David, you talked about me having too much faith in humanity. Here is something where I do have genuine faith. I think that when you ask people about things like performance incentives, they tend to give you honest answers, so long as you ask the right question.
People will manipulate their own behavior in order to achieve performance goals. But if you ask them, how is this goal causing you to change your behavior, they will often try to give you a good answer, because they don't want to be incentivized into doing bad things. If your performance targets are incentivizing people to rig the system and they would rather be doing something else, but they can't because of your incentive structure, then they're likely to be quite honest with you about how you are causing them to motivate their behavior.
Whatever system, just asking people, how is this making you behave, how are you reacting to it, what are you doing because the system is like this, will generally improve any system including a performance system.
David: Yeah, I agree, Drew. I think also just this idea of manipulating, and I don't want to be misinterpreted in this discussion because even in the example I gave earlier about significant incidents going from 20 down to five and changes in classification, there weren't people breaking classification rules. There's a lot of gray there. We've got numbers, but the classification of what was the potential consequence of this event could have resulted in a fatality, that's a massively wide gray area of judgment to argue and debate.
When it's not linked to performance bonuses and metrics, you might err on the side of saying, well, actually, this could have been a fatality, so we'll classify it as such. But then when you throw that in the mix and you go, actually, no, it wouldn't certainly have resulted in a fatality if something was different. We did have a few other controls that were there. When I say manipulate, I think it's just how people navigate the grade based on these systems. I actually don't think it's anything that's deliberately unethical going on.
Drew: I agree 100%. I think that's why we need to be very clear about when we are incentivizing people to navigate in a particular way because they're not being unethical. We're forcing them to make particular choices inside a gray zone by telling them what we want them to do through our incentive structures.
I was going to say, the question we asked this week was, ultimately, should we link safety performance to bonus pay?
David: I'm going to say yes, but how we do it matters.
Drew: I would say, in an ideal world, we wouldn't have conditional bonus pay. But given the world we're in, yeah, I agree. We should do it, but we should be careful about it and think hard about how we're doing it.
David: Awesome. That's it for this week. We hope you found this episode thought provoking and ultimately useful in shaping the safety of work in your own organization. Send any comments, questions, or ideas, for future episodes to feedback@safetyofwork.com.