On today’s episode, we discuss whether there is safety in numbers.
We use the 2019 paper, Safety in Numbers, to frame this week’s discussion.
Topics:
Quotes:
“A lot of statistically dodgy stuff gets published in some very, very good journals and some otherwise very good authors.”
“When something is psuedo-science, you tend to find that there are some studies that say that it works...until the very best studies show that the effect doesn’t work at all.”
“Whenever you use a concept of a rate instead of a raw number, you are assuming a linear relationship.”
Resources:
Elvik, R., & Goel, R. (2019). Safety-in-numbers: An updated meta-analysis of estimates. Accident Analysis & Prevention, 129, 136-147.
David: You're listening to the Safety of Work podcast episode 9. Today, we're asking the question, is there safety in numbers? Let's get started.
Hey everybody. My name’s David Provan. I’m here with Drew Rae and we’re from the Safety Science Innovation Lab at Griffith University. Welcome to the Safety of Work podcast. If this is your first time listening, then thanks for coming. The podcast is produced every week and the show notes can be found at safetyofwork.com. In each episode, we ask an important question in relation to the safety of work or the work of safety, and we examine the evidence surrounding it. Drew, what’s today’s question?
Drew: David, the question for today's episode is, is there safety in numbers? Usually, when I hear that phrase, it's some bad pun about matrix. In this case, it's really an old idea or maybe even urban legend that being part of the crowd is always safer than being alone. It works something like this. All those being equal, you’d expect that the number of people who get hurt during that activity goes up linearly with the amount of risk exposure. If 10 people ride a roller coaster and 2 of them get sick. Then if we have a hundred people riding a roller coaster, we expect 20 to get sick. If a thousand people ride the roller coaster, we'd expect 200 to get sick.
But the safety in numbers idea is that all of this is not always equal because the amount of exposure can also change the amount of risk. One person walking alone might have a high risk of being mugged, but 10 people walking together don't have 10 times the risk. They have a much lower risk.
The other extreme about this safe for one person to be on in their own bridge that dangerous for thousand people to crowd on to the same bridge. Not exactly a thousand times more dangerous but in fact, almost infinitely more dangerous.
The idea of safety in numbers is that there's a nonlinear relationship. Today, we're going to look at a paper that test one aspect of their relationship between risk and risk exposure. Then, we're going to have a bit of a broader conversation about how these type of thinking—thinking clearly about the relationship between exposure and risk—can help us to manage safety a little bit better.
David, before we get started, do you have any examples of your own where you can think of this nonlinear relationship between risk exposure and total risk?
David: Drew, there's a lot of examples and when you suggest this topic, I would've thought of, "Yeah, definitely safety risk or total risk doesn't always move in a linear way with exposure." I think lots of our processes and organizations are actually geared in assuming that's true.
For example, we see it a lot in the way that we aggregate risk in our enterprise risk frameworks within our organizations. We expand our existing risk, we add on new activity, and then we come up with our new level of risk. For example, if we own one production facility and we board a second production facility, would we have twice the risk over a particular incident in one of those facilities?
I also thought about it in relation to other activities. I had a look at some of the driving or the motor vehicle data, and I think the Australian road statistics are something like one fatality for every 200 million kilometers, but not every one of those kilometers is driven with the same level of risk.
If you're an organization, you could have a 100 million kilometers. If you added an extra activity, it may even be only an extra 100 kilometers of driving, but it could have a completely different risk profile with the rest of your activity.
Drew: Yeah, that's a really good example because that figures one fatality for every 200 million kilometers. Assumes that the moment you stepped out the door, that's the risk that you're exposed to. Whereas in fact, if it is someone driving for the first time, suddenly they're undertaking a huge amount of risk. Not just they share the fact that one fatality per 200 million kilometers.
David: Yeah, we see that with a lot of activities in organizations that have safety risks. A lot of these activities are unique. I think we intuitively know that, but I don't think our risk systems are maybe mature enough for us to help our organizations understand that.
We do some work in the general aviation space with some organizations. They might be very experienced at a particular type of aviation activity and then they introduced this one new different type of activity. When you're looking at the total risk, you could almost say that the order of magnitude increased their risk just by adding this one incremental risk of exposure.
Drew: The other example that I was thinking of is a little bit closer to the one we're going to look at in depth today. This is the idea of what sort of red lights are railway drivers most likely to run through?
A lot of statistical analysis just assumes that what matters is whether the light is red or green. Whereas, when people have done more a deeper analysis, they find that there's some lights that drivers always stop at because they expect them to be red but what really captures drivers out is if it's a signal that is almost always green and it's just sometimes red or it's almost always red and it's sometimes green.
When they're expecting one thing and they get hit with another thing is where most risk comes in. It's in the unusual rather than in the regular exposure to the risk.
David: That's always been the history of our protective level crossings as well. With train and vehicle interactions, with drivers just saying that they just didn't see the red signal because they drive through that crossing every single day and it's never red.
Drew: Let's get into the paper that we've got for today. It's got a bit of a boring title. The title of the paper is Safety-in-numbers: An updated meta-analysis of estimates. As it says on the tin, this is a 2019 paper that updates a 2017 paper which had a very similar title.
The first author of both papers is Rune Elvik. Rune is a transport safety economist. They’ve got a long career of using statistical analysis to examine the impact of various safety policies, particularly road safety policies. The 2019 paper was published in a journal called Accident Analysis & Prevention, which is a very good journal and the 2017 paper was published in Safety Science, which is also a reputable safety journal.
I should point it out here that normally, I give the author and the journal first up because they're pretty good sure hands just for giving the general idea of equality of the paper. When it comes to statistical papers, that's probably less the case. A lot of statistically dodgy stuff gets published in some very, very good journals and by some otherwise very good authors. Unfortunately, there are no real sure hands for trusting the results and statistical analysis. You need to get a bit down and dirty with the numbers.
That's not quite as bad as it sounds. You don't have to be a statistics expert. In fact, often that's the problem is they can get too hung up on the details and statistical methods. That distracts them from fairly obvious issues and problems. The trick I use is to focus on four things.
The first one is what are the independent variables? Sometimes there's just one, sometimes there's two or three. They're the basic things that we're comparing. Secondly, what's the dependent variable? That's the output of the study or the thing that we're comparing independent variable to. Thirdly, what's the apparent relationship between those two things? What are authors or the statistical analysis saying is the relationship between independent and dependent variables? What does that relationship mean or not mean? How much can we usefully extend from the statistical analysis back out into the real world?
Here’s a very good example. We can ask, does smoking cause lung cancer? The independent variable is whether or not you smoke. The dependent variable is whether or not you have lung cancer. The apparent relationship is that people who smoke a lot are more likely to have lung cancer than people who haven't smoked a lot.
What does that mean back in the real world? In that particular case, it meant that there was a lot of time and lots of research to get from the point of, there’s an apparent statistical relationship before there was a compelling argument that smoking actually caused lung cancer.
David: These matters of statistics are lies. Statistics, correlation, and causationis something that we need to be very wary of in safety, and social science more broadly. But we see these claims all the time in safety management where some would say, "Improving your safety culture will cause a reduction in injury rights or other correlations like a certain type of leadership behavior is related to an increase in safety climate scores."
Very little of these are claims that are made so commonly in safety are actually based in evidence. I just did it for a bit of fun. I just googled spurious correlations and crazy correlations. Listeners who are interested can run off and do that. You'll find that there's a 95% correlation between the crude oil exports of Norway with drivers killed in railway accidents.
For those who follow pop culture, there's a very strong correlation between people drowning in swimming pools or the number of people who drown in swimming pools and the number of films that Nicholas Cage appears in each year. We can actually find lots of relationships between lots of things and sometimes fool ourselves into thinking that we understand what's going on without actually truly knowing what's going on.
Drew: I find that really frustrating because some people fall into that trap and other people use that trap as an excuse to just dismiss all statistical data. I don't think either extreme is healthy. Some things which it's really useful to use statistics to unpack and to test, and the topic we're looking at today is a good example of that. Don't just take these as a broad rush hate towards statistics. Take it as a, be careful about what the statistics do tell you and be careful about what they don't tell you.
David: The statistics is one thing, Drew, those questions that you asked earlier. What's the apparent relationship? And then, what does this apparent relationship mean and not mean? It's probably that step that gets made from the statistics to the conclusions that we draw, which is a step as opposed to the actual raw statistics themselves, which is the step that we really need to make sure that we understand.
Drew: Let's look at the study. The study calls itself a meta-analysis. I should explain a little bit about what that is. There are three standard types of literature review that you see in journal articles. There's a very straightforward literature review, which just means that someone has gone out and collected lots of papers about the same topic, and then providing a synthesis of those various papers.
The second type is called a systematic review. The key thing about a systematic review is that there are rules that govern what papers are found and what papers are included and not included. Because of those rules, systematic reviews tend to be much more tightly focused around that particular question.
For a meta-analysis, that's one step tighter again, where we have a very, very specific thing that we are testing. We have often a claimed relationship between two variables or particular statistical model and we're looking for papers that are being published that directly speak to that very specific question. The rules will say it must have this as an independent variable, it must have that as the dependent variable, it must publish its effect size, it must publish the statistical method that you used.
Often, meta-analysis will not just summarized those studies, but it will actually combine mathematics, will combine the numbers of people included in each study, so that it creates almost a single bigger study that includes all of the data from those previous studies and reanalyzes them.
The advantage there is that it's like the way with telescopes. You have one really big telescope or you can have lots of little telescopes [...] around the globe, all of which feed together. You get this benefit of effectively having a globe-sized telescope looking at your data.
The disadvantage is that if any one of those studies has some mistake or misreporting it, you risk at contaminating the whole set. You risk accidentally double counting or securing your results by combining all the errors from the previous studies, not just all of the benefits from the previous studies.
This paper definitely doesn't fall into that trap because it doesn't do that mathematical combination. It's more of a very, very precise summary of all the previous analysis, comparing and contrasting them rather than directly adding them together.
The precise questions it's asking. The independent variables are the volume of motor vehicles—how many cars drive around the place—the volume of pedestrians, and the volume of cyclists. The dependent variable is the number of accidents.
The relationship it's testing is to ask, in order to predict the number of accidents, can you just use the volume of pedestrians and cyclists? Or is there some other factor that is pushing the number of accidents down as the number of pedestrians and cyclists increase, relative to the number of more vehicles? It's that mysterious effect pushing the number down that they have called the safety of number effect.
David: Drew, is it right to say here if I am riding my bike on a road—it's a country road, there's very few bikes out there, and I have a certain risk of an incident—versus if I'm on a road with 9 other cyclists, does this group have a 10 times greater risk of one person being hit by a car on a road where there's frequently lots of cyclists, versus myself out of my own on a country road?
Drew: Yeah. It's saying that if your risk of being hit is maybe 1 in 100 rides, then we would expect the risk of 10 cyclists to be 10 times that. If instead of being 10 times that, it's a little bit less than we say there's some other factor we've got to include in our model which is going to be the relationship between how many cyclists there are and how many cars there are.
It's a fairly precise question. The answer for this question is unusually for the papers we've done so far, David. It's an unquestionable yes. Yes, even when we do studies well, we find that there is a safety in numbers effect. Even more exciting, the bigger we make the study, the more we control for confounding factors. The more recent study the study is, the bigger the effect is. That's exactly what we'd hope to see for our a real effect.
When something is pseudoscience, you tend to find that there are some studies that say that it works, that the bigger the study is, or the better controlled the study is, or the more recent the study is, the smaller the effect is. [...] the very best studies show that the effect doesn't work at all. Either way saying the exact opposite.
David: It says that the old cliche of “there’s safety in numbers” which we've [...] is true. There's some evidence and fact behind that saying that there are safety in numbers.
Drew: Yes, indeed. The first thing we need to ask is, does this just happen for some statistical accident? Both papers include a fair bit of statistical analysis that I’m not going to try to summarize here. It does things like checks for publication bias. It checks to see whether there's some statistical effect that might be securing the result or one or two big studies that's making it seem like there's overall a good effect.
All those tests comes back clean. They're basically saying, there's no pattern here of bias. It's not that people who don't find for safety in numbers don't publish and people who do find safety of numbers do publish. This looks like it's a real effect that's being found by the studies.
The tricky things then is to say, what does that mean? Why is there safety in numbers? The others are pretty honest about the fact that they don't know. Statistical analysis often can't give you that causal explanation.
There's a few things that they put forward. It could be really straightforward. It could be what I imagined myself when I'm looking at these results that I imagined you, our listeners, are thinking of as well, which is, we need a lot of cyclists and pedestrians around if the drivers tend to expect cyclists and pedestrians, they tend to slow down, they tend to look out, they tend to be more careful.
It could be something totally backwards. It could be that when you have a safer city and safer infrastructure, that leads to more pedestrians and cyclists. It's not pedestrians and cyclists being safe. It's, there are pedestrians and cyclists because it is safe. A lot of those is likely to be the full story. One of the reasons we know is not the full story is that these different studies that are summarized sometimes look at a really specific location, like an intersection. Sometimes they look at one major cycling route. Sometimes they look at a whole city or a whole region.
The effect is weakest the more you focus on a single specific location, which is a little bit confusing and contradicts both of the main theories. There is also the problem that, if this was a single real effect, we ought to be able to give some rough idea of how big the effect is. In fact, the effect changes a lot between the studies. It's more variable than we really would expect to see, which probably means that there's no single explanation. There is, in fact, several different things going on.
Maybe it's true that drivers are more careful. Maybe it's also true that safety and infrastructure leads to more pedestrians. Maybe it's also true that pedestrians and cyclists behave perfectly when they are more than about, that maybe there's an explanation three, four, five and all of them contribute to this overall phenomena.
David: Yes, Drew. We thought now which is great, we've got a positive findings so we're giving our listeners that work for safety, which is safety in numbers. Now, if we're moving to practical takeaways, this is where we're going to generalize a little bit and maybe just throw our thinking on the table. This is are not takeaways that have come out of the paper. If, for instance, the safety in numbers effect is real, how might we think about that safety in numbers effect for how we approach safety management in our organization.
Drew, do you want to kick us off?
Drew: Okay. So, for the first time, I think we genuinely can take out of this study and that is, the study shows clearly, at least for this particular example, a very nonlinear relationship between exposure to risk and number of accidents. It's using, David, the start of the podcast.
A lot of our safety practices assume that there is a linear relationship. One ofthe key ones that I'd like our listeners to be aware of is that whenever you use a concept of a rate instead of a raw number, you are assuming a linear relationship.
When you talk about five accidents per hundred million kilometers driven, that's a rate. When you talk about the number of injuries per number of hours worked, that's a rate. Those rates are explicitly assuming that there's a linear relationship. This study gives you a very strong counter example that unless you have good evidence that the relationship is linear, you really shouldn’t assume there.
The practical takeaway here is just don't report rates, or at the very least, if you're going to report rates, report the raw numbers as well because the rate [...], both the numerator and the denominator, is not really sensible to divide them but at the very least say what the numerator isn't saying what the denominator is. You don't say that your [...] is 0.584. Say, “There were three accidents based on 30,000 hours worked.”
David: That's good insight because we all intuitively know that not all hours are created equal, not all exposure kilometers are created equal, but most of our reporting centers around rights and will be easy to say if we increase the amount of exposure, then we will have a linear increase in risk. I can't think of a specific example where it probably would be the case in safety.
The second one I want to talk about is that there could actually be a danger in rarity. This is something that I think is really interesting to think about in safety management. If you're more frequently performing a particular activity, you're more used to it, you know more, or you're more experienced about it, then you might come with a lower level of risk than if you do something quite rarely.
If we use the safety in numbers effect to say that if I do the activity once, I've got a certain amount of risk. If I do the activity 10 times, I don't have 10 times the amount of risk. I might learn how to do it really well and get really good at it. I might actually have not much more risk than just doing it once.
It's really interesting how we think about where we focus our safety management efforts in our organization. Do we focus on the things that we do all the time? Do we focus on the things that we do quite rarely? Our task that we do rarely may be safer to be performed by a professional contractor who does that type of task all the time.
Drew, do you think the safety in numbers effect can apply to how we think about the frequency of our organizational tasks?
Drew: This is a really useful concept to think about. Contractors are a perfect example to throw at this because we can mention an organization has a task that they want to subcontract for maybe just for business reasons. We can see that there could be two very different safety things going on.
One of them could be that we're delegating a task that is rare to ask, that we don't have the right equipment, or our people are not experienced in. But giving it to someone who does it everyday is probably equipped to do it, who understands the hazards, and who does it really well.
By subcontracting, we go also be giving it to a higher firm filled with people who've never done this before, where new people are constantly encountering the hazard in unexpected ways, and are, in fact, drastically increasing the risk.
Two business decisions that we could both label as subcontracting the risk. One of them, we subcontract the risk and it gets much smaller. The other, we subcontract the risk and it gets much larger.
David: Yeah, that's it. That's a good example, Drew. The third takeaway that we're thinking of is also, where in our organization do we have this lone cyclist or lone pedestrian type of activity? Straight away, I thought of things (quite simply for our listeners) like designated walkways around your facility and you're plant. I know we put a lot of effort into separating people and plant and designated walkways.
If your people are following a route within your plant or facility that operates as a plant and vehicles are used to seeing people in, then the safety in numbers effect would say that you are possibly going to have a lower risk than if you're people are just walking randomly around the facility.
Drew: The example that’s framed to mind for me was a couple of really tragic cases of people being inside cardboard compactors and getting crushed. Afterwards, the comments were just no one expected someone to be there. It is nothing but inherently dangerous in being inside and this equipment that is being turned off. It needs someone inside it to maintain it. The problem is that if someone's not even thinking that might be a possibility, it's totally a normal thing to do to plug the equipment back in and turn it on again.
David: I know we're generalizing quite broadly and meandering around different types of topics and now, maybe into areas of isolation and lock out, but thinking about what people expect to see in your organization, what risk they expect to face, where they expect to face them, and then what risk and things that they don't expect to see.
We can't assume that people who see things 10 times or experienced something 10 times, have 10 times more risk than people experiencing something once. In fact, like we said here, safety in numbers goes the more people see things, the more people experience things, the less risk it will be.
This not only sounds very intuitive to people but actually, my reflection would be, I've never really considered what it might mean for safety management, decision making, and our risk systems within our organizations.
Drew: That's really the main thing that we thought was interesting about the study that we like you, our listeners, to take away. It's just that way of thinking that how we expose people to risk. It's not just a straightforward case of more exposure equals more risk. It is we're thinking about how our systems encourage and manage familiarity with risk. That's something that we can actively manage in safety.
To try to capitalize on that idea, the risk doesn’t have to go up just because we’ve got more exposure. The risk can go down or at least not go up as much if we're careful about who gets exposed, how they get exposed, and how that familiarity and confidence with the risk works.
David: Drew, for those who want to engage with us for our safety of work LinkedIn group, is there anything that you'd be more interested in hearing more about or understanding more?
Drew: Sure. I actually have a lot of questions coming out of this, but I suspect some people who are out there listening already know the answer to. Some of the single ones is just that I'm interested in how much people use exposure as one of their metrics or things that they care about when they think about risk in their organizations.
Some people, for example, keep track of how much driving occurs. Do they have life exposure to risk as a safety measure? Do you try to measure and reduce the amount of work at heights, work in confined spaces, driving, or some other unsafe activity where you track the volume of the activity? It's a proxy for how safe the organization is.
David: Yeah, I know, Drew. I've been involved in a specific program that was titled Reducing Windscreen Time and the program specifically was about how to target if we're reducing driving exposure by 40%, in the hope that if the list number of journeys you were doing, the less incidents you would have based on your incident rate. They set a target to reduce your overall exposure to driving within the organization. It was quite a big program about Reduce Windscreen Time.
Drew: I'm interested in what things people are doing in that space. I'm also interested if people know if this research that says whether this is a good or bad idea. It seems logical to me that it is a good idea, but I'd love to know if you've done any direct measurement or know of any studies that talked about with this programs are successful by other measures.
David: You mean reducing exposure is a good thing to do?
Drew: Yeah. We started off with what we've got, add hazard by windscreen time, we introduced the windscreen time. You did that, in fact, reduce the number of accidents, that sort of information.
David: How about when you talked about practicing fire drills? The example that I'm trying to think of is if you are a firefighter. You need to try and learn how to be a firefighter by starting fires and learning how to put them out. That is increasing your risk because you're essentially being exposed to fire when you don't actually need to be exposed to fire.
Is that exposure actually building confidence, so that if you're going to face that situation in the future, you are able to manage the risk associated with that situation? That type of example becomes really interesting. If we reduce our exposure down to a certain level, do we get to the point where we become less confident with having to deal with that risk?
Drew: Yes. I'm really interested in what our readers think about this. Also, I think they're probably going to have to come back to this in a future episode, specifically about how we develop risk confidence and to have people of really interesting things. Feel free to pull us in that direction.
David: Yeah, great. That’s it for this week. We hope you found this episode thought-provoking and ultimately useful in shaping the safety of work in your own organization. Send any comments, questions, or ideas for future episodes to us at feedback@safetyofwork.com.