On today’s episode we discuss whether Virtual Reality is a more effective method of safety training.
We chose to use two papers to frame our discussion. Those papers are Construction Safety Training Using Immersive Virtual Reality and Comparing Immersive Virtual Reality and PowerPoint as Methods for Delivering Safety Training.
Let us know if and how you are using Virtual Reality in your business.
Topics:
Quotes:
“It was fairly targeted towards the outcome they want from normal types of training.”
“It does suggest that if we are going to spend more money on this...then the way to follow up is down that idea of simulating particular work tasks…”
“It’s like watching the Phantom Menace and then watching the Phantom Menace with 3D goggles and deciding that 3D goggles are no good, because they didn’t make it into a better movie.”
Resources:
Sacks, R., Perlman, A., & Barak, R. (2013). Construction safety training using immersive virtual reality. Construction Management and Economics, 31(9), 1005-1017.
Leder, J., Horlitz, T., Puschmann, P., Wittstock, V., & Schütz, A. (2019). Comparing immersive virtual reality and powerpoint as methods for delivering safety training: Impacts on risk perception, learning, and decision making. Safety science, 111, 271-286.
You’re listening to the Safety of Work Podcast Episode 19. Today, we are asking the question, "Is virtual reality safety training more effective?" Let's get started.
Drew: Hey everybody, my name is Drew Rae and I'm here with Dave Provan, and we're from the Safety Science Innovation Lab at Griffith University. Welcome to the Safety of Work Podcast. If this is your first time listening, then thanks for coming. The podcast is produced every week and the show notes can be found at www.safetyofwork.com. In each episode, we ask an important question in relation to the safety of work or the work of safety and we have a look at the evidence surrounding it. David, what's today's question?
David: Thanks for taking the load on the intro there, Drew, and mixing it up a little bit for Episode 19. The question for this episode is 'Is virtual reality safety training more effective?' A few of our listeners have asked questions along the lines of what new and emerging technologies are useful for safety, or is there anything innovative that you can talk about in the research in relation to safety. To be a little bit specific, we thought virtual reality is getting a bit of uptake in modern workplaces and in our general lives. We thought that might be a good place to start.
Last week, we spoke about Powerpoint and had good fun with the Powerpoint episode. This week, we thought if we said last week that Powerpoint might not really be effective because of the current information being presented to our learners, then virtual reality is sometimes being held to as a replacement for training, or enhancement of training. We thought hey, let's look at virtual reality and safety training and see whether it's more effective. In fact, one of the studies that we talk about today is a direct comparison between virtual reality and PowerPoint as a training medium.
Before we get to the study, let's do a bit of a background. Truth is, about 20 years of research into virtual reality and virtual reality systems have come a long way in the last 20 years. It’s actually a real mixed bag of results. I usually get away with just grabbing one paper and forming a question out of that and doing it, but I had to at least get two papers today, and I had to actually dig through a whole lot more just to try to make sure wasn't going to present the finding that wasn't representative of the actual total literature. How much reading did you do in the virtual reality space?
Drew: I think the problem we're facing here, David, is that virtual reality, a lot of the promise is not in industrial applications, it's in computer games. We're talking about a whole bunch of young, mostly male academics out of university who have discovered an excuse for playing computer games as part of their job. That's why we have so much research into virtual reality. I would argue it’s also why we have so much research into robotics. Researchers love having cool toys, and VR is a whole suite of good toys.
The first and most obvious one is the visual display. The idea here is that we're providing some sort of three-dimensional representation of an environment which is virtually created, pretty much almost exactly like it is in the computer game where we model the locations of objects and then we’ve got a couple of different types of technologies with different graphic fidelity to present those objects to the user as if they are sitting in the middle of that environment.
The difference between a computer game on a screen and the headset is that in VR, it’s more immersive. There is a sense of feeling that you are there, and that’s the goal and something they directly major is that feeling of being immersed, that feeling that you are in a real environment. You know it's fake but it feels more real.
We’ve usually got some way that you can maneuver around that environment. In movies and in books, you’ve got these really sophisticated spherical treadmills, you can just do any human action then it gets replicated. We don't have that technology yet, pretty much what you have got is some sort of hand controls. Sometimes they’re haptic gloves which can measure exactly where your hands are, sometimes it's more like a mouse, or a little device that you hold, that you can simulate walking forward with, and there are many that can fully capture your emotion. That's the goal, is to actually have you be able to move around as if you are in the environment.
David: I'm not a gamer but I do recall a few times during my Ph.D. in the middle of the night when I send you an email and you send me back a message saying that either had a pretty productive day and you're rewarding yourself with an hour or two of video games at midnight. I've never actually had a VR headset on but I’m going to assume that you have at some point in your life.
Drew: The honest answer is not good ones, sometimes you go along to places that are doing public demonstrations but VR doesn't exist in high quality for sitting at home. It's still quite an expensive technology. It's expensive both the hardware itself, and you have to have the latest, good quality because people learn from computer games what good looks like and they hate really being in badly simulated environments.
The other thing is the creating the environment that you are in, is actually a lot of work. It costs as much to make a good computer game as it does to make a movie. A lot of that is in the creation of the environment. Creating something that is real enough to be in is very expensive. A sort of central thing that's necessary to do research or to do training is that their users have to respond to the virtual environment in a very similar way to the way that they’d respond to the real environment. If the fact that it’s fake causes them to act like it’s fake, then it’s not going to be useful for any sort of training or for data collection.
David: We spoke about that, I think it was Episode 11 where we talked about the commercial sea skippers and that early computer simulation of making decisions to safety over productions and those trade-offs.
Drew: David, maybe you’d like to jump straight into the papers for today?
David: Yes, like I said I chose two papers, did a bit of work for this episode. The first study was done in 2013 and we'll talk about the method and the findings. Based on that, I thought I’ve got to go and get something more current that talks about what's happened between 2013 and now. I’ve got an experiment which had an expanded review section from 2019, just to make sure that we were going to represent the field like I mentioned earlier.
We talk about these two papers one at a time and then draw some general conclusions. Paper number one was titled Construction Safety Training Using Immersive Virtual Reality, it was authored by Sacks, Perlman, and Barak who are all based in Israel across two universities. Interestingly, the authors are from either the Faculty of Engineering, or the Department of Management. The Paper was published in the journal of Construction Management and Economics. We’ve got people who are not in safety departments in universities, are not publishing their safety research findings in a safety related journal, do you think this happens a bit?
One of the pros and cons of having different disciplines doing safety research and publishing safety research in different journals…
Drew: Obviously, it’s great to have safety cross-pollinated with lots of other fields of research. When it works well, I think that it’s really good for safety research. The risk is that from a research point of view, what we’re getting might not be up to date with the current field. In the case of VR, as someone using VR for safety, a VR expert who’s taking it into a new application who might not know enough about safety might be a little bit naïve about what they’re doing. Or are they someone from safety who’s just grabbing the latest VR stuff without really knowing what’s the best and appropriate ways to do VR research?
When we get people who know both, when they understand safety and they team up with someone from another field or they become an expert in another field, that’s the best, when we get to genuinely combine the parent field. But even then it gets a little bit harder for people consuming research because it’s hard to know where to find the latest stuff. If you want to learn about VR, do you search about VR plus safety, or do you just learn about VR, or do you go to a journal where they publish lots of VR stuff, or are you going to a safety journal and search for VR? It can be really hard to find and to be sure you’re finding good, recent stuff. That then compounds if researchers have that same difficulty, so they can’t build on research; they just end up reinventing things or going backwards.
David: Yeah, it can be hard to be confident if throwing your arms around a body of literature when you are looking into a different topic, given how many different disciplines and some of the concepts that we are trying to understand in safety can show up.
For this study, the researchers, the going in assumptions to their research was the ability to identify safety risks in the workplace. Is it acquired by people through their training and their experience? The researchers were already questioning traditional training methods that they saw in the construction industry. Their hypothesis was that providing safety training in a virtual reality safety training in a construction site environment would be more effective than traditional training methods. By traditional, they meant sort of face-to-face, classroom style training methods, written information, PowerPoint, and some of those things we spoke about last week.
I will run through this quickly, they basically built a virtual construction site so that they could compile a set of training scenarios. They based the scenarios that they designed on another study that had identified the most frequent types of construction accident scenarios. They had sort of three chapters to their training.
Section number one was just general site hazards, site safety requirement rules, and PPE; the types of things that you get in a typical safety induction. The second was safety for a specific job of casting concrete, so they actually designed this VR training to be ‘This is how you do this job and this is the safety requirement for this specific job on this site.’ The third was, again, another specific job of stone cladding work, and this was on the façade of the building, and basically it’s like ‘This is how you do stone cladding work, this is the process, and this is the safety requirements along the way.’
They had 66 participants, they split those up into 3 sets of 20 to 25. There’s a bit of differences between the groups. Two of the groups were construction workers, and the second group was 3rd year Civil Engineering students with limits in construction experience. They can probably groups that have been in the construction sites and groups that hadn’t been on construction sites before. They tested their safety knowledge immediately prior to the training, immediately after, and one month after.
Here’s some people who were doing some normal training, here’s some people doing some VR training, here’s what you knew before, here’s what you knew straight-after, here’s what you knew in a month’s time. The questions they sort of asked were just open questions about identifying hazards, and then some specific behavior questions like in this situation, how would you behave, and so on.
How do you feel about this design as a method?
Drew: One thing that I love about the papers last week and this week is they’re measuring training and they’re thinking hard about what’s the actual point of the training. We’re not talking about student satisfaction with training, we’re not talking about just giving them a quick multi-choice test, we’re saying what do they know beforehand, what do they know immediately afterwards, and have they actually learned that as opposed to just having encountered the information and it’s still sticking around in their brain. This is something you will find a lot in education literature; before, immediately, after. Then, wait a month or a couple of months then see if the information has stuck.
David: Yes, when they were doing the questioning, they were asking people, for example here is a situation you find, how would you behave in this situation? All the way to normal topic recall. It was fairly targeted towards the outcomes that they’d want from normal types of training. There’s three pages of statistical tests and results in this article, I don’t know if you want to go into the tests of statistical significance, a lot of statistics there. I don’t understand all of it, Drew understands a lot more than me.
In the interest of time for the podcast, what they basically found was there was quite a significant advantage in work-related training in terms of the VR training experience. For the stone cladding work, and for the casting concrete where people are going through a work process and having the safety information integrated into that work process, their knowledge both immediately after and one month after was greater than the traditional training methods.
Interestingly, Drew, there was no clear advantage for general site safety. This means from this study that VR is not necessarily a more effective way of comparing the general safety knowledge that we would normally compare in an induction. If only one of those work practices was actually significantly greater retention after one month, it’s interesting, it was not that compelling in the findings for me but the authors did strongly conclude that incorporating VR in construction safety training is recommended but I’m not so sure based on the results in the study.
Drew: Yes, I think this comes back to what I was saying before about researchers wanting to play with cool toys, you don’t want to publish your paper on VR and construction safety and conclude this is kind of a dead-end, let’s go back to preparing PowerPoints for people. You want an excuse to keep trying out the VR tool kit.
That’s one of the problems with academic publishing too, we get to see this as an electronic black and white PDF, and we’ve got no idea when they talk about using VR whether it didn’t work very well because VR is bad, or whether it didn’t work very well because actually they didn’t do a great job and they do need more money to produce a better VR training. If they have better VR training, maybe it will work. I think they are overstating their conclusion, but I do have a lot of sympathy. I’m not saying it’s bad, ‘give us more money and we’ll try to make it work,’ may in fact be the next step.
David: One of these things which I wrote in here, there was some better recall after one month of the training outcomes for the VR group, sort of across all three. The author sort of concluded that the VR experience sort of translates into better learning outcomes. I was kind of like if the researchers came along a month later and said, “Do you remember that cool VR thing we did a month ago, do you remember what was covered in that?’ Versus saying the same thing to a group of people who just had a normal kind of classroom thing, there’s a finding from another paper in 2006 that actually goes “Yeah, I can recall that information when you prompt me.” There’s kind of no information that suggests that they were applying that knowledge to their work situations in the one month in the two kinds of tests that they recall.
Drew: What I think is most interesting out of this is the difference between the general training and the work scenario training. I think this is the sort of thing that can be reasonably taken out of preliminary work practice, as it does suggest that if we are going to spend more money on this, we are going to do more research. The way to follow-up is down that idea of simulating particular work tasks rather than using our cool VR toy just to replicate generic hazard training.
David: We know the importance of the use of flight simulated training in the aviation industry and things like that where we can simulate different real potential operational scenarios, and we can actually expose people to risk in a way that doesn’t actually put them at physical risk and develop skills in that way. I think the Griffith Aviation School just purchased a whole lot of VR headsets so they actually help students with normal phases of flight type training, so you can have 20 students in a room doing VR flight-type training rather than one at a time trying to get time through a simulator.
I think there’s real application first to think about, like you said, in safety in relations to simulated work environments. Deciding that you’re going to invest money to replace your general induction training into a VR headset induction training probably isn’t going to change the outcomes of that training.
Drew: That’s probably a good point to move on to the next paper which we are doing a little bit of an accelerate forward in time, eight years I think. What does the slightly more recent research?
David: I kind of thought the VR technology has come a long way in eight years, so I thought maybe in that first paper the VR environment was just not real enough and we can’t see the environment that we’re used to. They had some limitations with the actual how immersed they could get the participants into that environment. I thought we’d go to a more recent paper. This second paper is titled Comparing Immersive Virtual Reality and PowerPoint as Methods for Delivering Safety Training: Impacts on Risk Perception, Learning, and Decision Making
Drew: I love it already from the title.
David: I thought you’d like me to bring you a paper with PowerPoint in the title, but clearly they were looking at comparing virtual reality and PowerPoint for delivering safety training, and what impact it had on risk perception, learning, and decision making. Five authors based across two universities in Germany, Leder, Horlitz, Puschmann, Wittstock, and Schuetz. Again, not from safety departments, but it was published in the Journal of Safety Science, Drew, which is a very reputable safety journal. Now, we’ve got researchers who are not in safety departments but they’re publishing their safety findings in a safety journals. You are associated with Safety Science, do you see a lot of submissions from outside safety departments coming towards Safety Science?
Drew: The honest answer is we do, and very often we just throw them back in the pond. What tends to happen is that people have this finding and they say ‘Well, it’s vaguely related to safety so let’s publish it in a Safety Journal, but the contribution it’s making isn’t an advance in safety, it’s an advance in measuring personality, or it’s an advance in doing VR, or it’s an advance in machine learning and its application to safety is kind of incidental to the main contribution it’s making. Sometimes we get stuff that actually says something genuinely new about safety, and I think this study is directly talking about how we train people in safety, so it’s obviously relevant within the scope of the journal.
David: I got the exact same response, Drew, when I’ve tried to publish one of my safety papers in a management journal, which was actually this is better off going back in a safety journal. I’ve seen that template response.
Here, we have two experimental studies. Study number 1, there were 53 apprentices, they were split up and they either did PowerPoint training or they did an immersive VR training. The researchers measured whether the learner’s conscientiousness, or the locus of control, moderated the effects of safety training. They found out in this study one which is called the preliminary study that people in the VR group had a change in risk perception in terms of the probability for their judgment of risk. They saw risks as not as risky after being in the VR environment.
This is something that was found in the first study too that I didn’t mention. If people have just done a traditional training, they saw things as more dangerous than if they have gone through the VR simulation and actually rated down in terms of probability and judgment for risk. Drew, do you think of why people might rate risks lower after they’ve been in a VR experience?
Drew: I think it’s because the presentation of the information doesn’t have context when it’s just on a slide. If you gave me a list of 10 things to worry about and 1 of them was falls. I might be imagining falling off a 10-story building, and then going to the VR environment and I am just looking down a meter, I’m not going to worry so much. You hear about electric hazards and then you go into the environment, the electric hazard is is the cord plugged in? I think having that sort of specificity and the context can give people… I doubt it’s actually less risk perception, I think it’s probably more accurate or more realistically risk perception, but that is just a personal guess.
David: The research has got some really interesting in this. They ran study two which was again a good lesson for people doing experimental study, designers don’t necessarily run one experiment, and then find something, and then publish it. That might be why psychology has gone through a bit of a replication process, at least in the study we did last week. People actually tried to replicate their own findings before they published.
These researchers, they ran a second study and they could not replicate the results in study one. After study two—we’ll talk a little bit about the method in detail, but they basically concluded as a spoiler which was I suppose quite good for the research, is to just say we did not find anything. They concluded that the costly procedure of developing immersive virtual reality does not seem justified for safety training because the less costly PowerPoint procedure with vivid film scenes did not fare significantly worse in respect to these perception of risk learning outcomes and decision making. I added in there about PowerPoint and vivid film scenes, Drew, because I want to actually be really clear in the design here as to why they ended up with it. Do you want to say something before I do?
Drew: No, I think what you’re about to say is exactly what I am interested in. I kind of suspect that they tried too hard to make the PowerPoint and virtual reality identical, except for whether it was VR, and as a result they were putting PowerPoint slides into a VR environment rather than creating an actual virtual experience.
David: We talked about research in the degrees of freedom and dilemmas, and they obviously really wanted to isolate this immersive VR experience. You are exactly right.
What they did, and they controlled it actually really tightly, they took high school students. Some were apprentices, but none had very much experience with the actual work or the machines involved in the training. The group of participants did not have individual experience with the work and experience with the risk that has compounded.
They made the PowerPoint basically contain as identical content as they could to the VR. They had vivid videos like you’d see in the movies, they had static sequences that were clipped out of the VR scenarios and things like that. Really, the only difference was one group was wearing the headsets and had an immersive experience, and one group was watching screens with PowerPoint slides with embedded videos and things like that.
Then, they went and measured these things like risk perceptions where they ask people to assess the probability and severity potential of accident scenarios, they ask people to record their knowledge of hazards in the training, and then they ask people to choose between a safe and an unsafe machine and why they thought it was safe and unsafe. They were testing kind of applied knowledge. Drew, in that context, they found the VR does not make a difference from a really, really good designed PowerPoint slide.
Drew: My takeaway from this is that they were trying so hard trying to be fair, and so hard to be careful about what was used in each format, that they did not use a lot of the possible potentials, compared to someone who is actually trying hard to make the VR work to the extent of using features in VR that aren’t available in other forms of training.
I don’t want to pretend that I know the answer here; whether VR can be of benefit or not. But, I think that by being so fair, they have not yet answered that question. It’s like watching the Phantom Menace and then watching the Phantom Menace in 3D goggles, and deciding that 3D goggles are no good because they did not make it into a better movie.
David: I think the question here would be if you had your current traditional safety training, we wouldn’t want you to form a conclusion from this paper that spending a lot of money on a VR environment isn’t going to be better than what you’re currently doing—we’re not saying that at all. There is a fair chance that it would be or could be better than what you are currently doing. What we are saying is if you invested the same amount into your PowerPoint as you would go in then invest into VR in terms of the content, then you wouldn’t necessarily have to take the step to going towards VR. I would not want to listen to run off and say, “We are not entertaining any ideas about VR because our existing PowerPoints are just fine.
Drew: The other thing that I think we should point out is the one thing that they didn’t measure was whether the PowerPoint itself actually worked. We only have this comparison of endpoints which is a fair way of doing it, given that these were students that supposedly did not know much beforehand. But it would have been good just to have that extra measure to check how much learning they get out of the PowerPoint. The answer from this study, one possible interpretation is that VR was as bad as PowerPoint. Another possible interpretation is that PowerPoint is as good as VR.
David: Let’s do some practical takeaways, I think both of us went and had a bit of a dig into the world of VR, just trying to make sure we were not missing something big and have one of our listeners comment something really obvious over the fence at us. Do you want to talk a little bit more about any practical takeaways that our listeners could take away?
Drew: The first big takeaway that we have to take, and this I think is fairly consistent with the board we are reading each time. At the moment, you can’t say there’s strong evidence that VR is going to significantly improve your training of general safety topics. If you are investing in VR, you are doing it as exploratory, innovative, testing out untried waters to see if you can get an improvement. You are not following in the footsteps of people who have done it and shown that it’s worked.
David: Yes, the caveat to that would be potentially if you are using your VR for simulated work environments, just to go back to what’s done with training pilots, flights stimulators, and creating real work scenarios. If you’re creating environments that provide people experience in different situations that’s directly relevant to their work tasks, it’s sort of a work training tool as opposed to safety concept or safety knowledge transfer tool, then that’s a slightly different question and we might have a different answer to that.
Drew: Yes, that would extend also to not just normal work situations but to emergency situations as well. There is a separate body of research on using VR to study evacuations and to train people in how to handle really stressful situations. We just haven’t looked at that literature for this episode. We are talking here about using this for your training people and things like identifying the hazards of the workplace, understanding what the risks are for those purposes. For that question, the answer is no benefit shown for VR, maybe with a “yet?” after it, and we are not answering the question about using it to train work tasks in emergency train situations.
David: I think that's probably a good step up point, Drew, to talk a little bit about broader literature. I had a bit of a look through, and there are definite applications of VR into the healthcare environment for handling, training of surgeons, and healthcare providers in emergency situations. There's quite a lot in disaster response, there's quite a bit in the fire and evacuation space, not only for simulating evacuations but in using the way that they would do evacuating in a VR environment to go back and inform building designs, evacuation plans and development, and things like that.
There's quite a lot in the VR research space as you can imagine in the last 20 years. We don't want anyone to walk away saying that David and Drew said that VR isn't useful.
Drew: No. I think there are two clear threads that you can look at to distinguish between those spaces. One of them is the use of VR to collect information. There's definitely a massive body of work, everything from medicine, to flight simulators, to emergency simulations. The idea is we create a very realistic situation, we look at how humans behave in that situation. It would be entirely unethical to actually push humans in that situation to study how they behave. We've no risk or we’ve very much reduced risk. We can study humans using VR, that's a clear application.
The second one is where we are using them for training. Again, it's putting humans into situations which are really hard to put them in. In aviation, the trouble is emergencies just don't happen often enough to put people into emergencies a lot. It would be unethical to cause emergencies just to train people. We give them those situations in a safer environment. That's the sort of thing where VR, not just has promise, but that promise has been realized in a lot of cases. It's the idea that just making training more exciting or making training more immersive as opposed to making it more specifically real. That's the one that hasn't been shown.
It's similar to literature, people can learn by playing games. The idea is to make it more real, make it more vivid, make it more exciting. People will learn more. People love learning using games, but they don’t necessarily learn anymore. People like being in VR, but they don't necessarily learn anymore.
David: Maybe that practical takeaway that we didn't say is be deliberate in what you measure as an evaluation of the effectiveness of the training. If it's a questionnaire that says, "Did you enjoy this training session?" By all means, if that's your measure of success, then VR is likely to get you potentially a better result. That's not necessarily the outcome of training that we're all after.
Drew: If that is your measure of success, then I'm providing people with good food and coffee at the training. It's much, much cheaper than giving them a VR headset.
David: And potentially keeping it short, letting them get on with their job as well.
Drew, what are some things we might like to know from our listeners?
Drew: I'd love to know how people are using VR. I've made a few jokes about it being your fun technology and a toy. The genuine thing is it's one of those solutions in search of problems to solve. I'm sure people are using it for interesting things, they're finding interesting problems to solve.
David: I'd love to know what sort of things people are using.
Drew: The other one that I'm interested in which we haven't covered on this episode, I'm not aware of a lot of research, so it's good to just know, is augmented reality. That's where you do things where you look through your mobile phone at the environment. It marks out the environment with certain features either through training or through automatic identification. I'd love to know what sort of things people are using in safety, what ideas they have, and how well it's working for them.
David: That sounds like an invitation for Cam Stevens to contact the show, Drew. He's now working for an AI technology company looking at safety solution designs. Cam, give us a low down.
Drew, at Christmas time, one of the other dads at my son's school just gave me a recommendation for a new VR headset that he got at Christmas time. He said that he lost about a week of his life exploring his new headsets. I think after this episode, I might have to go out and get myself one. Then, report back to our listeners in a future podcast about what I think of it.
Drew: David, I think we've established which one of us is more committed to spending time playing computer games. In the service of the podcast, if you get a hold of it, I'll be very happy to put in the work for you.
David: We'll do some kind of a controlled experiment between the two of us. Drew, today we asked the question is Virtual Reality safety training more effective? Do you want to have the go at the answer?
Drew: I think the answer is for generalist safety training, probably not. Unless what you're doing is providing simulations of real world scenarios, either work scenarios or emergency situations with the safety practices built in.
David: Great. That's it for this week. We hope you found the episode thought-provoking and ultimately useful in shaping the safety of work in your own organization. Join in the conversation on LinkedIn or send any comments, questions, or ideas of future episodes to us at feedback@safetyofwork.com.