Our discussion today centers around the Sarah Blakeslee one-page reference guideline for citing research sources (Blakeslee, Sarah (2004) "The CRAAP Test," LOEX Quarterly: Vol. 31: No. 3, Article 4.) The paper is not peer-reviewed, but it is a helpful and worthwhile reference to keep in mind when writing your papers. While Drew actually uses this guide with his students, research shows that even with the guidelines in front of them, many do not do the work and screen for these elements when using sources.
We will go through each letter of the amusing and memorable acronym and give you our thoughts on ways to make sure each point is addressed, and different methodologies to consider when verifying or assuring that each element has been satisfied before you cite the source.
Sarah Blakeslee writes (about her CRAAP guidelines): Sometimes a person needs an acronym that sticks. Take CRAAP for instance. CRAAP is an acronym that most students don’t expect a librarian to be using, let alone using to lead a class. Little do they know that librarians can be crude and/or rude, and do almost anything in order to penetrate their students’ deep memories and satisfy their instructional objectives. So what is CRAAP and how does it relate to libraries? Here begins a long story about a short acronym…
Discussion Points:
Quotes:
“The first thing I found out is there’s pretty good evidence that teaching students using the [CRAAP] guidelines doesn’t work.” - Dr. Drew
“It turns out that even with the [CRAAP] guidelines right in front of them, students make some pretty glaring mistakes when it comes to evaluating sources.” - Dr. Drew
“Until I was in my mid-twenties, I never swore at all.” - Dr. Drew
“When you’re talking about what someone else said [in your paper], go read what that person said, no matter how old it is.” - Dr. Drew
“The thing to look out for in qualitative research is, how much are the participants being led by the researchers.” - Dr. Drew
“So what I really want to know when I’m reading a qualitative study is not what the participant answered. I want to know what the question was in the first place.” - Dr. Drew
Resources:
The Safety of Work on LinkedIn
David: You're listening to the Safety of Work podcast, Episode 91. Today we're answering the question, how can we tell when safety research is C.R.A.A.P.? Let's get started.
Hey, everybody. My name is David Provan. I'm here with Drew Rae, and we're from the Safety Science Innovation Lab at Griffith University in Australia. Welcome to the Safety of Work podcast. In each episode, we ask an important question in relation to the safety of work or the work of safety and we examine the evidence surrounding it.
Today, we're going to be a little bit meta and talk about how safety practitioners, and anyone who listens to our podcast for that matter, can evaluate their safety science evidence for themselves. Drew, before we get started, we worked for 90 episodes to not have that E - explicit indicator on our podcast. Maybe let's not sound out our C.R.A.A.P. acronym and let our listeners kind of sound it out for themselves.
Drew: Yeah, okay David. I think that defeats the purpose as we get to it as to why the acronym exists in the first place. I feel a little bit uncomfortable with it myself, so I'm glad that you've slapped the restriction on.
David: I used to access podcasts through my work mobile phone and the organization used to always have security restrictions that you couldn't actually access any podcasts that had the E - explicit on it. For those listeners who are listening through work devices, it might make it impossible for them to access this episode.
Drew: For today, we're going to base our discussion around a set of guidelines, which from here on, I'm going to call the Sarah Blakeslee, the Blakeslee guidelines. This is based on a heap of tools that have existed throughout history, but this particular Blakeslee version has really caught on mainly because of the acronym.
Two disclaimers just upfront. First one, David, I think this is correct for both of us that we don't really strictly follow the guidelines ourselves. Hopefully, as we go through, listeners will see that we do pay attention to this sort of thing when we're following the podcast.
The second disclaimer is that even though I think the guidelines are a really good idea, when I went to research for the podcast, the first thing I found out is there's pretty good evidence that teaching students using the guidelines doesn't work, which is embarrassing me a little bit because I teach students using these guidelines. It turns out that even with the guidelines right in front of them, students make some pretty glaring mistakes when it comes to evaluating sources and that's just got worse once the Internet has existed.
As we go through, we'll try to point out some of those mistakes that people make as well and give you some tips for avoiding mistakes.
David: About the paper today, it's not a peer-reviewed published paper, but it's a single-page handout prepared by Sarah Blakeslee at the Miriam Library, hence why we’ll call this the Blakeslee acronym or criteria at California State University in 2004. It's been around a while.
Blakeslee produced this handout for a class teaching students how to evaluate sources. Like you said, Drew, this material is based on some quite long-standing criteria that libraries use. Her main contribution was to tweak the acronym, and I've got in your notes here Drew, to make it more memorable for students and more embarrassing for age professors to teach to their classes. Drew, how embarrassing is it for you to teach this to your students?
Drew: Until I was in my 20s, I never swore at all. I've had to teach myself as a social thing how to deliberately swear, and it's at the point where I'm a little bit more comfortable with it. I'm quoting someone else so I'm allowed to say it.
David: Ironically here, when we talk about research being C.R.A.A.P., it's actually a good thing when we go through these criteria, so it's not necessarily a bad thing. Drew, do you want to get us started? Let's go through. We've got five elements of research to discuss.
Drew: There are five elements and because it's just one sheet, basically, we can tell you everything that is on this sheet of paper as well as our own comments. We'll start off with Currency, which is the timeliness of the information. When was it first published or posted? Has it been revised or updated? Do you have a topic that requires things to be current or is it a topic that's okay for older sources? Just a basic sanity check, if it's got links, do those links still work or are they out of date?
David: When is new, I suppose, more recent research and information better?
Drew: As a general rule, we want any research to be fairly new. By new, what I usually use is the default on Google Scholar which is published in the last five years. Sometimes, if you can't find anything in the last five years, you might stretch it out to 10 years. Whether something's older than five years, you begin to become a bit suspicious.
David: Suspicious, not so much of the research at the time, but curious as to what's been published since.
Drew: Yeah. This really matters with things like literature reviews. If you're reading a different paper that's got a literature review in it or if it's a major analysis. Okay, this might have been stating of the art at the time it was published, but what's been written since? What's missing?
David: I think in terms of what's happening around the world at the moment, if you want to understand the latest immunology on COVID-19, you probably want a late 2021 paper, not a mid-2020 paper.
Drew: Exactly. The speed at which something is moving determines just how recent it has to be. Medical stuff, definitely you always want to be really recent because it's quite possible for one new study to basically discredit everything that's gone before. With something that's a bit more stable like economics or probably most of the safety science, one study isn't going to suddenly change the way you think about the world. It's not going to disprove everything that's gone before.
David: Yeah, if it's something like psychology, I guess a lot of the generally accepted principles in that space, some of those research papers are in the 50s, 60s, and 70s. I remember saying to people having done a psychology degree more than 20 years ago now, it will be a mistake to access anything that I learned during that degree.
Drew: Sometimes that old work just gets completely replaced or discredited. All of the recent work knows that and all of the older work just treats it as if it's absolute fact. It's really quite dangerous sometimes to use very, very old sources.
David: Would there be such a thing as something being too recent or too new?
Drew: There are a couple of times when being published really, really recently is a red flag. The first one is that if it's very new, it hasn't had a chance to attract critique. If you're not an academic yourself, if something is hot off the presses, that's the worst time to read it. You need to at least wait long enough to get a reaction from other people so that you know generally what to look out for. You let someone else do the hard work of evaluating it and deciding whether it's reputable.
David: I'm thinking now if you picked up that now-infamous research of the link between vaccines and autism when that was brand new, which was obviously redacted and there's a body of literature around that, but picking that up straight hot off the press with a very strong new claim, maybe something that you want to be cautious about.
Drew: Yeah, it's very tempting, particularly for newspapers or people whose job is to share information to want to be first with the news. Giving other people a chance to react sometimes is a good strategy. The other thing that really matters is when it's based on statistics. A lot of the ways in which things like accident and injury statistics are collected means that they don't really stabilize for a couple of years, and that's just to do with delays in how things are reported and synthesized.
We currently don't really know, for example, what the death figures are for 2020. Even though it seems like 2020 has long gone, those numbers are going to change for the next couple of years until they settle down. Generally, what happens is they end up being underreported. We see what we think are patterns and those patterns disappear as things smoothed out over a couple of years.
David: Yeah. Drew, are there circumstances when older research would then be better?
Drew: I was trying to think of good examples of this and it's not necessarily the exact age. It's really more how close a source is to the original. For historical information, you want to get as close to the source material as possible. Ideally, you want to actually pick up and read the source for yourself.
Your work often has multiple steps. If you read something today that is talking about safety practices in the 1920s, chances are what you're reading will be citing something which is citing something else, which is citing something else, which is adding something else, which is someone who actually saw the original. Every one of those steps has a chance to get corrupted. Ideally, what you want is someone who wrote in the 1920s about what they're doing in the 1920s because that person would know firsthand.
David: I think it was episode 17 or so when I interviewed Carsten Busch with the question, what did Heinrich really say? He staunchly made the point that most of what is being written now in relation to that is so far removed that it kind of misses the original point.
Drew: I think that's a thing that we've said on the podcast before as well is that when you're talking about what someone else said, go and read what that person said no matter how old it is. They might be out of date, but at least you won't be misrepresenting what the idea was.
David: Another podcast we've done on swiss cheese, if you want to read that, read Jim Reason's work in the '90s. Don't read commentary on it from 2021.
Drew: Basically, what this means in practice is if you're using a source, think about what you're using the source for and whether that means that you should be looking for something that is relatively recent or whether you're happy with something older. The older something is, the more important it is to read about the source itself. As in, look up the title of the author and see what's been said about that same source or topic recently. For papers, for example, look at who has cited that paper and what did they say when they cited it?
David: Yeah, perfect, Drew. That's currency, should we move on?
Drew: Okay, so the next topic is relevance. Relevance is about how well the information matches your needs. The most obvious one there that is oddly hard to drill into students is does the thing you're reading actually set out to answer the question that you want to be answered? It's really easy to accidentally pick up a source that is intended for a different purpose that doesn't really answer your question.
For example, the task that I give students is to find out whether playing violent video games makes teenagers more violent and you end up with all these things that aren't quite that. Does playing violent video games make the kids feel more aggressive, or does it make adults be more aggressive? Working out how close something is to actually answering your question.
Just reading directly from the original sheet just before we discuss, it says, who's the intended audience for the information? Is the information at an appropriate level? Not too elementary, not too advanced? Have you looked at a variety of sources before deciding to use this particular source? Would you feel comfortable citing this source? That's like an embarrassment test. If you're not going to admit that this is your source, should you really be using it?
David: Drew you mentioned that example that you gave your students, and I guess, that's a really big and important central question when you pick up a paper and think about this idea of relevancy. It's not just whether the source itself is okay like we talked about in the podcast. Is it reliable? Who are the authors? Is it the most suitable source for the question or the information that you're trying to draw out of the paper?
Drew: A good way to do that is just to think about where you would expect to find the best information, and rather than just doing a search on Google, go and actually look for that information. If it's the sort of thing that you'd expect to find on a government website, don't google it but go to a government website and do a search there. Is it true that most of our listeners are from Australia?
David: Not at all, Drew. About 50% are in Australia, 50% of the downloads are international.
Drew: For the 50% of Australian listeners, don't trust statistics anywhere unless you check the Australian Bureau of Statistics first. Because we have a really well-funded expert government statistics agency that can be trusted. Why not look at the people whose job it is to come up with good statistics before you trust a statistic published somewhere else. I can't really speak with authority to any other country. I'm sorry about whether your government statistics agency is trustworthy or not. I just know that ours is.
David: I think it's a good point, Drew. I think if you want to find something about a particular discipline, an example here might be machine learning. Go and look in a machine learning journal. Don't necessarily look for a machine learning paper within a safety science journal.
Drew: Listeners may have noticed David and I reference this occasionally. When we're talking about a particular topic, we'll say, we don't think that this topic is covered very well in the safety literature, so we've gone to an organizational science journal, we've gone to a psychology journal, and tried to find something there because we think that's where the reputable information is to be found.
David: Drew, what else do we need to think about when it comes to relevance?
Drew: The one other thing I would say is that this can be relative to a particular person. We're both academics, we're pretty comfortable looking up information in academic journals. We don't really have a lot of trouble reading academic articles. That doesn't mean that that's the best way for everyone to do it.
One of the ways you can select to see this is a trap. If you ever look up a technical topic on Wikipedia that you already know something about, you'll often find that the Wikipedia entry has good sources, but it was written by a 16-year-old from Iceland who lacked the technical knowledge to correctly read and interpret the source that they're using.
Just having a good source isn't very good if you don't understand it well enough. Sometimes, it's actually better to have something that's written in plain language that you trust rather than something that you don't fully understand enough to trust.
David: Yeah, Drew, you can speak a little bit for yourself on the comfort of reading academic papers. Anytime I see a statistics table inside a paper, I remember a number of times I've sent you some ideas for our episodes. You've come back and said, the statistics actually don't match the findings that the authors are claiming because of your knowledge and capability in the actual quantitative research methods, which my understanding is nowhere near yours of those types of things.
I think that's a good warning for people. If you read the author's claims in the discussion and can't understand the statistics for yourself, then you may not have something that's entirely relevant for you.
Drew: Yeah, that's going to be true for everyone just on different topics. There are a lot of medical papers, for example, where I literally just can't read the abstract or the method of the paper. It's referring to so many specialized techniques that I just don't understand at all. I have to rely on other people to summarize it and read the more plain language stuff for myself.
David: Speaking of relying on other people and I guess the authors of the publication or the source themselves, the A in the middle of the acronym stands for authority. Do you want to tell us a little bit about authority in relation to the usefulness of the source?
Drew: Sure. I think this one is really important and everyone does this in different ways with different levels of awareness that we're doing it. I would guess if I was to say that most of our knowledge, we don't really get in particularly reliable or rigorous ways for everyone. For everyone, I think our most common way of getting information is someone else tells it to us and it's just whether we happen to be good or bad at picking the right people to listen to.
Let's go for podcasts like Joe Rogan. If you look at what he does on his podcasts, he interviews really important famous people. He gets real experts and he gets nonexperts. Why is it that he gets off stuff so badly wrong? It's really just he doesn't know which ones to listen to. He doesn't know who's reliable and who's not reliable, so he uses information from an unreliable person to override a reliable person.
David: Nice one, Drew. Safety of Work podcast takes on the world's most popular podcast. Excellent.
Drew: Since we got to be giving shout-outs to podcasts that have got more listeners than ours in the hope of stealing their listeners. There's a legal podcast I listened to, Opening Arguments. Far better to production quality than we will ever have, David. Far far more reliable putting out episodes.
One reason that I listened to it is because I trust them as an authority. They tell me about stuff that I would not access firsthand, I would not read firsthand, I would not have the time or the expertise to go to, but I trust that they are reliably giving me the information.
David: On the same, we say Tim Harford's Cautionary Tales, just how well researched and credible as a journalist he is.
Drew: Now we've got a good link of people we can cross-list in this episode and see whose listeners we can drag in. I guess the point here is that it's not a logical fallacy to appeal to the authority of information because we can't evaluate everything firsthand ourselves.
What's a fallacy is if you use the authority to override problems with it and to overlook real problems with the data just because it's from someone with authority. How do you tell whether a source has authority? The big thing here, and this is a mistake that we particularly see undergraduate and graduate students making all the time, is you can never evaluate the authority of a source based on the source itself.
You see someone pick it up and they say, oh, this is by a person with a Ph.D., this is by someone with expertise in this topic, or this is by someone from Harvard University and they're just reading what the source is telling you. If the source isn't trustworthy, then you shouldn't trust the source to tell you just how much of an expert it is.
It's one of the simplest things that you can do to google the title of something that you're reading, to google the name of the author, and just see what crops up. If you're reading a paper about statistical analysis and you google the author, and the first 20 hits other publications that this person has written about statistical analysis, it's a pretty good sign that maybe they're an expert and can be trustworthy.
You google their name, the first 20 hits are either podcasts or videos they've produced about themselves or other people complaining about how shoddy their work is, that's probably a bad sign. Most of us just never even go to that point of googling other people's names to see what else they've done.
David: I guess all sources and academics are at various points in their career. Why does authority matter so much when we would normally be talking about taking the quality of the source based on the quality of the source? We do it in the podcast a lot. We talk about who the authors are in every episode and we'll mention their institutions, other work they've done, or their position. It's clearly important, but how does it help us with the quality of the information itself?
Drew: Okay, that's a fair point. That's an important question. Really all authority matters for everything that you can't independently check yourself. A good example is someone describing to you generally what is known about the state of the field they're writing in. You can't check that yourself if it's not your field. You could check each of the individual papers they cited, but you wouldn't know if those are the important papers or if they're cherry-picking data. All you know is what they put in front of you.
Whereas if someone is an expert in that field, then you can at least trust that they know what the field looks like, they know what's important, they know what's relevant. They might still deliberately cherry-pick, but at least they're not going to accidentally do it. That's where you have to rely on can they be trustworthy in other ways.
The same thing goes with statistical methods. I know you think that relative to you I know more about statistics, but I am not a statistician. I know some basic checks and the reason why I get annoyed at papers is that they fail even the basic checks that I know about. Beyond that point, I don't even know enough to analyze whether the statistics have been done correctly. What I need to do is trust the authority of that person. Are they an expert in this type of analysis?
Then you get to other things that are subtly hidden like if someone is doing ethnographic work. They're going out into the field, collecting data, and reporting it back. You can't see exactly what questions they asked, you can't see exactly what people told you, you can't see exactly how they've analyzed it, you're just trusting that what they're giving you is a fair representation. The fact that someone's got a good track record in this area helps fill in all of those gaps for you.
That's why on the podcast, we will often mention the authors of the papers and what else those authors have done. Often the first author of the paper might not have the expertise. They might be a field researcher doing their first study and that's fine. Everyone's got to write their first paper and their first paper might be brilliant. What we love to see is that the third author supervising them is someone who's an expert in the methods. He's making sure and basically vouching for the work done by the first person.
David: Yeah, Drew. The second A in our C.R.A.A.P. acronym is accuracy. Beyond the authority of the author of the source themselves, now we're starting to talk about the reliability, the truthfulness, and the correctness of the content in the source. Do you want to talk about what we look for in terms of accuracy?
Drew: This is the point where I think it genuinely gets hard and it gets hard not just in finding out the accuracy, but in distinguishing between accuracy and authority. I think this is one where Blakeslee actually gets it wrong in her detailed descriptions and see if you can spot when I list out what she's got under accuracy.
She says where does the information come from? Is the information supported by evidence? Has the information been reviewed or refereed? Can you verify any of the information in another source or from personal knowledge? Does the language or tone seem unbiased and free of emotion? Are there spelling, grammar, or typographical errors?
You look through that list, a lot of those are actually using authority as a marker of accuracy. They're not actually checking accuracy. They're saying based on where the source is from, can you trust it to be accurate, which is authority. Accuracy is hard.
David: Drew, I think that's a really good point, I think accuracy, we've got to maybe have some proxies for accuracy particularly when we don't have the personal knowledge to judge or even the detailed understanding of the research method. Maybe this authority becomes a bit of a substitute for accuracy, but are there things that we can specifically look at in the source to get a sense of accuracy?
Drew: I think there are a few basic things that anyone can do. The first thing that you can do is you can read the introduction into the method of an academic paper and find out what the authors actually did. You don't need to understand the details to know that in this paper they went out and interviewed a bunch of people, or in this paper they conducted a survey.
What you find very, very often is you read the introduction, you go looking for the method section, and you realize that the authors didn't actually do anything. All they’ve done is read a bunch of stuff, think about it, and write. Now, that is fine. There's nothing wrong with writing a paper like that, but that tells you immediately that the source you're reading contains no original facts. The authors haven't collected data using any method.
That means that you shouldn't be citing or relying on that source as a source of facts. You should be looking at it as a source of ideas, of analysis, and you should be going to other sources where people have actually done the work for the fact. That's the first basic thing that you can do.
The next thing you can do is if there is a method, then you can think about what the method can and cannot do. Different research methods, even without understanding the details, are only capable of doing certain things.
David, you've probably got your own examples here. For me, one of the big ones is that if you've conducted a survey that can only ever tell you what people are thinking or reporting about their own behavior. A survey can never tell you reliable facts about the world.
When someone conducts a survey and they say that 56% of people drive on the left-hand side of the road, you can't trust that at all. A survey can't measure how many people drive on one side of the road or the other. The most it can do is tell you that 56% of people tell you that they're driving on the left-hand side of the road. That becomes very obvious when it comes to 90% of people claiming that they never break the speed limit. We know that surveys can't actually tell us when they do or don't.
David: Yeah, Drew. I think an example, when I was doing a lot of work looking at influence, particularly around safety professionals in organizations and how they influence and other roles in organizations as well, you'd see a lot of survey questions where someone will be surveyed and asked how they influence. The paper would go on to claim that this is how people influence. It’s not at all. It's how they think they approach their role. That provides no facts about influence because they're not even talking to or observing the people who are meant to be being influenced themselves.
Drew: Yeah. I think we’ve got time here to just briefly talk through a few of the different types of research and what they are good for, not good for, and what to watch out for.
David: Perfect, Drew. I will defer to your expertise for this.
Drew: Broadly speaking, when we’re talking about research, we usually break it up into quantitative and qualitative. Fundamentally, there is no difference between quantitative and qualitative, except that quantitative uses numbers and qualitative generally doesn't. Anything else you hear about relative quality of quantitative versus qualitative or which is more scientific, all of those are just value judgments that we put over the top.
The difference is that quantitative needs to break the world up into things that we can count, whereas qualitative describes things using words. In safety, the main sorts of quantitative things that we do are we do lots of surveys where we’re asking people their opinions about things or asking them about their own behavior or the behavior of other people. That turns into a number because of how we ask the questions. We say, rate from a scale of one to five how you feel about something. Is this something that you do all the time, some of the time, none of the time, and we turn that into a number.
Second to have a number we use a lot of is injury statistics, how many people get hurt in a given period of time? Occasionally, but not nearly as often as I personally would like, we do experiments or we evaluate interventions. We've got some sort of endpoint of that experiment, which is expressed in terms of a number. David, does that cover the main ones? Any other types of quantitative research you see a lot of?
David: No. I think they're the ones, particularly surveys and injury statistics are the ones that show up over and over again in safety science.
Drew: For each of these, there are some really basic quality things that you can look for. If the survey is the one that I always look for and the one that I always complain about, including on the podcast, is just how closely the claims that they make match what they're actually measuring. Because people in safety love to measure someone's opinion and then make a claim about their behavior or even about an entire organization. That gap between I asked someone a question or a survey versus is the organization safe or not is a very, very big gap.
David: Those questions typical of culture surveys around, say, do I take risks when production schedules are tight? Fifty percent of people say yes. You can't then make the claim that 50% of people take risks when production schedules are tight. You can only ever claim that 50% of people report taking risks when production schedules are tight.
Drew: Yes. Because the reporting and the doing can themselves be influenced by different things, you can't even assume that one is just roughly good enough to tell you about the other one. The second main type of thing is injury statistics. The thing to look for here is just how vulnerable the statistics to variations in how things are reported, recorded, and collected. You've always got to look for these second explanations that actually it's not a real change in safety, it's a changing reporting. How well the studies manage that possibility tells a lot about how good an injury study is.
David: Yeah. I think particularly in something like this, I don’t need to make this a long conversation because we talked about injury statistics a number of times on the podcast, but even this idea of there are so many compounding variables in something like that, even trying to create some level of analysis between something that the researchers are looking at and something as broadly impacted as injury statistics is some of it is unlikely to be very accurate.
Drew: That's a good way of putting it. The third category of things where we've got something that's an experiment or an intervention evaluation. What we look for here is really just the markers as to whether this was conducted as a deliberate experiment and whether it was a well-designed experiment. You look for simple things like did they have a control group? Did they have one group today intervened with and another group that they didn't intervene with? Because without that control, it's never able to really claim what caused any change in the outcome.
You look for things like did they measure both groups before and after? Because if the two groups weren’t the same to start with, then it doesn't really tell you much that they're different to finish with. That difference could just be because they started off different. We look at things like the size of the experiment. Do they just have enough participants to reliably get the answer that they're looking for? Other explanations are the things that could have been causing any variation that we think we see.
David: I think if you do want to be reading a lot about experiments—and even understanding in detail the research methods, I think you gave me about six or seven examples or texts around research design and qualitative and quantitative research methods. Depending on how far our listeners want to go, it may actually be worth your while picking up a book or two on research methods. I know there's Creswell's Research Design and a few other books on the bookshelf. Even just leafing through a book like that to understand what good research design looks like would be a very useful thing to do.
Drew: It's funny, sometimes you can be reading a paper and they tell you what they did, and you're just nodding along thinking, that sounds fine. But then if you've ever had to do it yourself or you've ever read a book on how it's meant to be done, suddenly, you’ve got all these questions. That does sound fine but why did you do it like this when everyone else does it like that? You notice that you just happen to mention this little thing that we know that other people do.
David: Little things too about where samples are drawn from. Most say 400 people did this interview, but how they access the sample when you learn that they went to one professional association and they only spoke to safety professionals who are affiliated with the professional association. What does that tell you about the demographics and the representativeness of that sample? It's very easy to nod along and go, yeah, they surveyed 400 people and this and that, but you have to get to the next level to know whether or not the information or the source is as accurate as you want it to be.
Drew: There is a big difference between saying 100% of people really, really cared about safety, and 100% of people who clicked on a link which said, here is a survey about caring about safety. It sort of makes a bit of difference.
David: In qualitative, do you want to talk a little bit about qualitative research then? No numbers.
Drew: When we say no numbers, we're talking about things like ethnography, where you go to a place, you observe people, and you look at what they're doing. Surveys, but surveys without numbers. Surveys with more open questions where people can just write in their own responses. Probably the most common one is interviews.
Now, here's an anti-thing to look for is that the number of participants is always much lower in qualitative research. Whereas a moderate-sized survey, a quantitative survey might have 400 participants. A moderately sized interview study might have 10 participants. Just because there is so much more data from each participant in a qualitative study. There's so much more work to collect the data, there's so much more work to analyze it.
Don't go by just the raw numbers here and say that one study is better than another because it had 10 times as many participants. The thing to really look out for in qualitative research is how much are the participants being led by the researchers. What I really want to know when I'm reading a qualitative study is not what the participant answered. I want to know what the question was in the first place and how much that question drove them towards that particular answer.
I also want to know what the participants actually said in terms of specific examples and direct quotes rather than just the raw summaries given by the researchers. Whereas good quantitative research gives you really precise descriptions of the method, how the method was conducted, how it was controlled, and the variables were. A really good qualitative research makes you feel like you were there, like you're actually listening to the participants in their own voice telling you this. You can believe that the researcher is giving you a fair summary of what that data was.
David: When we published the professional identity paper during my Ph.D., and that was interview-based, there were four questions that I asked—four very broad open questions. That was enough to have interviewees talking for between 60 and 90 minutes off those four questions. Those four questions were listed in the method section of the paper.
If your source says we did interviews and doesn't actually list and tell you what questions they asked and how they asked those questions, then it's probably a little bit of a red flag to not know exactly how those interviews were carried out and how much the participants were led by the researchers during the data gathering.
Drew: Yeah. A double red flag is when you see the questions and the headings in the results directly match the questions. We asked people about these four topics and their answers revealed that they really cared about these exact same four topics, so you know the questions have defined what was found rather than the answers.
David: Great overview. The final in the acronym, the P, do you want to start us talking about purpose and the reason that the source exists in the first place.
Drew: The main reason why the purpose is here may lead to start distinguishing between academic and non-academic sources. Most academic work doesn't have a clear purpose other than to describe the academic work. You can try to read into it that maybe the authors have got an agenda, maybe the research was paid by someone with an agenda. Maybe they did this research to try to prove something. But reading those sorts of things is very hard. Often really what we want to do is just to understand why this was written in the first place? Was it written as an academic article?
Was it written really as newspaper post stories, an opinion piece, or is it a persuasive piece? It's not that something is inherently unreliable just because it's not objective, but we need to interpret it based on those feelings.
David: Yeah. I think even if we take, again, recent examples of the amount of information there is around the world for the topic of COVID, the thing is where are you going for your information. Something like a peer-reviewed journal paper, the purpose of publishing research around COVID in a peer-reviewed journal is to inform and teach.
Whereas reading about COVID from journalistic sources or even from government associations, the World Health Organization has a particular agenda, every government around the world has a particular agenda. Depending on what it is that you're trying to get out of that source, there are very much places to go and not go when it comes to this part of the quality of the information that you're getting.
Drew: Once you’ve found the source, I think one of the big mistakes that people make is using sources for information which is tangential to why the source was produced. The really common one that you see, even academics make this mistake all the time, is when you're writing a research paper, you usually start off by giving a couple of paragraphs of background or context just to explain to other people why you’ve written the paper.
One of the really sloppy things that people do is they have a fact and they're looking for a source that supports that fact. They pick up a paper and within the first couple of paragraphs of that paper, it says the same fact. They think, okay, this paper supports what I'm doing, I can cite it.
When people are writing those introductions, they're doing that same thing. They're not really concerned about the facts. This isn’t their key paper. They're just giving you a background. All they’ve done is just found sources that say the same thing. With these chains of pseudo-facts that go back through 20 generations of papers all citing each other and none of those papers was ever designed to find out the truth of the fact.
David: I think I've seen an example of these in, actually again, I’ll mention Carsten Busch earlier with that. If you read Carsten’s actual thesis on Heinrich and how Heinrich is represented in the safety science literature, he's got pages and pages of tables in there of authors who have cited some of these original Heinrich texts. He's actually got in the table how they were cited, in support of what particular idea, and then what the original source said. Actually, the accuracy levels, and even for very credible academics, the accuracy of the citation and the representation of the ideas were quite a long way off being as reliable as you'd like it to be.
I think that's because these chains, people find a citation in another paper and so they cite the original citation through the interpretation that the author has made.
Drew: Let me just give you one particular example of the statement, which has been a bugbear throughout my career. I’ll tell you the exact paper. It’s a paper published in 1965 at a system safety conference. The paper is called, Advances in Fault Tree Analysis. If you as a listener just want to hop online and search this paper, you'll find how often this paper has been cited. It is cited hundreds of times by people who obviously were never at that original 1965 conference.
I did my thesis on fault trees, and in trying to write that thesis, I was trying to find the original paper because all of these other people obviously thought it was important because they all cited it. As we talk here, David is looking up this paper, and just remembers that I did my Ph.D. at a time when the internet didn't really exist as a good source of academic papers. I couldn't find this paper. I did work out that there were probably four copies in existence. All within university archives that I didn't have access to.
We’ve got two explanations here. Either these hundreds of other authors had all managed to dig up one of these copies in one of these four university archives that I couldn't get access to, or they were all lying about having read this paper that they were citing. Eventually, many, many years later, a PDF appeared on the internet and I managed to get a copy. I'm pretty sure they were all just lying because it didn't really say what they said it said.
David: There you go. Is there anything else you want to say about purpose or any other aspects of this C.R.A.A.P. acronym for the quality of research or the quality of a source?
Drew: The one final thing I want to say about purpose is just this type of citing things just because someone else had said them is particularly how myths about the effectiveness of things spread. You very often get these claims that things work or don't work, are well known to be effective or proven to be effective, or a particular technique is good for a particular purpose. Those are the types of claims they get reported like this.
If you want to make a claim that safety culture interventions are good at reducing the rate of accidents, then you have to find a paper that actually studies the rate of accidents, not a paper that just happens to say that same stuff that you want to say. So often these kinds of effectiveness have no original source. They’re just someone who originally thought it was true or didn't care whether it was true or not and just set it as background to their right work.
David: Let's finish the episode the way that we always do with some practical takeaways. You've got a few here. Do you want to get us started? What can our listeners take away from what we’ve shared about the quality of source material?
Drew: The first one—and if this is the only thing that listeners take away from the episode, I would love it because it doesn't just apply to safety, it doesn't just apply to research, it applies to everything. You can’t fully evaluate a source just by looking at the source. The only way just to be a smart consumer of online information is to get into the habit of googling for the source, not googling for the topic that led you to this office in the first place.
When you think you've got a good source, find out some information about that source. Find out what other people have said about that same source. When you listen to a particularly good podcast episode, don't trust the stuff we said on the podcast episode, google what other people say about us and see whether they think we're reliable or not. Whether the internet is filled with criticisms about just how much [...] this episode about C.R.A.A.P. is filled with.
David: I mean we always give listeners the title of the article and all the articles that we're looking at in each episode, again, pick up those titles, put them into Google yourselves, go to the source, go to what people say as well, and see how well that aligns or doesn't align with what we've had to say about it.
Drew: The second one is a little bit related, which is just be careful of secondhand representation of sources. It doesn't mean you're not allowed to use secondhand sources. Just be aware that that's what you're doing. Just pay attention to whether what you're reading is a research article, is it a press release by the university about the research article, or is it a newspaper report based on a press release about the original article? Don't confuse one for the other. Check which one you are reading.
David: I think we've talked a lot about literature reviews for example on the podcast and how useful they are to get an overview of a particular domain, field, or even as we've done an aspect of safety science with something you always do. Your literature review should do this with every paper where Drew will come back to me and go, I thought this was going to be an easy one but I got tied up chasing down all of the citations in the introduction just to try to understand the background.
I guess that's another point here with the secondhand representation. If you pick up a literature review, if there are three or four things in that that you particularly want to take out of it, go to the three or four key papers that are cited, and just make sure that the literature to review has represented the original source consistently and accurately.
Drew: You could just use David’s strategy, which is to find a really pedantic friend. When you think you got a good source, send them the paper and just say, what do you think about this? Have someone else do the rabbit hole chasing for you.
David: Yes, I do play on your curiosity in some of these areas, Drew.
Drew: The third takeaway is just be careful of broad-brush categories for source evaluation. There are things that we say are good clues for good research. Things like is it peer-reviewed or is it published by an important university. Those things are clues, but they're never enough to judge the reliability of a particular source. There is peer-reviewed junk published in major journals by authors at prestigious universities.
We're talking here about the worst kind of junk, claiming that telepathy exists and is published in the biggest journals of all time like Nature and Science by authors at universities like Harvard University. Don't use [...] categories. At the other end, there is stuff that is reliable that are podcasts or comments on Reddit or on LinkedIn. We use these things as clues, but they are not definitive.
David: Great, Drew. Final takeaway.
Drew: The final takeaway is try to get into the habit of considering at least the broad method that the source uses to reach its conclusions. You don’t have to understand the details of the statistics. You don't even have to be able to pronounce the word statistics, just understand the paper is a statistical analysis and have some idea as to whether that method can match the type of needs you have for the information. Understand that the paper is doing a bunch of interviews and what you can and cannot from interviews.
David: Yeah. If you want to get some insights into some safety leadership and you want to get some ideas about what senior leaders in certain organizations think about certain aspects of safety leadership, then you might find a paper that’s got interviews or a survey of a senior leadership population in a similar industry to you. That might be incredibly useful because the broad method that the source is using is likely to give you the insights that you want based on what you're trying to take out of that source.
Drew, the question that we asked this week was how can you tell when safety research is C.R.A.A.P.? Your thoughts.
Drew: I think the answer is while you can’t use them as an automatic checklist, the Blakeslee guidelines that we covered here provide some useful prompts to the questions you should be asking and the things you’re going to be looking for.
David: I'm hoping that we can actually get this one-pager and link it in the comments when we post this episode.
Drew: It is very widely used, very widely available. If we can just possibly even take a screenshot of it and prop it into the episode.
David: Perfect. That's it for this week. We hope you found this episode thought-provoking and ultimately useful in shaping the safety of work in your own organization. Send any comments, questions, ideas for future episodes to feedback@safetyofwork.com.