Today we jump head-first into a research paper that we have both found to be really interesting and thought provoking.
It's Modelling the Micro-Foundations of the Audit Society: Organizations and the Logic of the Audit Trail by Michael Power. This paper gets us thinking about why organizations do audits in the first place seeing as it has been proven to often decrease the efficiency of the actual process being audited. We discuss the negatives as well as the positives of audits - which both help explain why audits continue to be such a big part of safety management in organizations.
Topics:
Quotes:
“We see that even though audits are supposed to increase efficiency, that in fact, they decrease efficiency through increased bureaucracy. - Drew Rae
“The audit process needs to aggregate multiple pieces of data, and then it has to produce a performance account, so the audit actually needs to deliver a result.” - David Provan
“We become less reflexive about what’s going on in terms of this value subversion - so we stop worrying about are we genuinely creating a safety culture in our business and we worry more about what’s the rating coming out of these audits in terms of the safety culture.” - Drew Rae
“Audits themselves are not improving underlying performance.” - David Provan
Resources:
Griffith University Safety Science Innovation Lab
Research paper: Modelling the Microfoundations of the Audit Society
David: You’re listening to the Safety of Work podcast episode 82. Today, we’re asking the question, why do we audit so much? Let’s get started.
Hi, everybody. My name’s David Provan. I’m here with Drew Rae, and we’re from the Safety Science Innovation Lab at Griffith University in Australia. Welcome to the Safety of Work podcast. In each episode, we ask an important question in relation to the safety of work or the work of safety, and we examine the evidence surrounding it.
Drew, today’s question was inspired by an email from a listener about safety culture certification. We know that there’s been a safety culture auditing regime of such, particularly in the nuclear industry for a long time. We joked on one of our earlier podcasts about the potential to see the rise of Safety II or Safety Differently auditing and certification. Do you think you could maybe say a little bit about the issues that were raised in this email and straight off the bat some of your own thoughts about this?
Drew: Sure, David. Whenever I see things like that, I just start out immediately ultra frustrated. I went through that whole cycle with this email where I don’t think the person sending the email was particularly sympathetic to the idea of doing the audits, but they were sort of asking questions about how to of it and what’s the effect of it, and understanding of thinking.
Now, this whole thing is nonsense. If you understand anything about safety culture, regardless of whether you agree or disagree with the theory, you shouldn’t be auditing it. It just literally makes no sense. I have to temper those ideas of frustration because obviously it does make sense. It makes sense somehow to someone, and that’s really fascinating.
How does it get to the point where it makes sense to the person who’s asking you to do an audit that makes sense to the person who does the audit, it makes sense the people who put up with the audits even though it doesn’t make sense according to any theory about how safety culture is meant to work.
I think that the problem of things that people do that seem to be just crazy is one of the most interesting things in sociology. Just understanding how people do to themselves things that they complain about, they gripe about it, but they’re the ones responsible for taking on the work and then doing the work.
There are two massive authors in this space. There’s James Scott who wrote the book, Seeing Like a State, and Michael Power who (I think) is probably most famous for his book, The Audit Society. Today, we’re going to look at some work by Michael Power in this space.
David: Drew, I think at one level it would be easy for our listeners and for me, at least, while audit seems to make rational sense, like this process of checking that things are working, that things are accurate, that things are reliable and as we expect them to be, it just seems like a rational thing today.
Even in this question, if we want to have a safety culture, surely it makes sense to have some sort of auditing and certification process to check if that’s what we’ve got. But as we’ll learn today, auditing itself, particularly anything involving groups of people, sometimes things that could be rational become very irrational very, very quickly.
Drew, I think we should just go straight to the paper. This is an amazing paper and you can actually get it just with the Google Scholar search. I will actually drop in the comments of this episode the link to that because both Drew and I, when we’re getting ready to record this, we wished we had left ourselves a few more days to read through this paper. It’s actually a really, really good piece of work. Drew, do you want to introduce the paper?
Drew: Sure. The author of the paper is called Michael Power. Anytime you look at Michael Power, you’ll see his work about The Audit Society. He’s been writing about auditing and why we audit for a good 20 years now. In fact, until David found it I wasn’t aware of this most recent paper. It’s in the journal called the Academy of Management Review, which is a very creditable journal, and the title is Modelling the Micro-Foundations of the Audit Society: Organizations and the Logic of the Audit Trail. A bit of a mouthful in the title. I think The Audit Society was a little bit catchier.
The paper really does what it says in the title, which is it’s a paper about theoretically modeling the way the processes behind a fairly common idea works. This is not an ethnography. There’s no data, there’s no interviews or observations or case studies. It’s just based on both literature that already describes some of these things happening and it’s just common understanding.
Pretty much everyone who reads the paper or reviews the paper or thinks about the paper will be working within an organization, so will have experienced some of the things that Power is trying to explain through the model that he’s producing. We hope as you go through this that you’ll recognize some of these processes that he’s talking about as things that you see happening, or at least will believe underlying some of the things that you see happening in your own organization.
David: Drew, for example, like the introduction this paper says sort of, and I quote, “Organizations now provide reports on their performance regarding diversity, sustainability, quality, security, data quality, customer satisfaction, employee engagement, and many other values.” Now, there’s no citation for this claim and no data, but it’s something that we all know to be true about the world.
Obviously, Power’s work goes back 20–25 years on this topic and perhaps longer, but it was very similar at a very similar time when we’re writing about Safety Work versus The Safety of Work. I think the style of this paper is a little bit similar to the style of paper that we wrote when we wrote that paper.
Drew, when we start trying to sort of what Power’s doing with all of this work, his make sense of the world, in that title of the paper, what he’s saying is what is the organization logic behind having all this auditing activity? There are definitely some things that need to be explained by whatever Michael Power says in this paper. Do you want to talk through what needs to be explained? What are we trying to explain?
Drew: The starting point which he takes to be quite obvious is just that lots of auditing happens. Not just growing in the number of them that we do but the scope of the types of things that we are conducting audits for seems to be growing. Things that previously we didn’t audit, we now audit. By audit—he doesn’t explain this in his paper but he says elsewhere—doesn’t mean strictly just something that is done by an auditor. There are other things that form the same basic process that we’ll go into a bit later, that you can clearly see the pattern happens in other organizational systems other than audits.
This growth in audits happens despite the fact that there are lots of very well-documented problems with audits, so he goes through basically five big things, and each one of these expands out into an entire body of literature. They’re not controversial claims.
The first one is what he calls a goal in detention displacement from core work to peripheral work. People focus on doing the audits and the administration instead of during the operational work.
David: The next one is about how this elevation of process over outcome. We audit the process as opposed to auditing and outcome. This reminds me directly of this Safety Work versus The Safety of Work. We can see, feel, and check the safety work, but we can’t necessarily audit the safety of the work, the outcome.
Drew: We see that even though audits are supposed to increase efficiency, that in fact they decrease efficiency through increased bureaucracy. Audits clearly require extra work, which is clearly overhead work. It’s not work that is fundamentally achieving the goals of the organization. Sometimes that is a huge expense that we take on.
David, I know you’ve made comments before about your previous roles about self-insurance and how the economics of that doesn’t stack up once you start factoring in the cost of doing the audits necessary to self-insurance.
David: Absolutely. The compliance effort is significant.
The fourth is this declining trust in professional judgment. We put faith in the auditing itself and we take that trust in that process more so than our trust in individual professional judgment.
Drew: Perhaps related to that is for the professionals themselves, a crisis of professional purpose. What are they doing? If they’re not being trusted, if they’re not getting to apply their judgment, if they’re not getting to achieve values that they achieve, they find themselves just managing systems and operating systems instead of doing the core functions that they care about.
David: Drew, I think those five things that we just mentioned there, people will see themselves in that. I mean, I saw myself in those things about process over outcome bureaucracy, the declining trust in professional advice, yet this seemingly dysfunctional practices of these things and particularly the contribution of auditing to creating this environment, all these pressure in organizations that are creating these things we’ve mentioned, the auditing persists and, in many cases, is being amplified.
Taking back to the question here is should we take it even further in audit culture all these other things that we mentioned that organizations report on? How does this paradox work if we know that it’s creating these things, yet we double down on the path?
Drew: I imagine our listeners have already got some ideas themselves about why we still audit. In the literature, it basically boils down to two overall theories. The first one is that audits play a role in the sort of like neoliberal governance framework. When we talk about values such as control, accountability, good governance, and due diligence, then audits play a central role in that. This is sort of like top-down theory that says we audit because other people make us audit.
Then there’s a bottom-up theory which says we adopt these practices because of their legitimizing and symbolic power, because they embody myths that we tell us about ourselves and about the way we operate as organizations.
Power just cites the existing literature and says we’ve got these two explanations. But they can’t possibly be enough because they still don’t explain the why. Why do people make us do it? Why do we keep in place the symbolic and legitimizing ideas if they’re genuinely ineffective? Even symbols and values that are powerful, still very quickly lose that power normally if that’s an unhelpful sense making. If it leads us into unproductive things, then we tend to give up our symbols and false beliefs. But that’s not happening with audits.
He says we need a model that explains why something can have so many negative effects and be so ineffective compared to what it says it does. Yet we don’t just keep it, but we tell ourselves myths, stories, and processes that help us keep it and help it expand. What’s the underlying engine that is driving that creation of a need for more audits.
David: What I liked about his framing for his model, he said that lots of these ideas tell us are broadly at a macro level why things might make sense, but they don’t explain why a seemingly autonomous organization would continue to do this to itself. That’s where he was saying that these theories let us down when we look at individual organizations, which is why I talked in the title of this paper about micro foundations. What’s the individual or the single organizational logic behind keeping this?
His explanation is to describe a model of audits that, in your words, sound a lot like an organizational virus. I love the way you like to characterize some of these models. I’ll just quote here: “By repeated enactment, this Logic of the Audit Trail is strongly performative of the conditions of its persistence and amplification by forming the disposition of organizational actors to reproduce, refine, and expand it in new settings.” It’s like doing audits creates the conditions in the organization that make people in the organization want to do more audits. Quite seemingly perverse when we get to the end of this episode.
If we go back to when an audit started, an audit started with producing a set of company accounts. I think I’m more informed of auditing so it comes out of accounting. Drew, do you want to just take us back and start us with maybe some of those origins and what we can understand by understanding those origins?
Drew: Okay. Actually, I’m not an accountant. I don’t know how accountants think, but this is the way sociologists talk about accounting. It’s the idea of creating representations of performance that you can then literally count. We take some sort of underlying thing that we value, we put on top of that some sort of representation that we can measure, and then we measure the thing that we can represent.
Power says it’s been well understood in sociology of accounting for a long time, that those representations very quickly take on a life of their own. In his words, “It constructs the reality or facticity of performance.” They go from being representations to becoming facts, becoming real things.
If we think about this in terms of safety, something like lost time injuries aren’t just a representation of the safety that we care about. They start like that, but then they become their own thing that needs to be managed. We still have to manage safety, but we also need to manage LTIs. It can get to the point where we take for granted that it’s worthwhile to reduce LTIs even if that has no impact on safety, just because LTIs are the reality now, that we need to manage, that we get judged by, that feed into other systems of abstractions.
David: Yeah, maybe LTI. I’m just thinking when you said, Drew, I think LTIs may be like tax. There’s a law that says if your sole purpose for making a decision in your organization is to avoid tax, then it’s an illegal decision and maybe we should do the same for LTIs.
Drew: It does sound almost corrupt, but that’s literally what happens with everything else we do in accounting. Instead of talking about the value of the organization, we talk about the share price. Things like profits and losses are paper representations of what’s actually coming in and out of the organization. Things like head count are abstractions of the people that are there and the actual cost of the people that are there. But they all have a management reality.
You can look at and manage an organization without ever touching any of the stuff that happens underneath. You can just do it entirely on the balance books because they have their own reality. The reason for that, the thing that gives it its power is that there are economic theories that tell us that that way of viewing an organization makes sense. Those aren’t just neutral representations. They’re representations that bring in lots and lots of theory about how economies and economics work. The same thing happens with any sort of metric.
David, I’ve built into my blue notes here a little bit of a rant about academia, if you don’t mind if I go down that rabbit hole for a moment as illustration.
David: Absolutely, Drew. Use the podcast for your own efficacy.
Drew: Fantastic. One of the things that we care about in academia are publication metrics. How many papers we publish, how much those papers get cited. Now, there are really good arguments that those are terrible ways of actually measuring research quality. But that’s beside the point.
Once universities start collecting publication metrics and they start using them in decision-making for things like promotions, grants, and who gets fired during a recession, then publications become valuable because they get used for those purposes. The fact that they’re being measured and monitored makes them important.
Whether or not you start off believing the publications are a good measure, you have to believe they’re a good measure. You have to manage them. You have to train other people to manage them. You have to supervise other people in ways of managing them. If you get successful, the last thing you want to do is change the system, burn it all down, and find a new way of measuring, because then all of your previous work to have good publication metrics would just go up in smoke. That’s the same thing that happens with any metric that gets used in accounting.
David: Then the strategy for you then must be to not do any original research at all. Just do a good quality literature reviews of popular topics.
Drew: That is indeed one of the strategies that does get used.
David: Very good. Just that people following on that it’s about citations, then writing good literature reviews is a good way of getting people to reference your material in the introductions of their papers. Very good.
We got these audits inside of organizations. We’ll talk a bit about the process and describe Power’s model for how this plays out in organizations. Organizations need to make themselves audit-ready. Organizations over time need to actually conduct and provide what audits need.
Also, employees themselves need to become auditable subjects, so we need to be able to check on the work of individuals. What does this mean, Drew? If we want to do audits, then organizations and our people need to be able to be audited.
Drew: Essentially, what it means is that we’re setting up our organizations with the types of systems and structures that can be observed, inspected, and evaluated. There’s this sort of key concept that gets used in this paper, the idea of an audit trace that we will get into a bit further down in our discussion.
David, if you don’t mind I might just jump to talk a little bit about the audit society and where Power’s original thinking comes from. I think once we talk about that it will be easy to talk about some of the other things.
David: Yeah, let’s do that. Good move.
Drew: The basic idea in the audit society is that there’s been an explosion in the amount of auditing and performance measurement throughout the 80s and 90s, and it’s particularly happened in public sector management. I think it’s fair to say that since that time it’s happened everywhere, that large organizations tend to operate now in the same way that public sector management used to operate. Power’s explanation is that the cause of this is that there is a neoliberal bundle of myths about good governance.
I think that myths is how most academics talk about neoliberalism. I don’t think it’s entirely fair, but I think a more neutral version would be that there are particular sets of values that get prioritized. Those values parallels out as transparency, efficiency, responsibility, auditability, and accountability.
For most people, I think you’d say, yeah, that sounds like a reasonable set of values for public sector governance. But the thing to note is that none of those values is about effectiveness or about serving the needs of the people who need those services.
That’s where the neoliberal bit comes in. To believe that those are good values you’ve got to believe that if you have those things, then market forces will take care of the rest. That you set in place the transparency, the auditability, the repeatability, the responsibility, and that leads to good government not just good governance.
But the risk is that the market forces don’t quite work like that. Decisions start to be seen as good because they’re made in the right way, using the right processes, the right records, not because they are actually the right decision. This is a decoupling of the goal of your process and the fact that you’ve done the process. That’s why the audit society is often seen as a sort of contrast to professional logic.
Your professional values are very much about making the right decision even at the expense of transparency, auditability, and accountability. You want your doctor to be telling you the most useful thing for you, not for them to be covering their back with the audit systems. But the audit system says the other way. Make them accountable and they will make the right decision.
David: Yeah, Drew. I think just as you mention the professional values there, I think of the hippocratic oath, like first do no harm. That says enough, but you’re right. If we worry about how other people are going to view our decision, then maybe that becomes more important than the outcome itself. Individuals may end up asking themselves the question of what decision can I justify versus (maybe) what is the best decision or even the right decision to make here.
Drew: To give neoliberalism its credit, I think most people who believe in it genuinely believe that in the long run having those sorts of accountability processes are what creates good decision-making. They genuinely believe that if the doctor thinks through—how am I going to explain this decision to the review board—that they will make better decisions than if they just focus on the patient in front of them.
But there’s a fair wealth of evidence that suggests that that’s not the case. When you have these systems, you do end up having a decoupling that you end up having prioritizing the processes rather than prioritizing the right outcomes.
David: I think we’ve seen in the last couple of years those big controversies in the technology sector around data privacy and so on, as well as some big issues in the finance sector involving things even in Australia like royal commissions. I think it’s clear that maybe some of those assumptions aren’t quite what plays out in reality. Maybe the rest of this discussion that we’ll have on this podcast will kind of explain why it plays out the way that it does.
Drew: Yup. Let’s leap into the foundations of what makes up an audit trail. Now, most of this will probably be fairly familiar to our listeners. I think it’s just useful the way Power explains how it all fits together and what conclusions we can draw from it.
There are a few fundamental things that you need to have the ability to audit things. The first one is that you can only audit things that have documents. You’ve got to have the creation of documents and records at the heart of any system that does audits. Often, those things didn’t previously exist. They were only started to be created for the purpose of having the audit.
The second thing is you’ve got an idea that transparency and traceability are the same thing and are important. We’ve taken the value of transparency and we’ve turned it into the practice of being able to link each step in a chain to the next and previous step in the chain. That’s the way an audit trail works. You take a particular document and you say, how does this link to the next document? How does this match the next document? You can follow through from this as work processes get transformed from lower levels in the organization to higher or across supply chains.
The final thing is the process, which is that we end up having to produce accounts of our performance. Those start off by individuals having to produce primary data, traces of the work that they’re doing. Even if their work isn’t on paper, they need to produce a piece of paper that keeps a trace of what they did. We then have to have systems in the company that aggregate those traces into more and more abstract forms, in a way that auditors can check each step, until eventually we report it up to the top of the organization.
David: So we combine a couple of those components of the audit trail. The first is these primary traces of performance, like you said these documents, these checklists, these artifacts. The audit trail demands the production of these things. But these things kind of reduce complexity and they become systemic in standardized representations of the way that work actually happens. It has to be like that so that the audit can first identify and then aggregate it all together. The audit process needs to do that. It needs to aggregate multiple pieces of these primary data.
Then it has to produce a performance account. The audit actually needs to deliver a result, if you like. All of these primary traces of performance have to be systematically aggregated to an organizational-level account. This is where we produce a ranking or a rating like satisfactory or unsatisfactory, or green or red. We’ve got these primary traces of performance, and then we’ve got the audit which produces a kind of a result or an account of what’s going on.
These two things together create this sequential logic of the audit trail. If we’ve got these primary documents, then we can create this outcome that we can then use to make sense of the world as managers.
Drew: We can make of this almost as the difference between quantitative and qualitative research. When you’re doing qualitative research, you can treat each individual thing as it’s own thing. You can be agnostic as to what counts as data. You can go and look at things, you can go and observe them, you can talk to people, you can check records.
When you are creating an auditable system, everything needs to be standardized to the extent that you can count it. That ability to count and check means that things need to be reduced to quite simple requirements. They can be marked as either true or false, yes or no, or literally count it up to produce a high level number.
There’s a quote I want to throw in here just directly because I think it sums up a lot of threads to what’s in the paper. He says, “Understood as a type, the process logic of the audit trail seems to lack any specific content or performance value, other than the imperative to produce performance accounts in accordance with the requirements of making performance traceable to primary performance data.” He then said, “The whole purpose of this is not to produce value for the organization. We don’t audit as a value-creating activity. We audited as an activity to produce performance accounts.”
That’s why, for example, we do inductions by PowerPoint, and risk assessment using checklists. Not because PowerPoint and checklists have performance values—they may, in fact, be worse than other ways of doing it—but they create primary performance data. They create a trace of what you’re doing in a way that other ways of doing those same activities don’t. An audit system requires you to be producing those frontline traces, and to be producing them in a form which can be easily aggregated.
That’s a challenge we make whenever we want to reform safety activities, whenever we want to get rid of clutter, whenever we want to make things more effective. If we break the ability to audit the activity, then that’s an unacceptable change inside an audit system.
David: Let’s talk about the process model of an audit trail where Power talks about how organizations take up these audits, and they grow, perpetuate, and continue.
First of all, you got this adoption of an audit process itself. Maybe use this example that we’ve used from this email. Say we decide that it might be a good idea to do culture audits. We adopt this idea of culture audits.
The first time we do these culture audits, no one’s got any of these primary data that we actually made. We can’t produce this really meaningful account and we probably get these quite poor results or difficult outcomes for the organization to deal with, because that organization is not audit-ready to be audited around safety culture and the individuals within the business aren’t auditable subjects. We have no way of counting, checking, and documenting that behavior that would match our audit criteria. But we adopt it nonetheless.
We increase the repetition and the routinization of the primary trace production so we know that the second time we’re going to be audited, we’re going to be asked for these things, these questions, and these documents. We pay little attention to actually creating a safety culture, but we pay a lot of attention to what we need to do to be able to produce for the audit trail next time we need to do the audit. It’s about how can we represent our world as opposed to what can we do about our world.
The more the facticity—I love that word; I’m going to use that in a sentence each week—of these performance accounts, which is satisfactory or unsatisfactory, exert a really strong influence over the powerful actors in the organization. We become less reflexive about what’s going on in terms of these value subversion, so we stop worrying about are we genuinely creating a safety culture in our business, and we worry more about what’s the rating that’s coming out of these audits in terms of the safety culture, so we focus on the representation of the world as opposed to the underlying world itself.
In that way, we don’t need this regulatory push to do audits anymore because we’ve got this really simplified view of the world that we can use to say whether we are getting better or worse ratings, and this whole cycle kind of reproduces itself. The organization draws conclusions from that and nothing has actually changed in the underlying reality of the organization, other than its ability to produce these primary artifacts. Did I work that example okay with the process?
Drew: Yeah. I was just imagining, actually. The first person in the organization who’s told to do this might be quite reflexive. They might say, yeah, but that way of measuring culture doesn’t work. That’s not really what culture is. They get told, okay fine. You tell us a good measure of culture and just make sure you’ve got it in place. They look around and they find that even though they disagree with the idea of using Likert scale surveys to measure culture, that’s the only thing they can do that’s going to produce a useful outcome.
They might fight back about that and then get told, okay, so you’ve got the worst safety culture in the group then because everyone else could comply with the audit and you couldn’t. Next time around, they work out how to comply. They’re still uncomfortable with it but they’re doing it. Next time around, that’s now the routine. That’s something they do every year, is around the safety culture survey. They get proud of the fact that their results were improving, so they want to have the survey in place to show the improvement.
David: Yeah, and the extension of that is the visible leadership visits, the behavior observations or the conversations, all those other things that go along with that, all the paperwork it produces, and all the stuff that can be shown the following year.
Drew, the outcome is that the performances become auditable in a way that seems entirely natural to the people in the organization. This all becomes really natural, like here’s the records for this, the records for that. The repetition of this audit trail creates an ever deeper rut in the road.
Every time you go through the cycle of generating this information, using it to judge performance, talking about it like that, and then moving on to the next cycle, it basically just creates this stronger, I kind of like deeper ruts in the road. I was going to try and come up with other words but I think we all understand what that means. It just self-reinforces itself. And then the desire to take that to other aspects of the organization and repeat that process like safety culture, diversity, or sustainability, becomes even stronger.
Drew: One of the key points about these ruts is originally there’s a decoupling of values, that people might fight back against the introduction of performance measures because they don’t think that it quite captures really what the underlying thing is. But very quickly, the values of the audit take over from other values.
Power presents a few different mechanisms of this transformational value capture. Part of it is in the power shift. The people within the organization who previously championed (I’ll say) the true values of the organization—I know that’s not a neutral term—the original values, are often held by the professional service providers.
If you can think that this is a hospital, then the people who really embody those values are the nurses and doctors. If you’re thinking of a child services organization, it’s the front line people who are visiting people’s houses. You’re dealing with families, hold the real values, and define the values of the organization.
But they lose their autonomy and license to do that. It gets replaced with power shifting to the systems instead of to those individuals. It’s now the systems that do the measurement that define the values. Ultimately, that type of transformation in itself just reinforces those types of neoliberal market values.
We’ve all heard it. Things that matter can be measured, and anything that cannot be measured doesn’t matter. That logic is directly a market logic built around the belief that things that can be counted, things that fit within the audit system, things that can be defined as processes, processes that can be measured, that’s what value is rather than other types of value.
David: I think there’s this example that you’ve got here and I’ve got a really specific story about it where it’s something like adversity. Really important for organizations are really, really current and rightly so important issues for business. It’s also a really sort of vague value as well.
What exactly do we mean? We can’t deal with uncertainty and vagueness in organizations, so we try to create a set of processes where we can audit whether the process is being followed, which is far more definite than this idea of improving diversity. But then, what happens is the idea of diversity gets redefined into whatever our processes say. When people talk (maybe) about diversity, they mean the processes and not anything else.
An example is an organization that really wants to increase its gender diversity in management. A really hard thing to audit because you can’t audit the outcome. You actually have to have some processes and documents. My decision, for example, that the last three persons shortlist for any single job in the organization needed to have two females and one male in that last three-person shortlist. That was the process, and that was the process that got audited. It was really interesting to see that that process over time got really, really well-followed, but the diversity of people appointed into positions didn’t change.
The audit result got better and better over the two or three years that that was being implemented, but the actual outcome, the value that the organization was trying to create from that process didn’t actually change. Is that it, Drew? Do you see that playing out?
Drew: Yes, exactly. We’ve got almost the exact same thing happening at the moment in our university about work-integrated learning. We start off with this really vague value which is about getting graduates ready for work outside universities.
In theory, every single thing we do could be getting people ready for life outside university. But there’s a particular government initiative, they’re going to set up measures, they’re going to change our funding based on our ability to demonstrate work-integrated learning.
We’re having lots and lots of meetings, not about how to get our graduates ready for work or what sort of skills they need, but how we set up processes that can measure and report on integration of work-integrated learning into our curriculum. The entire logic is based around what we can audit, rather than on what we think we can be doing.
There will be certain activities that we put in place because they’re easy to create traces that we can easily match in that audit trail towards the government priorities. And no way will we be asking if this is good for our students, because good for our students is not something that we can create a nice and easy trace for, or link to any of the requirements, or aggregate as measures across the university.
David: There’s this idea to show that even in this process, if you like, that audits end up going from push to pull. The first time someone is subject to a particular new type of audit, it becomes really resistant. People don’t like these new checks and balances, these getting asked questions about documents that they don’t have, going through a process which distracts them from what they call value-producing work for the organizations that they complain about. Why do we need to do all this extra stuff? All these systems don’t help me do my work. They don’t create value for the organization. It’s overly simplistic and not really a good thing to do.
This sort of dispositional type resistance where we don’t like it, but we can’t push back against it, like you can’t suddenly just opt out and say, you want to audit me next month? I’ve decided that I don’t have to be audited. Even though there might be this really underlying resistance to it, it doesn’t seem to stop the initial auditing activity from happening.
Drew: Do you want to keep going and talk about the other side of the coin, David?
David: Yes. You got this pull because what’s interesting is people want to know they’re doing a good job. People want to have clarity, simplicity, and certainty over what the organization wants them to do. In the same way that people resist this need to produce all of this stuff, there’s also this desire by people in organizations to make sense of their work by creating all of these primary documents, so they can show people the things that they’re doing. They like the idea about having evidence of what we do and the value that it has. Getting a good audit result is sort of a source of value to people in the contribution of their work.
Again, Drew, you mentioned academics before. Even I myself not being a full-time academic but someone who does take great joy in getting one of the papers cited, we complain that citations aren’t a great indication of good research, but it’s always nice to see your citations going up.
Drew: And David, I think you’ve done this with rooms full of people before, asked them whether they believe in LTIs, and then asked them whether they collected and reported LTIs. I think even people who think that LTIs are nonsense will still claim LTI improvements.
In fact, literally I think I might call out Sid Dekker on this one as one of the most vocal advocates against things like lost time indicators. Still, when he had a project where the introduction of Safety Differently principles had improved lost time indicators wanted to include that as one of the outputs.
David: Yeah, it’s compelling because it fits exactly what we’re talking about in this podcast here today. It matches the representations people have of the underlying value or outcome it’s trying to create.
Initially, there’s pushy strong and this pull is weak. If you want to start doing a safety culture audit, you should really have to push hard into the organization to make that happen. But as the organization conducted those audits over time, the push gets weaker and the pull gets stronger. The resistance changes from not wanting to produce these traces.
But then, people being very strategic and deliberate about how they manage them or even add additional primary traces which is to show even more—I don’t know if compliance is the right word—audit value that they’ve been able to create since the previous audit. So going beyond even the primary traces that the auditors are looking for because again, the audit score itself is an outcome that’s really, really valuable to people inside the organization, even if the underlying outcome that the organization’s trying to deliver is not changing.
Drew: I don’t know what you think about this one, but I think this is part of what explains why people are so insistent on having new metrics for safety. People are so captured in this framework that even if they hate things like LTIs, they don’t want to get rid of the same fundamental logic of measurement. They just want to replace LTIs with some Safety II–based metric that will slot into the exact same place in the organization.
We just change our audit questions slightly to ask for a different type of metric. We aggregate it in the same way, we report it in the same way, because even if we want to push back against the particular trace, we have completely bought into the logic of how we account for an audit.
David: I saw this in my research during my PhD. It didn’t matter what the intention was, coming from a board or a senior executive, in terms of providing a direction or instruction for the organization around safety. It didn’t matter what their intention was or what underlying outcome they were trying to create in the business. Every time that direction or instruction came down the hierarchy, it was always received as a compliance activity.
It was never received in a way that says, what’s the underlying outcome that is trying to be driven here? It’s, how can I report back to the board and the senior executive that this is being done? That was the only conversation, like your conversation about work-integrated learning earlier.
I think auditing is one of those like Power says, the micro processual foundations of it inside companies means that you could drop anything into that cycle with the very best of intention about what outcomes you’re trying to create, and this process will not create those outcomes for you.
Do you want to talk about other interesting aspects of the paper that we probably won’t have time to go into depth on, but we’ll sort of just mark out a sign post and then go on to the practical takeaways?
Drew: Sure. I should flag that this is about half the paper that we’ve compressed into six bullet points, not because it’s not good stuff in just as much depth as I think we’ve covered the main thing that we wanted to cover. The rest of the paper is interesting but makes some somewhat peripheral points.
I think one of the key things that it says now is that this process of auditing doesn’t just replicate creating those ruts in the road, but it also goes sideways. That once we’re brought into the logic of auditing some things, then we’re brought into the logic that everything needs to be audited. When we have a new value or a new concern, whether it’s the social impact of our work, or psychological safety, or employee well-being, then we end up creating those same structures and incentives that we already have with the existing things that we audit. We don’t just treat them as new topics outside of what we’re currently auditing. We just expand the audits to cover them.
There are also a few specific mechanisms that Power says apply to different organizations, and explain why some organizations are quicker or slower to adopt the audit society framework. The first one he says is that pre-existing work practices make a difference. Some organizations are already very into creating traces of their work. They make it really easy to slap an audit on top.
The example it gives there is pilot checklists. Pilots use checklists before they need to get audited on those checklists. They just thought it was a good idea. If the organization’s already doing lots of things that can easily be audited, then there’s much less push when you start introducing audits, and there’s much more pull because people are keen to demonstrate what they’re already doing.
The second thing he mentions is just that the introduction of digital technology means that it’s so much easier to create traces now. There are some things that we would never previously have audited that we are now starting to audit like customer satisfaction because it’s so easy for them to just click on the app and give you five stars. Instantly, you’ve got a trace. Hey, if we got a trace we should be doing something with that. We end up building it into our performance and appraisal structures.
David, jump in at any moment or I’ll just keep going.
David: No, I’ll let you keep going and there are a few points I want to make before we jump into takeaways.
Drew: The next one is that some organizations are predisposed to love the idea of auditing because they’re already needing to protect themselves. They’re worried about litigation, being sued, or regulatory action, then they might actually very welcome the idea of creating traces for self-protection.
In something like medicine, there’s much more pull than there was before because doctors see the direct value in having auditable traces of what they’ve told patients, and whether they’ve probably got informed consent for things. They’re much more receptive than they might otherwise be if they’re in a highly litigious environment.
The final one is that in some organizations, the shift towards audit gives power to particular people. The people who operate the systems have the power. If the people in the right spots are predisposed to seek that sort of power, then that’s going to really accelerate the process to the point where people might even use external events like an accident as reasons to introduce accountability for auditable processes, to make a power grab through putting in place and now operating that system.
We see that with some of the things like including sexual harassment under the safety banner. That’s just a clear grab to bring a new topic under the system to expand the power base of a particular organization.
David: Absolutely, so well worth the read this paper. In case we haven’t threatened the existence of anyone doing audits, I think to connect it back to our demonstrated safety in the safety workplace, the safety of work, but there’s a reason that organizations do this. This is why these processes like audits hang around. There may always be a need to do this, but just a couple of points, Drew, that seem very clear in my mind, and I’d be interested in your views about whether they are this clear.
Audits themselves are not improving underlying performance. Audits are a performative process not an instrumental process, so the audit doesn’t change the underlying thing that we are auditing. That’s point number one.
Number two, they’re not looking at outcomes or value creation in the underlying process. There was a quote in this paper that I thought was really good, and I read, “Organizations have a tendency to emphasize methods rather than goals as an important source, and this is an important source of disorientation in all organizations.”
This focus on the process and the document as opposed to the organizational goal, is a 1957 quote from what they called Selznick. They’re talking about 65 years ago and said, “Oh, be careful not to focus on process rather than the underlying goal of your organization.”
And a third is that, “As much as we might think there’s that value of transparency, accountability, and responsibility, that audits themselves and not providing open and transparent accounts. This idea of transparency is known to be paradoxical, and that audit practices themselves that are said to be very transparent, in fact lead to this opacity and specialist control. We know the primary data sources. We can manage and manipulate the primary data sources. The audit process is anything but a transparency exercise.”
Now, I’ve probably been pretty direct with those three points, Drew, but that’s sort of how I see the world now.
Drew: Let me throw a couple of contradictory bombshells at you then, David. A couple of arguments in favor of audits that don’t contradict your points (I think). The first one is that the displacement from goals to methods (I think) is a genuinely adaptive behavior in some circumstances. If you’re in a job with very, very high uncertainty and very, very low control over outcomes, then the ability to define the parameters of your work in terms of process is important for self-survival.
I’m thinking there of people like care workers dealing with children at risk. Sometimes the best thing that you can do is know that you have followed the process because you can’t protect everyone. Sometimes, people are going to be put into situations where they turn out to be at more risk than you realized. Sometimes, you’re going to take kids away from their parents and you’ll never know whether that was necessary, justified, or not. You’ll never have that uncertainty resolved. The ability to say, well, the best I can do is follow a process that we’ve all agreed on, that’s actually important for protection. I would argue that applies to a lot of safety work.
David: We’ve cited Wastell’s. I know it’s one of your favorite papers, Drew, Wastell’s The Fetish of Technique: Methodology as a Social Defence. We may even do a podcast about that. I’ll hear from you one day and we’ll do a podcast about it. Whether you’re trying to implement an IT project or manage accidents, I think that the idea that you can’t control the outcome, you can be judged against the process that you followed.
Drew: Yup. And the second one as I say it, adopting the worldview that comes with audits is adaptive for organizations. Even though this does come up from the bottom-up, it is still useful for aligning an organization with other organizations that think the same way. Your organization that didn’t need audits for its own sake might end up adopting them in choosing to adopt that same internal mindset and structure, so that they could talk the same language as their clients and partners. I don’t think that that is maladaptive for a single organization. Even if all of society goes to hell in a handbasket, it still makes sense for the organization to go along with it rather than fighting against the tide.
David: Awesome, Drew. Let’s move on to some practical takeaways I might lean on you a little bit here. This paper lays out very much a descriptive theory. Power’s done a great job of laying out your nod on the way through this paper, but it’s very much a descriptive theory. It describes the way the world is. It also makes this audit society seem inevitable by the outside or inside of an organization. It doesn’t really tell us what to do about it or in fact if we should do anything about it.
Drew, what do you sort of practical takeaways and advice that you do in relation to this inevitability of auditing, and perhaps the distraction of the core underlying value you’re trying to create?
Drew: David, one that I really forgot to put on this list that I think is important is don’t hang around sociologists for too long. It is a genuine path to existential despair once you understand just the inevitability of some of the mechanisms that create dysfunction. We see this with our own social science students. Understanding exactly how social problems work doesn’t make you feel good about yourself or hopeful for the future. Maybe don’t read too much of this stuff unless you completely find it interesting.
David: I have a social theory, a social science shelf on my bookshelf. I might steer away from it for a while. I do get lost a little bit during the PhD on some of that theory.
Drew: On a more hopeful idea, once you understand some of these mechanisms, that does empower you to be reflexive. If you can see it working, your simple understanding that it is going on around you gives you space to see where there is freedom.
One of the big ones is that even inside an audit framework, there’s a lot of choice about what primary traces are produced. You can say, hey look. It’s inevitable that any activity that we do has to produce an auditable record. That doesn’t mean it has to be a form. That doesn’t mean it has to be written down. That doesn’t mean it has to be a signature.
There can be other ways of producing traces that align much better with work. If we can design our activities to meet the needs—both of the audits and of the people involved rather than just blindly what’s most auditable—I think we can make a genuine difference for ourselves and for other people.
Secondly, similarly, is that when you see this happening you realize that maybe be a bit careful what you wish for in terms of what counts as safety. It might seem well-intentioned to deal with things like mental health, psychological safety, well-being, and sexual harassment under the same systems as safety. But before you get too ambitious, remember that it is fairly inevitable that those will experience the same capture and value decoupling that happens with your other systems. Ask yourself whether you want that to happen. There maybe for some of those things that yes, we think that would be a genuine improvement, but for others we may in fact lose the original value.
I think safety climate is a great example. The whole point of safety climate was meant to be a balance at more systems-focused and instrumental ways of thinking about safety. Once we let safety climate become captured in that same system, we lose everything that safety climate brings to the table. We ought to keep those things quite separate in how we think about and manage them. I very much think that if we care about things like employee mental health, that bringing it under the same safety systems could seriously compromise the values that we’re trying to preserve and protect.
David: Finally, I think this may not strictly be about audit itself. There are things like certifications, standards, and other processes like contractor prequalification and incident investigation, even risk management. Lots of unaudited activities can have the same underlying value decoupling and logic, that the process becomes maybe more important and decoupled from the outcome we’re trying to create. We see that with a lot of those safety practices.
Again, just like your first takeaway, knowing that this is happening and being strategic about that as a safety leader, as an organizational leader, means that you can work within that inevitable cycle that’s going to play out, and even find ways to use that cycle, maybe to more closely couple that cycle to the underlying outcome you’re trying to deliver or work with.
Drew: This has been one of our longer episodes and the paper is a bit of a long read, but I think it’s one of the ones where I’ve actually learned quite a lot from reading and thinking about the paper, and I hope our listeners have found a lot to think about and discuss with this further. I don’t want to just do this episode and have it just fall off the wagon.
I’d love to have some more discussion with anyone who has either read the paper or have some thoughts about maybe how these processes work for various things, particularly how they apply to expanding the scope of safety work in our organizations.
David: Yes, thanks, Drew. I’d love to as well, and I’m very happy. I think we’re probably about an hour, so thanks for hanging in there if you have made it all the way through. I think it has taken us 82 episodes to get to this topic of auditing. I suppose we took our safety clutter episode seriously, episode 80, so this one deserves the same sort of discussion because it plays such a central role in not just safety but organizational life.
So the question that we asked this week was why do we audit much?
Drew: For once, I think we have quite a very clear answer to that. Power explains pretty convincingly that we audit because every time we audit, it creates the organizational conditions that make us want to keep auditing.
David: All right, so that’s it for this week. We hope that you found this episode thought-provoking and ultimately useful in checking the safety of work in your own organization. If you want to contribute to our audit process and give us five stars on whatever podcast app you’re listening to, or maybe even a comment, please do that and send any comments, questions, or ideas for future episodes (just like this one) to us at feedback@safetyofwork.com.