When the workaround becomes normal: drift, “operator error” and CAPAs that change nothing on a GMP site

The quarterly quality review is running late, and the deviation trend is the last item before lunch. On the screen, a list sorted by reference number. Halfway down, three records from the same capping machine on the same filling line, months apart: torque out of range, operator did not follow the procedure, operator retrained. Each one was investigated, each one closed on time, each one approved. Nobody in the room reads them together, because nothing on the screen invites it. The meeting moves on.

Two floors down, on the night shift, a technician opens a guard on the same machine to clear a jammed cap without stopping the line. The door switch has had a small piece of metal taped into it since a commissioning snag that was meant to be fixed “next shutdown”. He did not put it there. He was shown it in his second week, by someone who had been shown it too.

None of these people is careless, and that is exactly what makes the problem hard.

A term borrowed from a launch pad

The phrase normalisation of deviance comes from the sociologist Diane Vaughan, whose 1996 book The Challenger Launch Decision reconstructed how NASA engineers and managers came to fly the Space Shuttle with joint seal erosion that the design never allowed for. Her finding was uncomfortable because it took away the villain. There was no single bad decision, only a long series of reasonable ones, each taking the previous acceptance as its baseline. Challenger broke up on 28 January 1986.

Seventeen years later the Columbia Accident Investigation Board, which consulted Vaughan, found the same pattern in the way foam strikes on the orbiter had been treated, and summed it up in a phrase worth pinning above any deviation desk: the acceptance of events that are not supposed to happen. Each foam loss after the first had been analysed, explained and judged acceptable, until the one that was not.

Process safety engineers have been making the same point about their own plants. Writing in ISA’s InTech in 2020, the functional safety specialist Paul Gruhn argued that the very low failure probabilities calculated for safety layers rest on random hardware failures, while real accidents come from slow drift in how those layers are actually maintained and operated. His warning sign was the manager who says the plant has run for years without an accident and must therefore be safe. Gruhn’s answer was statistical. On a GMP site we already have a tool for it, the quality system, provided we read it the right way.

What drift looks like in a pharmaceutical plant

Drift rarely announces itself as a compliance problem. It looks like experience, pragmatism, getting the batch out. And the forms it takes are strikingly similar from one site to the next.

• The temporary bypass. An interlock on an isolator door, a level switch on a buffer tank, a pressure check on a filter skid, overridden during commissioning or after a nuisance trip, with every intention of restoring it. The intention outlives the people who had it.
• The silenced alarm. A differential pressure alarm that sounds at every door opening is delayed, then raised, then acknowledged by reflex. A year later the room cascade has been reversed for minutes at a time and nobody has seen a reason to look.
• The paper shadow. A weighing or labelling system is slow at shift change, so operators jot values on a sheet and key them in later. The validated system now records what the paper says, not what happened at the balance.
• The recurring operator error. The same deviation, on the same step, with a different name on it each time. Closed as human error, corrected by retraining, back three months later.
• The expected deviation: “we always get one after the long weekend”. Once a team can predict a deviation, it has quietly become part of the process, unwritten and unvalidated.

Each of these starts as a small departure that is defensible at the time. The interlock really was tripping for no reason; the system really was slow. Drift sets in at the second step, the next time round, when the departure is no longer questioned because it was accepted before and nothing bad happened.

Why good teams do it

Spend long enough in plants, moving between engineering, automation, IT and quality, and you end up with the same honest answer: drift is a by-product of things organisations reward. Getting the line back up. Closing records on time. Not escalating every irritation. A shift lead who stopped the line each time the door switch misbehaved would be called rigid long before anyone called him rigorous.

The UK Health and Safety Executive described the mechanism plainly back in 1999, in its guidance Reducing error and influencing behaviour (HSG48). It separates errors (slips, lapses and mistakes) from violations, and calls a routine violation one where breaking the procedure “has become a normal way of working within the work group”. Among the reasons it gives: the rule seems too restrictive, it no longer seems to apply, it is not enforced, and new starters join a team where the shortcut is the norm and never learn that it is one. That last reason is our night-shift technician. He is not breaking a rule he knows. He is doing the job the way it was taught to him.

Two features of a pharmaceutical quality system make matters worse if nobody is watching for them.

First, records fragment the story. A deviation system is built around individual events, each with its own owner, deadline and approval. It is excellent at closing records and poor at noticing that five records are one problem. Drift lives in the gaps between reference numbers.

Second, knowledge is split between departments. Engineering knows the interlock was bypassed, production knows the workaround, quality knows the deviation trend, and each piece looks harmless on its own. Vaughan called the organisational version of this structural secrecy. On a site it is usually just three teams that never put their pieces on the same table.

What the guidelines already ask

Nothing new needs inventing here. The texts that govern a European site have asked for this for a decade or more. What tends to be missing is the habit of reading them as a defence against drift rather than as a list of record fields.

The revision of ICH Q9 deserves a word, because it speaks directly to this problem. ICH adopted Q9(R1) on 18 January 2023, and it has been in effect in the EU since 26 July 2023. Its new section on managing and minimising subjectivity names something every deviation board has seen: risk scores that creep downwards for a hazard the team has lived with for years. Familiarity is not evidence, and the revised guideline gives quality units a recognised basis for saying so in a risk review.

“Human error” is where the investigation starts

The sentence in Chapter 1 that matters most here is short: where human error is suspected or identified as the cause, it should be justified, having taken care that process, procedural or system-based problems have not been overlooked. Read carefully, it turns the usual order round. Human error opens the investigation; it cannot be the place where it ends.

In practice, that means asking a handful of questions before anyone fills in the root cause box.

• What kind of failure was it? A slip on a familiar task, a lapse of memory, a mistake of judgement, or a deliberate shortcut? Each points to a different fix. A slip calls for design (a confirmation step, a better HMI layout), a mistake for knowledge, a routine violation for a conversation about why the rule is being worked around.
• Could the next competent person have done the same? If so, the cause lies in the task, the equipment or the procedure, whatever name is on the record.
• Was the procedure usable as written, at that hour, with that equipment? A step that cannot be done in the time the line allows will not be done the way it is written.
• Has it happened before, anywhere on site? Search by equipment, step and symptom, not by reference number or by product.
• What did the people on the floor already know? The workaround is usually common knowledge on the shift long before it reaches a deviation report.

Retraining has its place when a person genuinely lacked knowledge. As a default CAPA it becomes a quiet form of drift in its own right: it closes the record, changes nothing in the system, and lets the same event come back under a new name.

Measuring whether a CAPA worked

ICH Q10 asks for the effectiveness of CAPAs to be evaluated; Chapter 1 asks for it to be monitored and assessed. Many sites meet this with a check, a few weeks after closure, that the action was implemented. Implemented and effective are two different claims.

What works is to write the effectiveness criterion when the CAPA is opened, in measurable terms, with a window long enough for the problem to have come back. No recurrence of the same failure mode on that equipment, or any similar one, over a defined number of batches or months. A trend indicator that has moved. An audit of what is actually done on the floor, not of the training records. If nobody can write the criterion, the root cause has probably not been found.

Where to start

You do not need a culture programme to begin. You need a few afternoons and the right people in the room.

1. Re-sort the last two years of deviations by equipment, process step and failure mode instead of by reference number. Repeats appear within the hour.
2. Count the human-error closures. Take a sample and test each one against the Chapter 1 sentence: was a process, procedural or system cause actually excluded, and how?
3. Walk the line with the validated documents in hand. Look for taped switches, keys left in panels, laminated cheat-sheets, sticky notes on HMIs, shelved or delayed alarms, paper logs beside electronic systems. Each one is a gap between the validated state and the real one.
4. Open a bypass and override register. Every inhibited interlock, forced signal or suppressed alarm gets an owner, a reason, a risk assessment and an expiry date, and the open list goes to management review.
5. Ask the night shift, without a clipboard, what they do that the procedure does not say. Then fix the procedure or the equipment, and thank the people who told you.

Five common mistakes

1. Treating a closed deviation as a solved problem. Closure measures administration, not recurrence.
2. Retraining as the reflex CAPA. If the system is unchanged, so is the probability of the event.
3. Setting a target to cut the number of deviations. It mostly cuts the number reported, and a falling count alongside rising batch-record corrections should worry you.
4. Looking for drift only in the data. Control charts catch process drift; behavioural drift shows up on the floor, in the gap between procedure and practice.
5. Punishing the person who reveals the workaround. Do it once and the next workaround will be invisible. A reporting culture is valuable precisely because it shows you what the records cannot.

Rules rarely fail on their own; attention does. The regulations already ask the right questions, and the work is to keep asking them about the things a team has stopped noticing.

References

• Diane Vaughan, The Challenger Launch Decision: Risky Technology, Culture, and Deviance at NASA, University of Chicago Press, 1996.
• Columbia Accident Investigation Board, Report, Volume 1, NASA, August 2003, chapter 8.
• Health and Safety Executive, Reducing error and influencing behaviour, HSG48, second edition, 1999.
• European Commission, EU GMP Chapter 1, Pharmaceutical Quality System, in operation since 31 January 2013, and Chapter 8, since 1 March 2015.
• ICH, Q10 Pharmaceutical Quality System, 2008, and Q9(R1) Quality Risk Management, adopted 18 January 2023.
• Paul Gruhn, “Bayesian analysis improves functional safety”, InTech, ISA, March/April 2020, pp. 28–33, the starting point of this article.