AI in the pharmaceutical industry: where it works, what it must prove, and who signs

Three demonstrations in one month: a camera that catches defects an inspector can miss, an assistant that answers questions on your procedures, a model that predicts a compressor failure. All three are plausible. Only one is likely to reach a batch record this year. The model has little to do with which one: what matters is what each can prove, and who signs when it is wrong.

This is the wide view: where artificial intelligence (AI) sits on a pharmaceutical chain, what a model must prove before it touches a decision under good manufacturing practice (GMP), and which text applies first.

Pharmaceutical manufacturing has always moved carefully with digital tools, for good reasons: every change on a validated line costs paperwork and tests. The guidance has moved anyway. The second edition of GAMP 5 (2022), the industry’s guide to computerised systems, added appendices on artificial intelligence and machine learning and on distributed ledger systems, and ISPE followed in July 2025 with a full GAMP guide on AI.

Six places AI sits on a pharmaceutical chain

• Discovery and research. In June 2025, Nature Medicine published a randomised phase 2a trial of a molecule whose target and structure came from generative AI: 71 patients, twelve weeks, a lung-function gain in one dose group. A real result, and one small trial. The European Medicines Agency (EMA) treats discovery as low regulatory impact while poor performance only costs the developer.
• Clinical development. Models support trial design, patient selection, endpoints and safety monitoring. The US Food and Drug Administration (FDA) says it has received more than 500 drug and biological submissions with AI components since 2016. The EMA expects frozen models and prospective testing on new data before a high-impact model reaches a pivotal trial.
• Manufacturing. Here a model meets the product: inspection of containers, soft sensors, predictive maintenance. The FDA chose a manufacturing example for its AI guidance: automated analysis of the fill level in every vial. The draft EU GMP Annex 22 is written for such uses. Predicting a compressor failure touches no batch record, so most sites start there; see use cases that survive a quality review.
• Quality and compliance. Deviation triage, search across procedures, first drafts. The draft Annex 22 keeps generative models and large language models out of critical GMP applications, and expects a trained person to own the output elsewhere. What such a tool is worth depends on the documents behind it, see the answer is only as good as the data.
• Supply chain. Forecasting and shortage signals are the oldest, least contested use. The methods are mature, and the arguments are about data quality rather than models.
• Pharmacovigilance. Case intake, duplicate detection and coding run with human review. The EMA is more permissive here: its reflection paper accepts models that keep learning, while leaving responsibility with the authorisation holder.

Six examples you will hear, checked

Presentations on AI in pharma reuse the same examples, often placed at the wrong point of the chain. Here are six that circulate, with what each actually did.

The pattern matches the map. The strongest results sit in discovery and in tools that support a person’s decision. The closer a model gets to a batch, the more it has to prove. Authentication, often presented as an AI application, is a matter of codes and databases; see how European verification actually works.

What the clinical evidence teaches a plant

The best evidence on how AI fails in the field comes from medicine, where models were deployed and measured. A 2022 study for the European Parliament’s STOA panel collects it (Lekadir et al., PE 729.512). Its examples are clinical; the reading for a plant is ours.

• One accuracy figure proves nothing. Testing a model on data that resemble its training data flatters it. A review cited by the study found only 31 of 516 published radiology AI studies (6%) evaluated on external data. A supplier’s figure on the supplier’s images is that kind of number.
• Performance collapses when the input changes. A deep-learning system for retinal scans had a 5.5% referral error on the device it was trained on, and 46.6% on a second device type. Retraining one stage with 152 scans from the new device brought it back to 3.4%. A pneumonia model scored an area under the curve of 0.931 at home and 0.815 at an outside hospital, where 1.0 would be perfect, and could tell which hospital an image came from in more than 99.9% of cases. Pneumonia was far more common at one of them, so recognising the hospital was a shortcut to the answer. On a line, that shortcut is a model that learns the format or the lighting instead of the defect.
• Bias comes from the data people labelled. The study’s examples are demographic. On a line, the subgroups are formats, container suppliers, lines, shifts and defect classes. A library built from what one inspector rejected teaches exactly that.
• People over-trust opaque tools. Responsibility drifts onto whoever used an assistive tool and cannot explain its output. The AI Act asks high-risk systems to keep their overseers aware of over-reliance, which it calls automation bias (Art. 14(4)(b)). The regulators’ good machine learning practice principles measure the human-AI team, not the model alone.

What a model must prove before it touches a GMP decision

Pharmaceutical people know how to handle a promising substance: phases, with a gate between them. The study brings together two phased schemes for evaluating AI, one built on drug development. Transposed to a site, they say who proves what, in which order.

• I. Feasibility. A laboratory result on a small test set under ideal conditions. It shows the task is possible. Ask what sits behind it: data sources, images per defect class, how labels were verified, and hits, misses and false alarms per class.
• II. Capability. Real conditions simulated on a bench: lighting drift, focus, position, every format, rare defects, artefacts such as droplets or glare. The study puts end users in this phase, watch operators work with the output before go-live, not after the first deviation.
• III. Effectiveness. The real line, which on a GMP site means performance qualification: does field performance match test performance? Expect local problems (camera mounting, lighting, handling) to be resolved with the supplier, each fix returning to the bench first.
• IV. Durability. Routine production with monitoring: results per defect class, a challenge kit of known units run to a schedule, periodic review. The study suggests re-evaluating against reference data monthly or quarterly.

Two rules decide whether this is evidence or paperwork. First, the test data must be independent: the draft Annex 22 asks that test data are never used in development, training or validation, with access control and an audit trail. Second, the acceptance criteria must be at least as high as the process the model replaces: so measure the human baseline first, per defect class and on enough units to mean something. For injections that is familiar ground: US Pharmacopeia chapter <1790> already expects an alternative to manual visual inspection to be equivalent or better.

Which text applies first

The AI Act (Regulation (EU) 2024/1689) sorts systems by risk: a few prohibited uses, high-risk systems with heavy obligations, transparency duties, and everything else. The surprise is what falls where. A model that inspects vials is usually not high-risk: medicinal-product law is not in Annex I, and manufacturing inspection is not an Annex III use. Systems that recruit, allocate work or monitor staff are high-risk (Annex III, point 4), and inferring emotions at work is prohibited except for medical or safety reasons (Art. 5(1)(f)).

The obligations arrive in stages. The prohibitions and the AI literacy duty of Article 4, which asks providers and deployers to make sure their staff understand the systems they work with, have applied since 2 February 2025. The high-risk obligations follow on 2 August 2026 for Annex III systems and on 2 August 2027 for AI built into regulated products. For a model deciding on product quality, the stricter text is GMP itself.

Accountability does not move. The EMA reflection paper is plain: the sponsor, the authorisation holder or the manufacturer must ensure that "all algorithms, models, datasets and data processing pipelines used are fit for purpose". The model signs nothing. A named person does: the reviewer who accepts the result, and in the EU the qualified person who certifies the batch.

Five ways it fails after go-live

Models rarely fail on day one. They fail quietly, months later, in five ways.

One line on the fourth row. An attack on a model is hard to see: the tool keeps working while its conclusions are confidently wrong. Most talk of AI and cyber risk is about AI as the attacker’s tool. Here it is the target, and quality monitoring is the first detector.

What to do first

1. Inventory the models you already have. Ask every equipment and software supplier one written question: does your product contain a trained model, and which version is installed? Sites regularly find a classifier inside a machine qualified years ago.
2. Write the intended use. One page per model: the task, the inputs, the limits, and the subgroups it must work across, formats, lines, suppliers, defect classes.
3. Measure the human baseline per defect class, before the pilot. Without it, "at least as good as what it replaces" cannot be shown.
4. Freeze a test set and build a challenge kit. Independent data the supplier never sees, and a physical set of known units for routine checks.
5. Write the monitoring plan with alert limits: input drift, results per defect class, cases the model refuses, human overrides. Give it a named owner and a review period.
6. Bring the model into change and configuration control (version, file fingerprint, thresholds, pre-processing, camera and lighting settings) as for any qualified equipment, so that a retrained model or a new camera is assessed for requalification like any other change. The frame is the one GAMP 5 Second Edition sets: justify the effort instead of piling up documents.

A model cannot hold a licence, carry a deviation or answer an inspector. Everything it does is signed by someone who can.

References

• Lekadir K. et al., Artificial intelligence in healthcare: applications, risks, and ethical and societal impacts, EPRS/STOA, PE 729.512, June 2022 (source of the clinical evidence and the phased evaluation): europarl.europa.eu729512_EN.pdf)
• European Commission, EudraLex Volume 4, draft Annex 22 Artificial Intelligence and revised Annex 11, consultation 7 July to 7 October 2025: health.ec.europa.eu
• EMA, Reflection paper on the use of artificial intelligence in the medicinal product lifecycle, EMA/CHMP/CVMP/83833/2023, adopted 30 September 2024: ema.europa.eu
• FDA, draft guidance Considerations for the Use of AI to Support Regulatory Decision-Making for Drug and Biological Products, January 2025: fda.gov
• Regulation (EU) 2024/1689 (AI Act), application dates in Article 113: eur-lex.europa.eu