AI/ML Governance and Validation in Regulated GxP Environments
AI governance frameworks compliant with Annex 22, EU AI Act Annex III, and GAMP 5 Second Edition — not shelf documents, but systems that hold up to inspection.
Why AI governance is an engineering issue
- An AI/ML system in a regulated environment differs fundamentally from a traditional GxP computerized system: it produces probabilistic outputs requiring specific validation approaches
- The concept of a 'validated system' is not permanent for an ML model: model drift requires continuous production performance monitoring
- GMP Annex 22 and EU AI Act Annex III define specific regulatory obligations for AI systems in pharmaceutical environments, already in force or in imminent enforcement
- Decision traceability — explainability — is not automatic: it must be architecturally designed into the system
- Responsibility for AI decisions in GMP contexts lies not with the model, but with the organization that validated and deployed it
Our operational approach
Dalia IA supports organisations through a modular AI governance service calibrated to the probabilistic nature of AI/ML systems and applicable regulatory requirements. Governing AI systems in pharmaceutical environments cannot be reduced to writing a policy: it is an engineering discipline requiring deep understanding of AI/ML technical mechanisms and applicable GxP regulations.
AI Governance Gap Analysis and Policy Framework
- AI Governance Gap Analysis: current state assessment against Annex 22, EU AI Act Annex III, and GAMP 5 Second Edition requirements
- AI Policy Framework: drafting or revision of corporate policies for AI use in GxP environments, including LLM-specific policies
- Identification of AI systems in use with risk classification (EU AI Act Annex III)
- Gap report with severity classification and regulatory mapping
Intended Use, URS, and Model Validation Planning
- Documented Intended Use definition: supported GxP process, input/output types, autonomy boundaries, model acceptance criteria
- User Requirements Specification (URS) adapted to the probabilistic nature of the AI system
- Validation Plan for AI/ML systems: risk-based approach, statistical performance testing (accuracy, precision/recall, F1-score), robustness testing on edge-cases
- Extended Supplier Assessment: model transparency evaluation and training data quality review
AI Change Control Framework, Explainability, and Model Monitoring
- AI Change Control Framework: formal management of retraining, algorithm updates, dataset changes, and threshold modifications
- Model drift monitoring: production performance KPI definition and escalation procedures for drift events
- Bias Assessment: training set representativeness analysis, disaggregated performance testing, low-performance area documentation
- Explainability implementation: SHAP, LIME, attention mechanisms — documented and justified per model type
- Inspection Readiness Review: AI documentation review for GMP/EU AI Act inspection readiness
Deliverables and outputs
- AI Governance Gap Analysis Report
- AI Policy Framework (including LLM policy)
- AI System Risk Classification (EU AI Act Annex III)
- Intended Use Document
- User Requirements Specification (URS) for AI/ML systems
- Validation Plan for AI/ML systems
- Bias Assessment Report
- Explainability Documentation
- AI Change Control Framework
- Model Drift Monitoring Plan
- Inspection Readiness Package (GMP + EU AI Act)
Regulatory references
Numbers that matter
Studio active since 2004
Pharma, biotech, medical devices
Multinationals and SMEs
No open major findings
Who this is for
- QA Manager
- Validation Manager / CSV Lead
- IT Manager in GxP environments
- Regulatory Affairs Manager
- Data Scientist / ML Engineer in life sciences
- AI Systems Owner in pharmaceutical manufacturing
Operational benefits
Operational AI governance framework
Not shelf documents, but systems that hold up to inspection. Dalia IA builds AI governance frameworks that integrate Annex 22, EU AI Act, and GAMP 5 Second Edition into a coherent, implementable approach.
Model drift management
Model drift — the phenomenon by which a validated model sees its performance degrade over time — is an AI-specific risk requiring continuous monitoring. Dalia IA designs and implements formal monitoring plans as part of system lifecycle management.
Anticipatory EU AI Act compliance
Organizations already operating under GAMP 5 and Annex 11 have advantages in EU AI Act alignment, but specific gaps exist. Dalia IA supports gap analysis and construction of the AI QMS required for high-risk systems.
LLM expertise in GMP contexts
Large Language Models in critical GMP processes require specific risk analysis and robust human-in-the-loop design. Dalia IA supports organizations in LLM risk assessment and compliant workflow design.
Why AI Governance is an Engineering Issue, Not Just a Policy Matter
An AI/ML system in a regulated environment differs fundamentally from a traditional GxP computerised system. A deterministic system always produces the same output given the same inputs. An AI/ML system produces probabilistic outputs that may vary based on training data, model parameters, and the distribution of production inputs relative to the training dataset.
The Regulatory Framework
GMP Annex 22 is the first formal European GMP normative response to AI in pharmaceutical environments. Core principles include: documented and approved Intended Use; mandatory human oversight for high-impact GMP decisions; formal lifecycle management; change management for model updates and retraining; and production monitoring against validation-defined acceptance criteria.
EU AI Act Annex III classifies as high-risk a range of AI system categories including healthcare applications. Requirements include a dedicated AI QMS, fundamental rights risk assessment, complete technical documentation, EU database registration, automatic operational logging, and designated technical responsible persons.
GAMP 5 Second Edition introduces AI/ML software as an autonomous category requiring risk-based application with statistical performance metrics, enhanced vendor assessment, and periodic reviews including model drift monitoring.
Deterministic vs Probabilistic Systems: Why the Difference Is Critical in GMP
Unlike deterministic systems (ERP, LIMS, MES), ML models can change behaviour without formal code modifications — simply because production data diverges from the training set. This requires continuous performance monitoring, more complex change control, and audit trails that capture input, output, and model context for every relevant decision.
LLMs in Critical GMP Contexts: Large Language Models present a particularly complex case. Their non-deterministic, prompt-dependent behaviour and susceptibility to hallucinations require specific risk analysis and, in most cases, a robust human-in-the-loop design with mandatory review of every output before use in production.
Model Validation Lifecycle
Intended Use and User Requirements must be formally documented before any development or vendor selection. A well-defined Intended Use specifies the supported GxP process, accepted input types, produced output types, autonomy boundaries, model acceptance criteria, and operability conditions.
Change Control for AI Models requires formal management of categories that do not exist for traditional systems: retraining on new data, algorithm or weight updates, training dataset changes, and classification threshold modifications. Model drift monitoring must be defined at validation time and implemented as a formal process.
Explainability, Traceability, and Bias Assessment
Explainability must be implemented with specific, documented, and verifiable methods (SHAP, LIME, attention mechanisms, counterfactual explanations). Bias assessment must include training set representativeness analysis, disaggregated performance testing, documentation of low-performance areas, and periodic review against production data.
Our Operational Approach
Dalia IA supports organisations through a modular AI governance service including: AI Governance Gap Analysis; AI Policy Framework drafting; Model Validation Planning; Intended Use and URS support; AI Change Control Framework design; Bias Assessment; and Inspection Readiness Review.
Each deliverable is audit-ready and structured to support GMP, EU AI Act, and sector-specific regulatory inspections.
Frequently Asked Questions
What is the difference between AI validation and traditional computer system validation?
AI system validation differs from traditional CSV in the probabilistic nature of the validated system. A deterministic system always produces the same output and can be validated with acceptance tests. An AI/ML system produces probabilistic outputs that may change over time without code modifications, requiring statistical performance testing, continuous production monitoring, and a change control process specific to events such as retraining.
What does the EU AI Act require of pharmaceutical companies using AI in GMP processes?
Companies using high-risk AI systems (EU AI Act Annex III) in GMP processes must implement an AI-specific quality management system, conduct a risk assessment including fundamental rights risks, ensure transparency and complete technical documentation, register the system in the EU database before deployment, maintain automatic system logs, and designate a technical responsible person for AI oversight.
How does GMP Annex 22 integrate with existing validation frameworks (GAMP 5, Annex 11)?
Annex 22 does not replace Annex 11 or GAMP 5: it completes them. Annex 11 remains the primary framework for computerised systems in European GxP environments; GAMP 5 Second Edition provides the risk-based validation methodology. Annex 22 adds AI/ML-specific principles: lifecycle management, model change control, explainability, and human oversight. The operational framework is an integration of all three.
Can an LLM be used in a critical GMP process?
Using LLMs in critical GMP processes is technically possible but requires a very rigorous approach. Non-deterministic behaviour and the hallucination risk of LLMs require robust human-in-the-loop design, a precise Intended Use definition excluding autonomous decisions on critical data, and validation documentation demonstrating risk control. Use of LLMs for low-risk support activities (document search, non-critical drafting with mandatory human review) is more straightforwardly validatable.
What is model drift and why is it relevant in GxP environments?
Model drift is the phenomenon by which an AI/ML model that was validated and performant at deployment sees its performance degrade over time, because production data changes its distribution relative to the training dataset. In a GxP environment, model drift is not just a technical problem: it is an event that may jeopardise the system's validated state and the quality of the supported process. Model drift monitoring must be defined at validation time as part of the system lifecycle management.
Ready to govern your AI systems compliantly?
Request a preliminary AI Governance Gap Analysis. We assess your current state and define the Annex 22 and EU AI Act compliance pathway.