How to deploy retrieval-augmented generation (RAG) in a regulated enterprise

A production-grade RAG system in a regulated environment is engineered as a sequence of governed stages, not a single pipeline. Each stage produces an artefact that an internal auditor or external supervisor can later inspect.

1. Source-corpus governance. Classify every document by sensitivity, residency and retention class before it touches the index. Maintain a written data catalogue, record provenance and lawful basis under GDPR Article 6, and exclude special categories of data unless an explicit Article 9 condition applies. Documents flagged for deletion under Article 17 must be removable from the index within the contractual erasure window.
2. Chunking and embedding strategy. Choose chunk size and overlap per document type, not globally. Regulatory text, contracts and clinical guidelines benefit from semantic chunking that respects clause and section boundaries; transactional logs benefit from fixed windows. Record the embedding model version with every vector so re-embedding can be triggered when a model is deprecated or a regulatory text is revised.
3. Vector store selection. Pick a store that runs in EU regions, encrypts data at rest with customer-managed keys, publishes a sub-processor list, and supports per-tenant or per-document access control. Verify the operator's ISO/IEC 27001 and, where available, ISO/IEC 42001 attestations [3].
4. Retrieval evaluation. Build a ground-truth set from real questions answered by subject-matter experts. Measure recall@k, mean reciprocal rank and context precision before any LLM endpoint is wired in. A retrieval layer that cannot return the right chunks will not be saved by a better generator.
5. Generation with citation enforcement. Constrain the orchestration layer so that every output cites at least one retrieved chunk and ungrounded responses are rejected or flagged for review. The Stanford CRFM literature on grounding shows that enforced citation reduces hallucination materially compared with free generation [4].
6. Human-in-the-loop review for regulated decisions. Any output that materially affects a data subject must route through a qualified reviewer before it takes effect. This is the operational reading of GDPR Article 22 and EU AI Act Article 14 and is not optional in credit, insurance underwriting, clinical triage or benefits adjudication [1][2].
7. Continuous evaluation harness. Run faithfulness, answer relevance and citation completeness tests on every release. Track drift on the embedding model, the source corpus and the regulator's published guidance. Treat regulator updates as code-pushing events: when EBA, EIOPA, ENISA or the AI Office publish, the corpus and tests update the same week [6][7].

The architecture is the same; the obligations are not. A regulated deployment has to demonstrate, on demand, how each output was produced and how each design choice was justified.

• EU AI Act Article 9 requires a documented risk management system that runs across the full lifecycle of a high-risk system, with explicit identification, evaluation and mitigation of foreseeable risks [1].
• GDPR Article 5 data minimisation applies inside the retrieval step itself. The pipeline should not fetch personal data the generator does not need; query-time filters and per-role access scopes are the correct controls.
• GDPR Article 32 security obligations cover encryption in transit and at rest, granular access control, immutable audit logs and tested incident response. Logs must be sufficient to reconstruct any single answer.
• GDPR Article 35 requires a Data Protection Impact Assessment before production deployment whenever the processing is likely to result in a high risk to data subjects, which is the default assumption for generative systems handling personal data.
• Article 22 plus EU AI Act Article 14 codify the human-oversight requirement: a competent reviewer with the authority and information to override the system must be in the loop for consequential decisions.
• Sector overlays. Banking deployments must satisfy SR 11-7 model risk governance, including independent validation and ongoing monitoring. Insurance deployments fall under EIOPA guidance and Solvency II Pillar 2 governance. Healthcare deployments must align with the Medical Device Regulation when the system informs clinical decisions, and with HITRUST or equivalent controls for protected health information [6].

Five failure modes recur across audits of production RAG systems and each is preventable with the right test in the evaluation harness.

• Silent grounding loss. A source document is updated, the old vectors stay in the index, and the system continues to cite an obsolete passage. Mitigation: every chunk carries a content hash and a source-document version, and stale vectors are evicted on ingest.
• Embedding drift on regulatory revisions. When a regulator amends a key text, the embedding of the new section sits at a different point in vector space. Mitigation: scheduled re-embedding on regulatory publication events, with a parity test against the previous index.
• Re-ranker masking citation. A re-ranker promotes a high-similarity but low-authority chunk and the citation no longer points to the document the answer actually relied on. Mitigation: rank-aware citation logging that records every chunk that influenced the final response.
• Prompt injection via uploaded documents. A user uploads a PDF whose content tries to override the system instruction. Mitigation: separate untrusted content from instructions at the orchestration layer, sanitise uploads, and follow the ENISA guidance on AI threat modelling [7].
• Cross-tenant retrieval leakage. A misconfigured filter returns chunks from another tenant or another department. Mitigation: tenant identifiers as a mandatory predicate on every query, automated tests that probe with adversarial queries, and per-tenant index isolation for the most sensitive corpora.

Evaluation in regulated settings is a continuous engineering practice, not a launch checkpoint. The minimum metrics matrix has six dimensions.

• Faithfulness. Share of generated claims that are supported by the retrieved context. Measured against a labelled ground-truth set.
• Answer relevance. Whether the response actually addresses the user's question rather than a tangent.
• Context precision and recall. Whether the retriever surfaced the right chunks and only the right chunks.
• Citation completeness. Whether every material claim in the answer points to a specific retrieved passage.
• Latency at p95. A regulated workflow that times out at the human-review step is unusable; latency budgets must include the orchestration layer, retrieval, generation and any guardrail step.
• Cost per query. Tracked per intent class so that the business can decide where to apply the system and where a deterministic lookup is cheaper.

Open-source RAG evaluation frameworks have matured to the point where regulated buyers can run these metrics in continuous integration. The frameworks differ in scoring model and reporting; the choice matters less than the discipline of running them on every release [4].