RAG for Compliance Docs: What Actually Works

You've pointed a RAG pipeline at product docs or an engineering wiki, and the answers come back close enough. Now your compliance team wants the same thing for 2,300 regulatory documents.

Test it with a question an examiner might ask: "What are our obligations under Regulation E for unauthorized electronic fund transfers?" Five passages come back. Three are from your wire transfer policy. One is from a vendor contract. None contain the answer.

The answer sits in Section 6 of your Reg E compliance manual, in a paragraph about "liability for unauthorized transfers." Your query said "obligations," but the document said "liability," and a general-purpose embedding model treats those as related but ranks other passages higher.

The gap between that demo pipeline and one that works for compliance sits in the pipeline mechanics: how you split the text, what model embeds it, and whether you can measure if the results are right.

The scale constraint

A mid-size bank's compliance library can run a few thousand documents. BSA/AML policies, SOX control descriptions, KYC procedures, vendor risk assessments, board resolutions, regulatory exam responses, internal audit reports. A single corporate KYC onboarding can generate dozens of documents across multiple jurisdictions. At a large institution, the full corpus can reach tens of thousands of pages.

A single BSA/AML policy can run dozens of pages, with a lengthy definitions section referenced by every subsequent paragraph. SOX control narratives describe processes spanning four departments and reference other controls by ID number. Regulatory exam responses cite specific findings from previous exams, which cite regulations, which cite interpretive guidance.

When an OCC examiner asks about "customer due diligence for politically exposed persons," your documents might use "enhanced due diligence," "PEP screening," "high-risk customer identification," or "heightened monitoring requirements." The semantic gap between how examiners phrase questions and how institutions write documents is where keyword search stops working.

RAG closes that gap by searching meaning instead of keywords. But the standard pipeline needs modification before it handles compliance text.

Why generic chunking fails

Most RAG tutorials split documents into fixed-size blocks, cutting every 512 tokens with 50-token overlap and embedding each chunk separately. For a knowledge base article or a product manual, that works well enough.

A cookbook where half the recipes say "for the base sauce, see page 42." Cut every page into same-sized strips, four per page, regardless of where recipes start and end. The strip with the sauce reference sits nowhere near page 42. A reader who gets only that strip finds a reference to a sauce they can never look up.

That's what fixed-interval chunking does to a BSA/AML policy. A paragraph saying "apply the enhanced due diligence procedures described in Section 4.2(c)" lands in one chunk. Section 4.2(c) lands in another. The retrieval system returns the paragraph with the cross-reference, and the LLM fills in what Section 4.2(c) might say. It fills in wrong.

A 20-row table mapping risk categories to required controls carries dense, structured information. Fixed-size chunking splits it mid-row. The resulting chunks contain cell values stripped of their column headers, which makes them useless as retrieval targets. A study of PDF parsers on financial documents found that tables spanning multiple pages are the single hardest element to preserve and that maintaining document structure correlates directly with answer accuracy.

Definitions create another failure mode. A compliance document defines "covered financial institution" on page 3 and uses the term 200 times across the next 77 pages. Every chunk that uses the term without the definition is semantically incomplete. The embedding captures the surface text but misses the meaning the definition gives it.

Chunks that carry their context

Section-aware chunking is the first fix. Instead of splitting at token boundaries, split at document structure boundaries like headings, section numbers, and numbered paragraphs. Each chunk maps to one logical unit of the document.

On examiner-style test queries, section-aware chunking alone can improve recall@5 by 15 to 20 percentage points over fixed-size chunking, consistent with Anthropic's research on contextual retrieval showing large gains from structure-aware chunking. The improvement comes from keeping logical units intact so that each chunk answers a question rather than containing a fragment of one.

Section-aware chunking alone isn't enough. Four additions close the remaining gaps.

Attaching metadata to every chunk (the document title, section path, effective date, and document version) gives the LLM context to weigh relevance and currency. A passage from a 2019 policy that's been superseded looks different from one dated 2025 when the metadata travels with it.

Storing both the chunk and a summary of its parent section prevents the LLM from misinterpreting a narrow passage out of context. Some implementations retrieve the parent chunk alongside the child, widening the context window at retrieval time rather than at chunking time.

Extracting tables as separate chunks with their full column headers and a one-sentence description works better than embedding raw cell values. Embed the description, not the cells. When the retrieval system matches the description, it returns the entire table intact.

Cross-reference resolution is the most labor-intensive improvement, and it eliminates the most common source of hallucination. Pre-process each document to build a reference map so "Section 4.2(c)" resolves to the chunk containing that section.

At query time, when a retrieved chunk contains unresolved references, pull the referenced chunks into the context automatically. This adds 50 to 150 milliseconds per lookup. The accuracy gain justifies the latency.

The embedding gap

General-purpose embedding models (OpenAI's text-embedding-3-large, Cohere's embed-v3, open-source models like BGE and E5) are trained on broad web corpora. They understand that "obligations" and "liability" are related, but they don't understand that "BSA/AML" and "Bank Secrecy Act/Anti-Money Laundering" are identical, or that "SAR" in a compliance context means "Suspicious Activity Report" and not "Specific Absorption Rate."

MTEB, the standard benchmark for embedding quality, tests retrieval across Wikipedia, news, and web data. It doesn't test for domain-specific terminology.

FinMTEB, the first finance-specific embedding benchmark, tested 15 models across 64 financial datasets. A bag-of-words baseline outperformed dense embedding models on financial semantic similarity tasks. Domain-specific models like Fin-E5 outperformed general-purpose models that rank higher on the standard MTEB leaderboard.

In financial passage retrieval tasks, domain-tuned embeddings typically gain 8 to 15 percentage points in recall@10 over general-purpose models of comparable size. The gain concentrates on queries that use abbreviations, defined terms, or regulatory shorthand, which are the patterns compliance queries rely on most.

The tradeoff is maintenance. A fine-tuned embedding model needs periodic retraining as regulations change and new terminology enters the corpus. General-purpose models get updated by their providers. For teams without ML infrastructure, adding a cross-encoder reranker on top of a general-purpose model (a second model that rescores the top 20 results using full query-passage attention) captures roughly 60 to 70 percent of the domain adaptation benefit without the fine-tuning overhead.

Evaluation without ground truth

Nobody hands you a labeled dataset of correct retrievals for compliance questions. Building one is the hardest part of the pipeline, and the part most teams skip.

Start with what already exists in your institution. Regulatory exam findings reference specific policies and procedures. If an OCC examiner cited Section 4.2 of your BSA policy in a finding, that's a labeled pair: the finding is the query, Section 4.2 is the correct retrieval target.

Audit reports work the same way. Internal questions from compliance officers, logged in your case management or ticketing system, paired with the policies they ultimately cited, are another source.

From these materials, build a test set of 100 to 200 query-passage pairs. That sounds small. It's enough to measure recall@5, catch regressions when you change chunking or embedding parameters, and compare alternatives with statistical confidence.

The metric that matters most is recall@k, which measures how many of the passages that should have been retrieved actually appear in the top k results. For compliance, k equals 5 or 10. A compliance analyst will review five passages but won't sift through fifty.

Precision matters less than recall here, and the gap between them can be stark on specialized text. Retrieval benchmarks on legal and regulatory documents consistently show precision far exceeding recall. Systems return relevant passages when they return anything, but miss most of the relevant passages in the corpus.

For compliance, that's the dangerous tradeoff. Missing a passage that contains the applicable regulatory requirement can survive undetected through to a regulatory response.

Frameworks like RAGAS can automate this evaluation on every pipeline change. A chunking parameter that improves average recall by 2 percentage points but drops recall on cross-reference-heavy queries by 10 points is a net loss in production.

What it costs to get wrong

A compliance RAG pipeline that returns wrong passages doesn't just waste an analyst's time. If the analyst trusts the system and cites a superseded regulation or an inapplicable policy section in a regulatory response, the institution faces examination risk.

OCC model risk management guidance applies to RAG pipelines used in compliance functions. The 2026 revision, SR 26-2, replaced the original SR 11-7 but carved generative AI out of its scope, creating a gap rather than a fallback. Institutions are expected to apply general governance and risk management practices to gen-AI systems pending dedicated guidance. A RAG pipeline that informs regulatory responses still needs documentation of retrieval and ranking methodology, ongoing monitoring of retrieval quality, and validation against your test set.

For every query, log what was retrieved, what was presented to the LLM, and what the LLM generated. When an examiner asks how you arrived at an answer six months from now, you need to reproduce the retrieval, even if the document corpus has changed since then. That requires versioned embeddings and timestamped chunk stores.

The economics justify the infrastructure. A compliance team spending 30 minutes per policy search saves half that time with a pipeline running at 80% recall@5. At 500 queries per month, that's 125 analyst hours recovered. The pipeline costs less than one analyst's monthly salary to run, and retrieval quality improves as the test set grows.

The chunking strategy and the embedding model are engineering decisions with measurable outcomes. Building the test set is harder than choosing either one, which is why most pipelines ship without it. Without that test set, every other pipeline improvement is a guess.