RAG in production: from POC to scale on VCF

A RAG POC takes a weekend. A RAG production system takes six months. The gap is not the model or the vector DB — it’s chunking that survives schema changes, freshness pipelines that don’t lose tomorrow’s tickets, eval suites that catch regressions before users do, and observability that points at the right layer when answers go wrong.

This article maps that gap on VCF and VKS. We use VCF Private AI Foundation for the runtime, but the patterns transfer. The goal is not “build the perfect RAG” — it’s “build a RAG that survives Monday morning.”

TL;DR

• Chunking and freshness are where POCs collapse — pin them before tuning the model.
• An eval suite is non-negotiable: golden queries, drift detection, alerting.
• Observability must span ingestion, retrieval, prompt, and answer — one trace ID end to end.

Chunking that survives schema changes

A POC chunker is forty lines: read the file, split every 512 tokens with a small overlap, embed, done. It demos beautifully because the demo corpus is a dozen clean PDFs. Production corpora are not clean. They are a Confluence space carrying three generations of page templates, a ticketing export where the useful content lives in custom fields, runbooks in Markdown, and vendor PDFs whose tables turn into alphabet soup once flattened to text. Fixed-window splitting treats all of that as one undifferentiated stream and destroys exactly the structure a retriever needs.

Structure-first chunking inverts the order. Parse each document into its natural units — a heading and its subtree, a ticket field, a table row with its header row re-attached, a code block kept whole — and only then pack those units into a size budget. A section that fits stays intact. A section that overflows splits at the next structural boundary down, never mid-sentence. The price is one parser per source type, unglamorous work that pays for itself the first time somebody asks a question whose answer is a table.

What bites six months later is identity. If a chunk’s primary key is a hash of its own text, any upstream change — a template migration, a renderer upgrade, someone normalising whitespace — shifts every boundary and the whole index looks new. You re-embed millions of vectors, burn a weekend of GPU time, and still cannot say which chunks genuinely changed. Derive the key from stable source identity instead, and keep the content hash beside it purely as a change detector.

# One chunk record. The id survives re-parsing; the hash detects real edits.
id: "confluence:SPACE-KB:page-41927:sec:install/prereqs:2"
source_system: confluence
document_id: page-41927
section_path: ["Install", "Prerequisites"]
ordinal: 2
content_hash: "sha256:1f0c…"          # re-embed only when this moves
embedding_model: "bge-m3@2026-04"     # index rebuilds are keyed on this
authz_scope: ["group:platform", "group:sre"]
source_updated_at: 2026-06-30T09:12:00Z

Metadata is not decoration, it is half the retrieval quality. Source system, document type, language, freshness timestamp and — critically — an authorization scope belong on every chunk, because filtering before the vector search is usually cheaper and always safer than filtering after it. This is where the vector store you picked shows its hand: pgvector gives you ordinary SQL predicates and joins against tables you already own, while Milvus and Weaviate expose their own filter engines with their own selectivity quirks. Neither is wrong; they fail differently under high-selectivity filters, so test with your real ACL distribution rather than a uniform one.

Chunk size itself deserves less agonising than it usually gets. Larger chunks carry more context and fewer of them fit in the prompt; smaller chunks retrieve precisely and fragment reasoning. Both effects are real, both are corpus-dependent, and the honest answer is that you should sweep two or three configurations against the eval set described further down rather than adopt whatever number a blog post recommends. What generalises is the shape: retrieve small, then expand to the parent section before generation.

Freshness pipelines on VKS

Every corpus has a decay rate, and it is almost never the one the business assumed. Product documentation moves quarterly; incident tickets move hourly. A single nightly job that re-indexes everything is simultaneously too slow for the tickets and wildly wasteful for the docs. Production pipelines run two paths side by side.

The incremental path is event-driven: source webhooks or change feeds land on a queue, a worker Deployment on VKS pulls batches, re-embeds only the chunks whose content hash moved, and upserts. The reconciliation path is a scheduled sweep that walks the source inventory and compares it with the index. The sweep exists because the event path will lose messages — a webhook retried into a full queue, an API outage, a source system that quietly stops emitting events for archived spaces. Without the sweep you find out months later, from a user.

Deletions are the silent killer. When a page is removed or its permissions tighten, nothing in the event stream necessarily tells you, and the orphaned chunk keeps being retrieved and cited. Write tombstones on every observed deletion, and let the reconciliation sweep hard-delete anything in the index that no longer appears in the source inventory. Treat “chunks with no live parent document” as an alerting metric, not a cleanup chore.

# Reconciliation sweep — deliberately boring, deliberately scheduled.
apiVersion: batch/v1
kind: CronJob
metadata:
  name: rag-reconcile-confluence
  namespace: rag-ingest
spec:
  schedule: "0 2 * * *"
  concurrencyPolicy: Forbid          # a slow sweep must not stack
  jobTemplate:
    spec:
      backoffLimit: 2
      template:
        spec:
          restartPolicy: Never
          priorityClassName: batch-low
          nodeSelector:
            workload: gpu-batch      # not the inference pool
          containers:
            - name: reconcile
              image: registry.internal/rag/ingest:2026.06
              args: ["reconcile", "--source=confluence", "--tombstone-orphans"]

Two VKS-specific details matter. First, batch embedding and online inference compete for the same silicon, and a re-index will happily eat the tail latency of your chat endpoint. Keep them on separate node pools, or on separate slices if you followed the GPU pooling approach, and give the batch workers a lower priorityClassName so preemption goes the right way when capacity gets tight. Second, changing the embedding model invalidates the entire index — vectors from two models are not comparable, no matter how similar the names look. Handle it as a blue/green: build a second collection alongside the live one, dual-write during the transition, cut over only when the eval suite says the new index is at least as good, and keep the old collection until you are sure.

Finally, give freshness a number. Corpus lag — the elapsed time between a source document changing and the change being searchable — is the one metric that makes this whole layer legible to non-engineers. Publish a p95 per source system, alert when it breaches, and you will stop having arguments about whether “the assistant is out of date.”

Evals: golden queries and drift

You cannot improve what you do not measure, and RAG is unusually good at hiding regressions. A change that improves the average answer while quietly breaking one document class looks like progress right up until the team that owns that class escalates.

Start with a golden set of real questions. Two to three hundred is plenty, and they must come from actual user traffic, support transcripts or interviews — not from a brainstorm, because invented questions are always better-formed than real ones. For each entry, record the expected answer and, more importantly, the source documents a competent human would cite. That second field is what makes cheap evaluation possible.

Then run evaluation at two levels, because they catch different failures and cost wildly different amounts.

Layer	What it catches	How it is scored	Cadence
Retrieval	wrong or missing chunks, bad filters, index drift	recall@k and MRR against expected sources — deterministic, no model call	every pull request
Generation	hallucination, ungrounded claims, missing caveats, tone	rubric scored by a judge model, sampled human review	nightly and pre-release
Production sampling	query drift, unanswered questions, new document classes	clustering of real queries, low-confidence and thumbs-down rates	weekly

Retrieval evaluation deserves most of your energy early. It runs in seconds, needs no GPU, produces a number that does not wobble, and in practice catches the majority of regressions — because when a RAG answer is wrong, the right passage usually was not in the context. Wire it into the same pipeline that already gates your platform changes; if you bootstrapped delivery the way the GitOps on VKS article describes, the eval job is just another required check before a chunker, prompt or retrieval-parameter change can merge.

Generation evaluation is where honesty is required. Judge models are useful and imperfect: they drift when the judge is upgraded, they reward fluency, and they are lenient about omissions. Pin the judge model and the rubric as versioned artefacts, re-score a fixed calibration subset whenever either changes, and keep a standing human review of a small random sample. Treat the judge score as a regression alarm, not as a truth claim about quality.

Drift closes the loop. Production questions migrate away from the golden set as users learn what the assistant is good at and as the business changes. Sample real queries weekly, cluster them, and promote the recurring ones that the current set does not represent. Every thumbs-down with a trace attached is a candidate golden entry — that is the cheapest source of eval coverage you will ever get.

Observability end to end

“The answer was wrong” is not an actionable report. Making it actionable requires a single trace ID that follows a request from the HTTP entry point through query rewriting, filtering, vector search, reranking, prompt assembly, generation and back out, with the intermediate state attached to spans rather than lost.

The attributes worth capturing are specific: the original and rewritten query, the filters actually applied, the chunk IDs returned with their scores, the number of chunks that survived reranking, prompt token count, model and revision, time to first token, total generation time, output token count, and whether a cache was hit. Log chunk IDs rather than chunk text — you can rehydrate the text from the store when you investigate, and you avoid duplicating potentially sensitive content into a log pipeline with different retention rules and a different audience.

On the platform side this is ordinary work you have probably already done. The Prometheus and Grafana stack on VKS covers the serving metrics; the RAG-specific additions are corpus lag per source, retrieval p95 separated from generation p95, prompt tokens per answer, GPU queue depth, and the latest eval scores exported as gauges so a dashboard can show quality and latency on the same screen. That last one changes conversations: quality stops being an anecdote.

Cost on-premises behaves differently from cost in a public cloud, and the difference is easy to get wrong. There is no per-token invoice, so the meaningful unit is GPU-seconds per answered question and, behind it, prompt tokens per answer. Prompt bloat is the dominant silent cost driver — every extra retrieved chunk lengthens prefill, inflates latency and consumes capacity you could have sold to another tenant. This is why a reranker often reduces total cost despite adding a model call: trading a small cheap model for a shorter context is usually a good trade, though how good depends entirely on your context lengths and hardware, so measure it on your own cluster before committing.

Caching pays well and is easy to get subtly wrong. An exact-match cache on normalised queries is nearly free and handles the surprising number of duplicate questions real users ask. A semantic cache is more powerful and more dangerous, because near-miss hits return plausible answers to slightly different questions. Whichever you use, the cache key must include the authorization scope and the corpus version — otherwise the first cache hit across a permission boundary becomes a data incident, and a stale entry keeps serving yesterday’s document after re-indexing. Correlating all of this with the underlying infrastructure signals is exactly what the observability foundations layer is for.

Conclusion

The uncomfortable truth about production RAG is that almost none of the hard work happens in the parts people find interesting. The model is a dependency you upgrade. The vector database is a datastore you operate. What determines whether the system is trusted a year from now is whether chunk identity survives an upstream migration, whether a deleted document actually disappears from the index, whether a bad change gets caught by a golden set before a user finds it, and whether a wrong answer can be traced to a specific retrieved chunk in under five minutes.

If you are starting today, build in this order: golden set first, retrieval eval second, tracing third, and only then tune chunking and prompts. It feels backwards — you spend the first two weeks building instrumentation instead of features — but it is the only sequence where every subsequent change is verifiable. Teams that invert it tend to spend those same two weeks later, in a worse mood, trying to work out which of eleven undocumented changes made the assistant worse.