A source-grounded RAG assistant that deflected the majority of tier-1 support tickets

Client	B2B SaaS company (client confidential — industry and architecture described with permission-safe anonymity)
Engagement	Design and build of a source-grounded RAG support assistant, via Upwork
My role	Sole architect and lead engineer — Rohan Jalil
Constraints	Every answer must cite retrieved sources or refuse; per-conversation token budget; escalations must carry full conversation context
Status	In production, answering live customer conversations
Stack	Python · LangChain · OpenAI API · pgvector · FastAPI · Postgres · Redis

01 Summary

02 The business problem

The client's support inbox received about 3,000 conversations a month, and most tier-1 questions had answers that already existed — spread across a public help centre, a stream of release notes, and internal runbooks the customers never saw. No single source was complete, and no agent could hold all three in their head.

In practice, agents copy-pasted from a shared document of saved replies. Those replies drifted out of date with every release, so customers received answers that were confident, polite, and wrong about the current product. Meanwhile a straightforward "how do I" question could sit in the email queue for hours before anyone touched it.

The goal was to answer the repetitive majority instantly and correctly, without ever letting an AI guess — and to hand the genuinely hard conversations to humans in better shape than the queue did.

03 Why was a linear chatbot insufficient?

Pointing a bare LLM chatbot at customers fails support work in four specific ways, and all four shaped this build:

• No grounding. A bare LLM invents plausible-but-wrong answers — the worst failure mode in support, because the customer cannot tell a hallucinated setting from a real one. This assistant may only answer from retrieved passages.
• No freshness. Model training data does not know last Tuesday's release. The answer source has to be the client's live documentation, synced continuously, not the model's memory.
• No citations. Support answers need to be checkable — by the customer, and by the agent reviewing an escalation. Here the model must cite the retrieved sources it used, and a validator rejects any answer that does not.
• No refusal behaviour. A chatbot always answers. A production support tool must recognise when the evidence is missing and say so — then hand over to a human rather than improvise.

04 RAG pipeline architecture

The assistant is a staged pipeline rather than a single retrieval call: each stage has one job, its own failure handling, and a measurable output. That separation is what makes the cite-or-refuse contract enforceable — the composition stage can be rejected by the guardrail stage without redoing retrieval.

# RAG support assistant — pipeline stage view
ingest/sync    help_centre + release_notes + runbooks → nightly diff sync
chunk/embed    section-aware chunking → embeddings → pgvector upsert
retrieve       hybrid search: dense (pgvector) + keyword → candidate passages
rerank         relevance rerank → top-k evidence set
compose        draft answer bound to evidence, inline source citations
guardrails     citation validator · confidence gate · token budget
deliver        respond with citations | escalate to human with full context

# contract: no evidence set → no answer. the model cites retrieved
# sources or the conversation routes to a person — never a guess

Three design decisions did most of the work:

• Cite-or-refuse as a hard contract. The composing model is instructed to cite, and a post-generation validator checks that every claim-bearing answer references passages actually retrieved for that conversation. Fail the check and the answer is discarded — the customer sees an honest handoff, not a guess.
• Hybrid retrieval from day one. Support queries are half conceptual ("how does seat billing work") and half literal (error codes, feature names, API fields). Dense embeddings handle the first kind, keyword matching the second; either alone missed real tickets in testing.
• Section-aware chunking. Chunks follow the documents' own heading structure, so a retrieved passage is a coherent instruction block with its source anchor — not an arbitrary 500-token window that splits step 3 from step 4.

05 Retrieval and memory strategy

Retrieval quality was treated as a data pipeline problem, not a prompt problem. A nightly sync job diffs the help centre, release notes, and runbooks against what is indexed: changed sections are re-chunked and re-embedded, deleted sections are removed, and each chunk carries its source URL, section anchor, and last-modified date. Stale answers were the original disease, so freshness is a first-class pipeline stage rather than a manual re-index chore.

At query time, dense and keyword results are merged and reranked, and only the top evidence set reaches the model — which keeps prompts small enough to fit the per-conversation token budget. Conversation memory is deliberately short-range: the assistant carries the current conversation's context so follow-up questions resolve correctly, but it builds no long-term profile of the customer. When a conversation escalates, that context — messages, retrieved passages, and the assistant's reasoning about what it could not ground — travels with it.

06 Human approval points

The assistant works the front of the queue; people keep authority over everything it cannot prove. Three gates make that concrete:

1. Escalation on missing evidence. When retrieval returns nothing the validator will accept, the assistant tells the customer a specialist is taking over and routes the conversation to an agent — with full context attached, so the customer never repeats themselves.
2. Release sign-off. No prompt, retrieval, or chunking change ships until the 250-question golden set has run and a human has reviewed the diff in scores and any newly failing answers.
3. Weekly drift review. Every week, sampled transcripts — resolved and escalated — are reviewed with the client's support lead. Documentation gaps found here feed back into the help centre, which improves the assistant without touching its code.

07 Production safeguards

• Citation enforcement in code. A validator checks every outgoing answer against the passages retrieved for that conversation; answers that cite nothing, or cite passages they were not given, are blocked and rerouted to escalation.
• Evaluation-gated releases. The 250-question golden set spans the real ticket distribution — billing, configuration, error codes, and questions the assistant should refuse. Releases ship only when grounded-accuracy and correct-refusal rates hold.
• Per-conversation token budget. A hard budget caps retrieval depth and composition per conversation; hitting it triggers escalation, not overspending.
• Idempotent sync with staleness alerts. The nightly sync uses bounded retries and idempotent upserts, and alerts if any source has not synced cleanly — so the index cannot silently rot.
• Drift monitoring. Escalation rate, refusal rate, and citation coverage are tracked weekly; a moving escalation rate is the early-warning signal that documentation or product behaviour has shifted.

08 Measured results

The deflection figure is an estimate with a plain basis: tickets closed without agent intervention, taken from the client's helpdesk reporting in the second quarter after launch, once the golden set and documentation feedback loop had settled. Time-to-first-useful-answer is the median across AI-resolved conversations; before launch, the equivalent wait was however long the email queue took that day — typically hours.

The quieter result is what escalation now looks like: agents open a conversation with the customer's messages, the passages the assistant considered, and its stated reason for handing over. Tier-2 work starts from evidence instead of from "could you describe the issue again?"

09 Technology stack

Pipeline	LangChain (Python) — retrieval, reranking, and cite-or-refuse composition stages
Models	OpenAI API — embedding model for dense retrieval; chat model for grounded answer composition
Retrieval	pgvector with hybrid dense + keyword search; section-aware chunks with source anchors and freshness metadata
Services	FastAPI, Postgres, Redis (caching and rate limiting), helpdesk integration via webhooks
Evaluation	250-question golden set run on every release; grounded-accuracy and correct-refusal thresholds
Observability	Per-conversation token and cost logs; citation-coverage, refusal, and escalation dashboards

---

Engagement details are published with client confidentiality preserved: the client is identified by industry only, and all figures are as measured by the client's own reporting during the engagement.