Skip to content

Glossary drawing · Bots, flows & AI

Retrieval-Augmented Generation(RAG)

Also called: Retrieval augmented generation · Grounded generation · RAG pipeline

Note 1 —A technique that retrieves passages from your own documents at question time and adds them to an LLM's prompt, so answers are grounded and citable.

Term
Retrieval-Augmented Generation
Abbreviation
RAG
Category
Bots, flows & AI
Level
Practitioner
Reviewed
Sheet no.
GLO-AUT-065
Drawn by
InfiQ Glossary

Drawing 1 · Interactive

Follow a customer question through a RAG pipeline

8 / 8
Policies & help docsPDFs, pages, sheets
Chunk + embed~300-token chunks → vectors
Vector indexvectors + metadata
Customer messagemessages webhook
Rewrite + embedstandalone question
Retrieve top-kfilter by language, rerank
Build promptrules + sources + question
LLManswers only from sources
WhatsApp replywith [1] source link
  • Policies & help docs to Chunk + embed: split
  • Chunk + embed to Vector index: upsert
  • Customer message to Rewrite + embed: text
  • Rewrite + embed to Retrieve top-k: query vector
  • Vector index to Retrieve top-k: nearest chunks
  • Retrieve top-k to Build prompt: top 5
  • Build prompt to LLM: grounded prompt
  • LLM to WhatsApp reply: cited answer

Hop 8 of 8

LLMWhatsApp replycited answer

Arrives at WhatsApp reply — with [1] source link.

Ingestion (top row) runs ahead of time; the query path runs for every WhatsApp message. Step through each hop to see what is passed along, from raw documents to a cited reply.

Hop 8 of 8: LLM to WhatsApp reply — cited answer.

What is retrieval-augmented generation (RAG)?

Retrieval-augmented generation (RAG) is a way of making a large language model (LLM) answer from your own material rather than from memory. When a customer asks a question, the system first searches a knowledge base — help articles, policies, product sheets, delivery rules — for the passages most likely to contain the answer, then places those passages in the prompt beside the question and instructs the model to answer only from them, citing where each fact came from.

A RAG system has two pipelines. Ingestion runs ahead of time: documents are cleaned and split into chunks of a few hundred tokens, each chunk is turned into vector embeddings, and the vectors are stored with metadata (source, language, product line, last updated) in a vector database or search index. The query pipeline runs on every message: embed the question, retrieve the closest chunks — often combining vector and keyword search, then reranking — assemble the prompt, and generate the reply.

RAG changes what the model sees, not the model. That is the key difference from fine-tuning, which retrains a model's weights on examples to shift its style or behaviour and is a poor way to teach facts that change. With RAG, fixing a wrong returns policy means editing one document; the next question retrieves the new text. It is the main tool for reducing AI hallucination in support bots, though not a cure: the model can still misread or ignore a retrieved passage.

Key takeaways

  • RAG retrieves relevant passages from your own content at question time and adds them to the LLM's prompt as grounding.
  • It changes what the model sees, not the model itself — unlike fine-tuning, which retrains the weights.
  • Editing a document changes the bot's answers on the next query once it is re-indexed; no retraining is needed.
  • Most bad RAG answers are retrieval failures: wrong chunks, stale documents or missing metadata filters.
  • When retrieval finds nothing relevant, the bot should say so or hand off rather than guess.

Operating sequence

How does retrieval-augmented generation (RAG) work?

  1. Prepare and chunk the content

    Clean the source documents, remove outdated versions and split them into self-contained chunks of a few hundred tokens with a small overlap, keeping headings so each chunk still makes sense on its own.

  2. Embed and index

    Convert each chunk with one embedding model and store vector, text and metadata — source URL, language, product, updated_at — in a vector index. Re-embed everything if you change the embedding model.

  3. Retrieve for each question

    Rewrite the customer's message into a standalone question if it depends on chat history, embed it, fetch the top 3–10 chunks with metadata filters such as language or product, and optionally rerank for precision.

  4. Generate a grounded answer

    Put the chunks in the prompt as numbered sources, keep the rules in the system prompt — answer only from the sources, cite them, admit when they are silent — and keep the reply WhatsApp-short.

  5. Measure and refresh

    Log which chunks fed each answer, review thumbs-down and handoff cases, and re-index when documents change. Track retrieval hit rate separately from answer quality.

Implementation

Retrieval-augmented generation (RAG) in code

MAX_DISTANCE = 0.35   # tune on your own questions
 
def answer(question: str, lang: str) -> dict:
    q_vec = embed(question)                    # same model used at ingestion
    rows = db.execute(
        """
        SELECT title, url, body, embedding <=> %s::vector AS distance
        FROM kb_chunks
        WHERE lang = %s AND archived = false
        ORDER BY distance
        LIMIT 5
        """,
        (q_vec, lang),
    ).fetchall()
    rows = [r for r in rows if r.distance < MAX_DISTANCE]
 
    if not rows:                               # nothing relevant: don't guess
        return {"text": None, "handoff": True}
 
    sources = "\n\n".join(f"[{i + 1}] {r.title}\n{r.body}" for i, r in enumerate(rows))
    reply = llm.generate(
        system=(
            "Answer the customer using ONLY the numbered sources. Cite them like [1]. "
            "If the sources do not answer the question, say you will connect them "
            "to the team. Reply in under 80 words, in the customer's language."
        ),
        user=f"Sources:\n{sources}\n\nQuestion: {question}",
    )
    return {"text": reply, "sources": [r.url for r in rows]}

Retrieve with a metadata filter, drop weak matches, and hand off when nothing relevant comes back. `<=>` is cosine distance in the pgvector extension.

Specifications

Retrieval-augmented generation (RAG): key facts and limits

Introduced
2020 research paper (Lewis et al.)
Typical chunk size
200–800 tokens with 10–20% overlap (starting point)
Chunks per answer
Commonly 3–10 after filtering and reranking
Embedding size
Commonly 384–3,072 dimensions, set by the model
What it changes
The prompt — not the model's weights
Freshness
Edit and re-index a document; the next answer uses it

Design intent

Why does retrieval-augmented generation (RAG) matter?

Support and sales questions on WhatsApp are about your business — your return window, the PIN codes you deliver to, this month's EMI offer — which no general-purpose model knows. RAG lets a bot answer them accurately and keep up as policies change, with citations an agent or auditor can check. It also keeps the context window small and the cost per reply down: instead of pasting a 200-page manual into every prompt, the bot sends the three passages that matter.

Failure modes

Common mistakes with retrieval-augmented generation (RAG)

  • Blaming the model for retrieval misses

    If the right passage never reaches the prompt, no model can answer correctly. Log the retrieved chunks and check them first whenever an answer is wrong.

  • Chunks that lose their context

    Splitting mid-table or stripping headings produces fragments like "within 10 days" with no subject. Chunk on document structure and prepend the title and section to each chunk.

  • Stale and duplicate documents

    Two versions of the returns policy in the index means the bot may cite the old one. Archive superseded documents and filter on an archived flag or a validity date.

  • Ignoring follow-up questions

    "And for sofas?" retrieves nothing useful on its own. Rewrite each message into a standalone question using the recent chat history before embedding it.

  • No path for "I don't know"

    Without a similarity threshold and an explicit instruction, the model fills gaps from memory. When nothing relevant is retrieved, say so honestly or hand the chat to an agent.

Example: A returns question in Hinglish at a Pune furniture store

A customer writes "sofa return kar sakte hai kya? 4 din pehle aaya tha". The bot rewrites it as "Can I return a sofa delivered four days ago?", retrieves the current returns policy and the large-item pickup page from the store's knowledge base, and replies in Hinglish that unused items can be returned within the policy window, with pickup booked from the order page — citing both pages as links. When the store later shortens the window for mattresses, the team edits one document and re-indexes it, and the bot's very next answer reflects the change. No model was retrained.

Recommended practice

Best practices for retrieval-augmented generation (RAG)

  • Store the source URL with every chunk and send it as a link, so customers and agents can check the answer.
  • Index Hindi and other Indian-language documents with a multilingual embedding model, or keep per-language indexes.
  • Combine vector and keyword search: order IDs, SKUs and PIN codes match better on keywords.
  • Build a golden set of 50–100 real customer questions and re-run it after every change to chunking, embeddings or prompts.

RAG FAQs

What is the difference between RAG and fine-tuning?+
RAG gives the model relevant documents at question time; fine-tuning retrains the model on examples. Use RAG for facts that change, such as prices and policies, and fine-tuning for a consistent style or output format. Many production support bots use RAG alone.
Does RAG stop hallucinations?+
It reduces them but does not eliminate them. A grounded model can still misread a passage or answer from memory when retrieval misses, so add citations, a similarity threshold and an explicit "I don't know" path.
Do I need a vector database for RAG?+
Not always. Small knowledge bases work with keyword search or a vector index inside a database you already run. Dedicated vector databases help at larger scale, with heavy metadata filtering or strict latency needs.
How big should RAG chunks be?+
A few hundred tokens with a small overlap is a common starting point. Tune it on real questions: chunks that are too small lose context, and chunks that are too large dilute relevance and inflate prompt cost.

See InfiQ pricing

Put RAG to work

InfiQ is an official Meta Business Partner. Start a 7-day free trial — no card needed — and see this running on your own number.

Start free trialBrowse all 550 terms