Fetch the most relevant passages from your own corpus and paste them into the prompt, so the model answers from evidence you control instead of from what it memorized.
ConceptWhat it is
RAG prompting — retrieval-augmented generation at the prompt layer — is the practice of fetching relevant text from an external source and placing it into the model's prompt before asking it to answer. Instead of relying on what the model memorized during training, you hand it a fresh, task-specific context window drawn from your own documents and instruct it to answer from that context, typically with a line like "Use the context below to answer."
It exists because language models have two stubborn gaps: they do not know your private or recent data, and they will confidently hallucinate when pushed past their training. RAG closes both by handing the model an open book: the answer is grounded in retrieved evidence you own, update, and can cite. The catch is structural: quality is capped by retrieval, so a good prompt over the wrong passages still yields a wrong answer.
How it worksThe mechanics
At query time the user's question is turned into an embedding and used to search a vector index (or a keyword or hybrid index) over your corpus; the top-matching chunks are ranked, optionally passed through a reranker, and concatenated into the prompt beneath an instruction such as "answer only from the context below and say you do not know if it is absent"; the model then generates a response conditioned on those chunks, often returning citations that point back to the source passages so each claim is traceable.
At a glanceSee it
The build-time side the runtime flow assumes — raw documents become a searchable vector index, and a skipped re-index silently ages every answer.
How top-k hits become the actual prompt — rerank, dedup, and order chunks to fit the token budget, since text buried mid-context gets ignored.
When to use itWhere it fits
- Answering over domain, private, or proprietary knowledge the base model never saw during training
- Facts change often — pricing, policies, product docs — and you need current answers without retraining
- You need traceable, citable answers where each claim can point back to a specific source passage
- Grounding is cheaper and faster to ship than fine-tuning when the goal is injecting knowledge, not behavior
When NOT to use itLimits & anti-patterns
- The task is reasoning, formatting, or style rather than facts — retrieval adds latency without adding value
- The needed knowledge already lives in the model or in a short, stable instruction you can simply hardcode
- Your whole corpus is small enough to fit in the context window — just paste it all and skip the pipeline
- You have no clean, chunkable, permissioned source to retrieve from, so retrieval would surface noise
Trade-offsAdvantages & costs
Advantages
- Grounds answers in your data and sharply reduces hallucination on in-scope questions
- Update knowledge by changing documents, not model weights — no retraining cycle
- Supports citations and auditability, which builds user trust and eases review
- Scales to corpora far larger than any single context window by retrieving only what matters
Trade-offs & costs
- Answer quality is capped by retrieval — if the right chunk is not fetched, the model cannot recover it
- Adds a retrieval hop, meaning more latency, more infrastructure, and more failure modes to operate
- Chunking strategy, embedding choice, and index tuning materially change results and need real iteration
- Over-retrieving inflates token cost and can bury the one key passage among distractors
ExampleIn the real world
A support team wants an assistant that answers questions about a product return policy that changes each quarter. They chunk the current policy PDFs and help-center articles, embed the chunks, and store them in a vector index. When a customer asks "Can I return an opened blender after 40 days?", the system retrieves the three most relevant policy passages, injects them into the prompt with an instruction to answer only from the provided text and cite the section, and the model replies with the correct 30-day window plus the opened-item exception — linking to the exact clause. When the policy is revised, the team re-indexes the new PDF and every answer reflects it immediately, with no change to the model.
ToolsHow to implement it
- LangChain and LlamaIndex — retrieval orchestration, chunking, and prompt assembly
- Vector stores such as pgvector, Pinecone, Weaviate, Qdrant, or FAISS for the index itself
- Embedding models like OpenAI text-embedding-3 or open-source options via Sentence-Transformers
- Cohere Rerank or cross-encoder rerankers to sharpen the retrieved set before it hits the prompt
Cost & effortWhat it takes
The prompting pattern itself is nearly free — a few lines of instruction and a slot for context. The real cost sits in the retrieval pipeline behind it: embedding your corpus, running and paying for a vector store, and the ongoing tuning of chunk size, retrieval depth, and reranking. Per query you pay for an embedding lookup plus the extra input tokens of the injected context, which can dominate spend if you over-retrieve. Expect most of the effort to land in data preparation and evaluation rather than in the prompt.