What is Retrieval-Augmented Generation (RAG) and why is it preferred over full fine-tuning for knowledge-intensive tasks?

Retrieval-Augmented Generation (RAG) augments an LLM's response by first retrieving relevant documents from an external knowledge source and injecting them into the prompt as context. Instead of relying solely on knowledge baked into model weights during training, the LLM reasons over dynamically fetched, up-to-date, and verifiable text passages.

RAG is preferred over full fine-tuning for knowledge-intensive tasks for several practical reasons: fine-tuning requires substantial labeled data, significant compute, and retraining whenever the knowledge base changes; RAG's knowledge can be updated instantly by changing the document store. RAG also reduces hallucination — the model is grounded in retrieved text it can cite — and enables attribution of answers to specific sources.

# Conceptual RAG pipeline (full implementation in Q08)
# 1. INDEX: chunk documents, embed each chunk, store in vector DB
# 2. RETRIEVE: embed user query, find k nearest chunks by cosine similarity
# 3. GENERATE: inject retrieved chunks as context, call LLM

SYSTEM = (
    'You are a helpful assistant. Answer the user question using ONLY '
    'the context provided below. If the answer is not in the context, '
    'say you do not know. Always cite the source document.\n\n'
    'Context:\n{context}'
)

def rag_answer(query: str, retrieved_docs: list[dict]) -> str:
    context = '\n---\n'.join(
        f"Source: {d['source']}\n{d['text']}" for d in retrieved_docs
    )
    from openai import OpenAI
    client = OpenAI()
    resp = client.chat.completions.create(
        model='gpt-4o-mini',
        messages=[
            {'role': 'system', 'content': SYSTEM.format(context=context)},
            {'role': 'user',   'content': query},
        ],
        temperature=0.2,
    )
    return resp.choices[0].message.content