Short prompts, long context: a small pattern for RAG

Most RAG prompts I read are almost the opposite of what works for me: a paragraph of instructions and three tiny snippets. I've been going the other way — one crisp instruction, more context.

I've been rebuilding a small internal RAG assistant this month, and I want to write down a pattern that has worked much better for me than what I used to do. It is not clever. It is just the opposite of what I had been doing before.

What I used to do

Long, careful system prompt. Detailed instructions. Personality. "If unsure, say so." "Cite your source." "Do not speculate." Etc. Then three or four short retrieved snippets. Then the user question.

What I got: outputs that sounded professional, followed the tone rules, cited sources — and were often wrong on the facts. Because the actual answer wasn't in the snippets I had retrieved, and the model had learned to smooth over that gap.

What I do now

Short system prompt. Bigger retrieved context. Sharp dos and don'ts, right next to the retrieved block, in the same slot.

[system]
You answer questions using ONLY the <sources> block below.
Rules:
- If the answer is not in <sources>, reply exactly: "Not in sources."
- Every sentence in the answer must be traceable to one source.
- Cite as [n] where n is the source number.

[user]
<sources>
[1] {80-line chunk}
[2] {80-line chunk}
[3] {80-line chunk}
[4] {80-line chunk}
[5] {80-line chunk}
</sources>

Question: {the user question}

Three concrete changes:

• Fewer instructions, more context. The system prompt is now four lines. Almost all "guidance" is either baked into the format of the sources block or into an escape hatch ("Not in sources.").
• Bigger snippets. Instead of 20-line chunks I now use 60–100-line chunks. The retrieval is a little coarser; the model has enough surrounding text to actually understand what it's reading.
• An explicit "I don't know" phrase. "Not in sources." is a boring phrase, easy to detect in downstream code. It converts hallucinations from a quality problem into a routing problem.

Every time I've asked a RAG model to be more careful with words, I got smoother lies. Every time I've given it a bigger, cleaner reading pile, I got fewer.

What broke, briefly

Latency went up. The bigger context is more tokens to encode. Cost went up a similar amount. This was fine for my internal tool; it would not be fine for a high-volume customer-facing surface without more work.

The other thing that broke was my summary evaluation script, because I now have an easy pass-through class ("Not in sources.") that my old graders were scoring as a failure. I had to add a rule that "Not in sources." on a question we know is unanswerable is a pass. This is worth doing anyway.

A note about "confidence"

I have stopped asking models to self-report a confidence score. In my experience, self-reported confidence is nearly uncorrelated with correctness at this scale, and it lulls downstream code into trusting a number that was, essentially, made up. If I need calibration, I compute it externally — from source retrieval scores, from downstream verification steps, from a separate lightweight grader.

Try this if you are stuck

If you are debugging a RAG pipeline that "sounds right but is often wrong," try this ordering:

1. Shrink the system prompt to four or five lines.
2. Grow the retrieved chunks by 3× and cut their number in half.
3. Add an explicit "Not in sources." escape hatch, and make it a boring, exact string.
4. Re-run your eval and see what moved.

In three separate projects this ordering has been the single biggest lever. Your mileage may vary — but it costs a lunch to try.