Context vs. instructions: putting the right thing in the right slot

A recurring failure mode I keep seeing: instructions in the system prompt, examples in the user prompt, retrieved documents inline. Same tokens, wrong slots.

Over the last six months I have watched half a dozen teams debug the same problem. The prompt is roughly the right length, the model is roughly the right size, and the output is still wrong more often than they can tolerate. Nine times out of ten, when we sit down and look, the issue is not the words — it's the slots.

By "slots" I mean the three or four containers that a modern LLM call is actually made of: system prompt, user turn(s), retrieved context, tool schemas. Each of them behaves differently. Each of them has a different weight in the model's attention. Once you internalise this, a lot of "prompt engineering" becomes "put the right thing in the right slot."

The slots, briefly

System prompt. Persistent rules. What the assistant is, what it must never do, what the output format is. High weight, low volume. If you find yourself writing a paragraph in here, it probably belongs in the user turn.

User turn. The specific task, right now. The current question, the current input, the current constraints for this one call. This is where the model spends most of its attention.

Retrieved context. Big, task-specific knowledge that would blow the system prompt if pasted there. Docs, transcripts, database rows. This is background material, not instructions — the model treats it more like a library than like a boss.

Tool schemas. Machine-readable contracts. If the model has to call something, the schema goes here, not in prose.

The failure mode

The most common cross-wiring I see: instructions in the retrieved-context blob. Something like: "When answering, always cite the source. If unsure, refuse." pasted at the top of every retrieved document. It looks correct — the model can see it, right there. But because it's inside a big block of retrieved noise, it gets attended to with the same weight as any random sentence from any random paragraph. The model ignores it about a third of the time, and no one can figure out why.

Move the same sentence into the system prompt and adherence jumps. Not because the words are more magical — because the slot has more weight.

Words carry weight, but slots carry more.

A quick heuristic

For every sentence in a prompt I ask myself two questions:

• How often does this change? If never — system prompt. If per task — user turn. If per user or per session — either the system prompt (for a small stable amount) or an injected context block.
• Is this a rule or a fact? Rules go into the system prompt, in imperative form. Facts go into retrieved context, in declarative form. If they get mixed, the model handles both badly.

Retrieval is not "extra prompt"

The single biggest change in my mental model this year has been to stop thinking of the retrieved context as "part of the prompt" and start thinking of it as "the reading pile." The prompt asks a question; the reading pile is where the answers are. If I mix the two — put questions inside the reading pile, put facts inside the instructions — the model gets confused about which mode it's in.

Practically, that means:

• Retrieved chunks get a consistent wrapper. Same delimiter every time. Same metadata line at the top of each chunk (source, date, section).
• The system prompt refers to the retrieved chunks ("use only the sources in the <context> block below") but does not contain any of them.
• If I need the model to prefer some sources over others, I put a ranking hint in the system prompt (a rule), not in the chunk text (a fact).

A concrete before/after

Before:

[user] Here are 8 support tickets. Please summarise each into
one line with the customer name and the outcome. Do not
invent outcomes if unclear. If unclear say "unclear". Reply
in JSON. Ticket 1: ...

After:

[system]
Role: support summariser.
Output: strict JSON array of {name, outcome}.
Rules: never invent outcomes. If missing, use "unclear".

[user] Summarise the tickets in <tickets> below.
<tickets>
[1] ...
[2] ...
</tickets>

Same words, roughly. Reordered into the right slots. In my eval set, this reordering alone moved a "missing outcome hallucination" rate from about 7 percent to about 1 percent — no change to the model, no change to the retrieval.