Prompt Engineering Patterns for LLM Apps

A clever prompt can win a demo. Shipping an LLM feature to thousands of users requires something sturdier: prompts that are structured, versioned and tested like any other part of the system.

Why separate instructions, context and data?

Mixing the task description with user input invites prompt injection and unpredictable behavior. Keep a stable system instruction, wrap retrieved context in delimiters, and clearly label untrusted user data.

const prompt = [
  { role: 'system', content: INSTRUCTIONS },
  { role: 'user', content: `<<DATA>>\n${userInput}\n<<END>>` },
];

How do you constrain the output shape?

• Ask for structured JSON and validate it with a schema (Zod) before trusting it.
• Use function/tool calling so the model returns typed arguments instead of prose you have to parse.
• Always handle the case where the model returns malformed output — retry with a repair prompt.

What belongs in a system instruction?

The system prompt is your product spec expressed as prose. It earns its length only when every line changes model behaviour:

• Role and scope: what the assistant is, and just as importantly, what it must refuse.
• Grounding rules: answer only from provided context; cite it; escalate to 'I don't know' otherwise.
• Output contract: exact JSON shape or tool-call names, with field descriptions the schema validator will enforce anyway.
• Tone and length budgets: 'two sentences maximum' beats hoping the model guesses your UX constraints.
• Failure protocol: what to return when data is missing — a defined fallback string, never free-form improvisation.

How do you validate structured output?

Parse nothing by hand. Ask for JSON matching a Zod schema, validate at the boundary, and route failures into a repair loop rather than letting bad shapes flow downstream:

const Extraction = z.object({
  summary: z.string().max(280),
  priority: z.enum(['low', 'medium', 'high']),
  dueDate: z.string().date().nullable(),
});

const raw = await complete(prompt); // model output
const parsed = Extraction.safeParse(JSON.parse(raw));
if (!parsed.success) {
  // one bounded repair attempt, then surface a typed error
  const repaired = await complete(repairPrompt(raw, parsed.error));
  ...
}

Why version and evaluate every prompt?

Store prompts in code, give each a version, and run them against a fixed set of example inputs on every change. A prompt tweak that helps one case often quietly breaks three others — only an eval set catches that.

What does a production prompt review look like?

• The diff shows exactly what changed in the instruction text — no silent edits buried in config.
• Eval scores for the new version sit next to the old one: same fixed input set, same judge rubric.
• Failure cases are attached: the malformed outputs the repair path now has to survive.
• Rollback is trivial — prompts live behind a version flag, so reverting is a config change, not a deploy.

Do few-shot examples still earn their tokens?

Usually yes — but spend them on format, not knowledge. Two or three worked examples showing input→output pairs stabilise structure far better than a paragraph describing the format, especially for edge-case handling like null fields or unusual dates. Keep examples short, cover the tricky branches (not three happy paths), and rotate them through your eval set so they can't silently overfit. If you're on a modern model with structured outputs or tool calling, prefer schema enforcement over few-shot for shape — and save the examples for judgement calls, like how to prioritise when two rules conflict. Re-measure after every model upgrade: few-shot sensitivity changes between versions more than any other prompt component.

How do you keep token cost and latency sane?

Every production LLM feature eventually meets its unit economics. Three levers matter, in order of leverage:

• Route by difficulty: send the 80% of requests with predictable patterns to a smaller, cheaper model and reserve frontier models for genuine reasoning — most teams cut spend by half without touching quality.
• Cache aggressively: identical inputs deserve identical answers. Hash the full prompt (system version included) and serve cached completions with a short TTL.
• Budget context ruthlessly: every retrieved paragraph you stuff into the prompt costs latency and money; retrieve fewer, better chunks and summarise long histories before they enter context.

Track cost-per-successful-task rather than raw token spend — a feature that costs twice as much per call but resolves tickets in one turn instead of three is cheaper.

Finally, instrument the model boundary like any external dependency: log latency percentiles, token counts and repair-loop rates per prompt version. A prompt that quietly started needing two repair attempts after a provider-side model update is a production incident — and without per-version metrics, it looks like users suddenly got worse at typing.

If a prompt change can't be reviewed in a pull request and scored against examples, it isn't ready for production.

Structured inputs, constrained outputs and continuous evaluation are what separate a reliable LLM feature from one that works until it embarrassingly doesn't.