Shipping an LLM feature without shipping a liability

We shipped our first model-backed feature eighteen months ago. It worked in the demo, it worked in staging, and it worked for about six hours in production before we found the first failure mode nobody had thought to test.

Nothing was on fire. That is the point. The failures were quiet, plausible and wrong, which is a considerably harder problem than a stack trace.

The hard part was never the model call

The model call is ten lines. Everything around it is the product: what you feed it, how you know the answer is any good, what happens when it is not, and how you turn it off without a deploy.

import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=1024,
    system=SYSTEM_PROMPT,
    messages=[{"role": "user", "content": user_input}],
)
summary = response.content[0].text

That is the easy ten lines. Here is roughly what the surrounding code needs to do before this belongs in front of customers.

Write the evaluation set before the prompt

This is the single change that mattered most, and it inverts the natural order of work. Writing the eval set first forces you to say what a good answer is, in concrete cases, before you have a prompt you are emotionally attached to.

Ours started as a spreadsheet with 40 rows. It is now 600 cases in a fixture file.

@dataclass(frozen=True)
class Case:
    name: str
    payload: dict
    must_include: tuple[str, ...] = ()
    must_not_include: tuple[str, ...] = ()
    max_words: int | None = None


CASES = [
    Case(
        name="refund_over_limit",
        payload={"amount": 5000, "policy": "standard"},
        must_include=("manager approval",),
        must_not_include=("approved", "processed"),
        max_words=60,
    ),
]

If you cannot write down what a good answer looks like, you are not ready to ship the feature. You are ready to run an experiment, which is a different thing with a different audience.

Keep a path that does not involve the model

Every model-backed feature we run has a fallback that is dumber, cheaper and always available. Not as a nicety: as the thing that runs when the provider is degraded, when a response fails validation, or when the flag is off.

The fallback is three lines of heuristics. It is meaningfully worse. It has also saved us twice, and it means the kill switch is a config change rather than a rollback.

Log the inputs and the outputs from the first day

You cannot debug what you did not record, and you cannot reconstruct it afterwards. Log the prompt version, the model id, the input, the raw output, the validation result and the latency. Redact the obvious fields, set a retention window, and talk to whoever owns privacy before you turn it on.

logger.info(
    "llm.completion",
    extra={
        "prompt_version": PROMPT_VERSION,
        "model": response.model,
        "input_tokens": response.usage.input_tokens,
        "output_tokens": response.usage.output_tokens,
        "validation": result.status,
        "latency_ms": elapsed_ms,
    },
)

Version the prompt like code

A prompt change is a behaviour change. Ours live in the repository, get a version constant, go through review, and run against the eval set in CI. A pull request that moves the pass rate down does not merge.

• Prompts in version control, never in a dashboard

• A version constant logged with every call

• Eval set runs in CI, with a pass-rate floor

• The diff of a prompt change is reviewable by someone other than its author

What we would do again

1. Write the evaluation set first, and grow it from real failures rather than imagined ones.

2. Build the non-model fallback before the model path, not after.

3. Log everything from the first commit, with retention agreed up front.

4. Put a flag on it, and test that the flag actually works under load.

5. Budget for the second version. The first one teaches you what the feature should have been.

The feature is still running. It is on its fourth prompt version and its second model. The eval set is the only part of it that has never been rewritten.