Structured Outputs and Prompts as Code

When you use ChatGPT or Claude in the browser, you read the answer with your eyes. If the format is a bit different every time, it is not a problem. You are a human. You understand.

Your backend is not a human. If your code expects JSON and the model answers "Sure! Here is the JSON you asked for:" before the JSON, your parser breaks. If one time the field is called category and another time Category, your code breaks. If the model invents a category that doesn't exist, your database has bad data.

In this post I talk about two ideas that changed how I build AI features:

1. Structured outputs: make the model answer in a format your code can trust.
2. Prompts as code: manage prompts with the same care as the rest of your code.

Part 1: Structured outputs#

The old way: "please answer in JSON"#

The first thing everybody tries is to ask nicely:

Classify this support ticket. Answer in JSON with the fields
"category" and "priority".

It works... most of the time. And "most of the time" is the problem. With thousands of requests per day, you will see:

• Text before or after the JSON.
• JSON inside a Markdown code block.
• Missing fields.
• Extra fields.
• Values that are not in your list.
• Invalid JSON (a missing comma or quote).

You can write a lot of code to clean these answers. But there is a better way.

The better way: schemas#

Modern LLM APIs let you give a JSON Schema. The model's output is then forced to follow that schema. There are two common ways to do it, depending on the provider:

• Structured output / JSON mode with a schema.
• Tool use (function calling): you define a "tool" with an input schema, and you ask the model to call it. The arguments of the tool call are your structured data.

Here is an example with Python and Pydantic. Pydantic generates the schema for you and validates the result:

from enum import Enum
from pydantic import BaseModel, Field

class Category(str, Enum):
    billing = "billing"
    technical = "technical"
    account = "account"
    other = "other"

class TicketClassification(BaseModel):
    category: Category
    priority: int = Field(ge=1, le=5, description="1 = low, 5 = urgent")
    summary: str = Field(max_length=200)
    needs_human: bool

Then you send TicketClassification.model_json_schema() to the model as the tool input schema or the output schema, and you parse the answer:

result = TicketClassification.model_validate(raw_output)

If validation fails, you get a clear error, not bad data in your database.

Always validate, even with schemas#

Some people think: "The API guarantees the schema, so I don't need to validate." I don't agree. Validate anyway, for three reasons:

1. Schemas don't check business rules. The schema says priority is an integer from 1 to 5. It can't say "billing tickets from VIP clients are always priority 4 or 5".
2. Providers and models change. Your code should not trust that a feature works the same way forever.
3. Defense in depth. Model output is untrusted input, like a form submitted by a user.

My rule: the model proposes, your code decides.

What to do when validation fails#

Plan this from the beginning. Options:

• Retry once, and include the validation error in the new prompt: "Your previous answer failed validation: priority must be between 1 and 5." Models are good at fixing their own errors with this feedback.
• Use a safe default, for example category = other and needs_human = true.
• Send it to a human queue.
• Log it with the input, so you can improve the prompt later.

Don't retry many times. If it fails twice, something is wrong with the prompt or the input.

Design tips for schemas#

Good schemas make the model's job easier:

• Use enums for closed lists. Never ask for free text when you have a fixed set of values.
• Use clear field names and descriptions. The model reads them. needs_human is better than flag2.
• Keep it flat when possible. Deeply nested objects increase errors.
• Add a "reasoning" field before the decision field when the task needs thinking. The model writes its reasoning first and then the decision, which often improves quality. You can log the reasoning and ignore it in your logic.
• Allow "unknown". If the model can't decide, give it a valid way to say so. Otherwise it will guess.

Part 2: Prompts as code#

When I started with AI features, my prompts were strings in the middle of a PHP class. Someone changed one sentence, deployed, and two days later we discovered that the quality of the answers was worse. Nobody knew which change caused it.

Prompts are not configuration text. Prompts are code. They decide the behavior of your application. So I treat them like code.

1. Keep prompts in their own files#

I store each prompt in a separate file, with a clear name and version:

prompts/
  ticket_classifier/
    v3.md
    v4.md
  post_chat/
    v2.md

Inside, I use a simple template system for variables:

You are a support assistant for a B2B fintech platform.
Classify the ticket inside <ticket> into one category.

Rules:
- If the ticket mentions an invoice, payment or refund, use "billing".
- If you are not sure, set needs_human to true.

<ticket>
{{ ticket_text }}
</ticket>

Benefits: you can read the whole prompt without searching in the code, and git shows clean diffs when someone changes it.

2. Version prompts and log the version#

Every AI call logs the prompt name and version:

{ "prompt": "ticket_classifier", "prompt_version": "v4", "model": "claude-haiku-5-5" }

Now, when quality changes, you can see if it happened after a new prompt version or a new model. Without this information, debugging AI features is almost impossible.

3. Review prompts in pull requests#

A change in a prompt goes through a pull request, like any other code. The reviewer asks the same questions as with code:

• Why is this change needed?
• Can it break other cases?
• Did you test it?

That last question takes us to the next point.

4. Test prompts with evals#

You can't write a normal unit test for a prompt, because the output is not always the same. But you can create a set of examples with the expected result and measure how many pass. This is called an eval.

CASES = [
    ("I was charged twice this month", "billing"),
    ("I can't log in after changing my password", "account"),
    ("The API returns 500 when I send a transfer", "technical"),
]

Run the cases with the old prompt and the new prompt, and compare the scores. If the new prompt is worse, don't deploy it. I explain this in detail in Testing AI Features with Evals.

5. Pin the model version#

Use specific model identifiers in production, not aliases like "latest". A new model can behave differently with the same prompt. Changing the model should be a deliberate change, with evals, not a surprise.

Common mistakes with prompts#

These are mistakes I made myself, so I can talk about them with confidence:

One giant prompt for everything. A single prompt that classifies, summarizes, translates and extracts data is hard to test and hard to fix. When one part fails, you don't know which instruction caused it. Split it into small prompts with one job each, like small functions.

Negative instructions only. "Don't be long. Don't invent. Don't use Markdown." Models follow positive instructions better: "Answer in two sentences. Use only the information in the context. Write plain text."

No examples. One or two short examples of a good answer (few-shot prompting) often improve the result more than a long paragraph of rules. But choose them carefully: the model copies their style, length and even their mistakes.

Secret logic in the prompt. If a rule is important for the business, such as "VIP clients always get priority 5", put it in code, not only in the prompt. Code is deterministic. Prompts are suggestions.

Changing the prompt and the model at the same time. If quality changes, you don't know why. Change one thing, run the evals, then change the next.

A small complete example#

This is how all the pieces fit together in a classifier adapter:

class ClaudeTicketClassifier(TicketClassifier):
    PROMPT = load_prompt("ticket_classifier", "v4")

    def classify(self, text: str) -> TicketClassification:
        raw = self.llm.structured_call(
            model=settings.CLASSIFIER_MODEL,
            prompt=self.PROMPT.render(ticket_text=text),
            schema=TicketClassification,
            max_tokens=300,
        )
        try:
            result = TicketClassification.model_validate(raw)
        except ValidationError as e:
            log.warning("classifier_invalid_output", error=str(e), prompt="v4")
            return TicketClassification.fallback()

        return apply_business_rules(result)

• The prompt is versioned and lives in a file.
• The schema defines the format.
• Validation protects the system.
• The fallback makes failure safe.
• Business rules have the final word.

And this class lives behind a port, as I explain in How to Integrate an LLM into Your Backend.

Summary#

• Don't parse free text. Use schemas (JSON Schema, tool use, Pydantic) to get structured output.
• Always validate the output. The model proposes, your code decides.
• Plan what happens when validation fails: one retry, a safe default or a human.
• Prompts are code: separate files, versions, pull requests, logs.
• Test prompts with evals before you deploy a change.

These practices are not exciting. But they are the difference between an AI demo and an AI feature that works every day in production.