Prompt engineering

Prompt Engineering

The report-generation prompt didn't arrive at its current two-stage form on day one. It was rewritten repeatedly, and every rewrite was a response to a concrete failure an editor or a log actually surfaced: a hallucinated fact, a road code slipping into broadcast text, an event silently dropped because the model judged it unimportant. This chapter follows that history, not from memory but from the git record itself, and explains why each change happened using the real prompt and backend code behind it. The full text of every version referenced below, not just the excerpts quoted here, is kept in full in this workflow's own prompt_history/, for anyone who wants to read a version end to end rather than the excerpt discussed here.

To make each failure concrete rather than described from memory, the historical prompt versions were actually replayed for this chapter against realistic test records, all through the same model, so that the wording of the prompt is the only thing that changes between a "before" and an "after" example. The raw inputs and outputs are kept in full in this workflow's own prompt_examples/results.json.

Reusable Pattern

Splitting one overloaded LLM call into narrower stages, each with a smaller rulebook and a narrower job, tends to make instructions easier for a model to reliably follow than piling more rules onto a single call. Constraining the output further, having the model choose which of a few fixed sentence templates applies and fill it in, rather than composing freely, cuts down on stylistic drift and phrasing inconsistency too.

How We Improved the Prompt, Iteration by Iteration

Rather than just presenting the current prompt on its own, we think it's more informative to explain why it looks the way it does by walking through the whole process that got us here: the working version we started from, the concrete problems each version ran into once it saw real traffic records, and the change made in response to each one. The sections after this one go deeper into the mechanics of the current design; this section is the story of how we got there.

The first working version of this call was small: one short system paragraph telling the model it was writing Slovenian radio traffic bulletins, one user template listing the incidents to cover, and, from the very first commit, output constrained by OpenAI's Structured Outputs so the model could only return JSON matching a fixed schema rather than free text. On well-described input records this worked about as well as it looks like it should.

The first real problem showed up on the records that weren't well described. One of the test records replayed for this chapter is about as sparse as a real accident report gets:

street: R2-409, Krško - Šentjernej
cause: Nesreča
description: (empty)

Against the original prompt, that produced:

Na regionalni cesti R2-409 med Krškim in Šentjernejem je zaradi prometne nesreče oviran promet. ("On regional road R2-409 between Krško and Šentjernej, traffic is obstructed due to a traffic accident.")

Nothing in the input said traffic was obstructed. The model filled the gap with the most plausible-sounding consequence of an accident, stated it as fact, and even tagged the item oviran_promet. The fix, three weeks after the first commit, was a short RULES block:

RULES:
- Use ONLY information explicitly present in the input records.
- DO NOT infer or guess anything (times, delays, causes, weather, etc.).
- If data is missing, omit it or say it is unknown.

The parenthetical list isn't decorative; "times, delays, causes, weather" names the exact categories of detail the failing outputs had actually been inventing, spelled out because a blanket "don't hallucinate" hadn't been enough on its own. Replayed against that version, the exact same input produces:

Na regionalni cesti R2-409 Krško–Šentjernej je prijavljena nesreča, dodatni podatki niso znani. ("An accident has been reported on regional road R2-409, Krško–Šentjernej; further details are not known.")

Same input, but now the model states only what's actually there and marks the rest as explicitly unknown instead of inventing something plausible.

That fixed invented facts, but it didn't touch how the text was worded, and once the anti-hallucination version had been running for a while, a second problem became clear: the output was accurate, but it often didn't read like something anyone would actually say on air. Road codes carried straight through from the source data, and separate facts got stitched together with a comma rather than composed into one sentence:

street: R1-204, Krško - Šutna
cause: Nesreča
description: prometna nesreča, en pas zaprt.

Na R1-204 Krško–Šutna je prometna nesreča, en pas je zaprt. ("On R1-204 Krško–Šutna there is a traffic accident; one lane is closed.")

The raw road code R1-204 survives into broadcast text, and "there was an accident" and "a lane is closed" sit side by side as two spliced facts rather than one clause. The fix this time was a much larger, deterministic rule block: numbered SECTIONs listing what must never appear in output (road codes, parenthetical direction labels, database-style formatting), a canonical road-naming table, and worked WRONG/RIGHT transformation pairs for the model to match its own output against, for example:

WRONG (parenthetical): "Gorenjska avtocesta (smer: proti Avstriji)"
RIGHT (natural prose):  "Na gorenjski avtocesti v smeri proti Avstriji..."

Replaying the same accident record against that version:

Na cesti Krško–Šutna je zaradi prometne nesreče zaprt en pas. ("On the Krško–Šutna road, one lane is closed due to a traffic accident.")

The road code is gone, and the sentence is now one cause-leads-to-effect clause instead of two facts glued together.

That should have been the fix, but a deterministic rule block that big comes with its own cost: by this point the system message ran past a hundred lines of numbered SECTIONs, a worked-example table, and a mandatory event-ordering list, and a single call didn't reliably apply all of it, every time, on top of everything else it was doing. The ordering rule states plainly that a wrong-way-driving report must be listed ahead of every other event type. Replaying the same four-record batch used above, which included one, against this version, the wrong-way-driving item came back listed last — in the same order the input records happened to be given, as if that one line of the rulebook had simply gone unread. Nothing was misstated; a rule that was right there in the prompt was just never applied.

The same overload showed up again in a second, subtler shape: not a rule going unapplied, but the model confidently stating something it was never actually told. The clearest case of this, from the records replayed for this chapter, is a wrong-way-driving report whose only road identifier is the generic highway code H3 between Škofljica and Ljubljana:

street: H3, Škofljica - Ljubljana
cause: Vožnja v napačno smer
description: vozilo vozi v napačno smer proti Ljubljani.

Nothing in the input names the road any more specifically than that. Run twice, in two otherwise identical calls against that same rule-heavy version, the model gave two different specific answers:

Na južni ljubljanski obvoznici med Škofljico in Ljubljano v smeri proti Ljubljani vozi vozilo v napačno smer. ("On the southern Ljubljana ring road, between Škofljica and Ljubljana, in the direction of Ljubljana, a vehicle is driving the wrong way.")

Na ljubljanski vzhodni obvoznici med Škofljico in Ljubljano vozilo vozi v napačno smer proti Ljubljani. ("On the eastern Ljubljana ring road, between Škofljica and Ljubljana, a vehicle is driving the wrong way toward Ljubljana.")

Southern ring road, eastern ring road — a confident, specific, unverifiable guess, and a different one each time, for a fact the input never actually stated. This is the same failure as the very first example in this chapter, just wearing a road name instead of a traffic condition.

The fix was structural rather than one more rule: splitting the single call into two. Each half now has a narrower job and a shorter, more focused set of rules to hold in mind while doing it, which is exactly what a skipped ordering rule or an over-confident guess had been evidence against. The first stage handles exactly one record at a time, with nothing else in its context to blend it with or borrow a plausible-sounding detail from, and its instructions are explicit that for both the road class and the event type, leaving the field unset is the correct answer whenever the record doesn't clearly support a specific value, rather than something to avoid. Replaying the same wrong-way-driving record through the first stage of the split version, the road field comes back empty and the text reads:

Na hitri cesti med Škofljico in Ljubljano vozilo vozi v napačno smer proti Ljubljani. ("On the expressway between Škofljica and Ljubljana, a vehicle is driving the wrong way toward Ljubljana.")

A generic "hitra cesta" (expressway) instead of a specific, unsupported ring-road name — not because the model got better at knowing Slovenian ring roads, but because being told explicitly that leaving the field unset is an acceptable, correct answer, in a task narrow enough to leave little else to reach for, gave it a way out that the earlier, all-in-one call never offered. The second stage, meanwhile, works only from the first stage's already-normalized fragments and never sees the raw source text at all, so it can spend its whole attention budget on ordering and grouping rather than sharing that budget with sixteen other rules about phrasing and road names.

This is the version that actually held up. Splitting "read every raw record and figure out what's actually true" from "order and phrase the ones that are true" gave the first half of that job a scope narrow enough that it stopped needing to guess, and gave the second half a short enough rulebook that a mandatory instruction was less likely to get lost in it. The next section goes into exactly how that split works and what each stage's contract with the other actually looks like.

The Final Prompt

Before the split, a single call had to do three things at once: decide what each raw record meant, decide how several records on the same road should be grouped, and write the final Slovenian sentence. The rule-heavy version already required a fixed event ordering (wrong-way driving first, then full closures, then accidents on motorways, and so on down to local roads), but ordering, merging, and phrasing all happened inside one pass, which made it hard to fix one without perturbing the others.

The fix was to split this into two calls with a narrow contract between them. Stage 1 takes each raw record independently and normalizes it into semantic fields plus short Slovene phrase fragments, explicitly forbidden from ordering, grouping, or comparing records against each other:

You generate normalized traffic event items for Slovenian radio broadcast.

Use ONLY information explicitly provided in the user input.
Treat EACH input record independently.
Do NOT order, group, merge, deduplicate, or compare records with one another.
Your task is ONLY to convert each raw input record into one normalized stage-1 item.

Stage 2 receives only those normalized items, never the raw records, and its job is the opposite: order, group, merge where compatible, and write text_sl, but never invent a fact that wasn't already present in a stage-1 fragment:

You assemble final traffic reports for Slovenian radio broadcast from
already-normalized stage-1 items.

Your job is ONLY to:
- order them correctly,
- group them when appropriate,
- merge compatible items when appropriate,
- preserve traceability,
- and return the final report JSON.

This buys two things a single call couldn't. Stage 2 can compare items pairwise for grouping and duplicate-collapsing, something that's awkward to ask a model to do while it's also busy interpreting each raw record on its own terms. And stage 1's output becomes an inspectable intermediate artifact in its own right: a report keeps a record of both calls separately, so when a final sentence is wrong, it's possible to tell whether stage 1 misread the source record or stage 2 mishandled a correct fragment.

The more interesting effect of the split, though, is on consistency, and it comes from what stage 1 is actually asked to produce. It isn't asked to write a sentence at all — it's asked to fill a fixed set of narrow, separately-defined fields: lead_phrase, cause_phrase, location_phrase, direction_phrase, effect_phrase, and an optional secondary_phrase for anything that genuinely needs a second sentence. Each field has its own job and its own rule for what belongs in it, rather than being left to whatever order the model feels like composing in. Where the road name goes is a good example: a ROAD-FIRST RULE spells out exactly where it belongs and where it doesn't:

When the road is known from the input, you MUST use a road-level pattern and put
the canonical road name in lead_phrase. The junction, km marker, or micro-location
goes in location_phrase — NOT in lead_phrase. Only use a micro-location pattern
when the road itself is genuinely unknown.

Turning "write a good sentence" into "fill in these specific fields, each with its own rule" is a smaller, more repeatable task than open-ended composition, and it's a large part of why the final report reads consistently across hundreds of different records: the road name always ends up in the same structural position relative to the rest of the sentence, because there's a rule for exactly that, applied once per field rather than reconstructed from scratch every time.

Stage 1 doesn't just sort facts into these fields, it also classifies each item's effect_phrase into one of three grammatical shapes, copular ("oviran promet", following "je"), lexical ("promet poteka po enem pasu", carrying its own finite verb), or an event noun ("zastoj"), and picks a matching surface_pattern. Stage 2 then realizes each pattern with a fixed template rather than improvising:

1) road_level_copular
   "[lead_phrase] je [cause_phrase] [location_phrase] [direction_phrase] [effect_phrase]."

2) road_level_lexical
   "[lead_phrase] [cause_phrase] [location_phrase] [direction_phrase] [effect_phrase]."

NEVER write:
   "[lead_phrase] je [cause_phrase] ... [effect_phrase]"
   if effect_phrase already contains a finite verb such as "poteka".

That last rule exists because a single-call model, given a fragment like "promet poteka po enem pasu v vsako smer", would sometimes still reach for "je" out of habit, producing an ungrammatical "je promet poteka...". Naming the two effect types explicitly and giving stage 2 a template per type removed that class of error rather than describing it as one more thing to avoid.

This field-by-field structure is what actually produced the phrasing-consistency improvement seen earlier in this chapter. One of the replayed records describes roadworks obstructing traffic on the gorenjska avtocesta:

street: A2, Ljubljana - Karavanke
cause: Dela
description: dela med priključkoma Naklo in Podtabor proti Avstriji, oviran promet.

The single-call version, composing one sentence directly, produced:

Na gorenjski avtocesti so med priključkoma Naklo in Podtabor proti Avstriji dela in je promet oviran. ("On the Gorenjska motorway, between the Naklo and Podtabor interchanges toward Austria, there are roadworks and traffic is obstructed.")

two facts ("there are roadworks" and "traffic is obstructed") stitched together with "in" (and), rather than sorted into a cause field and an effect field. The two-stage version, forced to put "dela" in cause_phrase and "oviran promet" in effect_phrase and then realize them through the fixed copular template above, produced:

Na gorenjski avtocesti je zaradi del med priključkoma Naklo in Podtabor proti Avstriji oviran promet. ("On the Gorenjska motorway, traffic is obstructed due to roadworks between the Naklo and Podtabor interchanges toward Austria.")

The Structured Output Contract

From the very first working call, this prompt has been schema-constrained rather than free text. The current stage-1 schema is a strict JSON Schema passed as response_format (via the Responses API's text.format), with a couple dozen fields covering everything from the road name to each phrase fragment. One field from it, event_type, is worth looking at on its own because it shows what a schema can enforce that a prompt instruction alone can't:

"event_type": {
    "anyOf": [
        {
            "type": "string",
            "enum": [
                "napačna_smer", "nesreca", "zastoj", "pas_zaprt",
                "popolna_zapora", "napoved_zapore", "okvara", "dela",
                "vreme", "ovira", "omejitev_tovornih", "drugo",
            ],
        },
        {"type": "null"},
    ]
},

event_type is a single, mutually exclusive classification of what the record fundamentally is, independent of which road it's on: an accident, a queue, a lane closure, roadworks, a breakdown, an obstacle, and so on, with drugo as a last resort. It's deliberately a different axis from tags, which is a set of attributes that may co-occur on the same item (an accident can also be tagged oviran_promet), and from cause_phrase, which is free text for a cause distinct from the event itself. An enum field like this can't drift the way a free-text instruction can; the model either returns one of the listed values or the call fails schema validation. road_class, surface_pattern, and tags all use the same pattern. The request itself is built the same way for both stages and for both providers:

def _build_api_request(*, model_name, instructions, user_prompt, schema_bundle, reasoning_effort):
    request = {
        "model": model_name,
        "instructions": instructions,
        "input": user_prompt,
        "text": {
            "format": {
                "type": "json_schema",
                "name": schema_bundle.get("name", "traffic_report"),
                "schema": schema_bundle["schema"],
                "strict": schema_bundle.get("strict", True),
            }
        },
        "store": False,
    }
    if reasoning_effort:
        request["reasoning"] = {"effort": reasoning_effort}
    return request

Moving Judgment Out of the Prompt

No instruction inside a prompt is ever fully enforced. A model follows a system message the way someone follows a strong, well-argued suggestion, not the way a program executes a constraint, and most of the time that's close enough: a clearly worded rule gets followed closely enough to work. But some of this prompt's rules turned out to need something firmer than "the model usually does this", and figuring out which ones is really what this section is about: the parts of the model's judgment that could be replaced by a deterministic check were pulled out of the prompt and re-implemented as plain code, both because code doesn't skip a rule under load the way a model occasionally does, and because its behavior can be verified directly instead of taken on trust.

The clearest case of this is how zastoji (traffic queues) get handled. The rule-heavy version reported a zastoj only "if delay ≥ 10 min", a threshold stated inside a wall of prose the model had to remember and apply consistently, correctly, across every zastoj in every batch, call after call. That threshold no longer lives in the prompt at all. It's now a plain deterministic check that runs before the LLM ever sees the records: a low-significance zastoj is dropped from the record set at the source, so stage 1 is never even shown it and never has to decide anything about it:

def filter_zastoj_records(records, *, ah_min_delay_minutes, grl_min_length_km):
    """
    Drop low-significance "zastoj" records per editorial policy. Records
    whose cause is not "zastoj" always pass through unchanged.
    """
    grl_min_length_m = grl_min_length_km * 1000
    kept = []
    for record in records:
        if (record.get("cause") or "").strip().lower() != "zastoj":
            kept.append(record)
            continue

        text = " ".join(p for p in (record.get("description"), record.get("additional_description")) if p)
        if _is_ah_road(record):
            minutes = _extract_delay_minutes(text)
            if minutes is not None and minutes >= ah_min_delay_minutes:
                kept.append(record)
        else:
            length_m = _extract_length_meters(text)
            if length_m is not None and length_m >= grl_min_length_m:
                kept.append(record)
    return kept

The motivation was reliability, not accuracy: a numeric cutoff like "at least 10 minutes" is exactly the kind of judgment a model can't be guaranteed to apply the same way on every record, in every call, especially once a batch runs to dozens of records at once. Moving it into code trades a suggestion the model might follow for an enforcement it can't skip, which is what a threshold like this actually needs in order to be stable. The prompt text itself reflects that move in stages: an intermediate version still described a specific delay and length threshold in its own text even after filtering had started moving to code, while the current version no longer states a number at all, since the threshold is now admin-configurable rather than hardcoded anywhere.

The same principle shows up again in how stage 1 handles a record it can't turn into a sentence at all, through a boolean field every stage-1 item carries, usable. When a record is structurally impossible to turn into a broadcast sentence, too little information, nothing coherent to say, stage 1 sets usable = false, leaves the semantic and phrase fields empty, and gives a short machine-readable omit_reason instead of trying anyway. That item never becomes a candidate sentence in stage 2; only its source id carries forward, so it can still be accounted for in the final report's bookkeeping even though it produced no text. Unlike the zastoj threshold, though, this judgment can't be made fully deterministic, because "is there enough here to say anything coherent" isn't a number to check, it's a read on the record's content. This is the one place in the pipeline where the model is still explicitly allowed to drop something, which is exactly why it's had to be constrained the most carefully: dropping an item is easy to get away with silently, and easy to justify after the fact with something that sounds like an editorial judgment call rather than a genuine structural gap in the input.

Since every usable event is now supposed to reach the final report by default, silently dropping one during stage 2's grouping and merging step became a real risk worth guarding against explicitly, not just asking the model not to do. The prompt gained an explicit coverage invariant that stage 2 must satisfy for every stage-1 item:

You MUST process every supplied stage-1 item.

For every stage-1 item where usable = true, its source_ref MUST appear exactly
once in either:
  1. one items[].source_refs list, if the item is included in the final report, or
  2. unmatched_inputs, only if a specific editorial omission rule below allows omission.

Default behavior: INCLUDE every usable item.
Do NOT omit a usable item merely because the report is long.
Do NOT omit queues/zastoji on avtoceste, hitre ceste, or major roads.

That's quoted from the current prompt version, which also adds an explicit rule that queues on motorways and expressways must never be silently dropped, a direct response to real under-reporting once every usable event started needing an accounted-for outcome. But asking, however forcefully, is not the same as guaranteeing, and this is the one invariant important enough that the backend checks it independently of what the model claims. ensure_coverage, run after every stage-2 call and again after any post-processing, recomputes the set of source ids actually used across the final items, and reconciles it against the full input set without trusting the model's own bookkeeping:

def ensure_coverage(report, records):
    all_ids = set(_valid_input_ids(records))
    items = report.get("items") or []

    used_ids = set()
    for item in items:
        used_ids.update(str(x) for x in (item.get("source_refs") or []))

    unmatched_ids = set(str(x) for x in (report.get("unmatched_inputs") or []))
    used_ids &= all_ids
    unmatched_ids &= all_ids

    missing = all_ids - used_ids - unmatched_ids
    if missing:
        unmatched_ids |= missing

    report["unmatched_inputs"] = sorted(unmatched_ids) if unmatched_ids else None
    ...
    return report

Anything the model forgot to place, whether in an item or in unmatched_inputs, gets swept into unmatched_inputs here rather than vanishing. post_process_report runs a companion check in the other direction: if an item's text_sl fails a plain-text sanity check (too short, matches a known vague pattern like "na cesti X omenjene ovire"), the whole item is dropped and its source_refs are moved to unmatched_inputs instead of letting a broken sentence reach broadcast. The same file also does mechanical cleanup the prompt would rather not spend rule-budget re-explaining every call, stripping any road code that slipped through (ROAD_CODE_PATTERN), collapsing "(smer: ...)" labels back into prose, removing meta-commentary placeholders like "(ni podatkov)". None of this replaces the prompt's own prohibitions on this kind of text; it's the backstop for the fraction of cases where a strict schema and a well-worded rule still weren't enough.

Put together, this is the whole shape of the usable story: the model is allowed to say a record can't be turned into a sentence, but not to say a record isn't worth including, the numeric judgment behind that distinction was moved into code once it became a threshold rather than a coherence check, and even the one judgment the model is still allowed to make gets independently re-checked against the full input set afterward rather than trusted at face value.

Why This Matters, and What's Next

Read across this whole history, this prompt's evolution is really the history of figuring out which failures a well-written instruction can reliably prevent, and which ones need a schema constraint, a deterministic filter, a structural split, or a post-hoc check in code instead. Numeric thresholds like the zastoj significance cutoff moved out into config-driven backend code. Structural guarantees like coverage stayed described in the prompt for the model's benefit, but got independently re-verified in code because a broadcast system can't afford to just trust an LLM's self-report. And the wrong-way-driving road-naming example earlier in this chapter is a reminder that even a genuine architectural fix, splitting one overloaded call into two focused ones, doesn't retroactively guarantee every narrower rule survived the move intact.

The exact prompt text at each stage of this evolution is kept in full in prompt_history/, so future prompt changes can be judged against what was actually tried before rather than institutional memory. The next two chapters build directly on this design: Report Generation covers how both the cloud and self-hosted halves of the provider fallback are actually used in production, and Local Generation Model covers how the self-hosted model that now serves this same two-stage contract as the primary path was trained. Model Comparison & Evaluation puts both paths side by side against this exact schema and these exact prompts.