Level 2 201 · for practitioners

Asking well, using answers

A decision model's answers are only as good as the questions. And once you have answers, how you combine them matters as much as the model. This page collects what held up across several months of a real inbox, graded blind against labeled concepts.

How the evidence was gathered

Each concept ("someone is waiting on a reply", "a payment failed") got hand-made labels, blind to any model score. Every question phrasing was graded with AUC against its own concept. AUC is the chance that a random true example scores above a random false one: 0.5 is a coin flip, 1.0 is perfect ranking.

What a decision model judged well

Nine rules for phrasing

1. Prefer a Choice whose options name the near misses. It was the best single question on every concept tested. "Deadline / event / offer / informational / none" works because each wrong answer has somewhere to go. Describe none well: with a vague none, accuracy on true negatives was 50%.
2. Exact Nouls are the best calibrated. One exact payment Noul had 88% precision at 0.5. Use them where a threshold has to mean something.
3. Abstract wording ranks but over-fires. "Involves a deadline" averaged 0.31 on negatives. It fails outright when the abstraction changes the concept: "personal" isn't "someone is waiting" (0.744).
4. Consequence and surprise framing depends on the concept. It won for hidden charges and waiting-on-reply, and lost for deadlines and payment failures. Test it; don't assume.
5. Don't decompose a concept the model already grasps. Surface-fact sub-questions lost by 0.03–0.08 AUC. Decomposition fits conjunctions of checkable facts, such as spam signals.
6. Grade every question against its own concept first. One question looked like chance against the target label but scored 0.89 against its own concept. It was a good question pointed at the wrong target.
7. Plain propositions beat custom true/false criteria. Custom criteria made one Noul fire at about 0.9 on noise and another read about 0 on invitations.
8. Batching is free. Send every question in one request with the shared state once. On a local model with a shared-prefix cache, six questions over about 10k tokens took 4.8 s instead of 16.9 s.
9. Phrasing doesn't transfer between models. Framings tuned on hosted Jev fell by up to 0.20 AUC on another model. Re-grade any phrasing ladder per model.

A worked example

Here's the shape of the contrastive Choice that won on deadlines. The near misses (an event, an offer) are named so that "there's a date in it" doesn't pull everything toward deadline:

"deadline_kind": {
  "type": "choice",
  "instructions": "What does this message ask of the reader, if anything, with respect to time?",
  "criteria": {
    "deadline":      "The reader must do something by a stated date or time, or lose something.",
    "event":         "A scheduled happening the reader may attend; nothing is lost by not acting.",
    "offer":         "A promotion or sale that expires; ignoring it costs nothing.",
    "informational": "Dates appear, but nothing is asked of the reader.",
    "none":          "No date, time or time pressure is involved at all."
  }
}

What it can't do, so code must

• Measure. Score ranks but doesn't size: Spearman 0.28 on amounts, 0.15 on time-to-deadline. Extract the parts and do the arithmetic in code.
• Know what isn't in the message: who is known, what is open, what the human has ruled. The worst misses were all of this kind. Code has to supply those facts.
• Route over a weak taxonomy. A 9-way area Choice was 60–65% top-1 on every model tried, so the taxonomy was the problem, not the model.

Using answers: features, not judges

The tempting design is to set a hand-picked threshold on each answer. The better design feeds every answer, plus the rule facts code already knows, into a small learned combiner (gradient-boosted trees). Same safety, about twice the filtering:

The same held on an unrelated public dataset. On TypeSafe's wine-review cookbook, 38 questions combined by a model predicted critic scores with an error of 1.75 points, against 2.20 for asking the score directly. And a small local model nearly matched hosted Jev when many questions were combined (+5% error), while trailing badly as a single judge (+23%).

Gates, facts and rules

As a gate, modestly

Escalating a filed message whenever any concept Choice had P ≥ 0.5 caught 1 of 2 filed actions, at a cost of 2.8 points of filtering. Useful, not magic.

Facts beat more questions for the remaining misses

• "A known person is never filed" caught 3 of 3 filed known-person messages, at a cost of 3 messages.
• The human's rulings and the list of open matters explain most of the rest. Those are facts code can supply. No model can infer them from one message.

Rules are facts, not overrides

In the email pipeline, rules ran in parallel with the model and contributed facts. Only bad input and embedded instructions (prompt injection) could force a review. Other rules could only keep mail out of the zero-token buckets, and the model's category never suppressed mail on its own. An earlier "not bulk-looking" override saved zero important messages over three evaluation sets and cost filtering.

Thresholds belong to a (model, rubric) pair

A number like 0.62 means nothing on its own. It means something for one model, one runtime and one set of questions. Changing any of them means refitting:

• Runtime: a faster prefill step on the local model shifted probabilities by up to 0.043, enough to flip 1 decision in 282.
• Tokenizer: token counts differed about 2.5× between two runtimes of the same model family.
• Backend swap: a wine combiner trained on hosted answers and fed local ones rose from 1.96 to 2.13 error. Trained on local and fed hosted, 1.86 rose to 2.12.

In an agent loop

• One flat Choice over many activities came back flat (top probability ≈ 0.2), with nothing to act on. One narrow Score per option, combined in code with explicit weights, gave a stable ranking.
• Let the model judge which target matters and let code derive the steps from data. The decision model never plans a chain.
• A decision model can choose steps on a web page. Given an accessibility snapshot, one request per page picked the page state and the next control: 25 of 25 fixtures, against a keyword baseline's 18, and 8 of 8 pages built to break keywords (baseline: 1). Code kept perception, execution and verification.