Evidence Measured October 2–8, 2026
Four ways compared
We ran the same harnesses against four ways of getting a typed judgment: Jev (TypeSafe, hosted), OpenAI's Decisions API, open-weight decision models you run yourself, and a small general LLM (Claude Haiku 5.5) asked to fill in the same answer shapes. Same questions, same labels, same grading.
The contenders
| Way | What we ran | Price | Kind |
|---|---|---|---|
| 1. Jev | jev-1.13.0, hosted | $0.042 / M input tokens; output free | Decision model |
| 2. OpenAI Decisions API | gpt-6-luna on POST /v1/decisions (public beta), through a Jev-shaped adapter | $0.10 / M input; output free | Decision endpoint |
| 3. Open weights | APUS-OpenJev-v1-4B (MLX 8-bit, M4 Pro) and AWS Strands Decider 2B (NVIDIA L4) | $0 per call; your hardware | Decision model |
| 4. Small general LLM | Claude Haiku 5.5, effort low, structured outputs, through a Jev-shaped adapter | $0.10 / M input, $0.50 / M output | Language model |
A note on "Luna." gpt-6-luna is two things at OpenAI: the only model behind the Decisions API, and OpenAI's cheapest general-purpose text model on Chat Completions and Responses ($0.10 in / $0.50 out per million, released September 22, 2026). Called through /v1/decisions, it behaves like a decision model: deterministic, input-only pricing. Called as a chat model, it's in the same family as Haiku 5.5, a small general LLM. We measured the Decisions endpoint.
The scoreboard
Bold is the best in each row. "—" means not run.
| Harness | Jev | Decisions API | APUS 4B | Decider 2B | Haiku 5.5 |
|---|---|---|---|---|---|
| JevBench public, 231 tasks (accuracy) | 0.866 | — | — | 0.723 | 0.970* |
| Wine, 38 questions → model (RMSE, lower is better) | 1.86 | 1.84 | 1.96 | 2.05 | 1.80 |
| Wine, ask the score directly (RMSE) | 2.31 | 2.34 | 2.83 | 2.25 | 2.01 |
| Mail: action deadline (best AUC) | 0.980 | — | — | 0.932 | 0.964 |
| Mail: payment failed or ending | 0.997 | — | — | 0.984 | 0.995 |
| Mail: someone waiting on a reply | 0.986 | — | — | 0.973 | 0.986 |
| Mail sorter: filtered at ~1 miss (of 244 actions) | 58.9% (1) | — | 46.8% (0) | 44.1% (1) | 57.1% (2) |
| Browser page chooser (12 synthetic pages) | 12/12 | 10/12† | — | 7/12 | 12/12 |
| Latency p50 per request | 0.59 s | 0.30 s | 5.5 s (Mac) | 0.40 s (L4) · 2.1 s (Mac) | 2–3 s |
| Same answer on re-ask | Yes | Yes | Yes | Yes | 33% of the time |
| Pinned model version | Yes | No | Yes | Yes | Yes |
| Cost per request, Jev = 1× | 1× | 2.4× | ≈0× | ≈0× | 5–7× |
* The JevBench public set is on GitHub, so a general model may have seen it. The wine and mail sets are private and don't share that risk. † One of the two misses was a forbidden click: it unticked "Account alerts and security notices" instead of the marketing boxes. Mail was not run through the Decisions API for this comparison. Wine RMSE is 5-fold × 3 cross-validation on the same 1,189 wines; mail AUCs are on a 564-message probe set.
What each one is
1. Jev: the reference
Strongest on real judgment where it matters most to us (mail concept detection and the sorter), fast, deterministic, pinnable, and the cheapest per token of any hosted option. Weak spots match its published jaggedness: temporal and numeric reasoning (0.27 on that JevBench family), long policies (0.63), multi-hop (0.83). Keep arithmetic and dates in code and it's hard to beat.
2. OpenAI Decisions API: a near-twin, faster, a bit pricier
- Ties Jev on wine: −0.02 RMSE with 38 questions (95% CI −0.07 to +0.02), +0.03 directly.
- Agrees with Jev more than any challenger (median per-question correlation 0.91 on Score, 0.87 on Noul), yet adds information: Jev plus Luna answers together gave 1.81, against 1.86 for Jev alone.
- Fastest hosted option measured: p50 0.30 s, 45 requests per second without a single 429.
- Accepts images in the input; Jev is text-only.
- Risks: no pinned model id (answers can move under a frozen threshold without notice), a different wire format (needs an adapter), it rejects one-option Choices, and the one forbidden click on the page chooser.
3. Open weights: free, private, and (in our tests) a step behind
- APUS-OpenJev-4B on a Mac: about 5% worse than Jev with 38 questions combined, 23% worse as a single judge. Unbiased but noisier. 5–18 s per message. As a sorter feature source it filtered 46.8% at zero misses, respectable.
- Strands Decider 2B: well calibrated (ECE 0.049) and fast on a GPU, but below Jev on every judgment task here and weakest as a feature source (oracle ceiling 15 points below APUS). It compresses degree: its Score answers cluster toward the middle. Fine for easy concepts (payment failures, waiting-on-reply) and for asking the score directly.
- Honest verdict: in early October, open weights were "mid": worth it for privacy or zero marginal cost, not for quality.
That may already be out of date. Since our runs, Strands shipped Gemma 4 sizes (the 26B-A4B reports 207/231 on JevBench, above Jev's 200), APUS added 9B and 35B-A3B sizes, and several 31B open models now score above Jev on JevBench's Capability board. We haven't run those on our harnesses yet, and the bigger ones need serious hardware: a 31B dense model wants roughly 17 GB at 4-bit and an 80 GB GPU at full precision. See Landscape and what it takes to run them.
4. Claude Haiku 5.5: the quality leader, at a price in money, time and repeatability
- Ties or beats Jev on every harness. Far ahead on JevBench (0.97) and on phrasings Jev handles badly: at or above Jev on 36 of 42 mail ladder questions. The first challenger to go 12/12 on the page chooser, and it answers the one-option Choice the decision endpoints reject.
- Best single ranker: "obligation" alone reached AUC 0.941 for ranking mail that needs action (Jev 0.799).
- Costs: 5–7× Jev per request (still under $0.001), 3–5× the latency, and answers that wobble: a third of re-asks identical, yes-probabilities moving 0.10 at p90, 13% of Choice argmaxes flipping.
- Its probabilities are numbers it writes down, not measured ones. It spreads them less than Jev (median Noul spread 0.16 against 0.22).
When to use a small LLM (Haiku, or Luna as a chat model) instead of a decision model
Reach for Haiku 5.5 or a Luna-class LLM when
- The judgment needs reasoning steps: multi-hop, long policies, dates, numbers, "probability" questions. Haiku scored 1.00 on JevBench families where Jev scored 0.27–0.76.
- Your phrasing is still rough. Haiku is far more robust to imperfect questions; a decision model rewards careful, contrastive phrasing.
- Volume is low (thousands a day, not millions) and a 2–3 s answer is fine. At $0.0004 a request, the money rarely matters.
- You want one vendor and one model family for the LLM tier and the judgment tier, or you need images or long context.
- You need a single strong ranker, not a feature set.
- The answer feeds a human or an LLM that tolerates a little noise, not a frozen threshold.
Stay with a decision model when
- A frozen threshold gates an action. You need the same answer every time.
- Volume or latency dominates: millions of decisions, an agent tick, a page step. Decision endpoints are 0.3–0.6 s and a fraction of the price.
- You're combining many questions as features. At 38 questions per item, 6× the price adds up, and the decision models already nearly match.
- You need calibrated probabilities rather than written-down numbers.
- The data must stay local: run an open-weight decision model.
A pragmatic hybrid: develop your questions against Haiku (it forgives rough phrasing and tells you whether the concept is learnable at all), then port the best phrasings to a decision model for production and re-grade them, because phrasing doesn't transfer between models unchanged. Or put Haiku one rung up: decision model first, Haiku for the watch band, a larger Claude for real escalations.
Choosing, in one table
| If you need… | Use |
|---|---|
| Best quality per dollar on bounded text judgments, pinned and deterministic | Jev |
| Lowest hosted latency, image inputs, or a second opinion that adds information to Jev | OpenAI Decisions API |
| Privacy or zero marginal cost | Open weights (re-test the October releases on your own data) |
| Robustness to rough questions, reasoning-heavy judgments, best single ranker | Claude Haiku 5.5 |
| Open-ended reading, writing, fixing the cause | Claude Sonnet, Opus or Fable, one rung up the cascade |