Structured by design
Decisions, ready to use
Jev supports choices, boolean judgments and scores. Several questions can share one state, such as a call transcript. No prose parsing is needed.
Research brief / 17 September 2026
Small decisions.
Big leverage.
Can a specialist classifier replace a general language model for the decisions our workflows make all day?
We measured Jev, GPT-5.6 Terra and GPT-5.6 Luna on the same call-classification task.
40 synthetic callsEnglish + GermanMatched input contentOur take: promising for a focused pilot.Jev was the fastest and cheapest configuration measured here; Terra matched the most reference answers. A representative real-call evaluation should test whether those tradeoffs hold in practice.
01 / Why this matters
A decision engine
inside the workflow.
Most automation steps do not need an essay. They need a reliable answer that another system can act on.
Economics
Small costs can scale
Classify at more steps: call outcomes, routing, eligibility or review priority. The measured costs below show what this exact task cost—not what every future workload will cost.
Human experience
Less waiting to act
A quick classification can route a workflow promptly. Full transcription, databases, business rules and any human review still add time.
Call transcript → Classify outcome + intent → Apply business rules → Route / record / review
02 / Measured response time
Matched content.
Transparent differences.
120 measured calls, one Windows host and one Gateway account. Model order rotated for each transcript. These are times to a complete response or terminal failure, not time to first token. Luna call-040 timed out at 30 seconds and remains included.
What the test controls
Identical recovered request objects, unchanged transcripts and decision rules, SDK version, client host, sequential requests and a 30-second deadline. Terra and Luna request low reasoning effort; Jev has no equivalent setting.
What it does not control
Provider hardware and routing, native evaluation versus text-generation interfaces, prompt serialization and model internals. This is the speed of the tested configurations—not a universal model ranking.
03 / Measured cost
What the calls
actually cost.
Known per-request Gateway charges, including billed model output. Luna has 39 cost observations; the timed-out request has unknown usage and billing. These figures exclude audio transcription, hosting, storage and human review.
Scale this exact short-call mix
An arithmetic illustration, not a forecast for longer real calls. Luna is a lower bound because one timed-out request has unknown cost. Uses measured cost per classification request, with both questions in each request.
Jev’s live catalog rate was $0.042 per million input tokens, with output free. Terra and Luna rates and actual token usage are retained in the metadata. Equal content does not imply equal billed tokens. Provider tokenizers, typed-question processing, output structure and adapter overhead differ. We report actual usage without padding requests to force equal counts.
04 / Classification quality
Speed only matters
if the decision holds.
Two independent three-way labels: new appointment booked, and commercial sales discussion. “Unclear” is a meaningful answer when the transcript lacks evidence.
Every case counts once per model. Missing or invalid outputs count as failures. This is a deliberately small functional suite, not an estimate of real-world production accuracy or statistical superiority.
A concrete blind spot
93% probability. Still wrong.
“Ich wollte einen Termin wegen [Aufnahme bricht ab]”
In this benchmark (call-028), this incomplete German call was labeled not sales with 93% probability. The expected answer was unclear because the topic was missing.
The transcript ends before the caller explains the purpose. The 93% value is the probability assigned to the selected label, not a confidence interval or a guarantee.
Design consequence
Give uncertainty a route
Incomplete audio and an unresolved topic should be able to trigger review. A probability threshold alone can miss a confident mistake.
- Keep “unclear” separate from “no.”
- Define booking and rescheduling explicitly.
- Calibrate thresholds on separate labeled data.
- Track missed positives and false positives independently.
05 / Inspect the evidence
Open a call. See every decision.
Synthetic transcripts and frozen expected labels. Filter to see where one or more models disagreed with the rubric.
06 / Fit, tradeoffs and recommendation
A specialist with a clear job.
Where Jev fits
- Repeated decisions with a bounded answer set.
- Routing and call-outcome classification.
- Several related questions over the same state.
- Workflows where latency and unit cost matter.
- Returning probabilities for later evaluation and calibrated routing.
Where it needs help
- No audio transcription: text must already exist.
- It does not produce the narrative or investigation a general assistant can.
- High-confidence errors still occur.
- Prompt and label definitions materially affect the outcome.
- Long real calls, load, drift and failure handling still need testing.
Recommendation: a focused shadow evaluation.Keep the current production workflow in place while measuring Jev on independently labeled, authorized examples. Select it for the narrow decision tasks it demonstrably handles well.
Define the decisions
Confirm what counts as a new booking, rescheduling and commercial intent.
Use realistic calls
Anonymized, authorized samples with full lengths, ASR mistakes and mixed intent.
Run beside production
Compare outputs without letting the new model take business actions.
Choose on evidence
Review errors, business impact, coverage, response time and total workflow cost.
07 / Method and provenance
Small test. Visible boundaries.
Reference labels and scoring
GPT-6 Astra generated and reviewed the reference labels; Python calculated exact-match scores. Reference labels and rationales were withheld from every model during inference. No model graded the responses afterward.
These reference labels are not independently human-validated ground truth. Shared model-family assumptions and synthetic case design can bias the evaluation. Human review or a blinded independent-provider review is needed before treating the labels as authoritative business truth.
Dataset and decision rules
This development dataset contains 40 synthetic transcripts: 27 English and 13 German. It is not an independent held-out evaluation. All models received the same transcripts, question definitions and answer choices, verified by request hashes.
Existing appointment reschedules count as no new appointment under the provisional rubric. A new booking cancelled before the end of the call also counts as no.
Transparent execution
One warmup per model is excluded. Each model then receives 40 requests. Order rotates J/T/L, T/L/J, L/J/T. Complete-answer timing includes network, provider processing and local validation, with a 30-second deadline. Luna call-040 timed out and is retained as a failed case (39/40 usable responses). It was not replaced, and its billing is unknown. No retries were requested. Providers are pinned to TypeSafe AI for Jev and OpenAI for Terra/Luna. Jev uses a native typed evaluation API; Terra and Luna use text-generation APIs with structured object output, so native API behavior differs.
Standard retention was used for wholly synthetic data. No private client records were sent. Credentials are not embedded in this report.