The question
Which model should sort incoming customer messages into the right support category? A missed category can send a customer to the wrong queue; a slow or expensive model adds friction to every request.
What we compared
Three hosted language models received the same prompt. A simpler classifier learned from 10,003 labeled training messages. All four were evaluated on the same 3,080-message Banking77 test set.
This compares two approaches: prompting a general model and training a model for a defined task. Their training differs, so the result does not establish which model is best in general.
Evaluation offline benchmark
The task-trained classifier led on accuracy.
Prompted models compared with a trained baseline on 3,080 banking messages.
Training differs across approaches. Local hosting cost and latency were not measured.
Evaluation data →Four approaches, one test set
| Approach | Correct category | Provider cost / 1,000 | Typical response |
|---|---|---|---|
| Trained classifier | 89.4% | No provider fee* | Not measured |
| Amazon Nova Lite | 70.5% | $0.05 | 498 ms |
| Meta Llama 3.3 70B | 72.2% | $0.40 | 1,417 ms |
| Mistral Large 2402 | 69.3% | $0.44 | 1,753 ms |
Correct category means one of 77 banking request types. Typical response is the median. Provider prices: September 16, 2026. *The classifier’s hosting cost and response time were not measured.
What the result means
The trained classifier led on this task. Among the hosted models, Nova Lite was the least expensive and fastest at the median, while Llama had slightly higher measured accuracy. A larger model did not automatically provide a better trade-off.
For a task with stable categories and labeled examples, a trained classifier deserves consideration. A deployment decision would also need its hosting cost, latency and performance on the intended users’ messages.
Try the hosted models
Choose a labeled message to check the answer, or enter your own to compare responses. This live demo runs the three hosted models; the trained classifier appears in the published benchmark only.
Compare a message
Live results appear here.
Methods and full benchmark
| Model | Accuracy, Fine | Accuracy, Coarse | Cost / 1k Messages | p50 Latency | p95 Latency | Brier Score |
|---|---|---|---|---|---|---|
| Amazon Nova Lite | 70.5% | 86.4% | $0.0492 | 498 ms | 3,308 ms | 0.2609 |
| Meta Llama 3.3 70B | 72.2% | 86.3% | $0.4025 | 1,417 ms | 9,465 ms | 0.2316 |
| Mistral Large 2402 | 69.3% | 84.1% | $0.4404 | 1,753 ms | 10,028 ms | 0.2403 |
| Local classifier (TF-IDF + logistic regression) | 89.4% | 94.7% | $0.0000 | not timed | not timed | not scored |
Evaluation Notes
Fine accuracy uses 77 categories; coarse accuracy uses 10 broader groups. Latency is reported at the median (p50) and 95th percentile (p95). A lower Brier score indicates better probability predictions on this evaluation.
Banking77 test split: 3,080 messages, one run per model. The local classifier used 10,003 training examples (seed 26); hosted models used a shared prompt on AWS Bedrock, us-east-1. Prices: Sep 16, 2026. Intervals use 1,000 bootstrap resamples; local fine accuracy: 88.3% to 90.4%. Local hosting cost and latency were not measured. Results apply to this task and test set. Sources: results.json and outputs/decision.json in the repository below.
Where It Runs Data, deployment and checks
/api/model-bench-run, calls AWS Bedrock (Converse API), us-east-1.data/prices.json, 2026-09-16.