All use casesThree Dials
built by giggit · model bench

Compare Models Before You Pick One

A bench that runs the same banking questions through several hosted models and reports accuracy, cost, and speed side by side, on real data.

Every number on this page is copied from results.json 3,080 real Banking77 messages · 3 hosted models

How It Works

  1. The same prompt goes to every model: sort one banking message into one of 77 categories, with a confidence score.
  2. Each model runs against the full Banking77 test set: 3,080 messages, not a sample.
  3. Accuracy is scored two ways: exact match on the fine category (77 options) and match on the coarse group (10 options) the fine category rolls up into.
  4. Cost is computed from each provider's published per-token price, at the token counts each model actually used.

Results

ModelAccuracy, FineAccuracy, CoarseCost / 1k Messagesp50 Latencyp95 LatencyBrier Score
Amazon Nova Lite70.5%86.4%$0.0492498 ms3,308 ms0.2609
Meta Llama 3.3 70B72.2%86.3%$0.40251,417 ms9,465 ms0.2316
Mistral Large 240269.3%84.1%$0.44041,753 ms10,028 ms0.2403

Full Banking77 test split, 3,080 messages, run once per model against AWS Bedrock (region us-east-1, prices as of 2026-09-16). From results.json in the repo below.

Accuracy by Model
Fine (77 categories) and coarse (10 groups) accuracy, percent correct on the full 3,080-message test set.
Accuracy by Model 0% 25% 50% 75% 100% Accuracy Model 70.5% 86.4% Nova Lite 72.2% 86.3% Llama 3.3 70B 69.3% 84.1% Mistral Large 2402
Fine accuracy (77 categories)Coarse accuracy (10 groups)
Cost by Model
Dollars per 1,000 messages, computed from each provider's published per-token price and the tokens each model used.
Cost by Model $0.00 $0.10 $0.20 $0.30 $0.40 $0.50 Cost per 1k Messages Model $0.0492 Nova Lite $0.4025 Llama 3.3 70B $0.4404 Mistral Large 2402

Three hosted Bedrock models are in this run: Amazon Nova Lite, Meta Llama 3.3 70B, and Mistral Large 2402. Two Claude models are priced in the pipeline (data/prices.json) but blocked at the AWS account level today and are not included here.

Run a Message

Type a banking-support message and send it to all three models.

Live runs switch on when the hosted keys are added.

Code

The bench, the prompt, and the scoring pipeline: github.com/alphan-ml/model-bench