All use casesOperations AssistantApproval DeskTransaction ReviewCustomer LifecycleResearch AgentModel BenchReorder Ranking
live checks · every six hours

A Scheduled Check on Five Live Systems, Every Six Hours

Every six hours a job sends five live systems a fixed set of reference inputs with the known answers: Model Bench, AREA, Fraud Radar, Reorder Radar, and Buyer Value Radar. The job scores the reply, compares the score with the one recorded when the model shipped, and appends one line to a public ledger. Operations Assistant and Approval Desk carry a receipt on every run instead. Every figure on this page is read from the ledger files the job writes.

Every line on this page is read from a ledger Fixed reference inputs we hold out of our training · the same endpoints the pages use Loading…

1Where Each System Stands

2The Latest Check, Replayed

$ giggit live-eval replay --latest▌

Each block is one check: the command, the release it hit, how many records it scored, the score against the recorded one, the timing. MATCH means the live score sits inside the stated tolerance of the recorded score. The newest check is replayed first.

3The Ledger

When (UTC)SystemReleaseRowsMetricRecordedObservedToleranceResultp50Errors
Loading…

The last 60 checks across all systems. Each system's full ledger is the ledger branch of its repository: fraud-radar · buyer-value-radar · reorder-radar · model-bench · area.

4How a Check Works

  1. A scheduled job in the system's own repository runs every six hours (GitHub Actions, cron). It calls the same public endpoint the demo page calls: AWS Lambda behind API Gateway for the three scoring models, the site's own functions for Model Bench and AREA.
  2. It sends a fixed set of records the model never trained on, with their known answers, to the same endpoint this website uses.
  3. It scores the replies with the same measure the model shipped with (ROC-AUC, WAPE, NDCG@10, accuracy, correct count) and compares the score with the recorded one, inside a stated tolerance.
  4. It appends one line to a public ledger. This page reads the five ledgers and replays the latest check. An error from the endpoint is written down as an error.