All use casesApproval DeskCustomer LifecycleOperations AssistantTransaction ReviewReorder Ranking
built by giggit · transaction review

Score Every Transaction Before It Clears

A model that scores each card transaction for review, built on real payment data with a time-ordered holdout. It ranks transactions by risk instead of flagging by rule, so a fixed review budget catches more fraud per hour reviewed.

Every number on this page is copied from the pipeline's own metrics file 590,540 real card transactions · time-ordered holdout

How It Works

  1. Transactions are sorted by time and split: the model never trains on a transaction that happened after one it is evaluated on.
  2. A LightGBM classifier trains on the earlier transactions, with class weighting so the rare fraud cases count as much as the common legitimate ones.
  3. Isotonic calibration is fit on a separate held-out slice, so a score of 20% means fraud actually happens about 20% of the time at that score.
  4. A review-queue operating point turns the calibrated score into a review flag: review the riskiest share of transactions, catch a known share of the fraud.

Results

ROC-AUC
0.894
PR-AUC
0.507
Holdout Transactions
118,108
Fraud in Holdout
4,064
Precision, 1% Review
88.7%
Recall, 1% Review
25.8%
Precision, 2% Review
70.4%
Recall, 2% Review
40.9%

Measured on 118,108 transactions the model never trained on, held out by time, not by random split. Reviewing the riskiest 1% of transactions catches 25.8% of the fraud in them at 88.7% precision; reviewing the riskiest 2% catches 40.9% at 70.4% precision. From metrics.json in the repo below.

Calibration

Transactions grouped into 10 equal-sized bins by predicted score. Mean predicted and observed fraud rate line up bin by bin.

BinTransactionsMean PredictedObserved Fraud Rate
111,8100.027%0.178%
211,8110.124%0.195%
311,8110.178%0.390%
411,8110.224%0.644%
511,8110.274%0.771%
611,8100.662%1.109%
711,8111.075%1.363%
811,8111.880%2.151%
911,8114.260%3.920%
1011,81125.634%23.690%

Score a Transaction

Pick a real holdout transaction and score it against the live model.

Examples are real holdout transactions the model never trained on. Scores run against the live model API.

Code

The model, the training pipeline, and the scoring service: github.com/alphan-ml/fraud-radar

What It Does

Risk Is Not a Rule

A fixed dollar threshold treats a $30 charge and a $3,000 charge the same. The model scores each transaction on the pattern of the whole card, not one field.

The Score Means What It Says

Calibration ties the number to reality: a transaction scored at 20% is a fraud roughly 1 time in 5, checked bin by bin against what actually happened.

The Review Queue Is a Dial

Review 1% of transactions and catch a quarter of the fraud at high precision, or review 2% and catch closer to half. The team sets the dial; the model supplies the ranking.

How It Was Built

1 CLIENT SIDE Card Transactions 590,540 labeled transactions, sorted by time 2 GIGGIT BUILT Time-Ordered Split Train on the past, hold out the last 20% by time, not at random 3 GIGGIT BUILT Calibrated Model LightGBM, class weighted, isotonic calibration on top 4 OWNER KEEPS The Review Queue Set the review rate A person clears or holds each flag
Client sideCard transactions590,540 labeled transactions, sorted by time
Giggit builtTime-ordered splitTrain on the past, hold out the last 20% by time, not at random
Giggit builtCalibrated modelLightGBM, class weighted, isotonic calibration on top
Stays with the ownerThe review queueSet the review rate. A person clears or holds each flag