The problem
A review team cannot inspect every payment. The useful question is how much fraud it can find within the number of transactions it has time to review.
How we approached it
A LightGBM model ranks transactions using payment and card-history features. Training uses earlier records; the final evaluation uses later transactions. Scores are calibrated on separate validation data, and review capacity determines how far down the ranked queue the team looks.
Choose a review capacity
Select a review budget to see the trade-off between fraud caught and legitimate transactions sent for review. These are measured results on historical data, not a live queue.
What Reviewing More Transactions Buys
Four measured review budgets on the 118,108 held-out transactions. Pick one. No line is drawn between the points: only these four were measured.
4,064 of the 118,108 held-out transactions are labeled fraud (3.44%). "Non-fraud transactions reviewed" = transactions reviewed minus true positives. From outputs/metrics_before_feature_pass.json, the baseline evaluation.
What the evaluation showed
Reviewing more transactions catches more fraud, but also sends more legitimate payments to the team. That makes the operating threshold a staffing and customer-experience decision, as well as a modeling decision.
Evaluation offline holdout; serving parity unverified
Risk scoring at a fixed review capacity.
Top 2% reviewed: 73.5% precision, 42.7% recall; 118,108 time-held-out transactions.
Offline serving-model results. Live historical-feature parity and deployment of the stronger challenger remain unverified.
Evaluation data →What still needs checking
The published results are offline measurements. Matching the live service’s historical features to the offline evaluation still needs verification. A later feature experiment improved the offline scores, but deployment of that stronger model is unverified.
Score one transaction
Choose a held-out transaction to inspect the deployed model’s response. This illustrates scoring; it does not reproduce the full historical evaluation.
Score a Transaction
Choose a held-out transaction to score with the deployed model.
Examples are real holdout transactions the model never trained on. Scores run against the live model API.
Methods, calibration and full results
Results
Measured on 118,108 transactions the model never trained on, held out by time, not by random split. Reviewing the riskiest 1% of transactions (1,181 of them) catches 26.2% of the fraud at 90.1% precision. The most any model could catch in that budget is 29.1% (1,181 reviews against 4,064 fraud cases), so the model fills 90% of the budget with real fraud. Reviewing the riskiest 2% (2,362) catches 42.7% at 73.5% precision, against a ceiling of 58.1%. The live review flag uses a cutoff fitted on validation data, not on the holdout; on the holdout it flags 1.65% of transactions at 80.1% precision and 38.5% recall. From outputs/metrics_before_feature_pass.json in the repo below, build d29088a, reported on this site as the serving model. The latest live deployment and training-versus-serving historical-feature parity require a separate verification.
One Feature Pass, Measured on the Same Holdout
- Added on Sep 19: count and mean-amount aggregates per card, per card and address, and per card and email domain; time since the card's previous transaction; amount relative to the card's mean; a card-level id from card, address, and account start day, with its counts. All fit on the training window only, with tests that the value for a transaction uses only earlier rows.
- Same split, seed, class weighting, and calibration. ROC-AUC 0.893 → 0.900, PR-AUC 0.526 → 0.553, Brier 0.0215 → 0.0206. At 1% review, recall 26.2% → 26.8% and precision 90.1% → 92.1%; at 2%, recall 42.7% → 44.4% and precision 73.5% → 76.5%.
- The two strongest features after the pass are the card id (28% of gain) and the new card-level id (23%).
- Scope: this is an offline feature comparison. The live demo and the other results on this page use the serving model shown in the table.
| Measure | Serving Model | After the Feature Pass |
|---|---|---|
| ROC-AUC | 0.893 | 0.900 |
| PR-AUC | 0.526 | 0.553 |
| Brier score (lower is better) | 0.0215 | 0.0206 |
| Recall at 0.5% review | 13.4% | 13.7% |
| Precision at 0.5% review | 91.9% | 94.1% |
| Recall at 1% review | 26.2% | 26.8% |
| Precision at 1% review | 90.1% | 92.1% |
| Recall at 2% review | 42.7% | 44.4% |
| Precision at 2% review | 73.5% | 76.5% |
| Recall at 5% review | 58.1% | 59.8% |
| Precision at 5% review | 40.0% | 41.2% |
| Live review flag: share flagged · precision · recall | 1.65% · 80.1% · 38.5% | 1.73% · 82.4% · 41.5% |
Before: outputs/metrics_before_feature_pass.json (build d29088a, serving). After: outputs/metrics.json (build 6377aa2). Same 118,108 held-out transactions.
Calibration
Transactions grouped into 10 equal-sized bins by predicted score. Predicted and observed fraud rates track each other across the ten bins, with observed rates running above predicted in the low bins. Each bin holds about 11,811 transactions. Weighted by bin size, the average gap between predicted and observed rates is 0.58 points (expected calibration error). The score is a calibrated estimate, not an exact probability.
| Bin | Transactions | Mean Predicted | Observed Fraud Rate |
|---|---|---|---|
| 1 | 11,810 | 0.088% | 0.144% |
| 2 | 11,811 | 0.145% | 0.263% |
| 3 | 11,811 | 0.161% | 0.372% |
| 4 | 11,811 | 0.177% | 0.491% |
| 5 | 11,811 | 0.272% | 0.838% |
| 6 | 11,810 | 0.658% | 0.957% |
| 7 | 11,811 | 1.071% | 1.651% |
| 8 | 11,811 | 1.745% | 2.633% |
| 9 | 11,811 | 3.763% | 3.666% |
| 10 | 11,811 | 26.049% | 23.393% |
Code
The model, the training pipeline, and the scoring service: github.com/alphan-ml/fraud-radar
Where It Runs Data, deployment and checks
fraud-score-transaction. Endpoint r3skxlusm4.execute-api.us-east-1.amazonaws.com.giggit-fraud-radar-models) at cold start./health. GitHub Actions with an AWS role; no stored keys.