credit-scoring

How it is built

Architecture overview

A credit scoring system turns what is known about an applicant or an existing borrower into a number that ranks people by the chance they will not repay, and into a reason a human can read. The number is usually a probability of default (PD) over a fixed horizon, mapped to points on a familiar scale. The best-known scale is the FICO Score, which FICO describes as ranging from 300 to 850 for most scores and as built from five groups of credit-report data: payment history, amounts owed, length of credit history, new credit and credit mix (source: myFICO, What's in your credit score). Two things make the problem unlike most machine learning. The label arrives late, because a loan has to be repaid or not repaid before the outcome is known. And the output is regulated: it has to be statistically sound, explainable per decision, checked for unfair effects and monitored for as long as it is used.

The idea that makes the system work is to separate the pipeline into stages that each have one job. Records are matched to a person, features are computed as they were known on the decision date, a model ranks risk, a calibration step makes the score mean the same probability next year, and an explanation step turns the model's arithmetic into reasons. A scorecard (binned features plus logistic regression) and a gradient-boosted model fit the same frame; the choice is between transparency by construction and accuracy plus explanation afterwards.

Parts:

• Identity and record matching — joins applicant, bureau and internal records into one person — Entity resolution, fuzzy-string-matching (no page yet)
• Point-in-time data — keeps what was known when, so a decision can be replayed — bitemporal-data (no page yet), Feature store
• Feature builder — turns raw fields into model inputs; for scorecards, bins them and replaces each bin with its weight of evidence — weight-of-evidence-binning (no page yet), Feature store
• Model — ranks risk — Linear and logistic regression, Gradient-boosted decision trees (GBDT)
• Calibration and score mapping — turns scores into probabilities that match observed default rates, then into points — Calibration
• Validation — out-of-time testing and ranking/accuracy metrics — Evaluation metrics, Time-series cross-validation (rolling origin)
• Explanation — per-decision reason codes — shapley-values-treeshap (no page yet)
• Fairness checks — compares outcomes and error rates across groups — fairness-metrics (no page yet)
• Monitoring — watches inputs, score distribution and calibration after launch — distribution-drift-detection (no page yet), Calibration

 application ─► match to person ─► features as of decision date ─► model ─► calibrated PD ─► points
 bureau file       (entity           (feature store, bitemporal)    (scorecard   (calibration)    │
 internal history   resolution)                                       or GBDT)                    ▼
                                                                                          reason codes
 outcomes (late) ──► labels ──► train / out-of-time test ──► fairness + calibration ──►  (SHAP or
        ▲                                │                      sign-off                 scorecard points)
        │                                ▼                                                    │
 monitoring (drift, matured cohorts) ◄── model registry + decision log ◄── decision ◄──────────┘

The decision log at the bottom is part of the design, not an extra: every score is stored with its inputs and the model version, so the same decision can be reproduced and explained later.

How the parts fit together

Identity and record matching

An applicant arrives as a name, a date of birth and an address typed by a person. Bureau files and internal accounts carry the same facts with different spellings, old addresses and typos. Entity resolution decides which records are the same person: it blocks to cut the pair space, scores pairs with a probabilistic model, and uses two thresholds so that near-misses go to review instead of being merged silently. Transitive closure is the dangerous step, because one false link merges two people's files and attaches their delinquencies to each other. String similarity on names and addresses (fuzzy-string-matching, no page yet) feeds the pair score. The output contract is a person id with a list of source records and a match confidence; low-confidence matches are not used as features.

Point-in-time data

Credit data is revised: a bureau corrects a record, a payment posts late, an address is fixed. A model trained on today's corrected values and then scored on what a bureau said at application time sees a different world in production. Storing both when a fact became true and when the system learned it (bitemporal-data, no page yet) lets a training row take the value as it was known on the decision date, and lets a dispute months later replay the exact inputs. The Feature store page describes the same rule for features: the training join is an as-of join, the newest value with a timestamp at or before the label time. For credit, the label time is the application date. Leakage here is quiet. A feature such as "number of accounts in collections" computed at the end of the performance window instead of at application will look excellent in validation and fail in production.

Feature builder and the scorecard path

For a scorecard, each variable is cut into bins and each bin is replaced by its weight of evidence, the log ratio of the share of good outcomes to the share of bad outcomes in the bin (weight-of-evidence-binning, no page yet). Monotone binning matters: if the weight of evidence rises with income, a regulator and an underwriter can read the model. The weights go into Linear and logistic regression, and every coefficient times a bin's weight becomes the points for that bin. Total points are the sum over variables, so the scorecard is its own explanation.

The scale is a definition. With score = offset + factor × ln(odds of being good), a factor of PDO / ln 2 makes the odds double every PDO points (points to double the odds). Lenders pick the offset and PDO once so that a score keeps its meaning across model versions.

Model: logistic regression or boosted trees

Logistic regression on weight-of-evidence features is the long-standing baseline. It is stable, needs few parameters, and its coefficients can be signed off feature by feature. Gradient-boosted decision trees (GBDT) usually rank better on tabular data with interactions and missing values, and the XGBoost documentation supports monotone constraints, which force the prediction to be non-decreasing or non-increasing in chosen features (source: XGBoost docs, Monotonic Constraints). That is how a boosted model is kept from learning "more debt, lower risk" in a sparse corner of the data. The trade is that the explanation is no longer in the coefficients and has to be computed.

Contract with neighbours: the model takes a fixed-order feature vector and returns a raw score; it does not decide, it does not know the cut-off, and it does not map to points. Keeping those out of the model lets the business change the cut-off without retraining.

Calibration and score mapping

A model can rank well and still output wrong probabilities. Lenders price and reserve with the probability, so it has to be right. Calibration fits a monotone map from raw score to observed default frequency on held-out, out-of-time data (Platt or isotonic scaling). The calibrated PD then goes through the points scale. After launch the same check is repeated on every cohort that has had time to mature: predicted PD against observed default rate per score band. Drift in calibration is the earliest reliable sign that a model is ageing.

Validation

The validation set must look like the future, not like a random sample of the past. Time-series cross-validation (rolling origin) trains on earlier vintages and tests on later ones, never the reverse; leaving a gap equal to the performance window prevents label overlap. Report ranking and accuracy metrics (AUC, average precision, log loss) with paired confidence intervals when comparing two models, plus the calibration table. Credit teams also quote Gini (2 × AUC − 1) and the Kolmogorov-Smirnov statistic; both come from the same ROC curve.

Explanation

When a lender declines an application in the US, Regulation B (12 CFR 1002.9) requires a notice that includes either a statement of specific reasons for the action or a disclosure of the applicant's right to request one within 30 days (source: 12 CFR 1002.9). For a scorecard the reasons are the variables where the applicant lost the most points against a reference. For a boosted model, shapley-values-treeshap (no page yet) attributes the prediction to features; the TreeSHAP algorithm computes exact Shapley values for tree ensembles in polynomial time (Lundberg et al.), where the generic definition is exponential in the number of features. The top contributions toward a decline become the reason codes. The contract is that reasons must come from the model that actually made the decision, at the same version and on the same inputs. Regulation B also defines an empirically derived, demonstrably and statistically sound credit scoring system (12 CFR 1002.2(p)), which is why validation evidence is kept with the model.

Fairness checks

Laws against discrimination in credit exist in many jurisdictions; the US example is the Equal Credit Opportunity Act and Regulation B. A model that never sees a protected attribute can still reproduce it through proxies such as postcode. fairness-metrics (no page yet) compare approval rates, false negative rates and calibration across groups; Hardt et al. define equalized odds and equality of opportunity as criteria on the error rates. The criteria can conflict when base rates differ between groups, so the choice is a policy decision recorded with the model, not a tuning knob.

Monitoring

distribution-drift-detection (no page yet) compares the distribution of each input and of the score now against the training window. The population stability index is the customary statistic; the usual rules of thumb read below about 0.1 as stable and above about 0.25 as a large shift, but these are industry conventions, not standards, and should be calibrated per portfolio. Input drift says "look"; calibration on matured cohorts says "retrain".

Data and control flow

Building and approving a model

1. Pull applications for a past period and their repayment outcomes from the loan ledger. The label is defined once, for example "90 or more days past due within 12 months of booking" (a convention, not a rule). Applications without an outcome yet are excluded and the exclusion is noted.
2. Match each application to bureau and internal records (Entity resolution); compute every feature as of the application date from the bitemporal store.
3. Bin and transform features, fit the scorecard or the boosted model on the oldest vintages, tune on the middle, test on the newest (Time-series cross-validation (rolling origin)).
4. Calibrate on the test vintage (Calibration); choose the points scale.
5. Compute reason codes on the test set and check that they match intuition and the monotone constraints.
6. Run fairness metrics by group; document the result.
7. Store the model, the binning table, the calibration map and the validation report as one versioned bundle; deploy.

Time goes to waiting for outcomes and to data work, not to fitting.

Scoring one application (online)

1. The application arrives with an id and an idempotency key so a retry does not score twice.
2. Entity resolution links the applicant to a person id; the bureau file is fetched (the slowest step, a network call to a third party; order of 100 ms to seconds).
3. Features are built from the bureau file, the application and the online feature store.
4. The model returns a raw score; calibration maps it to a PD; the scale maps it to points. These are microseconds to a millisecond for a scorecard or a few hundred trees.
5. Reason codes are computed.
6. A decision rule (cut-off plus policy rules, see the Loan origination system page) returns approve, decline or refer, and the full record (inputs, versions, outputs, reasons) is written to the decision log.

Monthly monitoring

1. Compute input and score drift against training.
2. For every cohort old enough to have an outcome, compare predicted PD with observed defaults per score band.
3. Recompute fairness metrics on recent decisions.
4. Raise a ticket if any check passes its threshold; retraining is a new run of the first flow, not an edit.

Scaling

Limits, in the order they bite:

1. Label delay. A 12-month outcome window means a new risk signal takes a year to show up in training labels. Hardware cannot fix this. Levers: shorter proxy labels (early delinquency) as a cross-check, vintage-by-vintage analysis, and honest monitoring of matured cohorts.
2. Selection bias. Only approved applicants have outcomes, so the training sample is a filtered population. The lever is reject inference (reject-inference, no page yet), and a small controlled amount of approvals outside the cut-off where policy allows.
3. Third-party calls. Bureau pulls dominate latency and cost per decision. Levers: pull only after cheap policy rules pass, cache within a window the bureau allows, and score with an internal model first.
4. Entity resolution grows with the number of records, since all pairs of 1 million records is about 5 × 10^11, so blocking quality decides whether it runs in minutes or days.
5. Explanation cost. TreeSHAP is cheap per row but not free; batch it for monitoring and compute reasons online only for declines.
6. Many models. Per-segment scorecards multiply governance work more than compute. Lever: versioned bundles, one validation-report template, shared monitoring.

Failure modes

Real implementations

The alternative: a pure scorecard (binning plus logistic regression) is the simpler and easier-to-audit design; a boosted model with post-hoc explanation usually ranks better. Many lenders run the scorecard as the decision model and the boosted model as a challenger, promoting it when it wins on out-of-time data and its reasons are acceptable to compliance.

Sources

• Regulation B (Equal Credit Opportunity): 12 CFR 1002.9, notifications (statement of specific reasons or disclosure of the right to one); 12 CFR 1002.2, definitions (empirically derived, demonstrably and statistically sound credit scoring system, paragraph (p)).
• myFICO, What's in your credit score (five categories) and What is a credit score (300 to 850 range for most scores).
• S. Lundberg and S.-I. Lee, A Unified Approach to Interpreting Model Predictions, 2017. paper
• S. Lundberg et al., Explainable AI for Trees: From Local Explanations to Global Understanding, 2019 (TreeSHAP). paper
• T. Chen and C. Guestrin, XGBoost: A Scalable Tree Boosting System, 2016. paper; monotonic constraints
• M. Hardt, E. Price, N. Srebro, Equality of Opportunity in Supervised Learning, 2016. paper