Calibration, Imbalance, and Decision Thresholds
Probability calibration asks whether predicted probabilities correspond to observed frequencies in the population being evaluated. A decision threshold turns scores into actions. Class imbalance describes the outcome mix. These ideas interact, but changing one does not automatically fix the others.
Start by defining the action and the consequences of false alarms and missed events. Evaluate probability quality separately from the operating policy, choose any threshold using appropriate validation data, and report final performance on data that did not select the policy. A threshold is a decision rule, not a repair kit for every model problem.
TL;DR
- Calibration asks whether predicted probabilities match observed frequencies in the evaluated population.
- A decision threshold turns scores into actions, and class imbalance describes the outcome mix; changing one doesn't fix the others.
- Start from the action and the costs of false alarms versus missed events.
- Choose thresholds on validation data, and report final performance on data that played no part in choosing them.
- Resampling the training data changes what scores mean; recalibrate for deployment prevalence.
Quick Example
These six fictional maintenance alerts demonstrate error accounting only. They are not personal eligibility decisions, measured equipment results, or evidence that the listed scores are calibrated probabilities.
At 0.5 there is one false positive and one false negative. At 0.3 there are two false positives and no false negatives. Under the stated fictional costs, the total changes from six cost units to two. That comparison does not establish the best threshold, a validated cost model, or expected production performance. Six invented observations are enough to explain arithmetic, not enough to authorize a deployment.
Core Concepts
Calibration evaluates the meaning of probabilities
A calibration curve groups predictions and compares the mean predicted probability with the observed positive fraction in each group. Scikit-learn describes this reliability-diagram interpretation and the need to separate data used to fit a calibrator from data used to fit the underlying classifier. Probability calibration
The curve is an estimate. Results depend on sample size, binning, the evaluated population, and the stability of outcomes over time. Sparse high-score bins can look reassuring or alarming because only a few outcomes fall there. Report the support behind a curve rather than treating a line near the diagonal as a certificate.
A model can rank examples usefully while assigning poorly calibrated probabilities. Conversely, acceptable aggregate calibration does not guarantee useful ranking or subgroup behavior. Choose evaluation measures for the actual use of the output. If downstream decisions rely on a probability meaning, discrimination alone cannot supply that meaning.
A threshold belongs to a decision policy
Scikit-learn separates predicting probabilities or scores from deciding which class to output. Its threshold-tuning documentation explicitly warns against training a classifier and tuning its threshold on the same data. That warning is about selection bias: a policy chosen to look good on one dataset needs an independent assessment. Decision threshold tuning
For a simple binary decision with correctly calibrated probabilities for the deployment population, constant false-positive cost C_FP, constant false-negative cost C_FN, zero cost for correct decisions, and no other constraints, compare the two expected costs. Predict positive when C_FP times (1 minus p) is no greater than C_FN times p. The resulting boundary is C_FP divided by (C_FP plus C_FN), with tie handling declared.
Under equal error costs those assumptions give a boundary of 0.5, even if the positive class is rare. Imbalance alone does not prove that 0.5 is wrong. If probabilities are uncalibrated, prevalence has shifted, costs vary, capacity is limited, or decisions have other consequences, that simplified rule may not describe the real problem. State which assumptions actually hold.
The toy example compares two selected thresholds empirically. It does not derive its 0.3 threshold from the cost formula or establish that its arbitrary scores satisfy the formula's probability assumptions. Keep the worked arithmetic and the theoretical decision rule separate.
Rare outcomes need visible denominators
When positive events are uncommon, predicting negative for everything can produce high accuracy. That is a baseline to expose, not a useful alerting success by itself. Report the confusion counts and relevant rates alongside the number of positives and negatives. A percentage without its denominator can hide how little evidence supports a decision.
Precision describes the positive share among predicted positives; recall describes the detected share among actual positives. Their operational value depends on the action. A maintenance team with limited inspection capacity may need to examine alert volume and missed-event consequences together. Do not substitute an abstract metric for the people and resources that must act on the alerts.
Best Practices
Write an evaluation plan before trying many threshold values. Define the population, time window, label availability, cost assumptions, and metrics. Keep model fitting, calibrator fitting where needed, threshold selection, and final assessment separated in a way appropriate to the data. Temporal applications may require a time-respecting evaluation rather than a random split.
Compare the proposed operating point with a useful baseline and nearby alternatives. Show what changes in false alarms, missed events, and workload. If small threshold changes cause large swings in a small sample, treat that sensitivity as uncertainty rather than hiding it behind a precise decimal threshold.
Inspect relevant subgroups or operating conditions when consequences differ, with enough support to interpret the estimates. Aggregate calibration can conceal uneven performance. Subgroup analysis still does not certify fairness or establish that a deployment is appropriate; responsible owners must assess the actual use and consequences.
Monitor whether data, outcome prevalence, labeling, or operating costs have changed. Revisit the policy when the conditions supporting it no longer apply. A different threshold may be appropriate, but investigate the cause before repeatedly adjusting a number to chase a drifting metric.
For high-consequence uses involving people, the simple maintenance illustration is inadequate. Domain, legal, privacy, fairness, and operational reviewers need the actual decision context, evidence, appeal or fallback process, and authority. No such deployment or independent review is claimed here.
Common Mistakes
Moving the threshold to fix calibration
Assuming oversampled scores match deployment
Choosing the threshold on the test set
Rejecting 0.5 by reflex
FAQ
Should imbalanced classifiers always use a lower threshold?
No. The appropriate threshold depends on the decision problem. Under calibrated deployment probabilities and equal binary error costs with no other constraints, 0.5 can be appropriate regardless of prevalence. Other costs, capacity, probability errors, or changed conditions can justify another policy.
Does calibration improve ranking?
Calibration concerns probability interpretation. Do not assume a calibration procedure improves every ranking measure or decision outcome. Assess the quantities that matter on appropriate held-out data and compare the resulting operating policy separately.
How do we choose a threshold responsibly?
Declare the action, costs, constraints, and evaluation design; select using validation evidence; then assess the fixed choice independently. Show counts, uncertainty, and limits. For consequential deployments, obtain the relevant qualified review rather than treating a numerical cutoff as approval.