A Threshold Is a Decision
Keep the model. Change the rule. See who gets a different decision and what those mistakes cost.
What changes when the scores stay the same, but the decision threshold moves?
A score is not a decision. A threshold turns a score into an action, and the costs of mistakes help determine which actions make sense.
Make a prediction.
Then move a control.
12 positive and 12 negative examples; neither side is perfectly separated.
| Predicted + | Predicted − | |
|---|---|---|
| Actual + | 9True positive | 3False negative |
| Actual − | 3False positive | 9True negative |
Same scores. Different consequences.
At 0.50, 9 of 12 positives are detected and 3 negatives are incorrectly flagged.
3 misses × 5 + 3 false alarms × 1 = 18 units
Changing the cost changes the consequences of errors. At a fixed threshold, it leaves the predictions and accuracy unchanged.
— means undefined: there are no positive predictions, so precision has no denominator.
Inspect every example and decision
| Case | Reference class | Score | Prediction | Outcome |
|---|
If missing a positive becomes more costly, which direction would you move the threshold? What would you measure before deploying that change?
Invented examples for learning. Scores are illustrative rankings, not verified probabilities. Any threshold chosen from these examples would still require evaluation on separate data.
An educational experiment with illustrative data. The controls run locally in your browser.
The idea,
in Python.
A compact implementation of the core concept. Open the code when you are ready to connect the visual to the algorithm.
Open the core implementation Python ↗
import numpy as np
# Illustrative scores: use validation data to choose a threshold.
y = np.array([1, 0, 1, 0, 1, 0])
scores = np.array([0.90, 0.80, 0.65, 0.45, 0.30, 0.10])
threshold = 0.50
pred = scores >= threshold
tp = np.sum(pred & (y == 1))
fp = np.sum(pred & (y == 0))
fn = np.sum(~pred & (y == 1))
tn = np.sum(~pred & (y == 0))
cost_fp, cost_fn = 1, 5
print({"TP": tp, "FP": fp, "FN": fn, "TN": tn})
print("Error cost:", cost_fp * fp + cost_fn * fn)Explain the change.
Test your reasoning.
- Start at a threshold of 0.50. Which type of mistake is more common?
- Lower the threshold. Explain the change in false positives and false negatives.
- Make a missed positive more costly. Separate what changes in the decisions from what changes in their evaluation.