A score is not yet an action
Imagine a model assigning a score to each example. To make a binary decision, we need a rule: predict positive when the score is greater than or equal to a chosen threshold.
Changing that rule changes predictions without retraining the model.
Open the interactive threshold lab →
The experiment
The lab contains 24 invented examples: 12 positives and 12 negatives. Each has a fixed synthetic score. These are teaching examples, not observations from a trained model.
At a threshold of 0.50, there are 9 true positives, 9 true negatives, 3 false positives, and 3 false negatives. Accuracy, precision, and recall are all 75%.
Lower the threshold to 0.30. More examples receive a positive decision. In this small example, that catches two additional positives and creates four additional false positives.
The scores have not changed. The decision rule has.
Now change the cost of an error
The lab also lets you assign a relative cost to a missed positive. False positives cost one abstract unit; false negatives can cost one, five, or twenty units.
At 0.50, with a false-negative cost of five, the total is:
3 false positives × 1 + 3 false negatives × 5 = 18 unitsAt 0.30, the same cost rule gives:
7 false positives × 1 + 1 false negative × 5 = 12 unitsThe lower threshold has lower accuracy here, yet a lower weighted cost under that particular assumption. If the costs are equal, the comparison changes.
What should you take away?
Separate three questions: how the model scores examples, how a rule turns scores into decisions, and how the context values the resulting errors.
The lab does not select an optimal threshold. In a real project, choose the rule with validation data and a justified decision context. Keep the final test data for evaluating the selected approach.
Download the synthetic example data, or use the evaluation notebook to work through the counts yourself.
Written for Insights by Ackley Will. About the author · Share a question or observation