Before comparing models
- Define the prediction task, target, and unit of observation.
- Separate training, validation, and final test data.
- Keep related observations together when splitting to avoid leakage.
- Fit learned preprocessing on training data only.
- Document class balance and the meaning of each label.
- Establish a baseline appropriate to the task.
When reading the results
- Report the sample count and class distribution.
- Inspect a confusion matrix with explicit label order.
- Use metrics that match the question and class balance.
- Review precision and recall when positive predictions matter.
- Inspect representative mistakes, not only successful predictions.
- Check whether a threshold was chosen on validation data.
- Report variability across runs when the comparison requires it.
Before making a claim
- Distinguish validation performance from final test performance.
- State which dataset and population the result describes.
- Explain important limitations and uncertainty.
- Avoid interpreting a high score as proof of deployment readiness.
- Keep the configuration, data version, and model checkpoint traceable.
A useful experiment record
Question:
Dataset and version:
Split procedure:
Preprocessing:
Baseline:
Configuration and seed:
Metrics:
Observed mistakes:
Interpretation:
Next experiment:Use the threshold lab to explore one part of this checklist interactively.
A standalone resource by Ackley Will. About the author · Share a question or observation