# Model evaluation checklist

By Ackley Will | https://ackleywill.com

## Before comparing models

- Define the prediction task, target, and unit of observation.
- Separate training, validation, and final test data.
- Keep related observations together when splitting to avoid leakage.
- Fit learned preprocessing on training data only.
- Document class balance and the meaning of each label.
- Establish a baseline appropriate to the task.

## When reading the results

- Report the sample count and class distribution.
- Inspect a confusion matrix with explicit label order.
- Use metrics that match the question and class balance.
- Review precision and recall when positive predictions matter.
- Inspect representative mistakes, not only successful predictions.
- Check whether a threshold was chosen on validation data.
- Report variability across runs when the comparison requires it.

## Before making a claim

- Distinguish validation performance from final test performance.
- State which dataset and population the result describes.
- Explain important limitations and uncertainty.
- Avoid interpreting a high score as proof of deployment readiness.
- Keep the configuration, data version, and model checkpoint traceable.

## A useful experiment record

```text
Question:
Dataset and version:
Split procedure:
Preprocessing:
Baseline:
Configuration and seed:
Metrics:
Observed mistakes:
Interpretation:
Next experiment:
```

Use the [threshold lab](/projects/threshold-lab/) to explore one part of this checklist interactively.

---
Resource text: CC BY 4.0. Attribution: Ackley Will, ackleywill.com.