Fit or Overfit?
Turn up model complexity and compare training with held-out validation. Find where extra flexibility stops helping.
Does a closer fit to the training points make a better model?
Flexible models can capture both signal and noise. Training error alone cannot distinguish them. Use validation comparisons to choose a model, and reserve a separate final test set.
Make a prediction.
Then move a control.
A flexible model can reduce training error while fitting sample noise. Compare its behavior on independent validation observations.
Validation helps choose. A test set checks the final choice.
You can explore degree, sample size and regularization with this validation sample. Repeatedly choosing settings for the same validation results can overfit that sample too. A real evaluation keeps a separate test set untouched until the model choices are final.
The dashed curve is known because we invented the data: y = sin(πx) + 0.35x + noise. Real datasets do not come with a visible generating curve.
Inspect the fitted model and observations
| Sample | x | Actual y | Prediction |
|---|
An educational experiment with illustrative data. The controls run locally in your browser.
The idea,
in Python.
A compact implementation of the core concept. Open the code when you are ready to connect the visual to the algorithm.
Open the core implementation Python ↗
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.linear_model import Ridge
from sklearn.metrics import mean_squared_error
import numpy as np
rng = np.random.default_rng(7)
x = np.linspace(-1, 1, 40).reshape(-1, 1)
y = np.sin(3 * x[:, 0]) + rng.normal(0, 0.2, len(x))
train = np.arange(len(x)) % 3 != 0
for degree in [1, 3, 9]:
model = make_pipeline(
PolynomialFeatures(degree, include_bias=False),
StandardScaler(), Ridge(alpha=0.001)
)
model.fit(x[train], y[train])
for name, mask in [("train", train), ("validation", ~train)]:
rmse = np.sqrt(mean_squared_error(y[mask], model.predict(x[mask])))
print(degree, name, round(rmse, 3))Explain the change.
Test your reasoning.
- Increase polynomial degree while watching training and validation error together.
- Raise the noise and see how the preferred complexity changes.
- Add training points. Explain why more evidence can change the comparison.