Regularization: Learn Beyond the Sample
Train paired neural networks in your browser. Compare measured learning curves, validation errors and the predictions that remain.
When does changing how a network learns improve its generalization?
Regularization changes the learning process, but its value must be evaluated. Compare a control and an experiment with paired data and initial weights. Separate dropout’s training behavior, AdamW weight decay and validation-based checkpoint selection.
Make a prediction.
Then move a control.
Fit the examples.
Question the generalization.
Train two small neural networks on the same data and initial weights. Change how one of them learns.
Ready. Both networks start with identical weights. Click Run to measure the result.
- Training RMSE
- —
- Validation RMSE
- —
- Epochs run / selected
- 0 / 0
- Training RMSE
- —
- Validation RMSE
- —
- Epochs run / selected
- 0 / 0
Learning curves
What the networks predict
Look inside dropout
An illustrative mask of 12 activations. These dots do not show the actual network weights.
The experiment, in PyTorch
How to interpret the comparison
The two networks receive the same training observations, validation observations and initial weights. The control always trains to the maximum epoch count without dropout, weight decay or checkpoint restoration. The experiment uses your selected settings.
Loss for gradient updates is mean squared error. The displayed metric is RMSE. Weight decay uses AdamW’s decoupled update and excludes bias parameters; it is not an L2 term added to this loss.
Early stopping checks validation RMSE every epoch, stops after the chosen patience and restores the best eligible checkpoint. The curve shows training history; the result cards and prediction curves show the restored model. If patience never expires, training reaches the maximum and still restores its best checkpoint.
Validation can guide choices, but repeated tuning can overfit validation too. This experiment has no final test set. Real projects reserve one for the final evaluation. Training is a reproducible synthetic experiment, not evidence of performance on an image dataset.
Training updates and metrics are computed in your browser on synthetic regression data. These results do not estimate accuracy on an image dataset.
The idea,
in Python.
A compact implementation of the core concept. Open the code when you are ready to connect the visual to the algorithm.
Open the core implementation Python ↗
import copy
import torch
from torch import nn
# The same experiment design; PyTorch and browser RNGs differ.
g = torch.Generator().manual_seed(17)
x = 2 * torch.rand(100, 1, generator=g) - 1
y = torch.sin(3 * x) + 0.3 * x + (2 * torch.rand(x.shape, generator=g) - 1) * 0.3
x_train, y_train = x[:20], y[:20]
x_val, y_val = x[20:], y[20:]
def run(dropout=0.0, decay=0.0, early_stopping=False):
torch.manual_seed(7) # Paired initial weights.
model = nn.Sequential(
nn.Linear(1, 24), nn.Tanh(), nn.Dropout(dropout),
nn.Linear(24, 24), nn.Tanh(), nn.Dropout(dropout),
nn.Linear(24, 1),
)
for layer in model.modules():
if isinstance(layer, nn.Linear):
nn.init.xavier_uniform_(layer.weight)
nn.init.zeros_(layer.bias)
weights = [p for name, p in model.named_parameters() if name.endswith("weight")]
biases = [p for name, p in model.named_parameters() if name.endswith("bias")]
optimizer = torch.optim.AdamW([
{"params": weights, "weight_decay": decay},
{"params": biases, "weight_decay": 0.0},
], lr=0.01)
best, wait, best_epoch = float("inf"), 0, 0
best_state = copy.deepcopy(model.state_dict())
for epoch in range(1, 601):
model.train() # Dropout active only here.
optimizer.zero_grad()
loss = nn.functional.mse_loss(model(x_train), y_train)
loss.backward()
optimizer.step()
model.eval()
with torch.inference_mode():
val_rmse = ((model(x_val) - y_val) ** 2).mean().sqrt().item()
if val_rmse < best - 1e-5:
best, wait, best_epoch = val_rmse, 0, epoch
best_state = copy.deepcopy(model.state_dict())
else:
wait += 1
if early_stopping and wait >= 60:
break
if early_stopping:
model.load_state_dict(best_state)
model.eval()
with torch.inference_mode():
train_rmse = ((model(x_train) - y_train) ** 2).mean().sqrt().item()
val_rmse = ((model(x_val) - y_val) ** 2).mean().sqrt().item()
return {"train_RMSE": train_rmse, "validation_RMSE": val_rmse,
"epochs_run": epoch, "selected_epoch": best_epoch if early_stopping else epoch}
print("Control:", run())
print("Experiment:", run(dropout=0, decay=0,
early_stopping=True))
# Validation guides choices. Keep a separate final test set untouched.Explain the change.
Test your reasoning.
- Run Early stopping and compare the epoch reached with the checkpoint selected. Explain which model the prediction curve uses.
- Explore dropout in training and evaluation modes, then compare a low and a high drop probability.
- Change one regularization setting at a time and repeat on a new sample. Does a smaller training–validation gap necessarily mean a smaller validation error?