What is overfitting and underfitting, and how do you prevent them?

Overfitting means the model has learned noise as well as signal. Training error is low, validation error is high, and it generalises badly. Underfitting means the model is too simple to capture the pattern — both training and validation error are high.

This is the bias-variance trade-off. Underfitting is high bias; overfitting is high variance.

How to detect it: compare training and validation performance. A large gap means overfitting; both poor means underfitting. Learning curves plotted against training set size distinguish them clearly — if validation error is still falling as you add data, more data will help; if the curves have converged and both are poor, the model is too simple.

Fixing overfitting:

• More training data — the most reliable fix when available.
• Regularisation — L1 (Lasso, which drives coefficients to zero and performs feature selection) or L2 (Ridge, which shrinks them).
• Simplify the model — fewer parameters, shallower trees, fewer features.
• Early stopping on a validation set.
• Dropout in neural networks; pruning and depth limits in trees.
• Cross-validation for reliable estimates.
• Ensembling, which averages away variance.

Fixing underfitting: a more expressive model, better features, less regularisation, or longer training.

Note: The most dangerous case is not overfitting to training data but overfitting to the validation set through repeated tuning. That is why a genuinely held-out test set touched once matters.