What is cross-validation and why do you need a train-validation-test split?

The three splits serve three distinct purposes:

• Training set — the model learns parameters from it.
• Validation set — used to tune hyperparameters and choose between models.
• Test set — touched once, at the very end, to estimate real-world performance.

The reason for three rather than two is subtle and important: every time you look at the validation set and adjust something, you leak a little information into your choices. After fifty experiments, validation performance is optimistic. The test set stays clean because you never optimise against it.

K-fold cross-validation splits the training data into k parts, trains k times each holding out a different fold, and averages the results. It gives a more reliable estimate than a single split — especially on small datasets — and it uses all the data for both training and validation across the folds. Stratified k-fold preserves class proportions in each fold and should be the default for classification.

Where standard cross-validation is wrong:

• Time series. Random splitting lets the model train on the future and predict the past. Use forward-chaining, where each fold trains on everything before a cutoff and validates after it.
• Grouped data. If one patient or customer has multiple rows, they must not appear in both train and validation, or you are leaking. Use group-aware splitting.