How do you find and handle duplicates, missing values and outliers in a dataset?
Duplicates:
- Find them by grouping on the columns that should be unique and filtering to
HAVING COUNT(*) > 1. First decide what "duplicate" means — a full row copy is different from two records for the same customer with different spellings. - Handle by deduplicating to the most recent or most complete record, usually with
ROW_NUMBER(). Investigate the cause: duplicates from a broken pipeline should be fixed upstream, not cleaned repeatedly.
Missing values — understand why before deciding:
- Missing at random can be imputed with a median or mode, or the rows dropped if few.
- Missing for a reason is informative. A blank "cancellation date" means the order was not cancelled, not that the data is missing. Imputing it would be nonsense.
- Adding a flag indicating the value was missing often preserves useful signal.
Outliers:
- Find them with the IQR rule, z-scores, or simply by sorting and looking at the extremes.
- Do not delete them by default. Decide whether each is an error (a typo, a test record, a sensor fault) or a genuine extreme value. Removing real high-value customers because they are statistically unusual destroys the most important part of the data.
- Options are correcting, excluding with documentation, capping, or transforming — and using a median rather than a mean where the distribution is skewed.
Note: Document every cleaning decision. An analysis where nobody knows what was excluded cannot be trusted or reproduced.





