Login to manage your account

Please enter a valid email address.
Forgot Password?
Please enter a valid password.
OR

Don't have an account yet? Sign up

How do you find and handle duplicates, missing values and outliers in a dataset?

Duplicates:

  • Find them by grouping on the columns that should be unique and filtering to HAVING COUNT(*) > 1. First decide what "duplicate" means — a full row copy is different from two records for the same customer with different spellings.
  • Handle by deduplicating to the most recent or most complete record, usually with ROW_NUMBER(). Investigate the cause: duplicates from a broken pipeline should be fixed upstream, not cleaned repeatedly.

Missing values — understand why before deciding:

  • Missing at random can be imputed with a median or mode, or the rows dropped if few.
  • Missing for a reason is informative. A blank "cancellation date" means the order was not cancelled, not that the data is missing. Imputing it would be nonsense.
  • Adding a flag indicating the value was missing often preserves useful signal.

Outliers:

  • Find them with the IQR rule, z-scores, or simply by sorting and looking at the extremes.
  • Do not delete them by default. Decide whether each is an error (a typo, a test record, a sensor fault) or a genuine extreme value. Removing real high-value customers because they are statistically unusual destroys the most important part of the data.
  • Options are correcting, excluding with documentation, capping, or transforming — and using a median rather than a mean where the distribution is skewed.

Note: Document every cleaning decision. An analysis where nobody knows what was excluded cannot be trusted or reproduced.

All Data interview questions

Login to manage your account

Please enter a valid email address.
Forgot Password?
Please enter a valid password.
OR

Don't have an account yet? Sign up as