What is a p-value and statistical significance, and how are they commonly misinterpreted?

A p-value is the probability of observing a result at least as extreme as yours if the null hypothesis were true. A small p-value means the data would be surprising under the assumption of no effect.

Statistical significance means the p-value falls below a threshold chosen in advance, conventionally 0.05. That number is a convention, not a law of nature.

The common misinterpretations — this is what interviewers are testing:

• It is not the probability the null hypothesis is true. It is the probability of the data given the null, which is a different conditional.
• It is not the probability your result is a fluke.
• Significance is not importance. With a large enough sample, a commercially meaningless 0.01% difference becomes statistically significant. Always report the effect size and a confidence interval alongside it.
• Non-significant does not mean no effect. It may mean the sample was too small to detect one — absence of evidence is not evidence of absence.
• Stopping a test when it becomes significant invalidates it. Repeatedly checking and stopping at the first significant moment produces false positives at a much higher rate than 5%. This is the most costly mistake in practical A/B testing.
• Testing many variables guarantees false positives. Twenty independent tests at p < 0.05 produce roughly one significant result by chance alone.

Note: Saying you prefer to report confidence intervals and effect sizes rather than a bare significant/not-significant verdict shows genuine statistical literacy.