In scientific research and data analysis, statistical significance helps us determine whether an observed effect is real or just due to random chance. At the core of this concept lies the null hypothesis (H₀) and the p-value, two fundamental components of hypothesis testing.

If you understand those two concepts, you will be able to debunk 80+% of what’s published on social media by referring to the results of studies published on PubMed.

The Null Hypothesis

The null hypothesis (H₀) represents the default assumption that there is no effect, no difference, or no relationship between variables. Researchers test this assumption to see if their findings provide enough evidence to reject it in favor of an alternative hypothesis (H₁ or Hₐ), which suggests a real effect.

Examples:

  • Drug Testing:
    • H₀: A new drug has no effect on blood pressure.
    • H₁: The new drug lowers blood pressure.
  • Marketing Research:
    • H₀: Changing a website’s design does not affect conversion rates.
    • H₁: Changing the website’s design increases conversion rates.
  • Education Study:
    • H₀: A new teaching method does not improve student test scores.
    • H₁: The new teaching method improves student test scores.

The Role of p-Values in Significance Testing

A p-value quantifies the probability of obtaining results as extreme as those observed, assuming H₀ is true. In most cases, researchers use a threshold of 0.05 (or 5%) to determine statistical significance:

  • p < 0.05 → The result is statistically significant (reject H₀).
  • p > 0.05 → Insufficient evidence to reject H₀ (results may be due to chance).

For example, if a clinical trial yields p = 0.03, we conclude that there is only a 3% probability that the observed effect is due to random variation — suggesting the drug may have a real impact.

There’s more

Having a high p-value makes a study inconclusive. However having a low p-value may not necessarily make it conclusive. For example, some studies may prove a statistically significative correlation, but that doesn’t imply practical significance or establish causation.

Examples:

  • A weight loss drug study finds that participants lost an average of 0.2 pounds over six months with p < 0.001. This is statistically significant but not practically significant

  • Studies show that ice cream sales and drowning rates increase together. There is correlation but not necessarily causation. The real cause is most probably hot weather, which increases both ice cream consumption and swimming activities, leading to more drownings.

See also: Not All Studies Are Equal: Short Guide to Scientific Research Quality.