Type I and Type II Errors

When we perform hypothesis testing, we make decisions based on sample data - but there’s always a chance we make a wrong decision about the population.

These mistakes are called Type I and Type II errors.

Hypothesis Setup Reminder

• H₀ (Null Hypothesis): “There is no effect or difference.”
• H₁ (Alternative Hypothesis): “There is an effect or difference.”

The test decides whether to reject H₀ or fail to reject H₀ - but since we’re working with samples, we can’t be 100% certain.

Two Possible Truths vs Two Possible Decisions

Type I Error (False Positive)

• You reject the null hypothesis when it’s actually true.
• You think there’s an effect or difference, but there isn’t.

Symbol: α (alpha)

Example:

• Concluding a drug works when it doesn’t.
• Detecting a pattern in data that’s just random noise.
• Believing a model improvement is real when it’s due to chance.

Analogy:

“You accused an innocent person.”

Controlled By:

The significance level (α) — usually set at 0.05 → meaning a 5% chance you’ll wrongly reject a true null.

Type II Error (False Negative)

• You fail to reject H₀ when it’s actually false.
• You miss a real effect.

Symbol: β (beta)

Example:

• Concluding a drug has no effect when it actually does.
• Failing to detect a real improvement in model performance.
• Missing a real correlation between two variables.

Analogy:

“You let a guilty person go free.”

Controlled By:

The power of the test (1 − β) — higher power = lower chance of Type II error.

Visual Summary

Imagine overlapping curves:

• The left curve = sampling distribution if H₀ is true
• The right curve = sampling distribution if H₁ is true

The critical region (α) is the area where you reject H₀.

• If your data fall there when H₀ is true → Type I error
• If your data fall outside when H₁ is true → Type II error

Increasing sample size narrows both curves → reduces both types of error.

Key Metrics

Goal:
Keep α low (e.g., 0.05) and power high (e.g., ≥ 0.8).

Real-World Examples

Balancing Errors

Reducing one type of error usually increases the other:

• Lowering α (making the test stricter) → fewer false positives, but more false negatives.
• Increasing α (being more lenient) → fewer false negatives, but more false positives.

The balance depends on context:

• In medicine → minimise Type I (avoid false claims).
• In fraud detection → minimise Type II (catch all fraud).