Proportion tests compare percentages, rates, or binary outcomes.

Examples:

• Conversion rate in A vs B
• Recovery rate in treatment vs control
• Pass vs fail rates
• Yes vs no survey responses

They answer questions like:

“Is the proportion of successes in one group different from another?”

Statsmodels provides:

• One-sample proportion test
• Two-sample proportion test
• Confidence intervals for proportions
• Test for equality of proportions across multiple groups

One-Sample Proportion Test

“Is the sample proportion different from a known value?”

Example:
A website has a target conversion rate of 10%. In a sample of 200 visitors, 28 converted (14%).

Example

Interpretation

• p < 0.05 → the observed conversion rate is significantly different from 10%
• Sign of the z-statistic shows direction (positive or negative)

Two-Sample Proportion Test (A/B Test)

“Is the proportion in group A different from group B?”

Example:

• Version A: 40 conversions out of 500 users (8%)
• Version B: 65 conversions out of 520 users (12.5%)

Example

Interpretation

• p < 0.05 → conversion rates differ
• Look at group means to see which is higher

Confidence Intervals for Proportions

Statsmodels makes it easy to compute confidence intervals.

Example

Other methods:

• "normal" — standard approximation
• "agresti_coull" — more accurate for small n
• "wilson" — recommended for general use
• "beta" — Bayesian interval

For A/B tests, you can compute CIs for each group and compare.

Multiple Proportion Test (3+ Groups)

Example:
Three marketing channels have conversion counts:

• Channel A: 30/300
• Channel B: 50/310
• Channel C: 40/290

Test whether all proportions are equal.

Example

Equivalent to a chi-square test of independence, but tailored for proportions.

Practical A/B Test Example

Imagine an A/B test where:

• A: 200 conversions / 2500 visits
• B: 260 conversions / 2480 visits

Run the test

Interpretation

• p < 0.05 → B performs differently than A
• Check proportions:

Testing Directionality

By default, proportions_ztest is two-sided.

You can specify one-sided tests:

• A > B
• A < B

Example (A less than B):

Example (A greater than B):

Effect Size for Proportion Differences

Cohen’s h

Interpretation:

• 0.20 → small
• 0.50 → medium
• 0.80 → large

Cohen’s h complements the p-value by quantifying the magnitude of the difference.

Confidence Interval for the Difference in Proportions

Statsmodels does not directly compute this, but you can do it manually:

Practical Guidelines for Proportion Tests

Use proportions tests when:

• Data is binary (success/failure)
• You compare conversion rates
• You analyse yes/no survey responses
• You compare event rates

Use chi-square when:

• Data is categorical with > 2 levels
• You work with contingency tables

Use z-tests when:

• Sample sizes are large (n > 20 per group)
• Expected counts are sufficient

Use Fisher’s Exact Test when:

• Small sample sizes
• Very low counts
• Expected frequencies < 5

Choosing the Right Proportion Test