Statistical confidence

Level of certainty in the results of a statistical test. It is generally expressed as a percentage, for example 95%, which means that if the test were repeated a large number of times under the same conditions, 95% of the tests would lead to the same conclusion. This is a central notion in frequentist analysis, used in the majority of classic A/B testing tools.

Frequentist approach:

• Statistical confidence is linked to the probability of wrongly rejecting the null hypothesis (type I risk of error).
• A 95% confidence level corresponds to a significance level (p-value ) of 0.05.
• This means that we accept a maximum 5% chance that the observed difference is due to chance.

Example: If variation B has a statistical confidence level of 96%, it is considered to significantly outperform version A with a margin of error of less than 4%.

Bayesian approach:

• Instead, we talk about the probability of one variation being better than the other (e.g. "variation B has a 92% chance of being better than A").
• This is not a rejection test, but a direct estimate of the probability of a real gain.
• This approach is often more intuitive for business teams, but depends more heavily on priors.

In CRO :

Whatever the framework chosen, statistical confidence is essential for :

• validate A/B test results,
• limit false positives (effects due to chance),
• make informed decisions based on reliable data.

Please note: high statistical confidence does not guarantee significant business impact. It must be crossed with other indicators (uplift, sample size, incremental value) to guide an implementation decision.

‍