.avif)
Statistical power
.webp)
Statistical Power: Probability that a test will detect a real effect, if that effect actually exists. In the context of CRO andA/B testing, a high power (generally set at 80% or more) means that the test has a high chance of correctly identifying a performing variation, and therefore reducing the risk of false negatives (not detecting a real effect).
Power depends mainly on three factors: sample size, confidence level (e.g. 95%), and Minimum Detectable Effect (MDE). A test with low power may miss important optimization opportunities, by wrongly concluding that a variation has no impact. This is why power calculation is a key step in the preparation of any A/B test, in order to guarantee reliable and exploitable decisions for optimization.
Similar definitions
MDE (Minimum Detectable Effect)
The smallest difference which, if it exists, can be reliably detected in a statistical test.
Uplift
The increase in conversions or value generated by a specific action, often measured in A/B tests.
Mann-Whitney U Test
A non-parametric test used to compare two independent samples to determine whether they come from the same distribution.
p-value
A statistical measure that indicates the probability of observing the data collected (or something more extreme) under the null hypothesis.
Frequentist
A statistical approach that estimates model parameters based solely on observed data, without using any a priori information.
.svg){kind=small}
