Mann-Whitney U Test

Non-parametric statistical test used to compare two independent samples, with the aim of determining whether they come from the same distribution. Unlike parametric tests (such as Student's t-test), it assumes no normality in the data, making it particularly useful in contexts where data are asymmetric, noisy or influenced by extreme values.

Application in CRO / A/B testing :

In a Conversion Rate Optimization approach, this test is ideal for analyzing non-binary metrics (e.g., average order value (AOV), revenue per session, time spent on a page) when :

• the distribution is highly skewed,
• outliers are frequent (e.g. very large baskets),
• or variances are unequal between groups.

Example:

An A/B test compares the average basket value between two variants. The data show a highly skewed distribution, with some very large baskets distorting the mean.
➡️ Rather than using a t-test, the Mann-Whitney test compares the true central tendency (relative position of values) without being biased by these extremes.

Advantages :

• Robust against non-normal distributions,
• Less sensitive to outliers,
• Suitable for continuous or ordered metrics that cannot be transformed into rates.

Please note:

It tests distribution differences, not necessarily averages. It is therefore useful to interpret it as part of an overall analysis, as a complement to other metrics and visualizations.

‍