A/A Testing Concept

Format: Understand why A/A testing (testing two identical groups) is necessary.

Concept: Before running an A/B test, first run an A/A test — both groups of users see the exact same version. If the A/A test also shows a "significant difference," it means your testing infrastructure has a problem.

Exercise: Consider the following situations and judge whether the difference is real or statistical noise:

1. After changing button color, conversion rate goes from 2.0% to 2.3% — significant?
2. Test results with only 100 users — reliable?
3. A test that ran for only 2 days before drawing conclusions — reasonable?