Recently, I was preparing to send an important bottom-of-funnel (BOFU) email to our audience. I had two subject lines and couldn‘t decide which one would perform better.
Naturally, I thought, “Let’s A/B test them!” However, our email marketer quickly pointed out a limitation I hadn’t considered:
At first, this seemed counterintuitive. Surely 5,000 subscribers was enough to run a simple test between two subject lines?
This conversation led me down a fascinating rabbit hole into the world of statistical significance and why it matters so much in marketing decisions.