How early is too early to stop a test? This answer can vary from site-to-site, but a good rule is to run a test for at least 2 weeks.
So, why at least two to three weeks?
There are a number of reasons, but here are a few main ones ones:
- Helps you account for email campaigns:If you email campaigns are weekly or bi-weekly, this traffic can have different motivation from the rest of your traffic sources. If your campaigns are a big enough contributor to your site-wide revenue, these can really skew your data.
- Gives you a minimum of 4 weekend days of traffic:Weekend shopping behavior is a lot different than weekday shopping behavior. Generally, more browsing and comparison shopping is done on the weekend.
- Allows returning visitors to adjust to the site:There are customers that have been to your site before who will return and see a new experience once your test starts. If you have customers who don’t allow cookies, then their experience could change with every visit. Allowing time for your return visitors to become acclimated to the new test requires some time because this segment of traffic tends to underperform early in the test.
Regardless of what your confidence levels say, these are all good reasons to let your test run for at least 2 to 3 weeks. I’ve heard stories of bigger companies not releasing test data for at least 21 days.
For someone new to testing, this is the hardest part to understand. Taking time to make sure you have good sample sizes for your different segments of traffic is important for making confident decisions on your changes.