In my previous Daily Concept, I discussed how to know when you have reached confidence by reading your data correctly. However, it covers how to help you decide when your current test is completed, not how to decide which tests to run. Today, we’ll try a pre-emptive approach: how do we make sure we run a test that won’t take months to achieve results?
There is a balancing act between the time required to run a test and achieving confidence in test results. You want to get the results of your test in as short of time as possible but you need a certain amount of traffic to get those results.
Testing high impact changes to your site will get you confident results faster!
Lets get started identifying high impact vs. low impact. Here are a few things to keep in mind:
First off: your analytics package is your friend. Start where things are working. If you test popular funnels on your site, you know you will have traffic and conversions to get you results. This could be high converting search traffic or your highest selling products and the paths used to get there. If you chase areas of your site that aren’t performing, often times you can be shooting in the dark and may never get any results. You risk running the test for months and never getting confidence. Focus on what is working first and try to make your strengths even stronger.
As a follow up to the first point: traffic, traffic, traffic! It takes traffic and conversions to get results. Make sure you are testing strong areas of your site and make sure you aren’t splitting that test into too many recipes (or combinations). For example, say you want to test your product page. You can test 5 different add to cart buttons, in 5 different colors, where different elements should be moved (such as price, product details, or the title), different colors, font sizes, whether to use tabs or not…the list goes on and on. If you run a large multivariate test (like this) with over 50+ combinations, your traffic will be split over 50 different ways. Some quick math: 100,000 visitors split over 50 recipes is 2,000 for each one. With a conversion rate of 1%, that’s roughly 20 conversions per recipe. On the other hand, those visitors split over 4 recipes (with the same conversion rate) equates to 250 conversions per recipe. You can make a much more accurate conclusion with 250 successes than only 20.
The last one, is BIG IMPACT (yes, a little over dramatic, but this is the key to this whole post after all). Test things that will cause big changes. Rather than testing all of the different little items on the product pages, try to focus on ones that will cause a big difference in the way the page is used. For example, instead of font sizes and colors, try removing the left navigation from the product pages (Amazon does it). How about a testing cross sell items ( such as related, most popular, or recently viewed items)? Or, what happens if you take them away completely? These changes will cause affect customer behavior and will have a bigger impact on your results. You will see this difference in larger percentage changes in your main KPI’s. If it’s positive or negative, you’ll learn something very quick and be able to create a more targeted second test based on what you learned from the first.
Keep these things in mind to help you make better decisions about what to test.
I’ll use a rought analogy in closing:
Picture a jar of loose change with a random assortment of pennies, nickels, dimes, and quarters. Let’s say there’s over $100 in the jar. If you lost 1% of that jar while you aren’t looking, would you notice right away? Probably not, unless you went through and counted that jar every day. However, if you lost 1% every day for over a month, you might start to be suspicious. It’d take two months before you were confident that your money was disappearing and you started figuring out why. However, if you started losing 50% every day, you’d notice pretty quick that something big was affecting your jar of loose change and you’d be fixing the problem by the end of the week!
Testing concepts with high impacts on customer behavior will require less time to achieve confident, accurate results.