{"id":"4729","date":"2020-06-22T07:00:00","date_gmt":"2020-06-22T14:00:00","guid":{"rendered":"https:\/\/www.pickfu.com\/blog\/?p=4729"},"modified":"2021-07-08T14:26:07","modified_gmt":"2021-07-08T21:26:07","slug":"shopify-a-b-testing","status":"publish","type":"post","link":"https:\/\/blog.pickfu.com\/shopify-a-b-testing\/","title":{"rendered":"Shopify A\/B testing: The ultimate guide"},"content":{"rendered":"\n

It’s thrilling to set up your own online store, especially on a platform like Shopify<\/a>, which makes it easy for sellers to create and maintain sleek e-commerce storefronts. <\/p>\n\n\n\n

But despite the ease with which you can share to social media, blog about your product, and curate a website, there’s an additional step you need to think about. <\/p>\n\n\n\n

To enjoy success, you need to do some Shopify A\/B testing. This will help make sure your customers are responding to what you’re selling and how you’re presenting it. <\/p>\n\n\n\n\n\n\n

What is Shopify A\/B testing?<\/h2>\n\n\n

Have you ever posted a Facebook poll to your friends, asking which version of a product they like better? <\/p>\n\n\n\n

That’s an A\/B test \u2014 also called a split test \u2014 where you figure out which version of something an audience likes better. By posting the popular version to your storefront, you’ll enjoy increased conversions.<\/p>\n\n\n\n

But the problem with your Facebook friends is that they aren’t an objective group. Your friends want to encourage you, and they’re probably not your target customers. <\/p>\n\n\n\n

With Shopify A\/B testing, your goal is to find out which version of your e-commerce storefront makes your customers convert.<\/p>\n\n\n

How to split test with Shopify <\/h2>\n\n\n

In a classic A\/B test, you direct a portion of your traffic to one version of the site and another portion to the variant. Various online software programs can help you run an A\/B test on Shopify: Shogun<\/a>, Convert<\/a>, and Dynamic Yield<\/a> are some of the top Shopify split-testing tools. <\/p>\n\n\n\n

\"\"
Shopify testing on Shogun.<\/a><\/figcaption><\/figure><\/div>\n\n\n\n

Keep in mind that these are all live split-testing tools, which means that they split traffic between two or more versions of a working web page and record data on which one earned the most conversions.<\/p>\n\n\n\n

You can also run manual split tests by posting one version of your homepage for a week and recording the data, and then switching to the variant for a week.<\/p>\n\n\n\n

Obviously, this is a tedious and more time-consuming way to conduct live testing. And whether you run manual tests or use a program, you’ll need to run tests for at least two weeks, often longer, for the best chance at getting conclusive data.<\/p>\n\n\n\n

\"Shopify<\/figure><\/div>\n\n\n\n

Another, faster way to split test is to run a poll entirely outside the Shopify marketplace. You’ll use the results from this poll to make changes on your site that you know <\/em>will make a positive difference.<\/p>\n\n\n\n

In this article, you’ll learn the benefits and drawbacks of both A\/B testing styles \u2014 plus tips to help you succeed.<\/p>\n\n\n

The benefits of live A\/B testing<\/h2>\n\n\n

The most significant benefit of live testing on Shopify is that you get cold, hard data from your shoppers. <\/p>\n\n\n\n

Say you run an A\/B test between two versions of your Shopify home page for a t-shirt business. The old page features a serif font and the new one tests out a sans serif font.<\/p>\n\n\n\n

Over the course of two weeks, your split test reveals that the sans-serif page drives more customers to your ‘Bestselling Tees’ page, which then leads to more people buying those t-shirts and greater revenue for you. <\/p>\n\n\n\n

You’d then direct 100% of your traffic there in order to reap the full benefit of your improved font.<\/p>\n\n\n\n

To make sure this testing is successful, follow these tips. <\/p>\n\n\n

Set a sample size before beginning<\/h3>\n\n\n

Many brands running live tests stop the test once they achieve statistical significa<\/g>nce, but this isn’t a good idea. <\/p>\n\n\n\n

In addition to reaching statistical significance, you also need to meet your predetermined sample size in order to achieve accurate results. <\/p>\n\n\n\n

Use an online sample size calculator<\/a> to punch in your baseline conversion rate and your minimum detectable effect, along with the desired statistical power and significance level. <\/p>\n\n\n\n

\"Screenshot
Source: evanmiller.org<\/a><\/figcaption><\/figure><\/div>\n\n\n\n

Only once your sample size per variation has been reached should you stop your test. You may or may not achieve statistical significance at all, depending on how much the variants affect your customers’ purchasing decisions. <\/p>\n\n\n

Run each variation for weeklong increments<\/h3>\n\n\n

Since you want data from each day of the week, avoid switching the variant every other day and instead change it every week. That way you collect data for Monday, Tuesday, Wednesday, and so on. <\/p>\n\n\n\n

This is important because maybe you send out a coupon each week on Monday and customers tend to use it on Tuesday. By testing for full week increments, you’ll get full data for each variant on those two days.<\/p>\n\n\n

Test for two full business cycles <\/h3>\n\n\n

For the most accurate results, you’ll want to test for at least one business cycle (usually a two-week time span) or two cycles (four weeks). <\/p>\n\n\n\n

If you don’t reach the sample size within that time frame, you still should stop the test and consider more beneficial ways to get insight on your webpage. (Hint: there’s a solution for that coming right up!)<\/p>\n\n\n\n

The sample size generally needs to be large in order to give you accurate results, as you can see in the screenshot above: 38,210 visitors per variation<\/em>. <\/p>\n\n\n\n

This means that if your website typically reaches 500 visits a week, you won’t meet your sample size and will either need to change the values (a lot) or try another option. <\/p>\n\n\n

What should you test?<\/h3>\n\n\n

Before you commit to such a large-scale, pricey endeavor as live split testing, you’ll want to make sure that what you’re testing actually matters. <\/p>\n\n\n\n

First, make sure that everything is loading well on your website across multiple device styles. <\/p>\n\n\n\n

If your page loads easily on an iPad but lags or otherwise frustrates iPhone users, what can you do to fix it?<\/p>\n\n\n\n

By getting a technical analysis of your site out of the way before testing, you’ll eliminate faulty software or loading from your future testing data. (And you’ll get more conversions before you even begin your test.)<\/p>\n\n\n\n

Another way to get ideas on what to test is to run on-site surveys. If your visitors are filling them out, what are they saying could be improved about their initial shopping experience? <\/p>\n\n\n\n

Focus your improvements and subsequent live testing on the most common or significant complaints. <\/p>\n\n\n\n

Once visitors convert to customers, ask those customers to fill out full-length surveys. <\/p>\n\n\n\n

These surveys will reveal further areas that could use brushing-up with improvements and live testing. <\/p>\n\n\n\n

You can also glean insights from usability testing<\/a>, where you observe the way users interact with your site from the initial loading of your webpage to the addition of a product to the shopping cart. <\/p>\n\n\n\n

If at any point the users get confused or irritated, it’s time to improve upon that area of your website and run a live test.<\/p>\n\n\n

Watch out for external validity threats <\/h3>\n\n\n

While the term external validity threats<\/em> sounds scary, it’s really just a reminder to be aware of circumstances surrounding your live test. <\/p>\n\n\n\n

Don’t, for example, start your live split test on December 15 and run it through mid-January. Chances are people are going to be purchasing a lot more in the initial week or two of the test and a lot less in January due to the holiday season. <\/p>\n\n\n\n

To run a test during this window would jeopardize the validity of your test. You could have an apparent winner by December 30 only to watch that winner drop in popularity in early January because people were done with their holiday shopping spree. <\/p>\n\n\n\n

\"\"<\/figure><\/div>\n\n\n\n

The same goes for any notable sales days: Black Friday, Labor Day, the first day you launch a sales campaign, the change of seasons, and more. <\/p>\n\n\n\n

Be aware of any significant events that contribute to increased or decreased shopping to reduce external validity threats. <\/p>\n\n\n\n

This doesn’t mean you can’t <\/em>test, but that <\/g>you need to take the circumstances into account when determining the significance of the winner. <\/p>\n\n\n

Shopify live A\/B testing: complicated but useful<\/h3>\n\n\n

As you can see, live A\/B testing on Shopify isn’t something to leap into when you’re just starting your business and don’t have much traffic, or when you don’t have thousands of weekly visitors. <\/p>\n\n\n\n

It’s often important to do once your business is established and growing. <\/p>\n\n\n\n

This is where a major downfall of live split testing comes in: it’s only useful once you’re already enjoying some success. <\/p>\n\n\n\n

What about those who need advice when they’re just starting, or who are seeing a lag in visits (and therefore conversions)? <\/p>\n\n\n\n

This is where a different kind of split testing comes into play. <\/p>\n\n\n

A\/B testing with PickFu<\/h2>\n\n\n

For those who don\u2019t yet have a ton of traffic on their Shopify storefront \u2014 or who simply want to get results in minutes or hours, not weeks \u2014 split testing on PickFu<\/a> is an excellent option.\u00a0<\/p>\n\n\n\n

It’s also a lot less complicated than live split testing.<\/p>\n\n\n\n

PickFu matches you with U.S.-based audiences who share valuable feedback on focused aspects of your Shopify store. <\/p>\n\n\n\n

This is another significant benefit because while you can change several things about your site to conduct a live split test, you can’t direct customers to that single aspect alone in order to see how they react. <\/p>\n\n\n\n

\"\"<\/figure><\/div>\n\n\n\n

PickFu helps you create polls and select the type of person who will respond to them.\u00a0For Shopify A\/B testing, you can test things like:\u00a0<\/p>\n\n\n\n