Split testing (sometimes called A/B testing or multivariate testing) is a method of gathering data to help you iterate on ideas and designs. You might use split testing to improve your website, book cover, or app icon, to name a few. Because split testing can often be confusing, and because so many terms are used interchangeably, let’s go over in detail everything you need to know to run an effective split test.
What is split testing?
A traditional split test directs a portion of your website traffic to one version of a web page and directs another portion of your website traffic to a different version of that page. In doing so, you can compare how website visitors in each group behave on each web page.
Split testing and A/B testing are used interchangeably. Both these terms mean that you are testing two versions of a website (Version A and Version B). If you add a third and fourth option, it’s now an A/B/C/D test.
Multivariate testing uses the same mechanism of an A/B test but compares more variables. A true A/B test only tests one variable (for instance, a blue call-to-action button vs. a red call-to-action button). But in a multivariate test, the headline, copy, call-to-action, and images on the website might all be different.
Even though A/B tests and multivariate tests are different, the term split testing is commonly used to describe either one.
Why split test?
Using split testing, you can see how various website layouts, imagery, copy, offers, or calls-to-action affect user behavior on your website. For instance, you could take your existing landing page (the control) and test it against a totally redesigned landing page. Then, using the data of the split test, you can answer questions such as:
- Are users staying on the site longer or are they more likely to abandon the site?
- Are they visiting more pages on the site?
- Are they more likely to fill out a form, buy something, sign up for a trial, or complete another conversion action you’re interested in?
How do you split test?
Generally, to conduct a split test, you direct half your traffic to Version A of your website, and the other half of your traffic to Version B of your website. You control the landing page URL through a software such as Google Ads, Facebook Ads, or a dedicated landing page software. If you’re technically savvy, you could also direct portions of your website traffic programmatically.
Limitations of split testing
Traditional website split testing has its limitations. For one, in order to gather enough data to make informed decisions, your website has to have quite a bit of traffic to test. If you’re using ads on Google or Facebook to drive this traffic, the effort to split test can become expensive. Split testing also takes quite a bit of time.
Steve Chou, who runs the e-commerce site Bumblee Linens, knows these limitations first hand. He would publish a listing on his site, wait several days, swap out the images for new ones, wait again, and then compare the results. The problem with split testing, he said, “is that it takes forever. Every test that I run usually takes at least three weeks or more. And I’d say that nine times out of ten, my tests are inconclusive.”
Besides the time required, a conversion pixel or some kind of tracking mechanism is needed, adding complexity and hassle to the tests, especially when selling on sites like Amazon, eBay, or Etsy. “Not only is this a major pain in the butt if you have multiple listings,” he said, “but if you’re lazy like me, you’re never going to do it.”
So Steve tried an alternative: PickFu. “I thought I’d give it a try just for fun,” he said.
He ran a poll on PickFu to test two featured product photos. The results took less than 20 minutes to come in. After featuring the winning photo from the PickFu poll on his site, sales of that listing increased by 209%.
Differences between split testing and PickFu polls
- Split tests use your actual website customers as test subjects, whereas the PickFu Panel is comprised of U.S.-based respondents who are paid a small stipend to answer your poll. With PickFu, you can use audience targeting to more accurately reflect your target market.
- Split tests take time to set up, and an even longer time to run (depending on your website traffic levels). Using PickFu, most polls complete in less than 15 minutes.
- When running a traditional split test, your test subjects are not aware that their behavior is being monitored. They do not know that there are versions of the website other than the one that they are seeing. In contrast, the PickFu Panel sees all the versions to compare and provide their written feedback.
- The cost of traditional split testing is sometimes unpredictable. In addition to the cost of the ads you may be running to split the traffic, if one version of your site has an adverse effect on your sales, you run the risk of losing revenue with a live test. With PickFu, the cost is a flat fee. Polling starts at $50 but changes with different options you can use to customize your poll.
Even though traditional split tests and PickFu polls differ in these important ways, it’s common parlance to use the terms split testing and A/B testing when referring to PickFu polling. Here at PickFu, we also call our offerings “split testing” or “A/B testing,” but we acknowledge the differences we’ve outlined here.
What can you split test?
Using the wider meaning of split testing described in the section above, what you can split test becomes much broader than only versions of websites. With PickFu, you can split test any creative idea. Some examples of what you can split test include the following:
- Business names
- Logo designs
- Website layouts
- Ad creative
- Mobile app user interfaces (UI)
- Mobile app icons
- Book cover designs
- Book titles
- Book descriptions (blurbs)
- Business ideas
- E-commerce product photos for use on Amazon, eBay, Etsy, and other seller platforms
- E-commerce product descriptions
- E-commerce product attributes (for example, which color of this product is more attractive?)
- Email subject lines
- Social media posts
What are the benefits of split testing?
Using PickFu to split test your ideas, it’s quick and easy to iterate new and improved versions of those ideas. The written comments help you understand how customers react. PickFu provides both quantitative data and qualitative feedback.
The feedback you receive has the potential to pay for itself. As Manny Coats, CEO and founder of Helium 10 notes, “PickFu.com is where I go when I want to quickly split test Amazon product images to actual Amazon Prime customers, and know which images customers prefer, before I make my product listing live. I get results within one hour with detailed ‘reasons why’ they voted a certain way. This has the potential to save thousands of dollars in lost sales from choosing the wrong image.”
In addition, PickFu can surface concerns of customers that you might not have foreseen. Author Mike Fishbein said, “I had a radical idea to use a 1970s-style motif for the [book] title and cover.” After brainstorming many ideas, he was most excited to title his book Pimp Your Book: How to Self-Publish a Bestseller on Amazon. “I thought it would add character and be attention-grabbing, [but] when I tested it on PickFu, I learned that readers found it unappealing and tacky.”
PickFu is faster than other split testing alternatives. As technical product manager Isabella Patton said, “We chose PickFu because of the quality of the users and the speed at which we get results! Other online survey services take up to a week to complete, and we just don’t have that time… Since it takes less than 15 minutes to get the survey results, we were able to alter our designs immediately and re-test them.”
How much does split testing cost?
Depending on your method of testing, costs vary. For instance, if you’re using a platform such as Google Ads or Facebook Ads to split test, you’ll pay for clicks. You also need to consider the time and effort required to set up a split test on your website.
On PickFu, a 50-response poll starts at $50. You can use available features and options to customize your poll and make it more valuable to your business. For instance, with audience targeting, you can only poll people who match certain demographic or behavioral traits, such as women, iOS users, Amazon Prime members, homeowners, or a combination of up to three traits. Use our Pricing Calculator to determine the cost of your poll.
Split testing best practices
- At the heart of a PickFu split test is the question you pose to poll respondents. This question must not positively or negatively bias the answers you collect. Read our guidelines on how to write unbiased questions.
- In addition, it’s important to understand when multivariate testing will work, and when you should stick to an A/B test. (Refer to the beginning of this article for definitions). For instance, if you’re testing different ads, you may want to vary up the design layout, imagery, copy, and call to action (a multivariate test) to see which general direction is the best to proceed. But if you’re interested in a specific question, such as determining which design layout works best for a book cover, you should keep the title, subtitle, and author information consistent in the options, leaving the design as the only variable you’re testing (an A/B test).
- Multivariate tests should not be confused with double-barreled questions, which are a common polling mistake. In a double-barreled question, the question asks about two disparate things. Double-barreled questions do not make good survey questions because respondents will often only concentrate on the one topic that means the most to them.
Example of a double-barreled question: Which title and subtitle do you prefer?
How to fix it: Double-barreled questions should be separated into two distinct questions and polled separately from one another:
• Which title do you prefer?
• Which subtitle do you prefer? (include the same title in all options if you’re interested to see how they work together) - Don’t just look at the vote tally. Sure, having a runaway winner in each poll can be great, but be sure to read the comments closely to understand why an option won over the other. Look for themes and recurrent reactions in the feedback. Use these insights to revise your ideas and re-test.
Ready to start split testing your ideas?
Visit PickFu now to create your first split test. It only takes a minute to set up, and you’ll start receiving unbiased answers to your questions immediately!
