Amazon split testing

With almost 500 million products on Amazon, how do you stand out from the crowd? Split testing is one of the most effective means to optimize your Amazon product listings and increase engagement and sales on your products.

See split testing in action Start your Amazon split test

What is split testing on Amazon?

Split testing on Amazon is an experiment between two or more variations to see which variation in the set performs best. For Amazon split testing, you might test variations of your product name, different Amazon listing photos, or versions of product copy. Split testing is also known as A/B testing because there is a version called A and a version called B. (If you add a third and fourth version, it’s called an A/B/C/D test).

Option A vs. Option B

Amazon split testing tools

Traditional split tests direct a portion of your Amazon traffic to one test variation and another portion to a different variation. After the test, you compare how shoppers behaved with each variation. For instance, on average, did one set of visitors browse for longer? Did one set convert at a higher rate?

Many split testing tools exist to help sellers run split testing on Amazon, including Splitly, Listing Dojo, and Cashcowpro. Amazon’s own split testing feature, Manage Your Experiments, allows eligible sellers to test the A+ Content of their brand’s listings.

However, the traditional split tests that these tools offer have some limitations:

  • For one, in order to gather enough data to make an informed decision, you have to accrue enough traffic to test.
  • In order to do this, you’ll probably need to pay to promote your products on Amazon, which can become expensive.
  • The tests also may take quite a bit of time, depending on the traffic levels. Splitly, for example, requires a minimum of seven days.
  • Further, because you’re using live traffic for your test, you risk seeing an adverse effect on sales or search ranking if a test variation performs worse than your current listing.

Split testing on Amazon using PickFu

PickFu is a polling software that enables you to quickly test elements of your Amazon listing with audiences that match your customer profile and target demographics. For example, you could test product names, copy, or logos, or try image split testing with variations of your main photo.

Differences between live split testing and PickFu polling

  • Traditional split tests use Amazon shoppers as they are shopping as test subjects.
    • The PickFu Panel is comprised of U.S.-based respondents who are paid a small stipend to answer your poll. A popular audience available to test on PickFu is Amazon Prime members. However, a PickFu split test is conducted outside of the Amazon marketplace, so there may be variances between the PickFu Panel and Amazon shopper demographics.
  • When running a traditional split test, Amazon shoppers are not aware that their behavior is being monitored. They do not know that there are other versions of your listing other than the one they are seeing.
    • On PickFu, the Panel sees all variations in order to compare them and provide their feedback. Not only does each respondent vote on the variation they like best, but they each also provide a written comment explaining why they chose their preferred option. These written comments provide insight into what works with shoppers to help guide your decisions.
  • Traditional split tests often take days or weeks to complete.
    • Using PickFu, most polls complete in less than 15 minutes.
  • The cost of traditional split testing on Amazon is often unpredictable. You need to pay for the testing software you’re using to split test, as well as the cost of the ads you’re running to build traffic.
    • With PickFu, the cost is a flat fee. Polling starts at $50 but changes with different options you can use to customize your poll.
  • Traditional split tests use live Amazon listings.
    • On PickFu, your product does not need to be live on Amazon. You can test product variations such as color options or design before going into production. Get a sense of what shoppers want before committing to inventory. Or try Amazon package testing by perfecting your packaging with PickFu’s audience.
  • Traditional split tests can only compare variations of your own Amazon listings.
    • Using PickFu, you can perform competitive analysis by polling your listing against competing listings. Gain insight into how your product fares against category leaders. Split testing product copy is especially effective with competitors.

The perks of testing outside of Amazon

Since PickFu polling happens outside of Amazon, you can run polls for idea validation before going into production or committing to inventory.

If you already have live listings, using PickFu means there’s no risk of losing sales if the new variant performs worse than the original. You don’t have to change anything in your listing until the poll results have shown it will perform better.

Another benefit of testing outside of Amazon is the ability to compare your products to a competitor or category leader. You can match your product directly against another to see how it fares with respondents. Since respondents leave written feedback along with voting, you discover what you’re doing well and can work that into the messaging of your listings and branding.

What can you split test?

Using PickFu, you can test every element of your Amazon listings, such as

Along with a voting system, PickFu allows for open-ended feedback. If you’re stuck on pricing your products, you can turn to respondents for their opinions. Try running a poll with questions such as:

  • How much would you pay for this product?
  • Which product looks more expensive?
  • Do you think $14.99 is a fair price?

We’ve found from previous polls that if you list a price in the details, people will leave their opinion on price anyway. You may see feedback such as, “I prefer the photo from Option A, but I wouldn’t pay that price for it.”

How PickFu protects the confidentiality of your business

When running your own e-commerce business, especially with proprietary products you’ve created, confidentiality is a top concern. Rest assured that PickFu understands this. We have several measures in place to protect the confidentiality of your ideas, products, and business.

Before taking part in any polls, every respondent signs a thorough non-disclosure agreement (NDA). Respondents are restricted from taking or sharing anything they see in a poll.

PickFu polls are private by default. This ensures that anything you share in the poll won’t be indexed by search engines and stays between you and the respondents. However, you have the option to include your poll in our public poll gallery, where sellers can get ideas on how to use polls and see how effective they are. You’ll get a $5 discount for sharing your poll in the public gallery.

Who are the PickFu respondents?

We use a variety of online services to find our respondents, all of whom are based in the United States and native English speakers. On the back end, we have developed proprietary quality controls that include internal rating systems and spam protection. We take quality seriously and are always working to ensure that respondents give insightful feedback.

While we have gathered a range of people to respond to PickFu polls, you have the ability to customize your audience. You can choose a general audience, which is a mix of genders, age groups, and interests.

Amazon split testing: screenshot of custom PickFu audience

You can also specify the audience who votes in your poll. One popular option for e-commerce sellers is polling Amazon Prime members. If you’re targeting your product to a specific kind of person, use filters like gender, age range, educational level, and income. For example, if you have a high-end product, you can target people who earn more than $100,000 per year.

You can choose multiple filters to get a specific audience, such as female homeowners or male mobile gamers in their 20s. We’ve also added categories that are especially beneficial for e-commerce users, such as cat owners or dog owners for pet products. Other information we gather from respondents includes:

  • exercise habits
  • cooking habits
  • if they use cosmetics
  • if they take nutritional supplements
  • the type of mobile device used
  • if they’ve donated to a Kickstarter campaign before

By combining different categories, you can niche your poll audience down to the same type of shoppers you are targeting with your products.

Our split testing software in action

When you create a poll on PickFu, start by inputting your question. For example, ask “Which image do you prefer?” or “Which design do you prefer?” You can input up to eight different options in a single poll, and these options can be images, text, links, or embedded videos.

If you have more than two options, you can either have respondents rank the options in the order they prefer or set up head-to-head matchups, which is a more complete way of polling.

You can also test product descriptions and entire product listings by including links in your polls.

Split tests of product photos

amazon split testing product photos with Option A being products laid out differently (products all bunched up) and Option B is shown as the winner with the products laid out properly. Which Amazon image do you like better?
A split test with 50 general population respondents that took 26 minutes.
amazon product photo split test - Another Amazon split test that shows baby clothes, some on-top in a pile (Option B) which one and Option A has all of the products in a line (which lost).
A split test conducted with 50 Amazon Prime members. Results took 30 minutes.
Main eCommerce product pictures for Yoga straps and bag, one with a woman modeling and another with only her leg.
Split test of exercise equipment with 50 female respondents. Can you guess which one won?

Split tests of physical product design

Split test of subscription box design
A 50-respondent poll for product design. Results took 24 minutes.
A 4-option product design poll with 50 female respondents. Results came back in under an hour.
Different designs surfaced on on olive oils one with a flower dislay and another with a cloudy display. Option A (Won), Option B (loss).
A 50-respondent poll for an olive oil dispenser. Results completed in only 13 minutes.

Split tests of product names and brand names

Split test with 50 females who were asked to rank three potential names for a natural skin care brand
Split test with 50 female respondents asked to rank three potential names for a natural skin care brand.
Split test of 4 names for an e-commerce store
A 50-respondent poll to rank names of an e-commerce store.
Fifty respondents who live in suburbs voted in this head-to-head poll for a high-end baby brand.

How long does a PickFu poll take to complete?

Once you’ve created and submitted a poll, we immediately find respondents to vote. Depending on your targeting parameters, this can happen quickly. If you choose a general audience or something simple like 50 female respondents, the poll may finish in less than 15 minutes.

If you are specific with your targeting, it can take longer for the poll to complete. For a Mother’s Day poll, we targeted mothers by combining female respondents with those who have children. This poll took 2 hours to complete.

There are other factors involved in the speed of poll completion. If you are running a poll at night or over the weekend when fewer people are online, it may take longer to finish.

On average, most polls finish in under an hour.

How PickFu compares to other Amazon testing tools

When you compare PickFu polls to other Amazon split testing tools such as Splitly or Listing Dojo, there’s one major difference that sets us apart: we’re people-powered. Thanks to our live respondents, business owners gain more insight with PickFu polls than they would with other split testing tools.

For example, one e-commerce seller was developing a garden product for people who live in apartments and don’t have much outdoor space. The product would allow these apartment dwellers to grow a garden in a compact space.

The seller created a poll to test two product names, Patio Harvest vs. Porch Farmer. Patio Harvest easily won the poll with 76% of the vote, but the real insights came from the written feedback.

Many respondents commented that apartments don’t have “porches,” while others said they don’t consider themselves “farmers.” Because of these reasons, respondents said the name Porch Farmer didn’t resonate with them.

A traditional split testing or keyword tool would never be able to give you such specific feedback. Product names, packaging, and color preferences are personal choices for consumers, so algorithmic feedback can’t provide this kind of insight.

How does PickFu pricing work?

Pricing for PickFu polls is simple. The base price is $50 for two options with 50 responses. This enables you to test images, text, links, or embedded videos.

The price goes up when you add additional options (up to eight per poll) or audience targeting parameters. As you build your poll, a pricing calculator gives you an updated price before you finalize everything and check out. You’ll always know exactly what your costs are before you pay.

On the other hand, traditional A/B testing with live Amazon listings can be unpredictable and expensive. You have to pay for the automation software to run your tests, along with running ads to build enough traffic to gather sufficient data.

PickFu may prove your assumptions wrong

As Amazon sellers and business owners, it’s easy to get tunnel vision. You tend to favor ideas that you, your spouse, or your friends like.

We’ve seen polls where we were sure we knew which option would win. For example, one seller ran a poll for a premium but affordable mattress. They were trying to decide between calling it Ebb & Flow or Budget Bed. Because we like alliteration, we were confident Budget Bed would win.

Amazon split testing: screenshot of PickFu poll for mattress name

Contrary to our assumption, Budget Bed ended up polling poorly. Thanks to the written feedback, we learned that respondents thought Budget Bed sounded too cheap and therefore thought it would be a low-quality product.

In the written feedback of your polls, you’ll see themes and commonalities arise. Your audience will tell you which way to go and what to choose for your products.

Want to see more e-commerce split tests? Explore our gallery of polls, then get started with your own Amazon split test.

Why would you run a split test on Amazon?

The purpose of running a split test on Amazon is to improve your click-through rate and increase sales. By taking a critical look at all aspects of your product listing, FBA sellers and brands work to satisfy the needs of shoppers as well as optimize for the A10 algorithm.

Should I run more than one split test at a time?

Running more than one test at a time is not recommended. However, you can run a multivariate test, which means in a single test you change more than one variable. The difficulty is that when you make multiple changes at once, it is difficult to pinpoint which change improved your KPIs. Usually, limiting your split test to a single change provides the most definitive results.

How many factors can attribute to an increase in conversion rates on Amazon?

Thousands of factors can change your Amazon conversion rate. Most split testing tools like PickFu look at factors within a seller’s control, including the main image, product title, and product description. However, externalities can also affect your conversion rate, including market competition, price, seller reputation, political climate, external marketing, seasonality, and promotional periods. Each factor in its own way can change the mindset of an online shopper, thereby increasing or decreasing your conversion rate.