With almost 500 million products on Amazon, how do you stand out from the crowd? Split testing is one of the most effective means to optimize your Amazon product listings and increase engagement and sales on your products.

See split testing in action Start your Amazon split test

What is a split test?

A split test compares how two versions of the same piece of content perform, whether that’s a website landing page, product image, infographics, book title, social media ad, or anything in between. There are various ways to conduct a split test, including running A/B tests on your online storefront or using a polling platform like PickFu.

Split testing is a way to gauge your target audience’s preferences. It’s useful if you test one detail or minor change at a time. You might assume that your readers prefer sans serif font over serif for your logo, only to test and find out the serif version gets a lot more love.

What is split testing on Amazon?

Split testing on Amazon is an experiment between two or more variations to see which variation in the set performs best. For Amazon split testing, you might test variations of your product name, main or secondary image, product description, Amazon bullet points, or product copy.

Split testing is also known as A/B testing because there is a version called A and a version called B. (If you add a third and fourth version, it’s an A/B/C/D test).

The traditional Amazon A/B testing method alternates traffic to each version and tracks which version gets more conversions.

Asset-3-480x270-partially-transparent-watermark-470x264
Option A vs. Option B

How do I split test a listing on Amazon?

There are various testing platforms you can use to split test on Amazon. Some are free tools, while others charge a fee. Those who run an Amazon business or participate in Amazon FBA can benefit from the tools we’ll discuss later in this article.

Amazon has its own A/B testing tool, Manage Your Experiments. To use this feature, you must own your own brand and have an eligible, high-traffic Amazon Standard Identification Number (ASIN) to test. Amazon allows brand owners to test three elements of a product’s detail page: product images, product titles, and A+ content including graphs and infographics. If eligible, you can get started with this feature on Amazon’s Seller Central.

What is the difference between A/B testing and split testing on Amazon?

A/B testing and split testing on Amazon are essentially the same thing, and the terms are often used interchangeably. The main distinction is that a true A/B test compares only two versions — version A and version B — of a single variable like a headline or a product image. Split testing can refer to a test comparing more than two versions and more variables.

For example, Amazon Manage Your Experiments allows you to A/B test two variations of one listing element. With PickFu, you can split test up to eight variations of your listing, each with a different product title, description, and main image.

How do I know if one variation of my listing is better than another?

If you use Amazon Manage Your Experiments or a traditional third-party split testing tool like Listing Dojo, it will let you know which variation performed better at the end of the testing period.

Most traditional split tests need at least seven days (but ideally more) to collect enough data and determine a conclusive winner. Some programs, including Manage Your Experiments, require you to run the test for a certain amount of time.

While you’ll know which variation got more traffic, you won’t necessarily know why. That’s because in a traditional split test, consumers don’t know they are participating and they don’t give feedback.

With PickFu, you get detailed written feedback from every respondent, usually within hours. This can help you understand not only which variation won, but why.

Both forms of split testing play an important role, but in an increasingly time-strapped world, testing tools that offer quick yet qualitative results can help you streamline your marketing strategy, netting you more conversions.

Amazon split testing tools

Listing Dojo and Cashcowpro are among the best Amazon A/B testing tools to help sellers run split tests of their brand’s listings on Amazon. For eligible sellers, there’s also Amazon’s own split testing feature, Manage Your Experiments.

Traditional split tests direct a portion of your Amazon traffic to one test variation and another portion to a different variation. After the test, you compare how shoppers behaved with each variation. For instance, on average, did one set of visitors browse for longer? Did one set convert at a higher rate?

However, the tests that these tools offer have some limitations:

  • For one, in order to gather enough data to make an informed decision, you have to accrue enough traffic to test.
  • In order to do this, you’ll probably need to pay to promote your products on Amazon, which can get expensive.
  • The tests also may take quite a bit of time, depending on the traffic levels
  • Further, because you’re using live traffic for your test, you risk seeing an adverse effect on sales or search ranking if a test variation performs worse than your current listing.

How many A/B tests does Amazon run?

If you use Amazon’s A/B testing tool, Manage Your Experiments, you can run one test for each eligible high-traffic ASIN at a time. Tests last for a minimum of four weeks and a maximum of 10 weeks, after which you receive detailed results that include sales and conversion rates.

These are valuable metrics provided by real customers in real time. The downside is that baseline four-week period of time. Waiting that long for results on small or large-scale differences between your variations can affect your sales.

With PickFu you’re testing outside of Amazon, so you can run as many A/B tests as you want without risking a drop in sales. Choose a target audience of Amazon shoppers and create a poll with two or more variations of your listing. You’ll receive your test results within hours, not weeks, so you can make changes to your listing much more quickly. This will result in improved Amazon listing optimization for all your ASINs — not just the high-traffic ones.

What are the benefits of A/B testing on Amazon?

A/B testing on Amazon gives you direct insight into how your buyers behave and what they want. With this invaluable information, you can make changes to your Amazon product listing and your ads, refine your digital marketing strategy, and attract more customers.

Whether you A/B test within Amazon, use a third-party split testing tool, or run a poll with PickFu, the data you gather from split testing can give you an advantage over your competitors.

How do I find the results of my Amazon split tests?

When your traditional split test is complete on Amazon or a third-party split testing tool, the results will be available within the program you’re using. Many of these tools provide graphs detailing the journey of each tested variant.

With PickFu, you’ll be notified via email when your test is complete. You can view the results on your PickFu dashboard.

Split testing on Amazon using PickFu

PickFu is a polling software that enables you to quickly test elements of your Amazon listing with audiences that match your customer profile and target demographics. For example, you could test product names, copy, or logos, or try image split testing with variations of your main photo.

Differences between live split testing and PickFu polling

  • Traditional split tests use Amazon shoppers as they are shopping as test subjects.
    • The PickFu Panel is comprised of U.S.-based respondents who are paid a small stipend to answer your poll. A popular audience available to test on PickFu is Amazon Prime members. However, a PickFu split test is conducted outside of the Amazon marketplace, so there may be variances between the PickFu Panel and Amazon shopper demographics.
  • When running a traditional split test, Amazon shoppers are not aware that their behavior is being monitored. They do not know that there are other versions of your listing other than the one they are seeing.
    • On PickFu, the respondent panel sees all variations in order to compare them and provide their feedback. Not only does each respondent vote on the variation they like best, but they each also provide a written comment explaining why they chose their preferred option. These written comments provide insight into what works with shoppers to help guide your decisions.
  • Traditional split tests often take days or weeks to complete.
    • Most PickFu polls complete within hours.
  • The cost of traditional split testing on Amazon is often unpredictable. You need to pay for the testing software you’re using to split test, as well as the cost of the ads you’re running to build traffic.
    • With PickFu, the cost is a flat fee. Polling starts at $50 for a 50-person panel and changes as you add options and customize your poll.
  • Traditional split tests use live Amazon listings.
    • On PickFu, your product does not need to be live on Amazon. You can test product variations such as color options or design before going into production. Get a sense of what shoppers want before committing to inventory. Or perfect your Amazon packaging with feedback from PickFu’s audience.
  • Traditional split tests can only compare variations of your own Amazon listings.
    • Using PickFu, you can perform competitive analysis by polling your listing against competing listings. Gain insight into how your product fares against category leaders. Split testing product copy is especially effective with competitors.

The perks of testing outside of Amazon

Since PickFu polling happens outside of Amazon, you can run polls for idea validation before going into production or committing to inventory.

If you already have live listings, using PickFu means there’s no risk of losing sales if the new variant performs worse than the original. You don’t have to change anything in your listing until the poll results have shown it will perform better.

Another benefit of testing outside of Amazon is the ability to compare your products to a competitor or category leader. You can match your product directly against another to see how it fares with respondents. Since respondents leave written feedback along with voting, you discover what you’re doing well and can work that into the messaging of your listings and branding.

What can you split test?

Using PickFu, you can run Head-to-head or Ranked polls to test every element of your Amazon listings, such as

You can also run competitive tests by linking to or taking screenshots of your Amazon product page and a competitor’s product page and asking respondents to choose the most compelling one.

Respondents submit their votes along with written feedback for every poll.

To get feedback on a single idea, run an Open-ended poll. Ask your target audience an open-ended question about whether or not an idea looks enticing (idea validation), what they think of a product’s functionality, or if anything’s missing from your PPC ads.

If you’re stuck on price points for your products, try asking questions such as:

  • How much would you pay for this product?
  • Which product looks more expensive?
  • Do you think $14.99 is a fair price?

We’ve found from previous polls that if you list a price in the details, people will leave their opinion on price anyway. You may see feedback such as, “I prefer the photo from Option A, but I wouldn’t pay that price for it.”

How PickFu protects the confidentiality of your business

When running your own e-commerce business, especially with proprietary products you’ve created, confidentiality is a top concern. We get it. That’s why we have several measures in place to protect the confidentiality of your ideas, products, and business.

Before taking part in any polls, every respondent signs a thorough non-disclosure agreement (NDA). Respondents are restricted from taking or sharing anything they see in a poll.

PickFu polls are private by default. This ensures that anything you share in the poll won’t be indexed by search engines and stays between you and the respondents. However, you have the option to include your poll in our public poll gallery, where sellers can get ideas on how to use polls and see how effective they are. You’ll get a $5 discount for sharing your poll in the public gallery.

Who are the PickFu respondents?

We use a variety of online services to find our respondents, all of whom are based in the United States and native English speakers. On the back end, we have proprietary quality controls that include internal rating systems and spam protection. We take quality seriously and are always working to ensure that respondents give insightful feedback.

While we have gathered a range of people to respond to PickFu polls, you have the ability to customize your audience. You can choose a random audience, which is a mix of genders, age groups, and interests. You can also specify the audience who votes in your poll.

One popular option for e-commerce sellers is polling Amazon Prime members. If you’re targeting your product to a specific kind of consumer, use filters like gender, age range, educational level, and income. For example, if you have a high-end product, you can target people who earn more than $100,000 per year.

Screenshot-2023-11-03-at-2

You can choose multiple filters to get a specific audience, such as female homeowners or male mobile gamers in their 20s. We have categories that are especially beneficial for e-commerce users, such as cat owners or dog owners for pet products.

Other information we gather from respondents includes:

  • exercise habits
  • cooking habits
  • if they use cosmetics
  • if they take nutritional supplements
  • the type of mobile device used
  • if they’ve donated to a Kickstarter campaign before

By combining different categories, you can focus your poll audience on the same type of shoppers you’re targeting with your products.

Our split testing software in action

When you create a poll on PickFu, start by inputting your question. For example, ask “Which image do you prefer?” or “Which design do you prefer?” You can input up to eight different options in a single poll, and these options can be images, text, links, or embedded videos.

If you’re testing more than two options, you can either have respondents rank the options in the order they prefer or set up head-to-head matchups, which is a more complete way of polling.

You can also test product descriptions and entire product listings by including links in your polls.

Split tests of product photos

Screen-shot-2022-02-17-at-3
This split test with 50 respondents finished in 26 minutes.
Screen-shot-2022-02-17-at-3
Results from a split test with 50 Amazon Prime members came in 30 minutes.
Screen-shot-2022-02-17-at-3
A product image test of exercise equipment with 50 female respondents. Can you guess which one won?

Split tests of physical product design

Screen-shot-2022-02-17-at-3
A 50-respondent poll for product design. Results took 24 minutes.
Screen-shot-2022-02-17-at-3
A 4-option product design poll with 50 female respondents. Results came back in under an hour.
Screen-shot-2022-02-17-at-3
A 50-respondent poll for an olive oil dispenser. Results completed in only 13 minutes.

Split tests of product names and brand names

Screen-shot-2022-02-17-at-3
Split test with 50 female respondents asked to rank three potential names for a natural skincare brand.
Screen-shot-2022-02-17-at-3
A 50-respondent poll to rank names of an e-commerce store.
Screen-shot-2022-02-17-at-3
Fifty respondents who live in the suburbs voted in this Head-to-head poll for a high-end baby brand.

How long does a PickFu poll take to complete?

Once you’ve created and submitted a poll, we immediately find respondents to vote. Depending on your targeting parameters, this can happen quickly. If you choose a random audience or something simple like 50 female respondents, the poll may finish in minutes.

If you are specific with your targeting, it can take longer for the poll to complete. For a Mother’s Day poll, we targeted mothers by combining female respondents with those who have children. This poll took two hours to complete.

There are other factors involved in the speed of poll completion. If you are running a poll at night or on the weekend when fewer people are online, it may take longer to finish. But typically, polls finish within hours — often less.

How PickFu compares to other Amazon testing tools

When you compare PickFu polls to other Amazon split testing tools such as Listing Dojo, there’s one major difference that sets us apart: we’re people-powered. Thanks to our live respondents, business owners gain more insight with PickFu polls than they would with other split testing tools.

For example, one e-commerce seller was developing a garden product for people who live in apartments and don’t have much outdoor space. The product would allow these apartment dwellers to grow a garden in a compact space.

The seller created a poll to test two product names, Patio Harvest and Porch Farmer. Patio Harvest easily won the poll with 76% of the vote, but the real insights came from the written feedback.

Many respondents commented that apartments don’t have “porches,” while others said they don’t consider themselves “farmers.” Because of these reasons, respondents said the name Porch Farmer didn’t resonate with them.

A traditional split testing or keyword tool would never be able to give you such specific feedback. Product names, packaging, and color preferences are personal choices for consumers, so algorithmic feedback can’t provide this kind of insight.

How does PickFu pricing work?

Pricing for PickFu polls is simple. The base price is $50 for two options with 50 responses. This enables you to test images, text, links, or embedded videos.

The price goes up when you add additional options (up to eight per poll) or audience targeting parameters. As you build your poll, a pricing calculator updates the price. You’ll always know exactly what your costs are before you finalize everything and check out.

On the other hand, traditional A/B testing with live Amazon listings can be unpredictable and expensive. You have to pay for the automation software to run your tests, not to mention run ads to build enough traffic to gather sufficient data.

PickFu may prove your assumptions wrong

As Amazon sellers and business owners, it’s easy to get tunnel vision. You tend to favor ideas that you, your spouse, or your friends like.

We’ve seen polls where we were sure we knew which option would win. For example, one seller ran a poll for a premium but affordable mattress. They were trying to decide between calling it Ebb Flow or Budget Bed. Because we like alliteration, we were confident Budget Bed would win.

We were wrong. Ebb Flow ended up winning the poll. Thanks to the written feedback, we learned that respondents thought Budget Bed sounded cheap and therefore felt it would be a low-quality product.

In the written feedback of your polls, you’ll see themes and commonalities arise. Your audience will tell you which way to go and what to choose for your products.

Want to see more e-commerce split tests? Explore our gallery of polls, then get started with your own Amazon split test.

Frequently asked questions

Why would you run a split test on Amazon?

The purpose of running a split test on Amazon is to improve your click-through rate and increase sales. By taking a critical look at all aspects of your product listing, FBA sellers and brands work to satisfy the needs of shoppers as well as optimize for the A10 algorithm.

Should I run more than one split test at a time?

Running more than one test at a time is not recommended. However, you can run a multivariate test, which means in a single test you change more than one variable. The difficulty is that when you make multiple changes at once, it is difficult to pinpoint which change improved your KPIs. Usually, limiting your split test to a single change provides the most definitive results.

How many factors can attribute to an increase in conversion rates on Amazon?

Thousands of factors can change your Amazon conversion rate. Most split testing tools like PickFu look at factors within a seller’s control, including the main image, product title, and product description. However, externalities can also affect your conversion rate, including market competition, price, seller reputation, political climate, external marketing, seasonality, and promotional periods. Each factor in its own way can change the mindset of an online shopper, thereby increasing or decreasing your conversion rate.


Guide-illustration
Mix and match over 90+ targeting traits to create your perfect lookalike audience today.
Get started

Learn more about PickFu for E-commerce. See how PickFu customers test Product descriptions, Physical product design, Product images, and Product listings.