Amazon split testing

Amazon split testing is an essential step in your product listing process if you want to stand out on the e-commerce platform. 

Sure, there are millions of products listed on the site. It seems impossible to even begin to differentiate yourself from the competition.

But it’s not.

There are lots of good listings on Amazon, but there are also plenty of typo-riddled, unclear Amazon product listings, too. With Amazon A/B testing, your product can leap out of the pack.

We can assure you that all the top brands run A/B tests before going live with their listing – and after.

There’s really no other way to know whether your listing content will appeal to your target audience.

Table of Contents

• What is a split test?
• Split testing with Amazon MYE
• Split testing with PickFu
  • PickFu in action
  • What is an Amazon Mockup Generator?
• Benefits of PickFu polls vs. live Amazon split tests
• Amazon split-testing with PickFu

What is a split test?

A split test measures how well two different versions of the same piece of content perform. You can test all sorts of things: website landing pages, product images, infographics, book titles, and social media ads.

Split testing on Amazon is an experiment between two or more variations to see which variation in the set performs best. You can test variations of your product name, images, and description, plus Amazon bullet points and product copy.

You can even test pre-launch elements – think product color and design variations. Or packaging design variations.

Depending on the split testing tool you use, you’ll either get quantitative data or qualitative data. (With PickFu, you get both. More on that in a second.)

Split testing with Amazon MYE

Certain Amazon sellers can run live split tests with Amazon’s Manage Your Experiments (MYE) feature. 

If you own a brand and are responsible for selling items under that brand name in the United States, you can use Amazon MYE. Basically, you must meet the criteria for the Amazon Brand Registry. If you don’t fit into this box, you’re out of luck.

For brand owners, here’s how Amazon MYE works. You’ll begin by picking the product you want to run your experiment with. Then, you must select an experiment type.

You’ll then submit a new version of your product listing to test against the original. You can change whatever you want, but we recommend only changing one feature at a time. If you want to test an updated version of a product’s Amazon A+ Content, for instance, only change the A+ Content section of your product listing. Not the bullet points, product description, or product image.

Once you go live with the test, Amazon will split your web visitors in half, with 50% going to the old version and 50% going to the new. Of course, you have to have enough traffic on your listing for this to actually work. If no one visits your listing, you won’t be able to run split tests with Amazon.

Even if you do have the amount of traffic you need, a traditional, live split test takes time to show results. You can set your Amazon A/B test up in two ways:

• Have it run until it reaches statistical significance – in other words, until one of the options clearly beats the other
• Have it run for a set number of weeks

Every week, Amazon will tell you which of the two versions is converting best – and how that might impact your sales.

So let’s say you split test two versions of a product title. After one week, you get your first results: Version B has a higher conversion rate than Version A by 7.23%. This means if you earn $300 a week with Version A, Version B can earn you an extra $21.69 per week – or roughly $86.76 more per month.

Of course, there’s no guarantee you’ll get enough data in your first week of testing to even show results. Most live split tests need at least two weeks, if not four or five, to show meaningful results.

Once your Amazon split test is done, you can manually publish the winning version. You can even set up Amazon MYE to automatically publish whichever version ends up performing better, saving you time and effort.

See our step-by-step guide to MYE for more details.

Split testing with PickFu

If you don’t want to spend time waiting for your MYE results to come in on your Amazon Seller Central platform, there’s another excellent tool for Amazon FBA sellers: PickFu, of course.

via GIPHY

PickFu is an Amazon product research tool that enables you to quickly test elements of your Amazon listing with audiences that match your customer profile and target demographics. For example, you could test product names, copy, or logos, or try image split testing with variations of your main photo.

Split testing with PickFu takes minutes, not weeks. No need to worry about traffic, either – PickFu brings the respondents to you.

This means you can test different variants before you even invest in creating new products. Or at the very least, before you launch them on Amazon. Amazon SERP optimization begins with product listing optimization, after all.

That way, you can go live with the best version of your product and listing right away. You can also test other assets with PickFu, like PPC ad copy and even product features.

PickFu in action

Top Amazon sellers, brands, and agencies use PickFu for listing optimization all the time.

This main image split-test poll is one key example. An Amazon seller asked 50 U.S.-based homeowners and Amazon Prime subscribers which product image they preferred for a hook rack.

The two images were nearly identical. They showed an identical product – a wooden bracket with silver hooks for hanging up things like coats, hats, and backpacks.

There was one key difference between the two images: Option B showed the product packaging. Option B did not.

The target audience of Amazon Prime users/homeowners picked Option B as the winner. Here’s why:

• “Given the opportunity, I typically prefer to see packaging included in a product display.”
• “I like [Option] B because, with the box in the ad, it makes me think that it will come already assembled and I won’t have to put it together.”
• “This image shows the box that it ships in, and the dimensions are on the box, too. More information is always more useful.”

Since the poll is such a close one, it’s worth looking at both sides of the debate. Respondents who voted for Option A preferred the more zoomed-in image, so if the seller zoomed in a little more, Option B might be even more of a winner.

The product packaging might need sprucing up, too. A respondent who ended up voting for Option A wrote, “I normally choose the one that includes the packaging but in this case, it does not enhance the image.”

The PickFu panel’s signature written feedback provides more than a spotlight on areas you need to improve. You’ll also discover what you’re doing well – and you can work that into the messaging of your listings and branding.

Starting an Amazon split-test poll is quick and easy with PickFu. Get started with one of our Amazon testing templates.

And psst, guess what? We’ve got even more tools to help you boost your Amazon business. One of our favorites is our Amazon Mockup Generator.

What is an Amazon Mockup Generator?

PickFu’s Amazon Mockup Generator lets you create mockups of your Amazon listing. Instead of changing details on Photoshop every time you set up a new test, you can use the mockup tool to quickly create variations – no graphic design skills needed.

You can pull ASINs from the detail page of an existing Amazon listing or use keywords for a hypothetical product. Use our mockup generator to test variations of your own listing, or to line your listing up next to your top competitors’ products.

Then, you can import your mockups directly into PickFu’s poll builder to split-test them with real consumers.

With the Amazon Mockup Generator, you can easily discover the best main image, title, and price point combo to stand out against the competition.

Benefits of PickFu polls vs. live Amazon split tests

Since PickFu polling happens outside of Amazon, you can run polls for idea validation before going into production or committing to inventory.

If you already have live listings, using PickFu means there’s no risk of losing sales if the new variant performs worse than the original. You don’t have to change anything in your listing until the poll results have shown it will perform better.

Here are a few key differences between these two types of Amazon split tests:

Traditional split tests use Amazon shoppers as they are shopping as test subjects.

The PickFu panel is composed of vetted respondents who are paid a small stipend to answer your poll. A popular audience available to test on PickFu is Amazon Prime members. However, a PickFu split test is conducted outside of the Amazon marketplace, so there may be variances between the PickFu panel and Amazon shopper demographics.

When running a traditional split test, Amazon shoppers are not aware that their behavior is being monitored. They do not know that there are other versions of your listing other than the one they are seeing.

On PickFu, the respondent panel sees all variations in order to compare them and provide their feedback. Not only does each respondent vote on the variation they like best, but they each also provide a written comment explaining why they chose their preferred option. These written comments provide insight into what works with shoppers to help guide your decisions.

Traditional split tests often take days or weeks to complete.

Most PickFu polls complete within hours.

Traditional split tests cannibalize your sales.

By definition, a worse version of your Amazon listing will always be showing while you run live split tests – meaning you lose out on conversions and sales. With PickFu, testing before you launch means you reduce this risk and avoid lost sales.

The cost of traditional split testing on Amazon is often unpredictable. You need to pay for the testing software you’re using to split test, as well as the cost of the ads you’re running to build traffic.

With PickFu, you only pay for the polls you run – and pricing is transparent up front. Tests start at $15 for a 15-person panel and change as you customize your poll options or audience targeting.

Traditional split tests use live Amazon listings.

On PickFu, your product does not need to be live on Amazon – meaning you can experiment with variations that the general public will never see. Test product variations such as color options or design before going into production. Get a sense of what shoppers want before committing to inventory. Or perfect your Amazon packaging with feedback from PickFu’s audience.

Traditional split tests can only compare variations of your own Amazon listings.

Using PickFu, you can perform competitive analysis by polling your listing against competing listings. Gain insight into how your product fares against category leaders. Split testing product copy and your main image is especially effective with competitors.

Amazon split-testing with PickFu

When running your own e-commerce business, especially with proprietary products you’ve created, confidentiality is a top concern. We get it. That’s why we have several measures in place to protect the confidentiality of your ideas, products, and business.

Before taking part in any polls, every respondent signs a thorough non-disclosure agreement (NDA). Respondents are restricted from taking or sharing anything they see in a poll.

PickFu polls are private by default. This ensures that anything you share in the poll won’t be indexed by search engines and stays between you and the respondents.

However, you have the option to include your poll in our public poll gallery, where sellers can get ideas on how to use polls and see how effective they are. You’ll get a small discount for sharing your poll in the public gallery.

Ready to get started?

Create your first Amazon split-test poll on PickFu today. Get ready to gain new insights, improve your click-through rate (CTR), and drive more conversions – all without taking the risks associated with a real-time test. 

FAQs

What Is A/B Testing?

An A/B test compares two versions of a webpage, product listing, or other content to see which one performs better. By showing each version to different groups, you can see which one gets better results and make decisions based on real data.

How can I conduct A/B testing on my Amazon product listings?

There are two main tools to use for conducting A/B testing on your Amazon product listings. Amazon’s Manage Your Experiments A/B testing tool is ideal for live testing. PickFu is a great choice for risk-free testing that doesn’t interrupt your live listing—or that lets you perfect the listing before you publish it.

Either way, you’ll create two versions of your product title, description, or main images. With Amazon’s Manage Your Experiments tool, Amazon will split your audience in half and send each half to one of the variations. You’ll receive updates on key metrics like conversion rate, sales, and customer engagement over a set period, typically a week or more.

With PickFu, you don’t test your actual customers. You run a poll to a target audience of your choosing, and they provide a vote and written feedback in return for a stipend. You’ll get both qualitative and quantitative data from PickFu’s Amazon split-testing polls.

After gathering sufficient data, you can publish the more successful listing.

Is Amazon FBA worth it?

Amazon FBA can be a great option for many sellers, but it really depends on your business and goals. With FBA, Amazon handles storage, shipping, and customer service, which can save you a lot of time and hassle.

That said, Amazon FBA startup costs can add up quickly, including buying inventory, shipping to Amazon warehouses, and FBA fees. Make sure to crunch the numbers on your product profitability and sales potential to see if FBA is the right fit for you.

Feature/Aspect	Amazon MYE A/B Testing	PickFu Split Testing
Who Can Run Tests	Available to brand owners in the Amazon Brand Registry	Available to any Amazon FBA seller
What You Can Test	Product name, images, description, bullet points, A+ Content	Product names, copy, logos, images, packaging, PPC ad copy, product features
How Long Tests Take	Takes days or weeks to show results	Results in minutes to hours
Traffic Requirements	Requires enough traffic on the listing	PickFu provides respondents, no traffic required
Type of Data Collected	Quantitative data (conversion rates, sales impact)	Both qualitative (written feedback) and quantitative data
Test Environment	Conducted on live Amazon listings	Conducted outside of Amazon marketplace
How Results Are Compared	Amazon splits web visitors 50/50 between versions	Respondents see all variations and provide comparative feedback
Risk Level	Potential risk of losing sales if new variant performs worse	No risk to live listings; testing done before changes are made
Respondent Feedback	No direct feedback from respondents	Detailed written feedback explaining choices

Mix and match over 90+ targeting traits to create your perfect lookalike audience today.

Get started

---

Learn more about PickFu for E-commerce. See how PickFu customers test Product descriptions, Physical product design, Product images, and Product listings.