Testing e-commerce listings against competitors: PickFu interview with EcomCrew

Mike Jackness and Dave Bryant are the minds behind EcomCrew, a site that gives entrepreneurs the type of e-commerce advice that produces real-time results. In their EcomCrew Podcast, the two share personal advice and interview businesses that can help their customers grow their brand.

After meeting PickFu co-founder John Li at Sellers Summit, Mike Jackness decided to host John on his podcast.

Read on for interview highlights, or listen to the entire interview here.

PickFu’s advantage over Splitly

To introduce PickFu, Mike talked about a truth everyone learns at one point or another: You go through life thinking you had things figured out — until you find a new way.

That’s what happened to Mike. Before he met John at Sellers Summit, he’d heard of PickFu before but never tried it. Now, he’s used it — and he loves it. He even wants to use PickFu before using a live split-testing tool like Splitly.

Mike noted that it was funny that he’d say that because at the moment of recording, he was sitting in the Jungle Scout offices (Jungle Scout owns Splitly) talking about split-testing images on PickFu instead.

He acknowledged that Splitly is one of his favorite tools and that he and Dave use it in their business, but being able to test before you have any data from Amazon is a real win. That’s why PickFu is so valuable.

The inception of PickFu

To start the interview with John, Mike asked him how PickFu got started.

PickFu started as a side project almost ten years ago when John and his partner were running a completely different business online. They were both developers by training and didn’t have a strong design background. They’d have tons of design-related debates that never got resolved.

John said, “So what do two developers do when they need to solve arguments? Create a software, of course.” He and co-founder Justin got the minimum viable product online in a weekend. It was ugly, but it worked. It helped them make decisions.

They tossed it up on some forums and didn’t think much of it. A PayPal button got added at some point. People kept using it, and it grew organically over time.

In the last three or four years, John and his partner refocused on PickFu. Only in the past year or so did they figure out how valuable it is to sellers online.

The most common uses for PickFu

PickFu has a panel of about 10,000 people who vote on their favorite version of something and provide feedback accordingly.

The most common uses are:

  • Testing main images for product listings.
  • Figuring out which titles and descriptions work best.
  • Competitive testing of the brand’s search results, i.e. pulling a screenshot of a bestseller on Amazon versus their own competing product and running a poll to Amazon Prime Members that asks, “Which one is better?”
  • Getting open-ended feedback with a question such as, “How could this landing page improve? What do you like and what don’t you like?” This gives you information on what you’re missing from the customer’s perspective.

People also test email copy, logos, and just about anything else you can think of.

John’s surprise poll for Wild Baby

Going back to the story of how he met John, Mike spoke about an email John sent him after they talked in Miami.

John sent not just a note but the results of a poll he’d created for one of Mike’s brands, Wild Baby. What’s amazing is that Mike and his team were in the process of re-designing the Wild Baby website, which John could never have known.

The poll John ran asked, “What would you change about this store homepage to make it more compelling?” He sent this out to females who had 1, 2, or 3 kids.

The responses varied, but went something like this:

  • “Nicer colors. Sorry but the taupe is very dull and boring.”
  • “The About Us should be listed in a brief paragraph to tell the customer about the website and products.”
  • “The website needs to appear more uplifting. Use more vibrant colors and fonts.”
  • “It’s very cute, but a little boring. Maybe make it a little more colorful. Add photos of some kind, or animation. More visually dynamic.”
  • “I would add more colors.”
  • “The colors are a bit drab.”

After reading all the comments, Mike saw a theme running through them: the colors needed improvement.

Wild Baby reinvented

Mike finished with the website re-designs and then decided to run another PickFu poll to see what commenters had to say about the changes.

This poll said, “We’re redesigning our online store. Is there anything missing or unclear in this design? Please ignore the fuzzy images and the dark surrounding borders.”

John added that it can help to specify what you want reviewers to ignore, especially if it’s something that will already be fixed in the final product — like fuzzy images.

If you ask your respondents to ignore something, they generally will and you’ll get the feedback you need.

The responses for this poll were starkly different than the first.

  • “It looks nice, but it’s a bit long. You have to do a lot of scrolling. Maybe get rid of the blog portion at the bottom.”
  • “I feel that all the bases are covered, the website is well-designed and easy to understand. I don’t think anything is missing.”
  • “It’s straightforward, products are front and center.”
  • “The design is to the point and it catches my attention.”

In short, there wasn’t much wrong with the site at all.

MIke appreciated how the PickFu polls he ran gave him an insight into what people are thinking in a way that would otherwise be impossible.

How Splitly and PickFu can work together

He again mentioned Splitly, a tool that split-tests images or titles or other content from your Amazon listing and gives you real data behind which one is performing better.

The downside of Splitly — which Mike does love — is that it’s more expensive than running a PickFu poll, and it takes a long time to get a definitive answer on something as simple as an image test. Weeks, even.

So Mike decided to split-test on PickFu some images that he had already run on Splitly. Splitly and PickFu both showed the same winner.

Because of this, Mike has decided to shortcut the whole process and split-test images on PickFu first, instead of the other way around. On PickFu, you can get a statistically probable answer for less money in a much shorter amount of time — from fifteen minutes to an hour or two.

Not only that, you get answers as to why poll respondents picked what they did.

John pointed out that it’s not an either/or situation. You can use PickFu in the launch phase and then get real data on your listing from Splitly. At its heart, PickFu is all about starting entrepreneurs off on the right foot and helping them make fewer bad business decisions.

Packaging matters

In another poll Mike ran, Amazon changed the images on them right as the poll went live, which can happen, frustratingly enough. The results were 50-50.

Even though the poll didn’t run with the correct images, Mike still read the comments from those who liked their product — because the product itself was the same.

Commenters said things like this:

  • “This one’s easy. First, it’s cheaper and I can see the packaging.”
  • “Seeing them in a bag is nice. It gives them some order.”

Mike thought it was neat to know that people appreciate the type of packaging they have.

John noted that in an image split-test, the choice that shows the packaging generally wins.

This poll drove Mike to make sure that when he can re-run this poll with the correct photos, he includes an updated picture with his brand’s updated packaging.

If you have a product on Amazon, run a PickFu poll with your biggest competitor. See why people might think their listing is better. If it’s because of this or that feature, then you know you need to highlight the product.

Pitting two competitors against each other

In another poll, Mike pitted two competitors against each other in a poll. This one was 50-50, which told Mike the other competitors all look the same. So he and his team asked themselves, “How can we differentiate ourselves from them?”

Next, they ran a poll about EcomCrew Premium. They asked, “What would you change on this page to make it more compelling?” about the EcomCrew Premium sign-up page.

The answers from 100 respondents were all over the place in many ways:

  • “I would use full-color photos instead of black and white.”
  • “They could have added more reviews.”
  • “Make more tabs, describing different areas of the webpage.”
  • “Make it more bright and full of color.”
  • “Too much text on one page.”

What do you do when the comments are so contrarian to what converts? You have to take what you can use. While Mike would be remiss to make the page smaller because he knows long-form content converts better, he did decide to take a second look at the colors on the page.

John jumped in to say that PickFu has a grouping mechanism where you can filter the responses by age, gender, and more. PickFu also does a word count of non-common words in the responses. This shows you what’s mentioned often. If you search “color,” that will only pull up the responses that mention color.

When Mike did this, he could more easily scan the color-related comments. Basically, the theme of all the comments? Add more color.

He decided to share this information with his designer. How could you possibly ignore that?

How much does PickFu cost?

PickFu starts at 50 bucks for 50 basic respondents — a fraction of a cost of one month of Splitly.

If you add audience targeting or other features, the price increases. There is a real-time pricing calculator as you build your poll to let you know what your final price will be.

PickFu does a lot of behind-the-scenes work to make sure its relationship with respondents is high-quality. When people aren’t responsive or thorough, PickFu ensures they don’t stay for long.

Mike closed the interview by reinforcing that PickFu has made some neat results in his business and he looks forward to using it for a long time to come.


Learn more: Optimize your product listings by testing design concepts, photos, and descriptions with a target audience of likely buyers.

Laura Melchor

Laura Ojeda Melchor (she/her) is a freelance writer whose work has appeared in Parents.com, Mom.com, Gardener’s Path, and of course, PickFu. She holds a Master of Fine Arts in Writing from Vermont College of Fine Arts. Her debut middle-grade novel, Missing Okalee, was published in the fall of 2021 by Shadow Mountain Publishing.