A/B tests: test a new version of your quiz

#How an A/B test works

An A/B test runs your quiz as it is (the original) against a copy with one change (the test version). Some of the original’s visitors get the test version instead, and Teacup Quiz counts the two separately. When one clearly does better, you keep it.

The test version only ever shows where the original is: in the same sections, product page buttons, popup and links. You don’t place it anywhere. If the original is on your home page and in the popup, visitors in the test group get the test version in both places.

For example, a skincare store could test a first question with pictures of skin types against the same question in words. A coffee shop could test its quiz with an email step against the same quiz without one.

#The three steps

1. Make a test version: a copy of a quiz that’s on your store.
2. Change one thing: the first question, the results or the look.
3. Start the test: visitors are split between the two. Keep the one that sells more.

Each quiz has at most one test version, and a quiz can be in only one test at a time.

#Make a test version

Any of these makes one:

• Open A/B tests, pick the quiz under Quiz to test, and click Make a test version.
• On the quiz’s analytics, click Make a test version in the A/B test card.
• In the quiz editor, click A/B test at the top. It opens A/B tests with that quiz picked; then click Make a test version.

You’ll see “Test version ready. Change one thing, then start the test.”, and the test version opens in the editor. It’s a copy of the quiz as last saved (including changes you haven’t published yet), named after it with “· test version”, such as “Skin quiz · test version”, with its own ID ending in -b. Making one again opens the same test version.

Quiz to test lists published quizzes that aren’t in a test and don’t have a test version waiting, with your main quiz first. When there are none, it says “Publish a quiz and put it on your store first: the test runs where it is.”

#Change one thing

The test version’s editor works like any quiz’s, with the badges Test version and Not started. Shoppers don’t see it until the test starts, and Start the test takes the place of Publish.

Change one thing, so you know what made the difference. Good candidates:

• The first question: its wording, pictures, or fewer answers.
• The results: more or fewer products, a personal result instead of products that fit, or a bundle discount.
• The look: another theme, colors or picture style.
• The welcome screen or the email step: on or off, optional or required.

“· test version” is only there for you: shoppers in the test group see the original’s name above the quiz, and your tracking events use it too. If you rename the test version, shoppers see the new name.

If you change texts in the test version and your quiz is translated, update its translations in Languages too. Otherwise shoppers browsing in another language see the older wording.

#Start the test

From the editor

1. In the test version’s editor, click Start the test.
2. Under Who sees this test version, pick the split.
3. Click Start the test.

Teacup Quiz saves what’s in the editor, checks the test version, publishes it and starts the test. You’ll see “Test started. Visitors now get the original or this test version.”, and A/B tests opens.

From A/B tests

A test version that hasn’t started shows on A/B tests as Ready to start. Pick Who sees the test version and click Start the test. Edit test version opens it in the editor, and Delete removes it after asking “Delete this test version?”

If some ways of answering the test version end with no products, Start the test asks first, from the editor and from A/B tests alike: “Some shoppers would get no products”, with Show me (the test version’s Check every answer) and Start the test anyway.

#How visitors are split

• Each visitor is picked at random the first time they meet the quiz. Their browser remembers the pick, so they get the same version on every page and on later visits.
• When a browser blocks storage (some private modes do), it can’t remember: the pick is made again on each page load.
• The same person on another device or browser is picked again, and may get the other version.
• Shoppers browsing your store in another language get the test version in that language when it’s translated.
• The page first shows the original’s opening screen, and the test version takes its place as soon as the quiz loads. On a slow connection, shoppers in the test group can see the original’s welcome screen or first question for a moment.
• Each version keeps its own numbers: views, starts, finishes, emails, carts and orders.

#Where tests show in the app

• Home hides test versions. The original’s card shows A/B test running or Test version ready, with an A/B test link.
• Analytics lists the test version as a quiz of its own, and it has its own analytics page. While a test runs, the A/B test card sits at the top of both quizzes’ pages, with See the results.
• A/B tests shows each running test with both versions side by side. See Reading A/B results.
• In the test version’s editor, the badge reads Test running, and See the results and Open the original take you to the test and the original. Duplicate, Unpublish and A/B test aren’t offered on a test version.

#While a test runs

• Changes you publish in the test version reach visitors right away and mix into its results. The editor reminds you: change it only if something is wrong.
• The same goes for the original: publishing changes to it changes what its visitors see during the test.
• Unpublishing the original, or deleting either quiz in the editor, ends the test and keeps the original. Deleting the original also deletes its test version.
• A test version in a running test can’t be deleted from A/B tests (“This test version is in a running test. End the test first.”). End the test first, for example with Keep the original.
• Shoppers who finish the test version count toward your plan’s completed quizzes, like any quiz.

#If you change to a plan without A/B tests

See Trial and changing plans.

#Tests started the wrong way round

With an older setup, a test could have the quiz on your store as the test version instead of the original, so the test got no visitors. Opening A/B tests turns such tests the right way round automatically. There’s nothing you need to do.