Reading A/B results and picking a winner

#The test card

Each running test has a card on A/B tests, titled with the original’s name. Under the title: the day it started, the split, and “picked once per visitor”, with a green Running chip. Then one sentence says where the test stands, and a table compares the Original and the Test version, each with its quiz name.

At the bottom, Original’s analytics and Test version’s analytics open each quiz’s own page, with every card described in Reading your analytics.

#What each row shows

Each cell shows a rate in large type and the count behind it in small type.

Visitors are counted as views: once per page load where the quiz was on screen or its popup opened. A shopper who sees the quiz on three pages counts three times, in both versions alike. Every rate is per visitor, not per start, so a version that gets more shoppers to begin scores higher too.

Difference compares the test version’s rate with the original’s: “+12%” means 12% higher than the original’s rate, not 12 percentage points. “Same” means less than half a percent apart. It stays grey until both versions have 200 visitors, then turns green or red. On Saw the quiz, it shows the split instead, such as “50% see the test version”.

Left their email also counts emails from your own test runs and from shoppers who declined analytics, just like Email rate on a quiz’s page.

#When counting starts

• Counting starts at 00:00 UTC on the day after you start the test, so every day counted is a whole day of the test. On the first day, the card reads “Collecting visitors: 0 of 200 for each version.”
• Both versions need 200 visitors before a winner is named. Until then, the card shows how many the version with fewer has, with a progress bar: “Collecting visitors: 120 of 200 for each version. Results mean something after that.”
• If neither version has had a visitor more than two days after the start, the card says “No visitors yet” and suggests checking where the original is with Put it on your store.

#How the winner is decided

Teacup Quiz compares one rate, depending on how many orders the two versions have together:

Orders are the real goal, but with only a few of them any difference is mostly chance, so finished quizzes stand in until there are 30. Added to cart and Sales per visitor help you judge, but don’t decide the winner.

Teacup Quiz then works out how certain it is that the difference isn’t chance, with a standard statistical test for comparing two rates. A version wins at 95% certain or more.

#What the sentence means

The same sentences name The original when it’s the one ahead.

#Use the test version

1. Click Use the test version.
2. Read what will happen, for example: “Skin quiz” gets the test version’s questions, results and look, and stays everywhere it is on your store. The test version is deleted.
3. Click Use the test version again to confirm, or Cancel.

The original takes everything from the test version: questions, answers, matching, results, design and translations. It keeps its own name and ID, stays your main quiz if it was, and stays in every section, button, popup and link. It’s published straight away and its bundle discount is updated. You’ll see “The test version is now your quiz. Test ended.”

#Keep the original

Click Keep the original. There’s no confirmation: the test ends at once, and you’ll see “Test ended. Every visitor gets the original again.”

The test version goes back to a draft and shows on A/B tests as Ready to start. Change something else in it with Edit test version and start a new test, or remove it with Delete. (A test started before test versions existed, between two separate quizzes, leaves the other quiz as it was.)

#Ended tests

Ended tests on A/B tests lists your 20 most recent tests: Quiz (the original’s name), Ran (such as “Oct 1 – Oct 15”) and Kept (Original or Test version). It records the outcome, not the numbers.

#Getting a result you can trust

• Run each test for at least one or two full weeks, even when a winner shows early. Weekday and weekend shoppers behave differently.
• Don’t edit the test version or the original while the test runs. Changes mix into the results, and you can no longer tell what won.
• Change one thing per test. If you change the first question and the look together, you won’t know which one helped.
• Use Half of visitors (50/50) unless the change is risky: an even split reaches 200 visitors per version soonest.
• Avoid starting or ending a test in the middle of a big sale, when shoppers act differently from usual.
• A quiet quiz can take weeks to reach 200 visitors per version. Test bold changes there; small ones won’t show.

When a test ends, test the next idea. See A/B tests to make a new test version.