Check every answer: the live quiz check

#What it is

Check every answer runs the quiz for every way a shopper can answer it and checks what that shopper would get. One way to answer is one answer to each question a shopper sees, or one combination of answers on a several-answers question. A quiz with 4 questions of 4 answers has 256 ways to answer. The check also looks at the quiz’s settings: jumps, rules, results, pages.

When it runs

• Live: a moment after every change, as long as the editor is open.
• Its findings show in the toolbar pill, as red and amber dots in Quiz steps, in a banner at the top of each question with problems, under routine steps and on result pages.
• On a brand-new blank quiz, problems stay hidden until you edit it, open the check or try to publish.
• It uses your products as the editor last loaded them. Publish always checks again with your products as they are now.

Open the full report with Check every answer under Before you publish, or by clicking the toolbar pill. While a new blank quiz’s problems are still hidden, the steps list says Run it before you publish under it, and the pill reads Check every answer.

#Reading the report

The first line says how many ways to answer were tried and how many different results they lead to: “We tried all 256 ways to answer. Shoppers get 9 different results.” When there are more than 50,000: “We tried 50,000 of the 120,000 ways to answer.”

When there’s no problem at all, a green banner says Every shopper gets products (best matches and routines) or Every shopper gets a result (result pages), with that line under it. Otherwise each problem is listed, red errors first, then amber warnings and blue notes. A problem tied to a question has Show question, which opens it.

Where shoppers land and Most common results

• Result pages: Where shoppers land lists each result with the share of ways to answer that lead to it. A result at 0% is never reached.
• Products that fit: Most common results shows the three most common sets of products, such as “Clear Gel + Calm Serum + Oil Toner”, and how many ways to answer lead to each, such as “96 of 256 ways to answer” (or “Every way to answer”). “(nothing fit: your backup products)” marks the backup, and “No products” a dead end.

When nothing can be tried

Some red errors break the quiz’s structure, such as a loop, a jump to a missing question or a rule with an empty tag. Until every such error is fixed, no way to answer is tried, and the first line counts 0. Fix the red items first; the rest of the report then appears.

#Errors (red)

An error means some shoppers would get a broken quiz. Every error stops Publish except the first one in this table, which asks you first. The messages use the quiz’s own names; the examples below use “Skin type?” for a question and “Oily” for an answer.

#Warnings (amber)

A warning never stops Publish, but each one is something shoppers may notice.

#Notes (blue)

A note is worth a look, but often fine as it is. Notes don’t count in the toolbar pill or in N to look at. So the pill can read Every answer works while the report shows a note and no green banner.

Shoppers who pick no preference

“Shoppers who pick no preference see your first 3 products, in store order.” Some ways to answer carry no Recommend more, Recommend less or Only recommend rule (a Never recommend rule is fine), for example a shopper who answers “Surprise me” everywhere. Answers with only Recommend less or Only rules keep their normal matching and aren’t counted here. The number is your How many products (“your first product” when it’s 1). In a routine it reads “the first product of each step, in store order.” Store order is by product ID, so older products usually come first.

In a best-matches quiz, Choose what they see opens Results at If nothing fits their answers. Choose a collection there, such as your bestsellers, and these shoppers see its products first, topped up with your first products in store order when it has too few they can be shown. Their Never rules still hold. The note then goes away, unless the collection has nothing these shoppers can be shown. Either way, they don’t count as a no match in your analytics.

If the note surprises you, some answers probably have no rules yet: give them Recommend more rules.

Products that never come up

“3 products never come up: no answer leads to them.” (or “This product never comes up: no answer leads to it.”) No way to answer shows these products. It’s only checked when every way to answer was tried. Under the note, up to 5 of them are listed, each with two fixes:

• Show it for an answer…: pick one of the quiz’s answers, and that answer gets a rule that recommends the product. Teacup Quiz picks the gentlest strength that gets it shown: Normal, Strongly or Very strongly. When showing it would push other products out of every result, it picks the strength that pushes out the fewest. If the answer can’t bring the product up, nothing is added and the row says “That answer can’t bring it up: its other rules rule it out, or no one gets there. Try another answer.” On a big quiz, the row says “Trying it on every way to answer…” while it works. Not offered in routine quizzes: a routine only shows products from its steps.
• Leave it out of this quiz: takes the product out of this quiz only, for products that don’t belong in it. It’s then listed under Results → Products in this quiz → Left out of this quiz, where Put back returns it.

With more than 5, the list ends with “And 12 more. Fix these first, and the next ones show up here.”

#What stops Publish, and what doesn’t

When Publish is stopped you see “Fix this before publishing: …” with the message, or, for several, “Fix 3 problems before publishing (see ‘Check every answer’).”

When the check has found dead ends, Publish, Publish changes and Publish again first open Some shoppers would get no products: “3 ways of answering end with no products: shoppers who answer like that see ‘We couldn’t find a match right now’. ‘Check every answer’ shows which answers lead there and how to fix it.” Show me opens this report; Publish anyway publishes the quiz as it is.

#Up to 50,000 ways to answer

The editor tries up to 50,000 ways to answer. Several-answers questions multiply fast: a question with 8 answers and no limit has 255 combinations (256 when it can be skipped), and three such questions make over 16 million ways to answer.

• Past the limit, the ways tried are the first ones in order, so the later answers of your first question may not be tried at all.
• Checks that need every way to answer, such as products that never come up and unreached results, are skipped.