Glossary

A/B testing has its own vocabulary, and some terms differ between tools. This glossary collects the terms used across the Split Test Pro documentation.

A

A/B/n test

An experiment with one control and multiple variants (B, C, D, …). Same statistical model as A/B; just more variants.

A/B test

An experiment comparing two versions of a page (a control and a variant) to determine which performs better against a defined goal.

Activation

The moment an experiment “fires” for a visitor, applying the variant’s CSS or JS and recording the page-view event. Usually happens on page load; can be deferred via event activation.

Activity

The tab on each experiment page (Results | Setup | Activity | Alerts) that lists what happened to the test — launches, pauses, schedule changes, its end, rollouts — with who did it and when. See Experiment Activity.

Alternate template

A second Shopify template for a page type, such as product.alternate.json next to the default product.json, rendered when the address carries ?view=alternate. Template tests compare one with the default. See Template Tests.

AOV (Average Order Value)

Total revenue divided by number of orders. A continuous metric available on Shopify experiments.

Archived

An experiment state for hiding completed or abandoned experiments from the main list. Data is preserved; the experiment just doesn’t appear in active views.

Assignment

The act of bucketing a visitor into a specific variant. Deterministic — based on a hash of the visitor ID and experiment ID — so the same visitor always gets the same variant.

B

Bayesian statistics

The statistical approach Split Test Pro uses. Computes posterior probabilities (e.g., “73% chance Variant B is best”) rather than frequentist p-values. See Bayesian Stats Explained.

Beta distribution

The probability distribution used to model conversion rates in the binomial case. Each variant’s conversion rate is treated as a Beta(α, β) where α = conversions + 1 and β = (sessions − conversions) + 1.

C

Conversion

The action that defines success for an experiment — a purchase, a signup, a click, a custom event. Each experiment has at least one conversion goal.

Conversion rate

Conversions divided by sessions, expressed as a percent. The most common primary metric.

Control

The variant a visitor sees if the experiment didn’t change anything. Usually labeled Variant A. The baseline against which other variants are compared.

Credible interval

A range with a stated probability of containing the true value of a parameter. “We’re 95% confident the true conversion rate is between 4.2% and 5.8%” is a credible-interval statement. Different from frequentist confidence intervals — see Bayesian Stats Explained.

Custom event

A merchant-defined conversion event, fired from your site via SplitTestPro.trackConversion(eventKey, opts). See Custom Events.

D

Declare winner

The action of formally ending an experiment with a chosen winning variant. Available when the leading variant crosses 95% probability to be best on the primary metric. See Declaring a Winner.

Device targeting

Restricting an experiment to one or more device classes (mobile, tablet, desktop). Set per-experiment, applies before variant assignment. See Device Targeting.

E

Event activation

Configuring an experiment to wait for a specific custom event before activating. Useful when the change only matters after a user interaction. See Event Activation.

Experiment

A single A/B test, with a name, hypothesis, targeting rules, variants, and conversion goals.

Exposure

A visitor session in which the variant actually applied. Distinct from assignment — a visitor can be assigned to a variant but never have it activate (e.g., with deferred activation that never fires).

F

Feature flag

A named on / off switch in your workspace that your backend checks through the evaluation API before showing a feature. A flag can be on, off, rolled out to a percentage of users or killed, and an experiment can set its value per variant. Created in Settings → Feature flags. See Feature Flags.

Funnel

The sequence of steps a visitor takes from arriving on the site to converting. On Shopify, the Web Pixel auto-tracks the full purchase funnel — see Shopify Funnel Tracking.

G

Global goals

Your testing program’s targets: the north-star metric it should move and the guardrails that protect other metrics while it does. Open them from the Testing Overview Coming soon (reach it at /experiments/overview while it is hidden from the navigation): Queue options → Global goals. See Global Goals.

H

Hybrid test

An experiment the pixel serves on your site whose variants also carry feature flag values, so your backend can branch on the same variant the browser was assigned. Your server passes the pixel’s visitor id and assignment string to the evaluation API. See Node SDK & Hybrid Tests.

Hypothesis

A testable statement of belief that motivates an experiment. Formatted as: “We believe [change] will [increase/decrease] [metric] because [reason].” See Testing Methodology.

I

ICE (Impact, Confidence, Ease)

A simple framework for prioritizing experiments. Score each idea 1–3 on each axis; test the highest sums first.

K

Kill switch

The Kill action on a feature flag. A killed flag serves false to everyone — including users inside its rollout and any running experiment — until you press Restore. See Feature Flags.

L

Launch checks

The live Experiment Summary beside the experiment wizard. It lists what each step still needs, and Launch Test stays disabled until every essential field is complete; a few more checks run when you launch. See Launch Checks.

Lift

The percent improvement of a variant over the control on the primary metric. A 5% lift means the variant performed 5% better than control.

M

Member

A team role with access to view and edit experiments, but not invite other team members or manage billing. See Team and Invites.

Modeled improvement

The Bayesian engine’s distribution of likely lift values, drawn in the Distributions view of the Results tab’s comparison card.

Monte Carlo sampling

The technique used to compute probability to be best — sample from each variant’s posterior distribution thousands of times, count which variant had the highest sample.

N

Normal distribution

The probability distribution used to model continuous metrics (revenue, AOV). Each variant’s mean is treated as Normal(µ, σ) where µ is the sample mean and σ is the standard error.

O

Overlap

Two running experiments that share visitors. The Results tab shows an Overlap banner with the share of shared visitors, and its Overlap view switch splits the results by the combination of variations each visitor saw. See Segmenting Results. For running tests side by side, see Running Multiple Experiments.

Owner

A team role with full permissions: experiments, billing, team management, workspace deletion.

P

Peeking

Repeatedly checking experiment results and stopping the moment they look favorable. Inflates false-positive rate. See Common Mistakes.

Posterior

The updated distribution of a parameter after observing data. In Bayesian terms, “prior + data → posterior.”

Preview URL

The page URL set on the experiment’s Variants step. It is the page Edit on site opens and the page screenshots are taken from. See Screenshots and Preview.

Primary metric

The conversion goal an experiment is judged on. Drives the “probability to be best” calculation and the Declare winner button. See Conversion Goals.

Prior

The distribution of belief about a parameter before observing data. Split Test Pro uses Beta(1, 1) (uninformative) for binary metrics.

Probability to be best

The probability that a variant is the best performer across all variants in the experiment, computed via Monte Carlo. The headline number on the Results dashboard.

Probability to beat control

The probability a non-control variant beats the control. Equal to “probability to be best” in two-variant experiments; differs for three+.

R

Redirect test

An experiment where one or more variants redirect visitors to a different URL instead of inline-modifying the current page. See Redirect Tests.

Rollout (feature flag)

The share of users, 0–100%, who get true from a flag that is on. Each user lands in the same place every time, so raising the percentage only adds users. Not to be confused with a winner rollout. See Feature Flags.

Revenue per Session

Total revenue divided by sessions. A continuous metric and the default primary metric on Shopify.

S

Sample Ratio Mismatch (SRM)

A critical anomaly: the actual traffic split deviates significantly from the configured split. Usually indicates broken assignment logic. See Anomaly Alerts.

Scheduled

The status of a test with a launch time set for later. Set it with Schedule under “When it runs” in the Variants step, or with the Dashboard’s Schedule button on a draft, then Launch Test. A scheduled test offers Reschedule and Cancel schedule in the ⋯ menu. See Scheduling.

Server API key

A secret key (stp_…) your backend sends to the evaluation API. Created and revoked in Settings → API keys, shown once, and never placed in a browser or app. Different from the public data-api-key in the install snippet. See Server API Keys.

Server-side test

An experiment with Where does this test run? set to Server-side: the pixel never serves it, your backend reads each user’s variant from its feature flags, and exposures and conversions arrive through the API. See Server-Side Testing.

Sequential testing

Running experiments one after another on the same surface, rather than in parallel. The right approach when tests would interact. See Running Multiple Experiments.

Session

A single visit by a single visitor, identified by a deterministic visitor ID. The denominator of the conversion rate.

Significance

The threshold at which we trust the result. Split Test Pro uses 95% probability to be best as its convention.

Splitty

The AI builder behind Create with AI. It opens on your own site in preview mode, suggests test ideas and builds the test — variants, targeting and traffic split — without you writing code. See Splitty AI Builder.

Surfaces

The named areas of your site (home page, product pages, cart, …), each defined by URL patterns in Settings → Surfaces. The experiments list shows a test’s surface as its Area, and a Test Plans roadmap runs one test per surface at a time. See Workspace Settings.

T

Targeting

The rules that determine which visitors and which pages an experiment applies to — URL targeting plus optional device targeting and event activation.

Template test

A Shopify test type where variant visitors see an alternate template for products, collections, pages or blog posts — all items of the type, or the ones you pick — while the control sees the default template. See Template Tests.

Theme App Extension

The Shopify component that injects variant CSS and JS into the storefront. See Shopify Theme App Extension.

Theme preview

Shopify’s way of showing an unpublished theme to a visitor, through preview_theme_id in the address and a cookie that keeps them on it. Theme tests use it, with Shopify’s preview bar hidden. See Theme Tests.

Theme test

A Shopify test type where variant visitors see one of your unpublished themes while your live theme stays published. See Theme Tests.

Traffic allocation

The percentage of visitors each variant receives. Defaults to even (50/50 for two variants); can be adjusted per variant.

V

Variant

One of the versions an experiment compares. Variant A is conventionally the control; B, C, D… are the treatments. Each variant uses one or more of the four test types: code (CSS and JavaScript), a URL redirect, a theme test and a template test.

Variant types

The four test types a variant can use, in order: Code · CSS & JS (visual and behavioral changes), URL redirect (full-page substitute), Theme test and Template test (Shopify only). A variant can combine them. See Variant Types Overview.

W

Web Pixel

The Shopify component that captures funnel events and attributes them to the visitor’s experiment-variant assignment. See Shopify Web Pixel.

Winner

The variant declared best after an experiment ends. The workflow is: leading variant crosses 95% → click Declare winner → apply the change to your theme. See Declaring a Winner.

Workspace

The container for one site or store’s experiments, team, and billing. See Workspaces.