Holdout group
People intentionally excluded from email so you can measure whether email caused the revenue.
Say it out loud
“A group we deliberately do not email, so we can prove the revenue came from the email and not from the customer.”
A holdout is a control group excluded from a campaign or a channel so their behaviour can be compared to those who received it. Without one, “email revenue” is last-click storytelling: it counts people who would have bought anyway and attributes them to whichever message they touched last.
They bought anyway. This is the part last-click attribution silently claims as email's.
(4.58 − 3.84) × 90,000. Six times smaller, and the only figure that is true.
Yours
Nobody does this for you.
What goes wrong
Running the holdout for one campaign and declaring a result. Incremental effects are small relative to weekly noise, so a single send is usually measuring the weather. Hold the group out for a period, not a message.
The dated rule behind this
A definition is not a citation. These are the pages with the primary source, the date it was published, and what to do about it.
- Klaviyo holdout groups need 400,000 profilesGlobal · Klaviyo
See also
Where this sits
Stop 8, it comes back as numbers. The reactions become a dashboard, and the dashboard is lying to you a little.