I Tried Shopify Rollouts. It's Not Ready Yet

Shopify shipped native A/B testing. It’s called Rollouts. It lives under Markets > Rollouts in your admin. It’s free.

I run a Shopify store. I’ve been paying for Convert.com on it. When Rollouts showed up on my plan, I figured I’d try it on a homepage test I’d been planning. I closed the tab without launching the test. Here’s why.

What it does

Rollouts is two things stitched together: scheduled deployment and A/B testing.

The deployment part is the boring useful one. You make changes to a copy of your theme, decide what percent of traffic sees them, and Shopify handles the split server-side. No JavaScript flicker, no anti-flicker snippet, no app overhead. You can schedule it to start at a date, end at a date, and auto-rollback when it ends. So if you want to push a Black Friday homepage on November 25 and let it expire on December 1 without anyone touching anything, you can.

You can do this for two things: your theme, or your checkout and accounts pages.

The A/B testing part is called an Experiment. You set up a control (your live version) and a treatment (your variant), split traffic between them, and Shopify reports which one converts better. That’s the part most people care about, and it’s also the part that needs you to be on the Grow plan or higher. Basic and Shopify plans get rollouts but don’t get Experiments. So if you’re on Basic and you turn on Rollouts hoping to A/B test your hero, you can’t.

I’m on Grow. I had access. I planned a test. Then I read the actual feature.

What it doesn’t do

Two things. Both are dealbreakers for the kind of testing I actually want to run.

It can’t exclude countries at the test level. My store is US-only. About 5% of our traffic comes from Asia, people who land on the site through who-knows-what, browse for a minute, and leave because we don’t ship there. Convert lets me exclude that audience from the test. Rollouts doesn’t. It lets me target a Shopify Market, which is a different thing. If I’d configured an Asia market in Shopify Markets, I could exclude that market. But our Asia traffic isn’t shopping a configured market. It’s incidental traffic landing on the US store, and there’s no way to keep them out of the test.

This sounds like a small thing. It isn’t. Real A/B tests resolve to small wins. A 5% lift is a win. A 3% lift is a win. If 5% of your data is noise from people who were never going to buy, you can’t see those wins. The variant that randomly gets more of the Asia traffic looks worse. You declare a winner based on noise.

The reporting is one number. “Control: 2.1%. Treatment: 2.3%.” That’s it. No Add to Cart. No Checkout. No purchases broken out separately. No revenue per visitor.

If I run a homepage test in Convert, I see whether the variant moved Add to Cart, moved Checkout, moved purchases, or moved none of them. That’s how I learn. If the variant lifted Add to Cart but not purchases, the hero worked and something downstream is killing it. If purchases moved but Add to Cart didn’t, I’m probably seeing variance. Rollouts collapses all of that into one number, and the only thing I can do with one number is shrug.

Without funnel data, you don’t know why anything happened. You just know it happened. CRO without why is just guessing.

How to actually use it

If you want to try it anyway:

1. Go to Markets > Rollouts
2. Create rollout
3. Pick what you’re changing, theme or checkout
4. Edit the variant in the theme editor
5. Set start and end dates
6. For a deployment: set Launch reach (what % of traffic sees it)
7. For an experiment: set control vs treatment split
8. Pick which Markets it applies to
9. Launch

Permissions: you need to be store owner, admin, or have the specific Markets > Rollouts permission on your store role. Plus the right permission for what you’re changing, Online Store > Themes for theme changes, View and edit checkout for checkout changes. If you’re handing this off to a developer, give them the Markets > Rollouts permission specifically. General theme access isn’t enough.

Where I’d actually use it

Honestly? Two scenarios.

Big visual changes you’re nervous about. You redesigned the homepage. You want to push it but you’re not sure. Roll it out to 10% of traffic for a week. Watch your real conversion rate (in Shopify, not in Rollouts). If it tanks, roll it back. This isn’t A/B testing, it’s a safety net for deployment. Rollouts is great for this.

Scheduled campaigns with auto-rollback. Seasonal sale, holiday landing page, BFCM hero. Push it on the start date, let it expire on the end date. You don’t have to remember to switch anything back.

For actual A/B testing, the kind where you have a hypothesis, want to know if it worked, and want to know why, Rollouts isn’t ready. Maybe in a year.

What I’d want Shopify to add

Three things, in order of importance:

1. Country-level exclusion at the test level, not just Market targeting
2. Funnel metrics, Add to Cart, Checkout, Purchase as separate numbers
3. Confidence intervals on the results so you know if 2.1% vs 2.3% is actually different or just noise

The third one is the most embarrassing omission, honestly. Every other testing tool shows you whether the difference is statistically significant. Without that, “Treatment won by 0.2%” is a coin flip you’re treating like a result.

The comparison, honestly

Neither tool is better in the abstract. They solve different problems and cost different amounts.

That last row is the one people underrate. When a variation loses, “it lost” is not a finding. Tools that pair the result with heatmaps or friction-filtered session recordings let you see what people did differently, which is the difference between one dead test and a hypothesis for the next one. Convert bundle both in every plan, and their heatmaps don’t count against the tested-users quota, which removes the usual reason people leave them switched off.

Two Shopify-specific things worth knowing

These apply regardless of which tool you pick, and they cost people real money.

Uninstalling an app may not remove its code. Shopify’s own docs warn that some apps inject code into your theme that isn’t automatically removed on uninstall. If you’ve cycled through testing tools, there may be dead snippets in your theme right now. Tools that install via an app embed block instead of theme file injection avoid this entirely, because there’s nothing of theirs in the theme to leave behind or to clash on a theme update.

Shopify’s pixel sandbox constrains what any tool can track. App pixels run in a web worker with no window and no document, which means a testing tool can’t just read the variation off the page. It has to be handed that information deliberately, through first-party storage or cart attributes. Worth knowing when you’re comparing what different tools can and can’t measure on Shopify. It’s a platform constraint, not a vendor shortcoming.

Should you cancel Convert or VWO?

If you’re paying for a testing tool and the only thing you use it for is splitting theme traffic 50/50 on the homepage, yeah, you could probably cancel and use Rollouts. Most stores I’ve seen using Convert use about 5% of what it does. They’re paying for features they never touch.

That’s a real criticism of how these tools get sold, not just of the buyers. Nobody needs six SDKs and a feature flagging layer to test a hero image.

If you’re running structured tests with audience exclusion, funnel analysis, custom events, or you’re testing anything outside the theme editor, pricing, product page elements without a theme variant, app behavior, keep paying. Rollouts won’t replace it. Not yet.

Don’t cancel your testing tool the day Rollouts shows up in your admin. That’s what Shopify is hoping you’ll do, and what every “Rollouts replaces VWO” agency post is trying to sell you on. It’s premature.

Want help figuring out whether Rollouts is enough for your store, or whether you need to keep paying for a dedicated tool? Book a Shopify CRO Audit. I’ll look at what you’re actually testing and tell you what fits.

Nikhil Sharma

I'm Nikhil Sharma. I write about Shopify, paid ads, email, and the systems I build for the DTC brands I work with.