Shopify shipped native A/B testing. It’s called Rollouts. It lives under Markets > Rollouts in your admin. It’s free.

I run a Shopify store. I’ve been paying for Convert.com on it. When Rollouts showed up on my plan, I figured I’d try it on a homepage test I’d been planning. I closed the tab without launching the test. Here’s why.

What it does

Rollouts is two things stitched together: scheduled deployment and A/B testing.

The deployment part is the boring useful one. You make changes to a copy of your theme, decide what percent of traffic sees them, and Shopify handles the split server-side. No JavaScript flicker, no anti-flicker snippet, no app overhead. You can schedule it to start at a date, end at a date, and auto-rollback when it ends. So if you want to push a Black Friday homepage on November 25 and let it expire on December 1 without anyone touching anything, you can.

You can do this for two things: your theme, or your checkout and accounts pages.

The A/B testing part is called an Experiment. You set up a control (your live version) and a treatment (your variant), split traffic between them, and Shopify reports which one converts better. That’s the part most people care about, and it’s also the part that needs you to be on the Grow plan or higher. Basic and Shopify plans get rollouts but don’t get Experiments. So if you’re on Basic and you turn on Rollouts hoping to A/B test your hero, you can’t.

I’m on Grow. I had access. I planned a test. Then I read the actual feature.

What it doesn’t do

Two things. Both are dealbreakers for the kind of testing I actually want to run.

It can’t exclude countries at the test level. My store is US-only. About 5% of our traffic comes from Asia - people who land on the site through who-knows-what, browse for a minute, and leave because we don’t ship there. Convert lets me exclude that audience from the test. Rollouts doesn’t. It lets me target a Shopify Market, which is a different thing. If I’d configured an Asia market in Shopify Markets, I could exclude that market. But our Asia traffic isn’t shopping a configured market. It’s incidental traffic landing on the US store, and there’s no way to keep them out of the test.

This sounds like a small thing. It isn’t. Real A/B tests resolve to small wins. A 5% lift is a win. A 3% lift is a win. If 5% of your data is noise from people who were never going to buy, you can’t see those wins. The variant that randomly gets more of the Asia traffic looks worse. You declare a winner based on noise.

The reporting is one number. “Control: 2.1%. Treatment: 2.3%.” That’s it. No Add to Cart. No Checkout. No purchases broken out separately. No revenue per visitor.

If I run a homepage test in Convert, I see whether the variant moved Add to Cart, moved Checkout, moved purchases — or moved none of them. That’s how I learn. If the variant lifted Add to Cart but not purchases, the hero worked and something downstream is killing it. If purchases moved but Add to Cart didn’t, I’m probably seeing variance. Rollouts collapses all of that into one number, and the only thing I can do with one number is shrug.

Without funnel data, you don’t know why anything happened. You just know it happened. CRO without why is just guessing.

How to actually use it

If you want to try it anyway:

1. Go to Markets > Rollouts
2. Create rollout
3. Pick what you’re changing — theme or checkout
4. Edit the variant in the theme editor
5. Set start and end dates
6. For a deployment: set Launch reach (what % of traffic sees it)
7. For an experiment: set control vs treatment split
8. Pick which Markets it applies to
9. Launch

Permissions: you need to be store owner, admin, or have the specific Markets > Rollouts permission on your store role. Plus the right permission for what you’re changing — Online Store > Themes for theme changes, View and edit checkout for checkout changes. If you’re handing this off to a developer, give them the Markets > Rollouts permission specifically. General theme access isn’t enough.

Where I’d actually use it

Honestly? Two scenarios.

Big visual changes you’re nervous about. You redesigned the homepage. You want to push it but you’re not sure. Roll it out to 10% of traffic for a week. Watch your real conversion rate (in Shopify, not in Rollouts). If it tanks, roll it back. This isn’t A/B testing — it’s a safety net for deployment. Rollouts is great for this.

Scheduled campaigns with auto-rollback. Seasonal sale, holiday landing page, BFCM hero. Push it on the start date, let it expire on the end date. You don’t have to remember to switch anything back.

For actual A/B testing — the kind where you have a hypothesis, want to know if it worked, and want to know why — Rollouts isn’t ready. Maybe in a year.

What I’d want Shopify to add

Three things, in order of importance:

1. Country-level exclusion at the test level, not just Market targeting
2. Funnel metrics — Add to Cart, Checkout, Purchase as separate numbers
3. Confidence intervals on the results so you know if 2.1% vs 2.3% is actually different or just noise

The third one is the most embarrassing omission, honestly. Every other testing tool shows you whether the difference is statistically significant. Without that, “Treatment won by 0.2%” is a coin flip you’re treating like a result.

Should you cancel Convert or VWO?

If you’re paying for Convert or VWO and the only thing you use them for is splitting theme traffic 50/50 on the homepage, yeah, you could probably cancel and use Rollouts. Most stores I’ve seen using Convert use about 5% of what it does. They’re paying for features they never touch.

If you’re running structured tests with audience exclusion, funnel analysis, custom events, or you’re testing anything outside the theme editor — pricing, product page elements without a theme variant, app behavior — keep paying. Rollouts won’t replace it. Not yet.

Don’t cancel your testing tool the day Rollouts shows up in your admin. That’s what Shopify is hoping you’ll do, and what every “Rollouts replaces VWO” agency post is trying to sell you on. It’s premature.

---

Want help figuring out whether Rollouts is enough for your store, or whether you need to keep paying for Convert? Book a Shopify CRO Audit. I’ll look at what you’re actually testing and tell you what fits.

Nikhil Sharma

I'm Nikhil Sharma. I write about Shopify, paid ads, email, and the systems I build for the DTC brands I work with.