After 50+ store audits, we’ve seen the same pattern kill more CRO programs than bad hypotheses ever will: founders running a $3K/month testing app against a store that has never shipped a single winning variant. The tool isn’t the problem. The absence of a testing practice is.
Then Shopify shipped Summer ’26 Editions on June 17, 2026 — and the calculus changed. Native A/B testing for themes and checkout configurations, built directly into the platform, no third-party app required. Every Plus merchant we’ve spoken to this week has asked the same question: do we still need Intelligems, Shoplift, or Convert? The honest answer is: it depends on exactly one thing.
TL;DR: Shopify’s Summer ’26 Editions added built-in split testing for themes and checkout (called Rollouts) across all plans. For most $1M–$5M stores that weren’t testing at all, this removes the only real barrier. For $5M–$10M stores running disciplined CRO programs with price testing and deep analytics, the native tool is not a replacement — yet. The framework below tells you which camp you’re in and what to do about it.
What Shopify Actually Shipped
Let’s get specific, because a lot of the coverage this week has been impressionistic. Shopify’s Summer ’26 “Everywhere Edition” included more than 150 updates, but two are directly relevant to conversion optimization. First, native A/B testing for themes and checkout configurations — called Rollouts — is now live under Markets > Rollouts in your admin. You can schedule tests, gradually release variants, and split-test themes and checkout configurations without installing anything. Second, Checkout Components reached general availability for Plus merchants, replacing the old checkout.liquid customization model with a composable, upgrade-safe component architecture.
These are not the same feature, but they’re part of the same platform direction: Shopify is absorbing what used to be paid add-ons into the core platform. Features that cost $300–$1,500/month as standalone apps — A/B testing, merchandising logic, checkout customization — are becoming native. That’s a real shift in the economics of running a Plus store.
What Rollouts does not include yet: price testing, post-purchase survey splits, deep segment-level analytics, or multi-page funnel testing. Those remain third-party territory. Keep that list in mind — it’s the crux of the decision.
The One Question That Decides Everything
Here is the framework stripped to its core: Were you actually running tests before this?
If the answer is no — if your CRO app has been installed for eight months and your last winning variant was the checkout button color you changed once — then Shopify Rollouts is almost certainly enough for your stage, and you should cancel the app today and put that budget toward a testing practice instead of a testing tool.
If the answer is yes — if you’re running 3–5 concurrent tests per quarter, tracking lift by cohort, and your CRO program has an operator who owns it — then you are probably not canceling anything yet. Native testing covers themes and checkout, not your full conversion funnel. You need to know which specific tests you’re running before making the call.
We’ve watched founders make both mistakes: paying $800/month for an app they open twice a year, and canceling a mature testing stack because a new native feature sounded like it covered the same ground. Both cost real money.
The Decision Tree, Branch by Branch
Branch 1: Are you testing at all right now?
If you haven’t run a completed A/B test in the past 90 days, you don’t have a CRO program — you have a CRO subscription. Cancel the app. Rollouts handles theme and checkout testing for free. Use the savings to hire a CRO specialist for one day a month to build a test backlog. That single change will do more for your conversion rate than any tool.
A $4M apparel brand we worked with last year was paying $1,100/month for a testing platform. In 14 months, they had run two tests — neither conclusive, neither acted on. When we cut the app and replaced it with a structured quarterly testing calendar, conversion on their PDP lifted 18% in the first cycle. The tool had never been the constraint. The process had.
Branch 2: What are you testing?
Native Rollouts covers: theme variants (layout, design, section arrangement), checkout UI configurations, and customer account configurations. If your test backlog is primarily theme-level — hero layout on the homepage, collection page sorting, trust signals above the fold — Rollouts handles this cleanly. You’re done. Cancel the app.
If your test backlog includes price testing, tiered offer testing by customer segment, post-purchase upsell splits, or buy-now-pay-later display logic, you need a third-party tool. Rollouts does not reach into pricing logic or segment-level funnel splits. Tools like Intelligems own price testing specifically and have no meaningful native competitor inside Shopify yet.
Branch 3: How sophisticated is your analytics layer?
Third-party CRO tools justify their cost partly through attribution modeling, statistical significance calculators, and segment breakdowns you can slice by traffic source, device, or customer LTV cohort. Rollouts is a scheduling and variant publishing tool — the analytics come from your existing Shopify analytics and whatever you’re running alongside it (GA4, Triple Whale, Northbeam).
If you have a clean analytics stack and someone who can read a significance test, Rollouts is enough for theme and checkout tests. If your analytics situation is patchy and you rely on your CRO tool to tell you whether a test is significant, you may need to solve the analytics problem before you can safely drop the app.
Branch 4: Are you on Plus?
Checkout Components — the new composable checkout customization model — is a Plus-only feature. If you’re on Shopify Advanced, you can use Rollouts for theme testing, but you don’t get the full checkout extensibility layer. For checkout-specific CRO work, non-Plus stores still need third-party tools or a Plus upgrade to access the full surface. Our breakdown of Plus vs. Advanced covers exactly when that upgrade pays for itself — checkout CRO capability is one of the clearest forcing functions we’ve seen.
The Objection: “Native Tools Are Always Worse Than Dedicated Apps”
Some agencies — particularly ones with rev-share arrangements on the tools they recommend — will tell you that native Shopify features are always watered-down versions of dedicated app functionality. Don’t run your store on that assumption.
It was true for email, until Shopify Email improved. It was true for subscription logic, until the native subscription APIs matured. The pattern is consistent: Shopify ships a native version of a category at 70% of the feature depth of the market leader, and the 70% covers the use case of 80% of merchants. The remaining 20% — the ones running sophisticated, volume-heavy programs — do need the specialist tool. But that remaining 20% knows who they are.
Here’s the real question: does your current CRO app give you capabilities in the 70% zone or the 30% zone? If it’s the 70% — theme tests, checkout layout experiments, gradual rollouts — Shopify just built your replacement. The app vendor’s job is to hold the 30%. If they’re not clearly operating in that 30%, the economics have shifted against them.
This is the same dynamic we wrote about in our app stack audit framework — Shopify’s platform keeps absorbing app functionality, and the apps that survive are the ones that compete on depth in their specific category, not breadth. Every Editions cycle is another audit trigger.
What to Do Monday Morning
- Pull your CRO app’s last 90 days of activity. How many tests ran? How many produced a statistically significant result? How many winning variants were actually implemented? If the answers are zero, one, and zero — you already know what to do.
- Map your test backlog against what Rollouts covers. Log into your admin, find Markets > Rollouts, and read what’s configurable. If your next three planned tests live inside theme and checkout, end the third-party subscription. If price testing is in the backlog, keep the app until native price testing exists.
- Run Checkout Components readiness check. If you’re on Plus and you have any active checkout customizations, verify they’re built on Checkout UI Extensions — not the legacy checkout.liquid file. Legacy checkout is being retired. The Checkout Components architecture is now the supported path, and it directly expands the surface Rollouts can test against.
- Set a testing calendar before you cancel anything. The founders who get value from native A/B testing will be the ones who treat it as a practice, not a feature. Block one day per quarter for a testing review. Assign an owner. Write down the hypotheses before you look at the tool.
- Revisit your full app stack through this lens. Rollouts is one data point in a broader pattern. Shopify’s Summer ’26 Editions also shipped native AI merchandising and B2B net terms — both of which put pressure on app-layer solutions in those categories. If you haven’t done an app audit since before this Editions cycle, the timing is right.
The Close
Metro School Uniforms didn’t get to 138.7% organic growth by adding more tools. The work that moved the needle was the decision to subtract what wasn’t earning its place, then go deep on what was. The same logic applies here. Shopify just gave you a free testing surface that most of your competitors won’t use correctly. The question isn’t whether the tool is good enough. It’s whether you’re going to build a testing practice around it — or keep paying for a practice you’re not running anyway.
If you’re unsure which camp your store sits in, we’re happy to take a look. Our thinking on CRO through subtraction is a good starting point — and it’ll tell you quickly whether your next move is a new test or a canceled subscription.
FAQ
What is Shopify Rollouts and which plans can use it?
Rollouts is Shopify’s native A/B testing feature, available in your admin under Markets > Rollouts as of the Summer ’26 Editions (June 2026). It lets merchants schedule, gradually release, and split-test theme variants and checkout configurations without a third-party app. Theme testing is available across plans; full Checkout Components testing is a Shopify Plus feature.
Does Shopify native A/B testing replace tools like Intelligems or Shoplift?
For theme and checkout layout testing, Rollouts covers the core use case and third-party tools become harder to justify on cost. For price testing, segment-level funnel analysis, and post-purchase experiment splits, Intelligems and similar tools still have functionality that Rollouts does not. Audit your actual test backlog before canceling any app — the decision depends on what you’re testing, not what the tool can theoretically do.
What is the difference between Rollouts and Checkout Components in Shopify Plus?
Rollouts is the testing and publishing mechanism — it controls when and to whom variant experiences are shown. Checkout Components is the underlying architecture for building custom checkout experiences on Plus, replacing the legacy checkout.liquid file. Rollouts can be used to test Checkout Component configurations, but they are separate features serving different functions in the stack.
When should a $1M–$10M Shopify Plus brand keep its third-party CRO app?
Keep your third-party CRO app if you are actively running price tests, segment-level funnel experiments, or post-purchase sequence splits — these are outside what Rollouts covers today. Also keep it if your testing analytics depend on the app’s significance reporting and your broader analytics stack isn’t yet clean enough to replace it. If neither of those applies, evaluate whether the cost is justified against what you’re actually running.
