Most founders walk into agency selection with the wrong filter. They look at portfolio aesthetics, pricing tiers, and how polished the deck looks on the discovery call. By the time they realize none of that predicts outcomes, they’ve already signed.
I’m Victoria Beltran, co-founder of Blackbelt Commerce. I run every discovery call we do. After 1,000+ Shopify projects — and after hearing what went wrong with the agencies founders came to us from — these are the 12 questions that actually tell you whether a Shopify Plus agency can deliver.
Before the Questions: What You’re Actually Evaluating
You’re not buying a service. You’re choosing a technical partner for the next one to three years of your business. The questions below aren’t a formality — they’re a filter. An agency that clears all 12 is real. An agency that hedges on four or more is not.
Also: you are allowed to push back on vague answers. “We have experience with that” is not the same as “here’s the client and here’s what we built.”
Questions 1–4: Credentials and Experience
1. Are you a verified Shopify Plus Partner? Don’t accept a verbal answer. Go to shopify.com/partners/directory and search the agency name. If the “Shopify Plus Partner” badge isn’t on their listing, they’re not certified at that level — regardless of what their website says.
2. How many Shopify Plus stores have you launched in the last 12 months? A certified Plus agency should be able to name specific projects without hesitation. If they start talking about their overall project volume instead of Plus-specific work, that’s a signal.
3. Can you show me a migration case study with before/after SEO data? A migration that loses SEO equity costs more to recover than the migration itself. Any serious Plus agency has a documented example. We wrote more on what to look for in the hiring guide for Shopify experts.
4. Who specifically will work on my account — and can I speak to them? Get a name, not a role. “A senior developer” or “our team” is not an answer. You want to know exactly who will be touching your store before you sign.
Questions 5–8: Technical Depth
5. What have you built with Checkout Extensibility? When Shopify deprecated checkout.liquid in 2024, the ability to customize checkout became exclusive to Plus Partners. If an agency can’t tell you specifically what they’ve built with Checkout Extensibility — upsell blocks, conditional fields, custom trust signals — they’re not doing Plus-level checkout work.
6. Show me a Shopify Flow automation you’ve built and deployed. Flow is a Plus-exclusive feature. A certified agency should have a concrete example: what triggered it, what it automated, what the business outcome was. Vague answers here mean they haven’t used it.
7. What are your Core Web Vitals benchmarks across your Plus client base? This is the one most agencies fail. Very few track this consistently across their portfolio. Speed affects both organic traffic and paid conversion rates — a Plus agency doing serious work should know what their clients’ LCP and CLS scores look like. If they don’t track it, that tells you something about how they define “done.”
8. How did you handle the Shopify Scripts to Functions migration for your clients? Shopify Scripts shut down June 30, 2026. Every Plus store using Scripts had to migrate to Functions before that date. An agency that completed this migration cleanly has deep platform knowledge. An agency that couldn’t handle it — or didn’t know it was coming — is not Plus-level. Our full writeup is in the Scripts to Functions migration guide.
Questions 9–12: The Engagement Model
9. Who is my main point of contact — and can I reach the founders when something is urgent? Account managers are a filter, not a feature. At Blackbelt Commerce, founders work directly with Cesar and me. We don’t use account managers. For a $100K+ engagement, you should have a direct line to the people responsible for the outcome.
10. Is this a fixed-scope, fixed-timeline proposal — or time and materials? T&M projects have a way of expanding. Know what you’re signing before scope creep becomes your problem. Fixed scope protects you. We cover what to expect on pricing in the true cost of a Shopify Expert post.
11. What does your staging process look like — do I get to review before production? Any serious Plus agency builds on staging with production-parity data before pushing live. If the answer is “we’ll push to production and fix anything that comes up,” that’s a red flag.
12. What does post-launch support look like — retainer, ad hoc, or handoff? Some agencies disappear after launch. Know before you sign whether you’re getting ongoing support, a defined retainer, or a clean handoff with documentation.
What Good Answers Sound Like
For Question 7 (Core Web Vitals), a green-flag answer sounds like: “Our Plus clients average a 78 LCP score on mobile. We benchmark at 30 days post-launch and quarterly after.” A red-flag answer: “We focus on design and development — speed is usually fine on Plus.”
For Question 9 (founder access), a green-flag answer: “Here’s my direct Slack handle and my business partner’s. Either of us can be reached same day.” A red-flag answer: “Your account manager handles day-to-day, but escalations go through our project lead.”
For Question 10 (fixed vs. T&M), a green-flag answer: “We work on fixed scope with a clear change order process. If something changes, we scope it and price it before we start.” A red-flag answer: “We prefer T&M because every project is different.”
The One Non-Negotiable
Everything else on this list is negotiable. The Plus Partner badge is not. If an agency isn’t listed in the Shopify Partner Directory with a Shopify Plus Partner designation, stop the conversation. No portfolio, pricing promise, or client reference changes what that credential unlocks — and doesn’t unlock.
Once you’ve vetted credentials, the next step is understanding what Shopify Plus Experts actually build — so you can evaluate whether what an agency is proposing matches what the platform actually supports.
Or: book a 30-minute call with me. We’ll run through all 12 of these together — and I’ll give you honest answers, including where our approach differs from other agencies and what that means for your specific project.