Agency Problems
Part of the Agency Problems cluster
How to Know If Your Ecommerce Agency Is Actually Working
An ecommerce agency is working if the one outcome metric they own has moved in your own platform data over the last 60 to 90 days — blended CAC and new customer ROAS for an ads agency, flow revenue percentage for an email agency — and they can tell you what they changed to move it. Both halves matter. A number that moved with no explanation might be the market. An explanation with no movement might be a story. This page covers the test, the metrics that count, and the ones that only look like they count.
Why Does the Report Look Good While Revenue Feels Flat?
You hired professionals so you would not have to second-guess them, and now you are second-guessing them anyway — while paying for the privilege. The discomfort usually starts as a mismatch: the monthly report is full of green arrows, and the bank account does not agree.
The mismatch has a mechanical cause. Marketing channels produce activity data continuously, so a report can always be filled with metrics that improved: reach is up, CPM is down, open rates are healthy, twelve tasks shipped. None of those are the outcome. They are inputs to the outcome, and an agency can optimize all of them for months while the cost of acquiring a real new customer stays exactly where it was.
The clearest example is blended ROAS, the headline number in most paid media reports. It includes returning customers who are attributed back to ads even when they would have come back organically. A report showing strong blended ROAS can coexist with new customer acquisition that loses money on every first order. The metric is not fabricated — it is just measuring a different question than the one you are asking.
Inputs are what the agency does. Outcomes are what the business gets. Reports are usually built from the first and read as if they were the second.
What Is the Real Test?
Two questions, answered from your own accounts, not the agency's deck.
1. Did the outcome metric move?
Open your own Ads Manager, GA4, or Klaviyo. Set the range to the last 90 days. For an ads agency, check blended CAC (total spend ÷ total new customers) and new customer ROAS. For an email agency, check flow revenue as a percentage of total email revenue against the 25–35 percent benchmark. Compare against the 90 days before the engagement started.
2. Can the agency name what they did to move it?
A working agency can answer in one email: what was changed, why, and what result it was expected to produce in what window. If the metric moved and they can explain it, the engagement is working. If it did not move and they cannot name what was tried, the engagement is producing activity.
Cross-check the attribution
Platforms grade their own homework. Check new customers from paid channels in GA4 or Shopify against what the ad platform claims for the same window. Large, unexplained gaps are themselves a finding worth raising.
What Does an Agency Not Doing the Work Look Like?
Beyond the outcome metric, the account itself shows whether work is happening. In an ad account, the telltale is creative: if no new ad variants have entered the account in 45 or more days while frequency climbs above 3 to 4 and ROAS declines, that is ad creative fatigue going unmanaged — the single most common execution failure in paid social engagements.
In a Klaviyo account, the telltale is build status and edit history: flows that were set up in the first 60 days and never touched again, or core flows missing entirely — welcome, abandoned cart, post-purchase, browse abandonment, win-back — months into a retainer. Both checks are visible under your own login in minutes, and both are facts, not feelings, which makes the conversation that follows far more productive.
One caution in the other direction: an account full of recent activity is not proof the agency is working either. Edits, tests, and new creative are inputs too. The pairing that matters is visible work plus a moving outcome metric — or visible work plus an honest explanation of why the metric has not moved yet and what happens next.
Benchmarks to Know
60–90 days
Fair assessment window for an established account, measured on your own data
25–35%
Of total email revenue from automated flows, the benchmark for an established list
3–4+
Ad frequency combined with declining ROAS: the creative fatigue pattern an agency should flag before you do
45 days
Without new creative entering an ad account is an execution gap worth raising
<10 min
Per channel to run the outcome check yourself, in your own accounts
1 email
A working agency can explain what they changed, why, and the expected result in one
Related Pages
Ad Creative Fatigue
The most common execution failure a paid social check uncovers: the same small set of ads running past the point the audience stopped responding. The full diagnostic for spotting it in your own account.
E1Email Flows Missing
The email-side equivalent: which automated flows an ecommerce store needs, and what it costs to run without them — the first thing to check in a Klaviyo account audit.
Frequently Asked Questions
What to Do Next
If you ran the check and the outcome metric has not moved, the next question is whether the problem is fixable inside the relationship or structural underneath it. If you are second-guessing what the numbers mean, having someone map the full revenue system is the right decision.
Free · 20 minutes · No pitch