The True Cost of Manual A/B Testing on Google Play (and How to Calculate Yours)
Fabio Salvador
Most ASO teams underestimate the true cost of manual experimentation. While design hours show up in the budget, the bigger cost is what gets left behind: tests that don’t happen, ideas that never get validated, markets that don’t get attention, and revenue that stays out of reach.
In this post, we’ll break down a simple way to calculate the real cost of your manual ASO testing cycle. By the end, you’ll have a number you can use in your next budget or business case for automation.
The five hidden costs of manual ASO testing
Most teams only count designer time. But there are five other hidden costs that often get missed.
First, there’s the backlog of untested ideas. Every ASO test you can’t run now gets pushed behind new ideas next month. Backlogs grow quickly. Many teams we talk to have 30 to 80 untested hypotheses, often sitting for over six months. The real cost isn’t just having a backlog—it’s losing confidence that these ideas will ever be tested, which means the team comes up with fewer ideas over time.
Second, slow testing cycles mean missed opportunities. A manual A/B test on Google Play can take up to 47 days from start to finish. If you can only run 6 tests a year but have 50 good ideas, you’re missing out on 44 chances to improve. Each slow cycle means you’re not building on wins as quickly as you could.
Third, there’s the time spent by PMs and ASO Managers. Designers don’t work alone. Each test needs input from product managers, ASO managers, and analysts. If your PM spends 4 hours per test and you run 24 tests a year, that’s 96 hours of senior time—often tens of thousands of dollars that never shows up in the ASO budget.
Fourth, limited testing means teams play it safe. When you can only run a few tests, you pick the easy wins and avoid bold ideas that could drive bigger results. The most valuable tests often get skipped, and this self-censorship is a hidden cost that adds up over time.
Fifth, if you have several apps, manual testing keeps learnings stuck in silos. A winning idea in one app rarely makes it to another unless someone takes the time to share and re-test it. At scale, this is a big cost—most teams skip it, so each app misses out on valuable insights from the rest of the portfolio.
The calculator framework
Here’s a simple model. Take 20 minutes with your team and fill it out for your setup.
Inputs you need:
- Number of apps in your portfolio
- Average hypotheses generated per app per quarter
- Average tests actually launched per app per quarter
- Average designer hours per test (briefing + creation + revisions)
- Average PM and ASO Manager hours per test
- Fully-loaded hourly cost for designer, PM, and ASO Manager
- Median time from hypothesis generation to decision (days)
- Estimated revenue impact of a 1% CVR lift on your top app
Step 1: Direct cost. Multiply the number of tests per quarter by the total hours (designer, PM, ASO Manager) and your blended hourly rate. This is what your finance team sees.
Step 2 — Backlog cost. (Hypotheses generated − hypotheses tested) × estimated effect size × revenue per 1% CVR. This is the value of the tests you didn’t run, discounted by the probability that any individual hypothesis would have produced a winning result.
Step 3 — Cycle time cost. (Industry benchmark cycle time of 7 days vs. your cycle time) × number of tests × revenue per 1% CVR. This captures the compounding effect of testing faster — every day saved in a cycle is a day the next test can run, building on the previous one.
Step 4: Portfolio learning cost. Multiply the number of apps (minus one) by the number of insights that could apply across titles and revenue per 1% CVR. This shows what you’re missing by not sharing wins across your portfolio.
Add up all four costs. The total might surprise you—and that’s the point.
A worked example: a 7-app portfolio publisher
Let’s run the math for a typical Segment A portfolio publisher: 7 apps, in-house design team of 4, ASO team of 3, mid-sized revenue base.
- Hypotheses generated per quarter: 12 per app × 7 apps = 84
- Tests actually launched per quarter: 2 per app × 7 apps = 14
- Backlog growth per quarter: 70 untested hypotheses
- Designer hours per test: 16
- PM + ASO Manager hours per test: 8
- Blended hourly rate: ~$80
Direct cost per quarter: 14 × 24 × $80 = $26,880 Direct cost per year: ~$107,520
That direct cost might seem reasonable. But let’s look at the hidden costs.
If each untested hypothesis had a 10% chance of producing a 2% CVR lift, and a 1% CVR lift on this portfolio is worth $200,000 in annual revenue, the conservative foregone revenue is 70 × 0.1 × 2 × $200,000 = $2.8M per quarter, or roughly $11.2M per year of revenue that the backlog is sitting on. A more aggressive assumption — that 30% of untested hypotheses would win — produces numbers an order of magnitude larger.
Either way, direct costs are in the six figures, and opportunity costs are even higher. The real cost of manual testing isn’t just design hours.
What automation actually changes
The argument for automating ASO experimentation isn’t “replace designers.” Designers are still essential for brand fidelity, creative direction, and the experiments where AI-generated assets fail the quality bar. The argument is: take the 60% of test creation that’s mechanical (variants, localized variations, version-based iteration) and let an autonomous system handle it, so designers spend time on the 40% that’s high-judgment creative work.
Pressplay automates the full loop — monitoring, learning, asset creation, testing, and kill decisions — running 24/7 across 50+ concurrent experiments per app. It integrates directly with Google Play Console (no simulation), respects brand guidelines through fine-tuned models, and shares learnings across titles in your portfolio.
The result for our clients has been measurable: Avakin Life lifted CVR by 57%, Lion Studios (AppLovin) saw a 20% install lift, and Wildlife Studios is projecting 12M additional installs annually from the same approach.
Ready to take the next step?
If you want to see the numbers for your own portfolio, book a 30-minute session with our team. We’ll review your current cycle, use your real data, and show you the gap. No demo, no sales pitch—just the numbers.