How Leading Portfolio Publishers Run ASO Experimentation (Lessons from the Field)
Fabio Salvador
We’ve noticed a clear pattern in mobile portfolios that take ASO testing seriously. Once publishers scale experimentation beyond three or four apps, they stop focusing on individual tests and start building a system. This is an operational shift, not a technical one. The teams that succeed share a few key habits that set them apart.
This post brings together what we’ve learned from working with portfolio publishers like Avakin Life, Lion Studios (AppLovin), Wildlife Studios, and Raiffeisen Bank International, across both consumer and fintech. We’ve left out any client-specific tactics to keep things confidential, and instead focused on the operating principles that work across portfolios. If you’re looking to scale ASO testing beyond a single app, these are the patterns to pay attention to.
Principle 1: Treat the portfolio as one experiment stream, not many
Most teams start ASO testing by working on one app at a time. Each app has its own roadmap and backlog, managed separately. This approach works at first, but it quickly breaks down as you scale.
A better approach is to treat your portfolio as a single learning system. When you run a test on App A, use what you learn to inform tests on every other app where it makes sense. Wildlife Studios, for example, expects to gain 12M extra installs each year by sharing learnings across titles. The real value comes from how quickly you can validate and roll out insights across your portfolio.
In practice, this means keeping a shared backlog of hypotheses, tagged by mechanic—like ‘social proof in screenshot 1,’ ‘character-led icon,’ or ‘benefit-led short description.’ Track the status for each app and mechanic. When something works on one app, queue it up for testing on every other app where it could apply.
Principle 2: Separate “what to test” from “who designs the test”
One of the biggest mistakes is letting designer bandwidth decide what gets tested. If designers are the bottleneck, you end up running the tests that are quickest to build, not the ones with the most potential impact.
The publishers running ASO experimentation well separate the two functions:
- Hypothesis prioritization — owned by ASO/growth leads, scored on expected impact and confidence, independent of design capacity.
- Asset production — owned by design (or augmented by automation), executed against the prioritized list.
This separation also makes automation possible. Once you decouple prioritization, asset production becomes a queue you can scale in different ways. Use in-house designers for high-judgment work, freelancers for producing variants, and automation for the bulk of variants that don’t need new creative direction.
Principle 3: Decide kill criteria before the test launches
This might sound simple, but most teams only set success criteria, not kill criteria. As a result, underperforming tests often stay live too long and waste valuable traffic.
The portfolio publishers we see operating well define, before launch:
- The minimum sample size and confidence threshold to call a winner
- The maximum runtime before the test is killed for inconclusive results
- The threshold at which an underperforming variant is killed early to limit downside exposure
Most teams skip this last point. If a variant is tracking 5% or more below control with enough data, you’re losing installs every day it stays live. Remove it quickly.
Automated systems can enforce these rules around the clock. PressPlay, for example, applies kill rules to all running experiments and acts within hours when there’s a clear negative signal. If you’re running tests manually, make sure the rule is written down before the test starts. Don’t negotiate it in the middle of the test.
Principle 4: Don’t conflate visual A/B testing with portfolio strategy
This is where we see senior ASO Managers and VPs of Growth diverge in their assessments of which experimentation programs are worth pursuing.
Visual A/B testing—like icon, screenshot, or feature graphic variants—is tactical. Each test aims to improve conversion rate on a single listing. Avakin Life, for example, saw a 57% CVR lift from systematic visual testing at scale.
Portfolio experimentation is strategic. It’s about deciding where to invest design resources, which markets to target, which mechanics to repeat, and what insights to build into your brand standards.
A strong experimentation program does both. If you only focus on visual testing, you’re missing out on most of the value.
Principle 5: Measure cycle time, not just outcomes
Most reporting on ASO experimentation focuses on win rate (the percentage of tests that beat the control) and effect size (how much they beat by). These are the right outcome metrics.
But for portfolio publishers, cycle time matters even more. This is the median number of days from when a hypothesis enters the backlog to when a decision is made and put into production.
For example, if you run 100 tests per year with a 14-day cycle time, you get 100 decisions per year. If you cut the cycle time to 7 days, you can make 200 decisions in the same year. Faster cycle times mean you can deploy and build on winners more quickly.
Cycle time is also the metric that automation moves most. Manual ASO testing has a floor on cycle time set by human briefing, design, and review steps. Automated systems can compress the floor by an order of magnitude.
Principle 6: Be honest about where AI helps and where it doesn’t
The best portfolio publishers use AI-generated assets where it makes sense—like background changes, layout tweaks, or localized copy on a proven template. They rely on designers for creative work, such as new visual concepts, brand icons, or hero screenshots that need original direction.
Here’s the reality: generative AI is great for expanding the number of testable variants on proven concepts. It can’t replace a designer for important creative decisions. The publishers who get the most out of AI use it to multiply their creative direction, not to replace it.
What to take from this
If you’re running ASO testing across more than three apps, these six principles are the playbook used by publishers who scale successfully:
- Run one portfolio-wide experiment stream, not isolated app roadmaps.
- Decouple prioritization from production so impact, not bandwidth, drives the roadmap.
- Define kill criteria up front and enforce them ruthlessly.
- Combine tactical visual testing with strategic portfolio decisions.
- Track and compress cycle time as a first-class metric.
- Use AI as a force multiplier, not a replacement, for great creative.
You don’t need PressPlay to do any of this, but automation makes it much easier to keep the loop running and share learnings across your apps.
If you want to see how this works in a real portfolio—how cross-title learnings are shared, how kill criteria are enforced automatically, and how cycle time drops with automation—book a working session with our team. We’ll walk through your portfolio model and highlight where you can get the most leverage.