π Growth Gems #9 - Gems from "How to Level Up Your A/B Testing Game"
Hi there,
Today's gems are mined from another presentation made during the App Promotion Summit byΒ Karan Tibdewal (Growth Consultant at Phiture):
How to Level Up Your A/B Testing Game
π#1
To get to the point where more testing = more growth being true you need to have a thorough process in place.
13:28
π#2
3 key goals of the testing framework are:
Communicate ideas from all teams and prioritize based on company objectives
Understand and define leading and lagging metrics
Centralize test results and communicate next steps
14:12
π#3
Use slides for your experiments: excel files are good for internal use but slides are better because they help get the buy-in from different teams and get them involved.
17:13
π#4
When it comes to goal metrics, you do not want to only optimize for revenue: you want "leading metrics" that feed in to revenue because there are a lot of variables that go into optimizing for revenue.
22:48
π#5
To know which ideas to prioritize, measure the Impact score with Impact = Reach x Relevance x Frequency.
24:15
π#6
Set up experiments for success by planning ahead of time and using a sample size calculator tool like the one from Optimizely (link here).
28:27
π#7
You should not have a "win or lose" mentality. Once you have a significant positive impact you want to scale up the test (apply to 100%) then you want to start iterating on it. Double down on high impact tests and keep iterating.
31:20
π#8
After 2 or 3 variations of a test idea that does not prove a significant positive impact, do not get too attached to your idea and stop iterating: it is not meant to be.
32:05
π#9
Do a reach audit where you assess how many tests you could run with your user base with your current engagement rate. Example: don't just look at how many people will receive that email and assume a 10% conversion rate. Instead, look at your past performance, look at how many users you can reach and use calculator to figure out time to reach statistical significance.
35:50
π#10
Onboarding and activation funnels are the main areas of improvement for almost all of the apps (even 5+ years apps!).
36:20
π#11
How long to run tests for? A week is often enough but it of course depends on volume. Usually the absolute difference between variants should be more than 200-300 to get proper test results (if you are very small). Do not go for tests that are more than 1-1.5 months.
49:55
π#12
How often do you revisit experiments? Once you have an experiment in place and you have maximized/iterated on it, it is good to have quarterly reviews.
52:32
Stay savvy!
βοΈ Sylvain