What is A B Testing |
| A/B experiment is a approach in which two variations of a change are shown to two different groups of users, and whichever one works better (however better is defined) is chosen over the variant that didn't work as well. It's a form of market testing made famous by search engines and now in common use across the tech industry. |
What are types of test in ab testing |
How to design ExperimentsClassic A/B Experiment - there is control A and a treatment (varience) B. This is most commonly usd method. Leap Experiment - It is variation of above. Multiple changs/treatment are added together to test whether newer version provide lift. It reduce experimentation time for each change. It has a downside that you do not know which change is contributing how much to lift?
Exploratory Cascade ExperimentIn this method one design multiple variations. Each variation has a single change. The variations build on top of each other and this allows for comparisons across set. (Some teams call each set a flight). It is useful when you have very large traffic and one can use it to try out multiple solutions or simply explore what users like. Exploratory Leap Experiment - In this we explore multiple different combination to determine which change bring benefit. Reverse Experiment - This is similar to above. Here you explictly remove feature to find its effect tect Example : remove price from webpage and see it if it drop conversion.
Intensity Experiment - This type of experiment is useful if hypotheis is that intensity of treatment will impact outcome. There are scenario where a variable has a linear impact on outcome (Example : 1 testimonial, comments on post might be good, 2 even better, 3 even, 100+ is awesome |
Pre req for Ab testing |
How to determine when we are ready for ab testing/ experimentation with trafficConsider these important factors
💡 Explanation: If you do not have enough traffic, you cant split it meaningfully.If you have enough traffic you can split into a and b set. To comppare effec in a and b set you need to have enough conversion /treatment effect. |
How much data I need for AB tesing |
Sample Size importance⛔ Myth: A statistically significant double digit conversion rate increase is awesone. 🎉 The bigger conversion uplift, is better for business! 🥳✅ Reality: If lift in conversion is quite big, it is likely to be erronous. 🧨 Check sample size, ypu may be running under power test. . 🧨 💡 Explanation: With low sample size result can appear statistically significant. Sample size is key. Here is example, if 10 users come to website and ab testing split traffic to a and b set. By chance if one user convert on set A an 2 users convert on set B, one may reach to conclusion that there is 200 percent lift. However one should not reach to such conclusion as one extra user can convert by chance. Such test are called under powered test and lead to wrong conclusion. While designing a b expriment it is important to ensure that experiment is not under powered. |