Visual Website Optimizer has a great post debunking the multi-armed bandit algorithm. This was made famous recently by a blog post titled: 20 lines of code that will beat A/B testing every time.
What we found out was that the reality is not as simple as what that blog post claimed to be. In short, multi-armed bandit algorithms do not “beat” A/B testing. In fact, a naive understanding of this algorithm can lead you to wondering what’s happening under the hood.
if you just talk about average conversion rates, multi-armed bandit algorithms usually perform better than A/B testing. But a fundamental point missing here is the concept ofstatistical significance.
what you gain by increased average conversion rate, you lose on time it takes to see statistical significance.