Why multi-armed bandit algorithm is not “better” than A/B testing

Visual Website Optimizer has a great post debunking the  multi-armed bandit algorithm.  This was made famous recently by a blog post titled: 20 lines of code that will beat A/B testing every time.

What we found out was that the reality is not as simple as what that blog post claimed to be. In short, multi-armed bandit algorithms do not “beat” A/B testing. In fact, a naive understanding of this algorithm can lead you to wondering what’s happening under the hood.

if you just talk about average conversion rates, multi-armed bandit algorithms usually perform better than A/B testing. But a fundamental point missing here is the concept ofstatistical significance.

what you gain by increased average conversion rate, you lose on time it takes to see statistical significance.

This entry was posted in Analytics and tagged , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s