Statistical significance & other A/B test pitfalls

Just read a very insightful post by good UX friend Cynnedd Bowles of Brighton based Clearleft and I think he makes an excellent point. Although I can’t say I fully agree with him.

Last week I tossed a coin a hundred times. 49 heads. Then I changed into a red t-shirt and tossed the same coin another hundred times. 51 heads. From this, I conclude that wearing a red shirt gives a 4.1% increase in conversion in throwing heads.

A ridiculous experiment (yes, I really did it) with a ridiculous conclusion, yet I sometimes see similarly unreliable analysis in A/B testing.

Read the full post here.