{"id":917,"date":"2013-02-05T22:52:31","date_gmt":"2013-02-05T22:52:31","guid":{"rendered":"http:\/\/www.nicholasgoodman.com\/bt\/blog\/?p=917"},"modified":"2013-02-06T19:04:15","modified_gmt":"2013-02-06T19:04:15","slug":"a-then-b-testing","status":"publish","type":"post","link":"http:\/\/www.nicholasgoodman.com\/bt\/blog\/2013\/02\/05\/a-then-b-testing\/","title":{"rendered":"At first, it’s A then B testing not A\/B testing"},"content":{"rendered":"

Had a great conversation today with Aydin Ghajar<\/a>; a guy who really knows his stuff when it comes to building viral loops for consumer services.\u00a0 The issue of rates\/conversions yield big differences in ultimate outcomes for viral systems.\u00a0 One of the things I’ve been grappling with thinking about A\/B testing in MVPs is the complexity this adds early on.\u00a0 After all, Lean is about doing things efficiently to learn what you need to match a customer\/market need.<\/p>\n

What happens when the basics to install the ability to learn how well A or B (or C or D) is non trivial?\u00a0<\/strong><\/em> A\/B testing that isn’t trivial website button color, text takes some effort (days, but still).\u00a0 My friend made an astute observation (paraphrased):<\/p>\n

At first, your A\/B testing while you’re still validating early assumptions is NOT parallel A\/B testing (50 visitors this week into TWO 25 visitor buckets) but rather sequential: A is done one week, evaluated.\u00a0 B is done week after, evaluated.<\/p><\/blockquote>\n

Why?\u00a0 First off, A\/B testing is complicated and best deferred until a bit later in MVP.\u00a0 Why else? At first, when you’re testing your early hypothesis your volumes are small enough that you’re decreasing the accuracy of and time to determine if A or B is true.<\/p>\n

In our example, if we get 50 users per week and we split them we’ll get 25 users in A and 25 users in B per week.\u00a0 Week one, we get 25 of each, and get an action or two in each.\u00a0 The statistic relevance or confidence of 25 user sample is LOW; it’s probably not enough to validate or invalidate the effectiveness or value of A or B (whatever you’re testing).\u00a0 So, next week you get another 25 in each.\u00a0 At 50 users, you’re confidence increases and now you’re ready to make a real determination on if A or B worked.<\/p>\n

What’s the problem?\u00a0 When you’re early on, getting things figured FAST is important.\u00a0 While not right for every thing, I’d much rather have tried A in one week (50 sample size), find the results, and move on to B.\u00a0 Same learning (faster in some cases) and more efficient (aka cheaper) initially.<\/p>\n

So, I’m in perfect agreement with my friend.\u00a0 A\/B testing happens sequentially at first, then in parallel once practical (implementation\/sample size).<\/strong><\/p>\n","protected":false},"excerpt":{"rendered":"

Had a great conversation today with Aydin Ghajar; a guy who really knows his stuff when it comes to building viral loops for consumer services.\u00a0 The issue of rates\/conversions yield big differences in ultimate outcomes for viral systems.\u00a0 One of the things I’ve been grappling with thinking about A\/B testing in MVPs is the complexity […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[13],"tags":[],"_links":{"self":[{"href":"http:\/\/www.nicholasgoodman.com\/bt\/blog\/wp-json\/wp\/v2\/posts\/917"}],"collection":[{"href":"http:\/\/www.nicholasgoodman.com\/bt\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.nicholasgoodman.com\/bt\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.nicholasgoodman.com\/bt\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"http:\/\/www.nicholasgoodman.com\/bt\/blog\/wp-json\/wp\/v2\/comments?post=917"}],"version-history":[{"count":4,"href":"http:\/\/www.nicholasgoodman.com\/bt\/blog\/wp-json\/wp\/v2\/posts\/917\/revisions"}],"predecessor-version":[{"id":921,"href":"http:\/\/www.nicholasgoodman.com\/bt\/blog\/wp-json\/wp\/v2\/posts\/917\/revisions\/921"}],"wp:attachment":[{"href":"http:\/\/www.nicholasgoodman.com\/bt\/blog\/wp-json\/wp\/v2\/media?parent=917"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.nicholasgoodman.com\/bt\/blog\/wp-json\/wp\/v2\/categories?post=917"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.nicholasgoodman.com\/bt\/blog\/wp-json\/wp\/v2\/tags?post=917"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}