Treatment Heterogeneity and Customer Value in Online Experiments: How A/B Testing Leads to Data-Driven Mistakes
Operations, Information, & Decisions Department; Faculty Adviser: Kartik Hosanagar
Randomized control trials—commonly referred to as “A/B tests” in retail and e-commerce—are the sine qua non of causal inference and data-driven decision making. Managers are increasingly turning to A/B tests to make objective decisions backed by statistical theory. In current retail practice, most A/B tests are conducted by randomly assigning customers to two groups (group A and group B), presenting each of them with a separate shopping experience, measuring an economically important outcome variable, and comparing the average of this variable between groups. However, there is a hidden shortcoming with this otherwise valid technique: while it is possible that version A outperforms version B on average, version B might actually outperform version A among the firm’s most valuable customers. By simply taking averages and ignoring this heterogeneity in customer response, managers are susceptible to choosing consequential business strategies that actually harm profits, based on what they presumed to be a foolproof statistical test.
It is currently not known how prevalent this problem is in e-commerce and there is no off-the-shelf remedy for identifying whether a particular test is susceptible to arriving at the wrong conclusion. This project will apply state-of-the-art tools from statistics and retail analytics to develop methods of robust inference that ensure the insights from A/B tests align with practitioners’ long-term strategic and financial objectives. This research is eminently relevant for any firm that relies on A/B testing as part of their decision-making process.