Estimating Causal Effects of Long-term Treatment in A/B Testing: Methodology and Application

Professor Huang, Shan   (Principal Investigator (PI))

24

2023-12-01

816500

Estimating Causal Effects of Long-term Treatment in A/B Testing: Methodology and Application

Causal Effects, Long-term Treatment, A/B Testing, Methodology, Application

Business Studies

ITS/311/22FP

Innovation and Technology Support Programme

2023

On-going

Randomized controlled experiments, also known as A/B testing, have become the goldstandard for technology firms to test the effectiveness of their new product developments.One lingering challenge of such randomized experiments at many technology firms isestimating the long-term impact of rolling out a new development indefinitely in thesystem. Conducting experiments for a long duration, however, could be risky and costly.Instead, experiments are typically conducted for a short time (e.g., two weeks).Nevertheless, it is the long-term performance of treatment that reflects the overalldevelopment strategy.This project aims to leverage the availability of such short-term experiments to estimatethe impact of long-term treatment, which remains an open question in academia andindustry. Specifically, we will develop mathematical frameworks to evaluate the causaleffect of long-term treatment, i.e., the different performance between the experimentalunits continuously exposed to the treatment and those exposed to the control for a longduration. We will then empirically verify and implement our methods on industrialexperimentation systems. Our proposed methods will largely improve the reliability andefficiency of A/B testing for decision-making. They also can apply to other areas, e.g.,evaluating the long-term treatment impact of government policies and clinical trials.