Episode Details

Researchers from Meta Platforms propose a novel offline evaluation task called impression share prediction to better anticipate how new ranking models redistribute traffic across different business objectives. Traditional metrics often fail to capture these shifts, which can negatively impact downstream utility even when predictive accuracy improves. To address this, the authors developed a structural causal model that identifies how model signals and delivery capacity interact to determine impression allocation. Their framework includes a Random Forest regressor for established models and a specialized encoder-conditioned architecture to handle the complex dynamics of newly introduced models. This system significantly reduces prediction error compared to standard baselines, particularly during the critical first hour of a model's deployment. Ultimately, this approach provides practitioners with vital visibility into a candidate model's allocation behavior before proceeding to expensive online A/B testing.