Episode Details

What’s up everyone, today we have the pleasure of sitting down with Sundar Swaminathan, author of the experiMENTAL newsletter and part time Marketing and Data science advisor?

Summary: After leading Uber's Marketing Data Science teams, Sundar shares insights that work for both tech giants and startups. Beyond uncovering that Meta ads generated zero incremental value (saving $30 million annually), they mastered measuring brand impact through geo testing and predicting LTV through first-week behaviors. Small companies can adapt these methods through strategic A/B testing and simplified attribution models, even with limited sample sizes. Building data science teams that embrace business impact over technical complexity, and maintaining curiosity, like when direct driver engagement revealed that recommending Saturday afternoon starts over Friday peak hours improved retention.

About Sundar

• Sundar started his career as a software developer at Bloomberg before managing $19 Trillion at the US Treasury as a Debt Manager
• He pivoted to growth marketing and data science consulting where he worked with DirectTV and an ed-tech AI startup
• He then made the mega move to Uber where he spent 5 years building Brand, Performance, and Lifecycle Marketing Data Science teams
• He moved over to a travel tech startup and helped them go from $0 to $100K MRR
• Today, Sundar is a marketing and data science advisor, he helps B2C founders and marketers 
• He’s also working on an upcoming podcast and has a newsletter where he shares frameworks, how-to guides to help B2C marketers

Marketing Incrementality Testing Reveals Meta Ads Ineffective at Uber

Performance marketing often reveals surprising truths about channel effectiveness, as demonstrated by a fascinating case study from Uber's marketing operations. When confronted with unstable customer acquisition costs (CAC) that fluctuated 10-20% week over week despite consistent ad spend on Meta platforms, Uber's performance marketing team, led by Sundar, decided to investigate the underlying causes.

The investigation began when the team noticed significant volatility in signup rates despite maintaining steady advertising investments. This inconsistency prompted a deeper analysis of Meta's effectiveness as a primary performance marketing channel. The timing of this analysis was particularly relevant, as Uber had already achieved substantial market penetration eight years after its launch, especially in major urban markets where awareness wasn't the primary barrier to adoption.

Through rigorous data analysis, the team implemented a three-month incrementality test to measure Meta's true impact on user acquisition. The test utilized a classic A/B testing methodology, comparing a control group receiving no paid ads against a treatment group exposed to Meta advertising. The results were striking: Meta advertising showed virtually no incremental value in driving new user acquisition, a finding that was validated by Meta's own data science team.

The outcome of this experiment led to a significant strategic shift, resulting in annual savings of approximately $30 million in the U.S. market alone. While this figure might seem modest for a company of Uber's scale, its implications were far-reaching when considered across global markets. The success of this experiment also highlighted the importance of data-driven decision-making and the willingness to challenge assumptions about established marketing channels.

Key takeaway: Established marketing channels should never be exempt from rigorous effectiveness testing. Regular incrementality testing can reveal unexpected insights about channel performance and lead to substantial cost savings. Marketing teams should prioritize data-driven decision-making over assumptions about channel effectiveness, even for seemingly essenti