Podcast Episode Details
Back to Podcast Episodes
Complete Course: AI Experimentation
"AI has been the biggest driver of change in experimentation I've seen in my career."
That's Frederic De Todaro, Chief Product Officer at Kameleoon (profitable SaaS with 2K+ customers).
Fred has been at Kameleoon for 12+ years. In that role, he's helped thousands of teams use AI to experiment faster and smarter.
In today’s episode he’s breaking down:
How AI changes experimentation
How to experiment with AI features
Last week, I covered how one aspect of this: vibe experimentation. Today’s video is the A to Z AI impact.
If you experiment at work, this episode is for you.
----
Check out the conversation on Apple, Spotify and YouTube.
* Mobbin: Discover real-world design inspiration
* Jira Product Discovery: Build the right thing, reliably
* AI Product Strategy Certificate for Leaders: Get $550 off
----
Timestamps:
00:00 How AI Changed Experimentation Overview
01:54 The 4 Steps of Experimentation Framework
14:12 ADS
16:00 How AI has Changed Experimentation
21:08 User Behaviour Models
26:56 Multi-Armed Bandit vs Contextual Bandit
30:05 ADS
31:55 AI Content Genration
35:13 How Vibe Coding Changes Experimentation
41:35 Live Demo From Idea to Running Experiment in 2 Minutes
43:36 Two-Minute Build Achievement
51:49 How to Measure AI Features Properly
54:17 Measuring RAG Systems 3 Key Metrics
01:07:18 Best Experimentation Company Booking.com
01:10:10 Biggest PM Mistakes in Experimentation
01:13:52 Ending
----
Key Takeaways
1. Build is the bottleneck. Most teams can't A/B test because developers are busy. AI removes this constraint anyone can now create experiments in minutes using plain English.
2. 80% of experiments fail. But with AI opportunity detection, you can drill into failed experiments to find hidden wins, like features that work great on mobile but fail on desktop.
3. Vibe coding meets experimentation. It's not enough to build prototypes quickly. You need to test them with real users at scale. Prompt-based experimentation bridges this gap.
4. Context is everything. AI performs best when it has access to your website's framework, design system, and past experiments. The more context, the better the ideas and implementations.
5. Humans still matter. PMs bring business context, data scientists ensure statistical rigor, and AI handles the grunt work. It's augmentation, not replacement.
6. Start simple with feature flags. You don't need to copy Booking.com overnight. Begin with feature flags, then rollouts, then full experimentation. AI makes each step easier.
7. Measure beyond usage. For AI features, track: How many prompts to success? Time from idea to live? How often do developers step in? These reveal true value.
8. Multi-armed bandits for speed, contextual for personalization. Use multi-armed whe
Published on 1 week, 6 days ago