Episode Details

In this episode, Gabriel (Founder) and Caine (CTO, first employee) discuss the history of our search engine, why now is the right time to build a full web search index, and how our scale makes us uniquely positioned to ship, learn and iterate quickly.

Disclaimers: (1) The audio, video (above), and transcript (below) are unedited and may contain minor inaccuracies or transcription errors. (2) This website is operated by Substack. This is their privacy policy.

Gabriel: Hello, welcome back to Duck Tales. I haven’t been here in a while. And I am Gabriel Weinberg, the founder of DuckDuckGo. And I have with me someone who I don’t think has been on Duck Tales at all yet, but you should know, Caine Tighe, who I know very well, who’s the first employee of DuckDuckGo and now our CTO. Caine.

Caine: Hi Gabe.

Gabriel: We’ve been working together for a very long time. And we’re here today to talk about something we’ve both been working on. Caine more than me, but I’m working on it some, which is our web search index. So as some background, first some background. DuckDuckGo started as a search engine, as many people know, and it was actually started by me. I was by myself for a few years. And the first thing I did was start crawling the web and building a web index.

Caine: Yeah, for sure.

Gabriel: But you know, I soon realized that that is very expensive, especially as one person. And there were other places to get a web index at the time. And what was more interesting was maybe adding value on top of the web index. So building other indexes, this was a time, this is the mid 2000s, you know, there weren’t, there obviously wasn’t AI, but there wasn’t even really many instant answers on search engines.

Caine: I mean, that’s what we were working on together at the very, very beginning. Like we were working on, you know, you had the knowledge graph. It wasn’t called a knowledge graph at a time, but you were doing all the structured content from Wikipedia and otherwise. We worked on some other smaller indices. So yeah. And then actually fun fact, in hiring our backend project is still based on some of the original spam and content farm crawling, like one of the projects is based on some of the spam and content farm crawlers that you originally wrote. So that lives on 15 years.

Gabriel: Yeah. So we were doing lots of indexing and lots of crawling. Yeah, exactly. Just not, you know, we started, but then we stopped doing a full web index, but just as examples, right? We started like the code that you were talking about indexing Wikipedia, which became our knowledge graph, you know, which is, powers a lot of answers, which also we used when we started working on AI answers. We’ve been doing local indexing for, you know, over a decade, local businesses and things like that. You know, then all sorts of kind of niche indexes that involve some crawling like lyrics and things like that. So indexing technology is not new to us, despite what some people say about it. Sometimes we do lots of search indexing, but we hadn’t been doing a full web index until relatively recently, last few years-ish. But now we are. And so the question is, the questions and why you’re here, and we’ll talk about it for a few minutes, is kind of why, what’s going on, how, all the main questions, which we’re obviously not all go