Podcast Episode Details
Back to Podcast Episodes
Malcolm Got in a Heated Argument with Eliezer Yudkowsky at a Party (Recounting an AI Safety Debate)
Malcolm recounts a heated debate with AI theorist Eliezer Yudkowsky on AI safety. He explains his belief that subsystems in an advanced AI would converge on the same utility function, while Yudkowsky insists no AI would subdivide that way.
Simone notes Yudkowsky's surprising lack of knowledge in physics and neuroscience given his confidence. They express concern his ideas ruin youth's outlooks and discuss hypothetical clapbacks. Overall they conclude that while well-intended, Yudkowsky's certainty without humility on AI risks is dangerous.
Simone: [00:00:00] What's
Malcolm: really interesting is that he actually conceded that if this was the way that an AI structured itself, that yes, you would have terminal convergence, but that AIs above a certain level of intelligence would never structure themselves this way.
Malcolm: So this was very interesting to me because it wasn't the argument I thought he would take. And that would be true. I, I will agree that if the AI maintained itself as a single hierarchy, it would be much less likely for its utility function to change. But the problem is... Is essentially no government structure ever created and has functioned that way.
Malcolm: Essentially no program ever created by humans has run that way. Nothing ever encoded by evolution has run that way. i. e. the human brain, any brain, any neural structure we know of. There are none that are coded that way.
Malcolm: So it is very surprising. So I said, okay, gauntlet thrown. Are you willing to be disproven?
Malcolm: , because we will get some more understanding into AI interpretability, into how AIs. Think in the near future. If it turns out [00:01:00] that the AI's that exist right now are actually structuring themselves that way, will you concede that you are wrong about the way that you tackle AI apocalypticism?
Malcolm: And then he said, and this is really interesting to me. He's like, no, I won't
Malcolm: I was also like, yeah, also, we could run experiments where we do a bunch of basically unbounded A. I. S. and see if they start to show terminal convergence.
Malcolm: Do they start to converge on similar utility functions? You know what they're trying to optimize for again? He was like, well, even if we saw that, that wouldn't change my views on anything, right? Like his views are religious in nature, which was very disappointing to me. Like, I thought that maybe he had more of like a logical or rational perspective on things.
Malcolm: That. And it was, it was really sad.
Malcolm: You know, we don't talk negatively about people on this channel very frequently, but I do think that he destroys a lot of people's lives. And I do think that he makes the risk of AI killing all humans dramatically higher than it would be in a world where he didn't exist.
Would you like to know more?[00:02:00]
Simone: Hello, Malcolm.
Malcolm: Hello. So we just got back from this wonderful conference thing we were at called Manifest. So we had gone out to SF to host a few pronatalist focused dinner parties and randomly we got looped in to something called Manifest, which was a conference for people who are interested in prediction markets.
Malcolm: But interestingly, we ended up meeting a bunch of people who we had known through like online stuff. Some were absolutely fantastic, like, Scott Alexander, absolutely. I never met him before in person. We'd communicated on a few issues, really cool guy. Would you say so Simone?
Simone: Yeah. Like super awesome.
Malcolm: Richard Hedania, a really nice guy as well. Robin Hanson, who we, we, we'd actually met him before. But. And o
Published on 2 years, 2 months ago