Episode Details

Please share this with anyone doing AI research with 3rd party providers so that they can ensure their research won’t be corrupted.

• When you ask OpenRouter[1] to give you tokens from a given model, OpenRouter sends your request to a random available provider.
• OpenRouter providers have variable quality. Ensuring that your provider is high quality is really difficult. 
• There is precedent for an AI safety paper accepted to NeurIPS having its core results entirely overturned by these issues.
• A review of influential AI Safety research codebases that use OpenRouter for their reported results found that 31/32 (97%) of them use OpenRouter unsafely.[2]
• Researchers who wish to do research using OpenRouter or similar providers should take precautions to minimize the risks to their research,[3] though the current selection of providers is insufficient for ideal scientific reliability.
• Replicators should test whether results hold up when these bugs are fixed.
• Core AI Safety codebases should make fixes so that downstream users are able to implement best practices.

This is a tangent I took for a few days during the Pivotal AI Safety Research Fellowship. I’m doing an AI Control project mentored by Adam Kaufman and James Lucassen of Redwood Research and partnered with Aniruddh [...]

---

Outline:

(02:04) OpenRouter Kills an AI Safety NeurIPS Paper

(03:29) Background

(04:15) Prior work/writing on the subject.

(05:38) No Guarantees from Fixing a Provider

(06:14) No Guarantees From Large N

(07:09) No Guarantees from "The Benchmark Didn't Move"

(08:19) No Guarantees from "Choose the Most Expensive Provider"

(08:54) No Guarantees from "Set a floor"

(09:12) When is Default OpenRouter Fine?

(09:50) Developing Best Practices

(11:45) Spotting Fingerprints

(13:12) Highlighting Some Impacted Work

(13:57) en-US-AvaMultilingualNeural__ Infographic titled "Provider Routing Inspector" with statistics and bar chart on OpenRouter reliability.

(14:10) Should The Community Do Something About This?

(14:15) Fixing Repos

(14:34) Replicating Research

(15:01) A New Org?

The original text contained 12 footnotes which were omitted from this narration.

---

First published:
July 23rd, 2026

Source:
https://www.lesswrong.com/posts/KsyoSAyBRXtwzSugg/not-pinning-your-openrouter-provider-might-invalidate-your

---

Narrated by TYPE III AUDIO.

---

Images from the article:

---

Listen Now