Episode Details

“What is neuralese and why is it bad?” by Linch

Published 2 weeks, 3 days ago

Description

What is neuralese?

To explain neuralese, we need to first understand chain-of-thought, one of the largest developments in AI in the last five years.

Right now AIs think broadly but shallowly in a single forward pass. The model gives you an immediate snap answer to a question you might be interested in.

They can be pretty smart in their snap answers,, but mostly they can’t do very advanced reasoning tasks like complicated math or programming:

.The solution that the frontier AI companies have come up with is called chain-of-thought. Basically the model runs one forward pass, writes down some intermediate thoughts in natural language in a journal, and then that's fed back into the model to run another pass. This loop is repeated until the model is somewhat confident it has the right answer (or it hits a cap on thinking time), and then it outputs the user-visible results (for example a chatbot's response to your question, or working code). The looping step is often called “recurrence.”

Natural-language chain-of-thought is a major advance in letting models reason for longer, but it also has an accidental safety benefit. Using natural language as a key recurrence step for a [...]

---

Outline:

(00:10) What is neuralese?

(03:25) Why is it bad?

(04:48) Is it in use today?

(06:07) Appendix A: OpenAI's response

(06:58) Total number of serial steps low

(07:59) Chain-of-thought monitoring isn't a perfect or long term solution anyway

(09:28) Aren't you afraid of manifesting the bad thing?

The original text contained 10 footnotes which were omitted from this narration.

---

First published:
September 2nd, 2026