Episode Details

To all my fellow researchers doing SLT, computational mechanics, one of ARC's programs, natural abstractions/condensation, proofs on NNs (or any interp on small models), this is for you.

Tensor transformers (ie replacing your MLPs & attention with bilinear variants) are performant and allow you to deploy the full power of linear algebra. In fact, our recent paper used generalized cosine similarity on the full tensor transformer. And yes, I mean cos-sim defined on the eg 9th order tensor, not individual vectors or matrices. This removed all the symmetries/invariances that weren't functionally relevant.

But tensor-variants don't generalize to "real models", right?

The architectures are very similar:

SwiGLU(x) = D(swish(Lx) ⊙ Rx) (used by DeepSeek-V3, Kimi K2, and Qwen3))
Bilinear(x) = D(Lx ⊙ Rx) (this is the tensor version)

Where D, L, & R are linear matrices. For reference:

MLP(x) = D(ReLU(Lx))

Due to the double-encoder/multilinearity, SwiGLU & Bilinear have no global Lipschitz constant (and other similar inductive biases). This means results like finetuning away the normalization might not generalize to these SOTA archs since this was only run on single-encoder MLPs.

For attn, the more SOTA tensor-arch is:

Bilinear_Attn = OV()

Compared to softmax attention, this does [...]

---

Outline:

(02:30) Frontier Models aren't the Only Thing That Matters

(03:42) My Extreme Pessimism (or Ignorance)

The original text contained 5 footnotes which were omitted from this narration.

---

First published:
September 2nd, 2026

Source:
https://www.lesswrong.com/posts/expsAXaBgiqgitFe6/if-you-re-interpreting-less-than-1b-parameter-models-you

---

Narrated by TYPE III AUDIO.