Episode Details

When a speech recognizer and a correction model do one job, can they be folded into a single set of weights? The answer depends entirely on a distinction most coverage skips: model merging and distillation are not the same operation. Merging combines parameters from models that share a pretrained ancestor — no training, no extra inference cost, same parameter count. Distillation trains a new model to imitate others, and it's the only route that works across different architectures. This episode walks through task arithmetic, weight averaging, re-basin permutations, and LoRA merging to find exactly where each one breaks, then follows the distillation route from FuseLLM to cross-architecture transfers between CNNs, transformers, and Mamba. Along the way: why an elaborate feature-alignment loss made a student model worse than plain output matching, and why you can't average a lesson into a model that never learned it. Key sources: • Model Merging in LLMs, MLLMs, and Beyond survey (Yang et al.) — https://arxiv.org/abs/2408.07666 • Donato Crisostomi, Model Merging — https://crisostomi.github.io/blog/2025/model_merging/ • Uncertainty-Based Gradient Matching (ICLR 2024) — https://arxiv.org/abs/2310.12808 Episode 5975 — myweirdprompts.com/n/5975 (permanent short link: myweirdprompts.com/586126)