Also Worth Noting - 2026-07-03

Also Worth Noting

02 [Inference] Morphing into Hybrid Attention Models Fixed-pattern heuristics for hybrid layer placement miss the interdependence between layers, and treating each layer's importance in isolation produces meaningfully worse results than learned selection. This work formulates hybrid layer selection as a global optimization problem, finding that the layers worth keeping as full attention under a joint configuration differ substantially from what per-layer scoring predicts. For teams converting dense Transformers to hybrid architectures to cut long-context costs, the selection method matters as much as the conversion itself. link

03 [Training] Denser $\neq$ Better: Limits of On-Policy Self-Distillation for Continual Post-Training On-policy self-distillation does not reliably prevent catastrophic forgetting during continual post-training. Self-distillation policy optimization (SDPO) accelerates in-domain specialization when teacher signals are stable, but collapses on out-of-distribution scenarios and still exhibits strong forgetting under continual training conditions. The "on-policy fixes forgetting" narrative that spread through 2025 is condition-dependent, and teams running sequential fine-tuning pipelines should not treat SDPO as a general-purpose safeguard. link

04 [Theory] WARP: Weight-Space Analysis for Recovering Training Data Portfolios A foundation model's domain mixture weights can be recovered from its public parameters alone, with no access to the training pipeline. WARP analyzes weight-space statistics to infer what proportion of training data came from each domain, operating at the corpus-composition level rather than the individual-sample level that membership inference targets. This exposes a transparency gap in standard model releases and gives auditors a concrete tool for interrogating undisclosed data recipes. link

05 [Application] Learning to Move Before Learning to Do: Task-Agnostic Pretraining for VLAs Vision-Language-Action models conflate two distinct learning objectives, and only one of them actually requires expensive expert demonstrations. Task-Agnostic Pretraining (TAP) separates physical competence from semantic alignment, first learning transferable motor priors from cheap, unlabeled motion data, then fine-tuning on far fewer expert demonstrations for task-specific semantics. Teams bottlenecked by demonstration collection costs for robot learning should treat motor pretraining as a separable, cheaper upstream stage. link

06 [Agent] AutoMem: Automated Learning of Memory as a Cognitive Skill Memory management in LLMs performs better when the model learns it as a trainable skill rather than executing a fixed retrieval policy. AutoMem promotes file-system operations to first-class actions alongside task actions, letting the model decide what to write, update, and delete, with both the memory structure and the model's proficiency improving through training. On multi-step tasks where context windows alone fail, the learned policy outperforms static retrieval strategies by a measurable margin. link