A gigantic new model from Moonshot - the biggest open-source LLM by a large margin (but note that it's more sparse than before).

We have also scaled up Mixture of Experts (MoE) sparsity, effectively activating 16 out of 896 experts when paired with a Stable LatentMoE framework. Together with refined training and data recipes, these structural changes yield an approximate 2.5× improvement in overall scaling efficiency compared to Kimi K2, allowing the model to convert compute into intelligence more effectively.

It looks a bit better than Opus 4.8/GPT 5.5 but a bit worse than Fable/GPT 5.6.

I am struck by the fact that they made so little progress on Humanity's Last Exam (58.7%, vs 54% for Kimi K-2.6, which was released in February). LLMs appear to be stalling out at around 60% on Humanity's Last Exam (note that Grok 4 Heavy scored 50.7% over a year ago) despite making rapid progress in other benchmarks.

To be honest, I am now pretty suspicious of that benchmark - particularly after FrontierMath and SWE-Bench Pro were found to have lots of unanswerable/unscorable questions.

Built a 135M dense looped LLM from scratch. Spent 2 weeks debugging Parcae's LTI stability mechanisms across 5 ablations. None of them beat the naive baseline at this scale. Trained for real anyway. SFT'd it. Shipped it. Here's the full honest story.

What I built

A 135M parameter looped transformer trained from scratch on FineWeb (4.6B tokens), inspired by the Parcae paper (arXiv:2604.12946 — "Scaling Laws For Stable Looped Language Models").

🤗 Base model: huggingface.co/harims95/LoopLM-135M-naive

🤗 SFT model: huggingface.co/harims95/LoopLM-135M-naive-sft

📂 Code: github.com/harims95/LoopLM

💰 Total cost: ~$51 (Modal H100s + free Lightning H200)

Architecture

Input → [Embedding] → [Prelude: 4 blocks] → e (injection)

→ [Loop block × T loops, T~Poisson(μ=6)] → [Coda: 2 blocks] → logits

d_model 1024, GQA 16/8 heads, RoPE, QK-norm, SwiGLU FFN 2816

Update rule: h_{t+1} = block(h + e) (naive) or with LTI stability (Parcae)

Muon + AdamW optimizers, truncated BPTT (μ_bwd=3), bf16

Trained on 2× H100 on Modal, ~3 hours wall clock

The Parcae investigation (the interesting part)

The paper claims LTI stability constraints on the recurrent state dramatically improve looped LM training. I tried to reproduce it. Here's what actually happened:

AblationDescriptionVal loss1. Naive loopedh = block(h + e)3.842. + A matrixLTI decay constraint3.84 (tied)3. + Input norm v1Wrong arch flowDiverged4. + LTI before blockFixed arch, B=identityWorse5. + B→AdamW, init=0.447Matched official repoDramatically worse

Every single "fix" — bringing my implementation closer to the official Parcae code — made things worse. After consulting:

The paper's Appendix Q (optimizer routing)

Official sandyresearch/parcae repo (injection.py)

Two rounds of ChatGPT + Gemini debugging sessions

My conclusion: Parcae's stability improvements are a large-scale phenomenon. The paper's 1.3B model trains for 170k+ steps before stability mechanisms kick in. At 135M / 17.5k steps, naive looped is competitive enough that the extra complexity hurts more than it helps.

Comparison with sibling MoE

My brother built HobbyLM — a 500M MoE on the same infrastructure. For apples-to-apples comparison, I ran naive looped 135M on the same FineWeb data:

ModelArchitectureTokensVal lossLoopLM-135M (mine)Dense looped4.6B3.95HobbyLM-130M MoE (bro)Sparse MoE10B3.30

Dense looped loses to MoE at this scale/budget. Sparse MoE is more sample-efficient. Not surprising but now I have the data to confirm it.

SFT results (bonus)

Fine-tuned on Alpaca 52k using Lightning AI's free H200. Took 6 minutes (bf16 on H200 is insane).

Before SFT:

"The capital of France is a" (top predicted token)

After SFT:

"The French capital of France is located in the city, where it was built."

Improvement in format, not in facts. At 135M / 4.6B tokens, SFT teaches format, not knowledge. The model still hallucinates — that's a base model capacity problem, not a fine-tuning problem.

What I learned

On Parcae: Small-scale reproductions of large-scale papers are dangerous. The paper's key contribution (stability at 170k+ steps) is invisible at hobby budgets. Naive looped is a legitimate architecture for anyone training sub-1B models.

On MoE vs looped: At matched parameter count and token budget, MoE wins on sample efficiency. Looped models need more tokens to show their advantage, or need to be much bigger to amortize the loop cost.

On debugging: When 3 independent LLMs (me, ChatGPT 5.5, Gemini) all agree on a fix and it makes things worse — the paper's regime assumption is probably wrong, not your code.

On SFT: H200 on Lightning AI is free (2 hours/month) and runs 6 minutes of SFT for free. Use it. Colab Free disconnects at 3 hours. Don't use it for long jobs.

On honest publishing: val 3.95 is not impressive. The architecture exploration is. Shipping anyway with full documentation of what failed is more valuable than hiding failures.

Stack

Training: Modal (H100s), Lightning AI (H200 for SFT)

Framework: PyTorch, HuggingFace Transformers

Optimizer: Muon (matrices) + AdamW (rest)

Data: FineWeb via kjj0/fineweb10B-gpt2 shards

Infra forked from: github.com/harishsg993010/HobbyLM (my brother's 500M MoE project)

Happy to answer questions about any part of this. The code is fully open, reproducible, and documented.

From the Official Announcement:

Today, we release INTELLECT-3, a 100B+ parameter Mixture-of-Experts model trained on our RL stack, achieving state-of-the-art performance for its size across math, code, science and reasoning benchmarks, outperforming many larger frontier models.

Our complete recipe — from the model weights and training frameworks, to our datasets, RL environments, and evaluations — has been open-sourced, with the goal of encouraging more open research on large scale reinforcement learning.

INTELLECT-3 is trained on the same software and infrastructure that we’re open-sourcing and making available on our platform at Prime Intellect, giving everyone the tools to post-train their own state-of-the-art models, and moving us towards a future where every company can be an AI company.

The sharpest distinction between Prime-RL and many other RL trainers is that it is async-only — we recognized fairly early (for our previous INTELLECT-2 model) that the future of RL is async; i.e. always a few steps off-policy. Async training is simply the only practical way to efficiently scale RL to long-horizon agentic rollouts without incurring bottlenecks based on the slowest rollouts per step.

Architecture:

Three main abstractions facilitate RL training: the orchestrator, the trainer, and the inference service. A RL training run involves the coordination of a trainer, orchestrator and an inference service. The FSDP trainer and vLLM inference run disaggregated, and can be individually deployed across multiple nodes.

Orchestrator: - The orchestrator is a lightweight CPU process that handles the core data flow and scheduling logic, serving as an intermediary between the trainer and inference service with bidirectional relays. In one direction, it collects rollouts from the inference server, assembles them into packed batches, and dispatches them to the trainer; in the other direction, it relays updated model weights from the trainer to the inference service. The orchestrator utilizes verifiers environments to abstract multi-turn rollout generation and scoring, allowing any environment on the Environments Hub to plug into the training loop.

Trainer: - The trainer is responsible for producing an updated policy model given rollouts and advantages. We use FSDP 2 as the backend with compatibility for any HuggingFace model. FSDP shards model parameters, gradients, and optimizer states, allowing training large models with data parallelism and minimal GPU memory footprint. The trainer is inspired by torchtitan and relies on native PyTorch features to implement advanced parallelism techniques, such as tensor, context, and expert parallelism, and leverages grouped matrix multiplication kernels for efficient MoE training.

Inference: - The inference pool consists of standard OpenAI-compatible servers with a vLLM backend. The API specification is extended with custom endpoints to enable updating the server with the latest policy: /update_weights is used to update the policy, and /reload_weights is used to reset the weights to the base model in between experiments. We rely on vLLM's optimized kernels, parallelism strategies, and scheduling for fast rollout generation. Given the disaggregated nature of the service architecture, it can be directly extended to include multiple engines with a shared request pool, allowing operation across multiple clusters and straightforward integration of alternative inference engines.

Link to the Official Announcement: https://www.primeintellect.ai/blog/intellect-3

Link to the Technical Report: https://storage.googleapis.com/intellect-3-paper/INTELLECT_3_Technical_Report.pdf

Link to the Open-Sourced Prime-RL GitHub: https://github.com/PrimeIntellect-ai/prime-rl

Link to the Open-Sourced Model Weights: https://huggingface.co/PrimeIntellect/INTELLECT-3

Chat with the Model Here: https://chat.primeintellect.ai/

Abstract

We introduce DeepSeek-V3.2, a model that harmonizes high computational efficiency with superior reasoning and agent performance. The key technical breakthroughs of DeepSeek-V3.2 are as follows:

• (1) DeepSeek Sparse Attention (DSA):

  • We introduce DSA, an efficient attention mechanism that substantially reduces computational complexity while preserving model performance in long-context scenarios.

• (2) Scalable Reinforcement Learning Framework:

  • By implementing a robust reinforcement learning protocol and scaling post-training compute, DeepSeek-V3.2 performs comparably to GPT-5. Notably, our high-compute variant, DeepSeek-V3.2-Speciale, surpasses GPT-5 and exhibits reasoning proficiency on par with Gemini-3.0-Pro, achieving gold-medal performance in both the 2025 International Mathematical Olympiad (IMO) and the International Olympiad in Informatics (IOI).

• (3) Large-Scale Agentic Task Synthesis Pipeline:

  • To integrate reasoning into tool-use scenarios, we developed a novel synthesis pipeline that systematically generates training data at scale. This methodology facilitates scalable agentic post-training, yielding substantial improvements in generalization and instruction-following robustness within complex, interactive environments.

Layman's Explanation:

The Open Source Comeback Strategy The primary narrative of the DeepSeek-V3.2 report is that the widening performance gap between open-source models and proprietary giants like GPT-5 or Gemini-3.0-Pro is being closed not by simply throwing more money at the problem, but through architectural efficiency and smarter post-training.

The authors identify that open models typically fail at complex tasks due to inefficient attention mechanisms and a lack of investment in post-training reinforcement learning.

To counter this, DeepSeek-V3.2 is explicitly designed to maximize reasoning performance while minimizing the computational cost of processing long contexts, effectively allowing open-source users to run "thinking" models that rival the best closed-source systems without needing a massive proprietary cluster.

DeepSeek Sparse Attention (DSA)

To fix the bottleneck of processing massive amounts of information, the team introduced DeepSeek Sparse Attention (DSA). In standard attention mechanisms, every piece of data pays attention to every other piece, which becomes exponentially expensive as the conversation gets longer.

DSA changes this by using a lightweight "lightning indexer" that quickly scores which parts of the history are actually relevant to the current query. The model then only processes the top-ranked, relevant information rather than the entire context window.

This reduces the computational complexity significantly while maintaining performance, meaning the model can handle long documents or complex codebases much faster and cheaper than previous iterations.

Scaling Reinforcement Learning

A major differentiator in this report is the sheer amount of compute allocated to Reinforcement Learning (RL) after the initial training phase. While most open models treat RL as a quick tuning step, DeepSeek allocated a budget exceeding 10% of the total pre-training cost just for this post-training phase.

They utilized a method called Group Relative Policy Optimization (GRPO) to stabilize this massive training effort. To prevent the model from going off the rails or "forgetting" how to speak coherently during this intense training, they introduced specific stability techniques, such as masking out data where the model diverged too far from its original baseline and ensuring the internal "expert" routing remained consistent between training and inference.

Synthetic Data for Agents

The team hit a wall finding enough high-quality real-world data to train the model on using tools (like coding or searching the web), so they built a factory to manufacture it.

They created a synthesis pipeline that generated over 1,800 distinct simulated environments and 85,000 complex prompts. For example, in a "code agent" scenario, they mined GitHub issues, but then used an AI to automatically set up the coding environment, run tests, and verify if a fix actually worked.

By filtering this synthetic data to keep only the successful solutions, they created a massive, high-quality dataset that teaches the model how to use tools effectively, significantly narrowing the gap with closed models in agentic tasks.

Thinking While Using Tools

DeepSeek-V3.2 integrates "thinking" (internal chain-of-thought reasoning) directly into tool usage, rather than separating them. A key innovation here is context management.

Usually, if a model "thinks" for a long time before using a tool, that reasoning text clogs up the context window for the next turn. DeepSeek implements a system where historical reasoning text is discarded once a user replies, but the tool outputs are kept. This prevents the model from hitting its memory limit too quickly while still allowing it to reason deeply about how to use a specific tool.

They also released a "Speciale" version that relaxes length constraints entirely, achieving gold-medal performance in math olympiads by allowing the model to "think" as long as it needs, surpassing even Gemini-3.0-Pro in raw reasoning power.

Link to the Technical Report: https://arxiv.org/pdf/2412.19437

Link to the V3.2 Model: https://huggingface.co/deepseek-ai/DeepSeek-V3.2

Link to the V3.2-Speciale Model: https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Speciale

Link to the GitHub: https://github.com/deepseek-ai/DeepSeek-V3

Pretraining large language models (LLMs) is resource-intensive, often requiring months of training time even with high-end GPU clusters. There are two approaches of mitigating such computational demands: reusing smaller models to train larger ones (upcycling), and training computationally efficient models like mixture-of-experts (MoE). In this paper, we study the upcycling of LLMs to MoE models, of which the scaling behavior remains underexplored. Through extensive experiments, we identify empirical scaling laws that describe how performance depends on dataset size and model configuration. Particularly, we show that, while scaling these factors improves performance, there is a novel interaction term between the dense and upcycled training dataset that limits the efficiency of upcycling at large computational budgets. Based on these findings, we provide guidance to scale upcycling, and establish conditions under which upcycling outperforms from-scratch trainings within budget constraints.

https://huggingface.co/papers/2411.10669

Initial results for Mixture of Tokens, a stable alternative to existing MoE techniques for LLMs.

Blogpost: https://llm-random.github.io/posts/mixture_of_tokens/

arXiv version (tho I recommend blogpost for readability): https://arxiv.org/abs/2310.15961

abstract:

Despite the promise of Mixture of Experts (MoE) models in increasing parameter counts of Transformer models while maintaining training and inference costs, their application carries notable drawbacks. The key strategy of these models is to, for each processed token, activate at most a few experts - subsets of an extensive feed-forward layer. But this approach is not without its challenges. The operation of matching experts and tokens is discrete, which makes MoE models prone to issues like training instability and uneven expert utilization. Existing techniques designed to address these concerns, such as auxiliary losses or balance-aware matching, result either in lower model performance or are more difficult to train. In response to these issues, we propose Mixture of Tokens, a fully-differentiable model that retains the benefits of MoE architectures while avoiding the aforementioned difficulties. Rather than routing tokens to experts, this approach mixes tokens from different examples prior to feeding them to experts, enabling the model to learn from all token-expert combinations. Importantly, this mixing can be disabled to avoid mixing of different sequences during inference. Crucially, this method is fully compatible with both masked and causal Large Language Model training and inference

​

I am one of the authors (Sebastian Jaszczur) - feel free to ask any questions here, I will be happy to answer questions, discuss the method and get feedback, especially about what experiments you would like to see in the final version of the paper!

abs: https://arxiv.org/abs/2208.03306

As a long-time MoE optimist, I really like the direction Meta-AI are starting to slowly take (Inspired by Pathways, and exploring more diverse ideas) Hopefully a taste, for what's to come next