LongCat 2.0: A 1.6T Model Trained Without Nvidia

A Frontier Model Built on Chinese Hardware

LongCat 2.0 comes from an AI lab under Meituan, China's equivalent of DoorDash. The model is 1.6 trillion parameters with 48 billion active, released under an MIT license, and it was trained on over 50,000 AS6 super pods — custom Chinese AI accelerators, not Nvidia GPUs. The lab's blog reportedly compared the immaturity of the AS6 software stack to Nvidia's ecosystem, implying they've built a working alternative from scratch.

The training run reportedly had no rollbacks or unrecoverable loss spikes, which the creator called a strong signal that the Chinese hardware stack is functioning well at scale.

• Note the scale: LongCat 2.0 is roughly tied with DeepSeek V4 Pro in size, making it the second-largest openweight model behind Kimi K3.
• Watch for follow-up papers: the creator expects more systems-level disclosures from LongCat given how much infrastructure they built independently.

Sparse Attention Built for Hardware, Not Just Theory

LongCat's attention mechanism, called LongCat Sparse Attention, is a variant of DeepSeek Sparse Attention — the current benchmark for efficient long-context attention. The core problem LongCat addresses is that sparse attention's token-selection step (the indexer) can itself become a bottleneck if it isn't hardware-friendly.

They made three specific improvements: streaming-aware indexing for contiguous memory reads, cross-layer indexing so layers share a search result instead of recomputing it, and hierarchical indexing that searches coarse blocks before zooming into individual tokens.

• Cross-layer indexing: LongCat shares one index across two layers (more conservative than GLM 5.2's four-layer sharing), using cross-layer distillation during training to keep shared indexes reliable for every layer that reuses them.
• MTP extension: multi-token prediction steps also share a single indexing path, avoiding the cost of re-searching a million-token context at every draft step.
• Hierarchical search: token retrieval works block-first, then token-level, similar to how a developer skims files before narrowing to a specific line of code.

Engram Embeddings: A New Kind of Memory

The most distinctive design choice is engram embedding, where roughly 135 billion parameters are dedicated to learning representations of short token sequences — up to five tokens long — rather than just single tokens. This gives the model an early, built-in sense of common phrases and code patterns, similar to DeepSeek's engram concept but implemented differently: as a separate embedding-side memory table rather than a new transformer block.

LongCat's team reasoned that their mixture-of-experts setup had already passed the point of useful returns from adding more experts, since the model is already about 97% sparse. Redirecting that parameter budget into engram embeddings gave a better return while keeping engram parameters under 10% of the total model.

• Where it helps most: heavily pattern-based content like code, where recognizing common multi-token structures early saves compute in later layers.
• Training stability: LongCat also used the Muon optimizer, which reportedly helped prevent unrecoverable loss spikes during training.
• Post-training: the model uses multi-token on-policy distillation (MOPD) to strengthen reinforcement learning capability.

The Payoff

For anyone tracking the AI hardware race, LongCat 2.0 is evidence that a frontier-scale model can now be trained and served entirely without Nvidia GPUs, using custom Chinese accelerators, without sacrificing much competitiveness. Readers building infrastructure or evaluating openweight models get a concrete architectural playbook — hardware-aware sparse attention and engram embeddings — worth watching as other labs adopt similar ideas.