Kimi K3 Explained: Inside the Biggest Open AI Model Ever — and the Piece It Dared to Delete

China's Moonshot AI just released Kimi K3, and the headline number is easy: 2.8 trillion parameters, weights free to download, which makes it — in the words of researcher Sebastian Raschka, whose architecture analysis is currently the most-discussed AI post on Hacker News — "by far the biggest open-weight model right now." A hundred outlets have covered the launch. Almost nobody has explained what's actually different inside it. That's what this piece is for — no math required.

2.8 trillion parameters — but it only wakes up 104 billion at a time

Kimi K3 is a mixture-of-experts model, and the easiest way to picture that is a hospital with 896 specialists on staff. When a patient (a token of text) walks in, they don't see every doctor — they're routed to the 16 most relevant specialists, plus 2 general physicians who see everyone. That's why a 2.8-trillion-parameter model only activates about 104 billion parameters per token, according to the official model card. You get the knowledge of the whole hospital while only paying the salaries of the doctors in the room. If mixture-of-experts is new to you, our plain-English guide to how LLMs actually work covers the foundations.

The piece it deleted: the model has no sense of word position

Here's the genuinely surprising part. Almost every LLM you've used relies on positional embeddings — effectively a GPS coordinate stitched onto every word so the model knows "dog bites man" differs from "man bites dog." The industry standard, RoPE, is baked so deep into modern models that it's practically assumed.

Kimi K3 threw it out. The model uses NoPE — no positional embeddings at all — throughout the entire network, which Raschka notes makes it the first frontier-scale model to do so. The model isn't told where any word sits; it has to infer order from how attention flows between words. Counterintuitively, this appears to help at extreme lengths: hard-coded position signals degrade as sequences stretch, and K3's context window runs to 1,048,576 tokens — roughly a dozen novels in one prompt.

The rest of the engine room, briefly

• Two kinds of attention, layered: 69 of the model's 93 layers use Kimi Delta Attention (a fast, memory-light mechanism inherited from Kimi Linear), while 24 use gated multi-head latent attention — a heavier, more precise mechanism. Cheap layers do the bulk work; expensive layers do the thinking.
• Attention residuals: layers get to reuse and reweight earlier layers' attention patterns, at a cost Raschka pegs at roughly 4% extra training compute and 2% at inference.
• LatentMoE: the expert layers themselves are compressed — a design K3 borrows from Nvidia's Nemotron 3 Ultra — shrinking the giant matrices that dominate a model this size.
• Born quantized: K3 was trained quantization-aware with MXFP4 weights, meaning the low-precision, cheap-to-serve version is the native one — not a lossy afterthought.
• Fully multimodal: text, images, and video flow through the same model, via a compact 401-million-parameter vision encoder.

Moonshot's self-reported benchmarks are frontier-tier: 93.5 on GPQA Diamond, 91.2 on BrowseComp, 88.3 on Terminal-Bench 2.1. Treat vendor numbers with the usual caution — but the download stats are independent, and the model passed 99,000 Hugging Face downloads almost immediately.

Why the design matters more than the size

Every choice above points the same direction: maximum capability per unit of compute. That's partly philosophy and partly necessity — Chinese labs operate under chip export controls, and efficiency is how you compete at the frontier with constrained hardware. It's the same force we covered in this week's AI market rotation, where cheaper architectures are rewriting Wall Street's assumptions about how many chips the future needs — and it sharpens the open-versus-closed contest we mapped in our analysis of the open-weight race.

One honest caveat: "open" here means open weights under Moonshot's own Kimi K3 license, which carries usage conditions — download-and-run freedom, not a fully permissive MIT-style grant. And at 2.8 trillion parameters, "you can run it" means data-center clusters, not laptops. The ideas, though — NoPE at scale, hybrid attention, born-quantized training — are the part every lab will be studying, and those are free.

FAQ

What is Kimi K3?

A 2.8-trillion-parameter open-weight AI model from China's Moonshot AI — currently the largest openly downloadable model — with a one-million-token context window and native text, image, and video support. It activates about 104 billion parameters per token via a mixture-of-experts design.

What are positional embeddings, and why does it matter that K3 has none?

Positional embeddings tell a language model where each word sits in a sentence. Kimi K3 uses NoPE — no positional embeddings anywhere — making it the first frontier-scale model to rely purely on attention patterns for word order, which appears to help at very long context lengths.

Is Kimi K3 open source?

The weights are freely downloadable on Hugging Face under Moonshot's own Kimi K3 license, which includes usage conditions. That makes it open-weight rather than open source in the strict licensing sense.

Can I run Kimi K3 myself?

Not on consumer hardware — 2.8 trillion parameters requires multi-GPU server clusters even with its native MXFP4 quantization. Most people will use it through hosted providers; the architecture's efficiency tricks mainly lower costs for them.