The download is about 1.4 terabytes. In July 2026, China's Moonshot AI published the full open weights for Kimi K3 — 2.8 trillion total parameters in a sparse mixture-of-experts design, 896 experts with roughly 16 active per token — the largest open-weight model release in history. Anyone can take it. Almost no one can run it well. That tension is where AI infrastructure now lives: the contest has moved on from training the biggest model to a harder, less glamorous question. Who can actually run one?

For years, the frontier lived behind somebody else's API: your data went off to a closed model on a distant server, and you accepted whatever came back. In 2026 that arrangement cracked. Organizations can now pull down genuinely frontier-class models, inspect the weights and run them on hardware they own. Taking the weights costs nothing; running them is another matter.

On paper, Kimi K3's scores travel well. It topped the Arena.ai Frontend Code Arena with a 1,679 Elo rating and ranked fourth on the Artificial Analysis Intelligence Index at 57.1. The model ships with a 1-million-token context window, though it was initially limited to 131K tokens for local inference, and the download runs about 1.4 terabytes using MXFP4 quantization, 594 GB compressed. Self-hosting demands a minimum of 1.4 TB of fast GPU memory, and 64 or more accelerators are recommended. Its hallucination rate on the AA-Omniscience benchmark hit 51%, and generation ran at 32.6 tokens per second against an industry median of 70.9. The weights are open. Running them well is still an elite engineering exercise. Open describes the download, not the difficulty.

In a sparse MoE model, the full set of experts must sit in memory — all 896 of Kimi K3's — while each token is routed to only a small subset, roughly 16 in this case. Memory holds the whole library; computation reads a few shelves at a time. That is how 2.8 trillion parameters can be served at all — and why the memory figures are so punishing while the per-token arithmetic is comparatively merciful. A 51% hallucination rate and 32.6 tokens per second are not footnotes to the release; for an organization planning a deployment, they are the release. Downloading the weights confers possession, not performance. The gap is engineering: serving stack, quantization, hardware.

Precision is the price of admission

Quantization is the tool closing that gap. It reduces the numerical precision of a model's weights, so the model needs less memory and less compute — and modern post-training quantization has become remarkably good at keeping the capability while cutting the cost. The cleanest example available is NVIDIA's NVFP4 release of Alibaba's Qwen3.6-35B-A3B, prepared for fast inference with vLLM. Precision fell from 16 bits to 4 bits, across the weights and activations of the linear operators inside the MoE transformer blocks. Disk size and GPU memory shrank by about 3.06x. And on MMLU, GPQA and SciCode, the quantized build landed close to the full-precision BF16 baselines.

No single format won the argument. AWQ and GPTQ stay popular for NVIDIA GPUs. GGUF — the Q4_K_M preset especially — dominates CPU and consumer-GPU deployment through llama.cpp. FP8 and NVFP4 aim at the data center. The guidance settling out of the ecosystem in 2026 is pragmatic: llama.cpp when portability matters most, quantization compatibility checked carefully, stability valued over novelty. Boring advice. Also correct. The wrong format choice shows up fast — in speed, in memory, or in scores that quietly sag. Four-bit quantization only counts as a win when the benchmarks hold, and here precision was cut deep into the network's linear operators, memory fell roughly threefold, and MMLU, GPQA and SciCode barely moved. This is also why open-source models closing on proprietary systems in code-generation benchmarks matters more as an operations story than a leaderboard story — a score only counts once the model fits on hardware someone actually owns.

Serving is a discipline of its own

Downloading a model is half the job. Serving it to many users at once is the other half — and for that half, vLLM has become the enterprise standard. Built at UC Berkeley and released as open source under Apache 2.0, vLLM introduced PagedAttention, which divides the KV cache into fixed-size memory blocks. External fragmentation disappears; internal fragmentation falls to under 4%. Far more concurrent users get served per GPU. It now supports more than 400 model architectures across NVIDIA, AMD (ROCm), Intel Gaudi and Google TPU hardware, and its OpenAI-compatible API server lets existing applications point at a self-hosted endpoint with minimal changes. That is the difference between a model one person can experiment with and one an organization can put in front of thousands of users — the load that arrives when multi-agent systems move from pilots into production.

The OpenAI-compatible API quietly decides how expensive switching is. An application written against a commercial endpoint can be pointed at a self-hosted vLLM server with minimal changes — the requests look the same, the responses look the same, only the address changes. That converts self-hosting from a rewrite project into a configuration project. It is also why the serving layer, not any individual model release, is becoming the durable asset: models churn every few months, the interface stays put.

Alternatives carve out their niches. SGLang's RadixAttention is optimized for prefix reuse, cutting time-to-first-token hard when queries share system prompts or conversation histories — the shape of agentic coding workflows and long-context RAG pipelines. Prefix reuse sounds minor until you price it: at production volume, recomputing a shared beginning every time is pure waste. For edge and laptop deployment, llama.cpp runs quantized GGUF models on everything from Apple Silicon down to plain CPUs. Weights alone don't make a product, either. Ollama reduced local deployment to a single command. LangChain and LlamaIndex supply the orchestration pieces — prompt templating, retrieval-augmented generation, tool calling — that turn a weights file into a working assistant. LiteLLM offers a unified API gateway, letting applications swap between self-hosted and commercial models without rewrites. Models get superseded every few months; the interfaces persist, and switching costs stay low. Enterprises standardizing on open models in 2026 are, in truth, standardizing on the interfaces rather than on any single model.

The legal geography of a prompt

The most powerful driver of adoption may not be technical at all. Use a closed API, and every prompt travels to servers under someone else's legal jurisdiction. For governments, banks, hospitals and defense-adjacent industries, that exposure is becoming unacceptable. Self-host the weights and inference traffic never reaches Moonshot's Beijing-based servers — which matters, because the API route sends all prompts to servers subject to China's National Intelligence Law, Cybersecurity Law and Data Security Law. Downloading the weights removes that exposure at the inference layer. For the institutions concerned, it can decide whether a model gets used at all.

The same logic applies symmetrically — and everyone in the market knows it. European banks, Gulf sovereign funds and Asian governments are all investing in domestic AI infrastructure to keep their data home. Open weights are the enabler: data sovereignty is impossible with a model you can only reach through somebody else's API, a constraint that sits awkwardly beside the governance frameworks now hardening into law around AI. Some of the new capacity is even being built outside the hyperscalers altogether, as inference spreads onto open, decentralized networks already being compared seriously with the cloud giants.

The release came as the White House was accusing Moonshot of distilling Anthropic's models, and as Beijing weighed export controls on AI model weights. A White House accusation and export controls weighed in Beijing describe a release window politics can close. If open weights at frontier scale become something governments ration, then the organizations that downloaded, evaluated and operationalized them early hold an asset latecomers may not be able to acquire at any price. Sovereignty, in that light, is not only about where inference runs today. It is about whether the capability remains available to run at all.

Trust, but benchmark

How do engineers know a quantized model is still good? The community has converged on a ritual. Capability benchmarks — MMLU for broad knowledge, GPQA for graduate-level science questions, HumanEval for code generation — run against both the full-precision original and the quantized build. A good 4-bit quantization keeps its scores within a few percentage points of the original. Arena-style human preference rankings — LMArena chief among them — add a complementary signal: real users blind-testing which model's answers they prefer. And for deployment decisions, the metrics that matter are operational — tokens per second, time-to-first-token, and VRAM footprint on the target hardware. NVIDIA's reported NVFP4 results on Qwen3.6 — close to BF16 baselines on MMLU and GPQA while cutting memory about 3x — are exactly the kind of evidence that moves a model from interesting to deployable.

The ritual has a logic worth stating. Running the same suites against both versions turns quantization from an act of faith into a measurement: the question stops being "is 4-bit safe?" and becomes "how many points did this model, on these tests, actually give up?" A few percentage points is the accepted toll — small enough to buy back in memory and speed. Human preference rankings catch a different failure, the answer that scores fine and reads wrong — which is why LMArena-style blind testing sits alongside the numeric suites. The honest caveat is that benchmarks measure what they measure. A model can hold its MMLU score after quantization and still degrade on long-context coherence or tool-calling reliability. Serious teams evaluate on their own workloads rather than public leaderboards. The failure mode is quiet: nothing crashes — the answers just get slightly worse, in ways a leaderboard won't show.

The playbook, meanwhile, has hardened into consensus. vLLM is the default substrate for production serving; llama.cpp with GGUF covers edge and offline deployment; four-bit variants serve where cost rules; and for regulated industries, self-hosting open weights is increasingly the only architecture that satisfies capability and compliance at once. The frontier is no longer defined by who trains the largest model, but by who can run the largest models, where needed, under their own control. The argument has moved from access — who may use the strongest models — to operations: memory budgets, serving stacks, evaluation rituals, and the legal geography of inference.

And the sovereignty argument is practical rather than merely political, because open-source AI models have closed on proprietary systems in code-generation benchmarks.