The decision you don’t realize you’re making
You sit down to wire Gemma 4 into a local agent loop — a Claude-Code-style tool-using harness, a long-context code reviewer, an offline research assistant. Google has handed you four architectures from the same release. The contest framing nudges you toward an obvious read:
-
E2B and E4B (effective-parameter) models for phones, browsers, and a Pi 5.
-
31B dense as the on-prem workhorse.
-
26B MoE as the efficient one — the mixture-of-experts variant that activates a fraction of its parameters per token and claims better quality per FLOP.
The cheap reflex, in 2026, is to reach for the MoE. It’s the architecture every frontier lab is paying for. Mixtral, DeepSeek V3, Qwen3 — the production direction is one-way traffic. If the open release at your scale gives you an MoE option, you pick it. That’s the move.
That move is wrong for the workload most of you will actually build.
For interactive local agents — multi-turn tool use, code editing, long-context reasoning chains where each step is read by a human or fed into the next step — the 31B dense is the right default, and the 26B MoE is the wrong one. The reason is not size, quality, or open-weights ergonomics. It’s that agents fail on the tail, not the mean. Routing variance lives in the tail. That’s the entire argument; the rest of this post is the math, the mechanism, and a benchmark you can run this weekend on whatever hardware you have.
The tail is the workload
A common mistake when reasoning about local-model deployments is to optimize the average. Tokens per second, mean time-to-first-token, throughput at batch 1 — these numbers are what every llama.cpp benchmark in your feed reports.
For a chatbot that gets one user message and emits one reply, the mean is fine. For an agent, it isn’t.
Consider a 30-step tool-use loop — a typical ReAct-style agent doing local code search, file edits, test runs, and a final summary. Each step is a generation call. Suppose each call has a 3% probability of exceeding a soft timeout (say, the human watching the screen gives up and Ctrl-Cs). The probability that at least one step in the loop blows the timeout is:
1 – (1 – 0.03)^{30} approx 0.60
A 3% per-step tail becomes a 60% chance of a visibly broken loop. Push the per-step tail to 1% and you’re still at 26% loop-level failure across 30 steps. The mean latency could be excellent; the user experience is broken. Agents are tail-bound systems, the same way low-latency trading, search ranking, and request-level web serving are. Jeff Dean’s “tail at scale” is, in fact, about exactly this — when you fan out many requests, the slowest one dominates. An agent fans out across time, not machines, but the math is identical.
This is the lens the architecture comparison has to be read through. Not “which model gets more right answers per FLOP,” but “which model gets right answers reliably under the user’s patience budget.”
How MoE routing makes a tail
Every MoE forward pass routes each token through a small subset of the available experts. In Mixtral 8×7B, that’s top-2 of 8 experts per token per MoE layer. In DeepSeek V3, it’s top-K out of hundreds, with shared experts plus routed experts. The architectural details vary; the routing mechanism does not. A learned gating network emits expert probabilities; the implementation activates the top-K.
Several consequences fall out of this design — and they are not bugs, they’re load-bearing features of why MoE works at all. They just bite a local agent harder than a batched cloud workload.
Active parameters per token are a distribution, not a constant. Google’s naming for the MoE — gemma-4-26B-A4B, “26 billion total, 4 billion active” — describes an expectation across tokens, not a per-token guarantee. For a prompt that lands hard on a small number of experts — say, code in an unusual language, or a domain-specific token distribution — the same experts get hit on every layer, creating utilization hotspots. The total work across the layer is the same; the load balance is not. On a multi-GPU deployment this is fine, because the all-reduce across experts amortizes it. On a single-machine local deployment it shows up as per-token latency variance.
Expert utilization is non-uniform. A closer look at Mixture-of-Experts in LLMs (Lo et al., 2024) finds that routers tend to select experts whose outputs have larger norms, that expert diversity changes layer by layer, and that the final layer routes differently from intermediate layers. The implication for an agent’s tail isn’t subtle: which experts your tokens actually hit is a function of what the tokens are, and adjacent steps in an agent loop — where one step’s output becomes the next step’s input — can land in substantially different parts of the expert configuration space, with substantially different latencies. For a batch-mode service the variance washes out across many users; for a single agent loop, each user is the batch.
Load-balancing losses pull the model toward uniform expert use, but they don’t guarantee it. Training-time auxiliary losses (Shazeer et al., 2017; Fedus et al., Switch Transformer) penalize over- and under-used experts so the model learns a roughly even distribution. Roughly even at the dataset level is not even on your specific prompt. The auxiliary loss is a soft prior; the runtime gating is what determines your tail.
The KV cache is mostly arch-agnostic, but the activations on top of it aren’t. For Gemma 4 specifically we don’t yet have the per-architecture cache numbers I’d want, but the general principle holds: the KV cache footprint at long context will be similar for dense and MoE in the same family (same attention layout, GQA, same hidden size), but the per-token activation cost will diverge. The MoE wins on raw FLOPs per token. It does not win on predictable FLOPs per token.
The dense model has none of this surface. Every token does the same work. Routing variance is zero because there is no routing. The p99 looks like the p50 plus the inherent system noise of your kernels, your scheduler, and your memory hierarchy. That’s the property the agent loop wants.
The MoE memory myth
A second piece of the cheap reflex says: “26B MoE is smaller than 31B dense, so it fits better in my VRAM.“
It doesn’t.
MoE memory residence is all parameters, not just the activated ones. Every expert has to be loaded somewhere — either in GPU memory, or paged in from system RAM, or paged in from disk. The “26B” in Gemma 4 26B MoE is the residence footprint; the “activates a fraction per token” describes compute, not memory. On a single consumer GPU, both models compete for roughly the same memory budget, and the dense model is often the easier fit because:
-
Quantization is cleaner on dense. Dense weight matrices quantize uniformly; MoE quantization has to contend with expert-utilization skew, where rarely-routed experts get less calibration signal and degrade harder under aggressive bit-width reduction. Sub-1-bit MoE compression is feasible — see QMoE for the existence proof on Switch-Transformer-c2048 — but it requires custom kernels. The boring 4-bit path on the dense model is the path that already works in every local inference engine you’ll touch (LLM-FP4 is the well-trodden version).
-
Offloading hurts MoE more. If you spill experts to CPU or disk, you pay the offload cost per token, conditional on routing. A dense model that spills paid the cost predictably; an MoE that spills pays it stochastically, and the stochasticity is — again — the tail.
-
Compilation is cleaner on dense. llama.cpp, MLX, and vLLM all support both, but the dense path has had more attention. Fewer corner cases in expert routing, GQA, and KV layout interactions. If you’ve ever had a custom kernel mis-handle expert dispatch, you know.
The 26B MoE will run on your machine. So will the 31B dense. Which one runs predictably is the question.
When the MoE is correct
I’m not arguing the 26B MoE is a bad model. It’s the right model for several workloads — they just aren’t the local-agent workload most readers building this weekend will actually have.
The MoE is the right pick when:
-
You’re serving many concurrent users. Batching across requests amortizes routing variance. Hot experts become warm; cold experts get exercised. The per-token expectation is what you see.
-
You’re throughput-bound, not latency-bound. Background processing, RAG indexing, bulk classification, synthetic data generation at scale. Anywhere the user isn’t waiting on a single response.
-
You can afford a per-request retry budget. If a slow step is recoverable by re-issuing, the tail flattens out. Most agent loops can’t recover gracefully — the partial state is already entangled with the world (a file was edited, a test was run).
-
You’re deploying on accelerators with native expert-parallel support. Multi-GPU or TPU deployments where the all-reduce overhead is dominated by the FLOP savings. This is exactly where the MoE was designed to win, and it does.
The Gemma 4 26B MoE is, on its merits, an excellent open-weights MoE. Pick it for the workloads it was designed for. Don’t pick it because the marketing material says “more efficient” and you didn’t ask “efficient at what.”
The decision matrix
| Workload |
Pick |
| Single-user local agent loop (tool use, code edit, multi-turn) |
31B dense |
| Local long-context Q&A (one or two large reads, then short outputs) |
31B dense |
| Local RAG service with many concurrent users |
26B MoE |
| Local batched synthetic-data or evaluation pipelines |
26B MoE |
| Background indexing, classification, summarization at scale |
26B MoE |
| Fine-tuning on a narrow domain (small dataset, limited compute) |
31B dense (MoE fine-tuning collapses expert specialization at small data scales) |
| Mech-interp / SAE / circuit work on an open MoE |
26B MoE (it’s the interesting object) |
| Phone, Pi 5, browser deployment |
4B effective (you don’t have a choice) |
| Cloud or multi-GPU server deployment |
26B MoE (this is where it wins) |
Two things to flag explicitly. First, the fine-tuning row: MoE fine-tuning is a research subfield of its own because the router and the experts have to be tuned coherently, and small datasets can collapse expert specialization. If you have 5,000 examples and want to teach Gemma 4 your codebase’s conventions, the dense model is the more forgiving target. Second, the mech-interp row: a 26B MoE with open weights is a genuinely exciting interpretability artifact — see the GemmaScope lineage on Gemma 2 dense — and if your “agent” is actually a circuit-analysis pipeline, the MoE is the more interesting object. That’s a different essay.
The benchmark you should run this weekend
Don’t take my word for any of the above. The whole point of running models locally is that you can measure. Here is the experiment, three ways, calibrated for the hardware most of you actually own.
What to measure: not tokens per second. Measure per-step p50, p95, and p99 latency across at least 200 generation calls, with prompts that approximate your real workload (short turns, mixed lengths, occasional long-context reads). If you can run an actual agent loop end-to-end, even better — record the per-step distribution and synthesize the loop-level success probability.
Apple Silicon (M2 / M3 / M4):
# Build llama.cpp with Metal backend
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp && cmake -B build -DGGML_METAL=ON && cmake --build build -j
# Community-quantized GGUFs (Google ships safetensors; unsloth ships GGUF)
huggingface-cli download unsloth/gemma-4-31B-it-GGUF
gemma-4-31B-it-Q4_K_M.gguf --local-dir .
huggingface-cli download unsloth/gemma-4-26B-A4B-it-GGUF
gemma-4-26B-A4B-it-Q4_K_M.gguf --local-dir .
# Benchmark: 200 generations of 512 tokens, log per-call timing
./build/bin/llama-bench -m gemma-4-31B-it-Q4_K_M.gguf -n 512 -r 200 -o json > dense.json
./build/bin/llama-bench -m gemma-4-26B-A4B-it-Q4_K_M.gguf -n 512 -r 200 -o json > moe.json
The 26B-A4B suffix on the MoE is Google’s naming for “26B total params, ~4B activated per token” — the very expectation-vs-distribution gap discussed above. llama-bench gives per-call timings; pipe through jq to extract the per-call latencies and compute your own quantiles. For a richer view, MLX-LM gives cleaner kernel timing on Apple Silicon and lets you verify the dispatch overhead directly.
Consumer NVIDIA (RTX 4090, 3090, A6000):
# vLLM with the official safetensors, served on the same machine
pip install vllm
python -m vllm.entrypoints.openai.api_server
--model google/gemma-4-31B-it --quantization awq
--port 8000 &
# Drive it with a benchmark harness — 200 requests, log per-request latency
python -m vllm.benchmarks.bench_serving
--backend openai-chat --endpoint /v1/chat/completions
--base-url http://localhost:8000
--model google/gemma-4-31B-it
--num-prompts 200 --dataset-name sharegpt
--result-filename dense.json
vLLM’s benchmark suite reports p50/p95/p99 directly. Repeat with --model google/gemma-4-26B-A4B, compare the p99 columns, and that’s your answer.
A note: vLLM’s MoE kernel quality varies by version. Check the supported models page before assuming both architectures get equal-quality dispatch. This is itself part of the dense-by-default argument — the ecosystem is more battle-tested on dense.
Pi 5 / phone (Gemma 4 E4B):
The MoE-vs-dense decision doesn’t apply at this tier — you’re running the E4B effective-parameter model, not because you chose to, but because it’s what fits. The interesting benchmark here is different: measure token-level latency *variance* on the same prompt set across multiple runs. Edge silicon shares compute with the OS scheduler, and your real tail will be dominated by OS jitter, not architecture. Use time plus a wrapper that logs to a CSV:
# On Pi 5 with llama.cpp built for ARM
for i in $(seq 1 200); do
/usr/bin/time -f "%e" ./build/bin/llama-cli
-m gemma-4-E4B-it-Q4_K_M.gguf
-p "$(jq -r ".prompts[$((i % 50))]" prompts.json)"
-n 256 -no-cnv 2>>latencies.csv
done
The lesson on the edge is the same as on the workstation: optimize for the experience the user actually has, which is the tail.
What to take away beyond this release
Gemma 4 forces a clean architectural choice at the same scale tier — dense versus MoE, 31B versus 26B. The next open release will too. The release after that. The “always pick MoE because it’s more efficient” reflex was correct for cloud training and cloud serving, and it has been pattern-matched into local deployment where it doesn’t belong.
The lens that holds up across releases is not parameter count and not FLOPs per token. It is the latency profile of your workload. If a single user is waiting on a single response chain, you want predictable tokens, not cheap tokens. If many users are sharing a pool of compute and the throughput is what’s billed, you want cheap tokens at the expense of any single user’s tail.
The open-weights ecosystem is mature enough now that you don’t have to pick on vibes. Two llama-bench runs and a quantile calculation will tell you the answer for your hardware and your prompts. Run them before you commit to an architecture.
The dense model is the default. The MoE is the optimization. Treat them in that order.
Submitted to the DEV.to Gemma 4 Write-About challenge, May 2026, with some help from AI. Code samples have been validated against the relevant tool documentation; concrete benchmark numbers are intentionally omitted because the right benchmark is your benchmark, on your hardware, with your prompts.
