Running a capable agent on your own machine is mostly a memory problem. Nemotron 3.5 Bolt is NVIDIA's attempt to make a large model fit a 24 GB budget: a 74-billion-parameter model with 9 billion active parameters, published on 2026-10-15 with open weights.[1][3] What makes it worth a close read is that NVIDIA's technical report documents the compression recipe, the memory budget line by line, and the accuracy lost along the way.
This Model Watch reads that report, the research page, the Hugging Face model card and NVIDIA's announcement posts, all opened on 2026-10-20. We did not run the model, so nothing here is our own testing, and every benchmark figure is NVIDIA's.
What was released
| Spec | Detail |
|---|---|
| Model | Nemotron 3.5 Bolt 74B-A9B 24GB, published 2026-10-15[3] |
| Architecture | Hybrid Mamba-Transformer mixture of experts, 74B total and 9B active parameters[3] |
| Parent | Nemotron 3.5 Super VL, 120B total and 12B active[3] |
| Inputs and outputs | Text, images and video (MP4, up to 2 minutes) in; text out[4] |
| Context length | Up to 256K tokens per the model card; some deployment recipes use less to fit memory[4] |
| Licence | OpenMDW License Agreement, version 1.1, per the model card[4] |
| Target hardware | NVIDIA RTX Spark and DGX Spark; benchmarks run on an RTX Spark laptop with 128 GB[1] |
| Companion files | A BF16 checkpoint, a GGUF build, and a "LiLiCorr" speculative-decoding drafter[1] |
NVIDIA's announcement says a smaller sibling, Bolt E4B, was due on 2026-10-23 and that RTX Spark laptops were arriving 2026-10-16. Both are stated plans; we did not confirm either.[2]
How a 120B model becomes a 24 GB model
The report describes two compression stages followed by a speed fix.
1. Pruning experts. The parent has 512 routed experts per mixture-of-experts layer and selects 22 per token. A learned router, using NVIDIA's Star Elastic framework, chooses how many experts (from 240 to 496) and how wide each layer's expert is (1,536 to 2,560, against 2,688 in the parent). Distillation from the parent trains the router and the student together, and the number of experts selected per token falls from 22 to 9.[1] The hidden width and the Mamba, attention and mixture-of-experts layer counts stay the same as the parent's, which keeps the vision head compatible.[1]
2. Quantising to about 2 bits. The expert weights are stored in a llama.cpp format called IQ2_XXS, at 2.0625 bits per weight. Mamba projections use NVFP4, and attention, embeddings and the output head stay at 8.5 bits (Q8_0). NVIDIA says this takes a 148 GB BF16 checkpoint to 22.1 GB, an 85.1% reduction, and that the quantisation is "lossy by design."[1] It recovers accuracy with quantisation-aware distillation: 26 billion tokens of training against a BF16 teacher, then a short second stage against a newer teacher.[1]
3. Speculative decoding. A separate 1.13-billion-parameter drafter proposes several tokens at once, which helps because at low concurrency the speed limit is how fast weights can be read from memory. NVIDIA reports 84 tokens per second on its coding speed benchmark with the drafter.[1]
NVIDIA's reasoning for the order is a design claim, not a result we can check: the report says it is "more Pareto-optimal to quantize a larger network to lower bits, than to have a smaller model at a higher bit-width."[1] Its evidence is its own ablations and prior literature it cites.
The memory budget, line by line
The report's Table 2 is unusually specific. At batch size 1 and 131,072 tokens of context:[1]
| Component | Size (GB) |
|---|---|
| Routed-expert weights (IQ2_XXS) | 17.05 |
| Mamba projections (NVFP4) | 2.47 |
| Embedding, output head, attention, latent MoE and other weights | 2.49 |
| Weight subtotal | 22.01 |
| KV cache, SSM cache and activation buffer | 0.87 |
| Speculative drafter | 0.96 |
| Total, text only | 23.83 |
| Vision encoder and projector (listed separately) | 1.68 |
Two readings follow. First, the 24 GB figure leaves 0.17 GB spare at 128K context, so a longer context or a second application competes for headroom. Second, the report says the vision encoder and projector "are listed separately and excluded from this target, which covers text-only serving."[1] Adding the 1.68 GB to the 23.83 GB total gives about 25.5 GB. That sum is our arithmetic, not a number NVIDIA states. A buyer sizing hardware for image or video input should plan for more than 24 GB.
What the compression costs
NVIDIA's Table 7 reports each stage's score, so the cost of each step is visible. All values are NVIDIA's, and the report notes that accuracy for the other models was measured in BF16 while Bolt 24GB is 2-bit.[1]
| Benchmark | Super VL 120B (BF16) | Bolt 74B (BF16) | Bolt 24GB (2-bit) | Qwen 3.6 35B-A3B | Gemma 4 26B-A4B |
|---|---|---|---|---|---|
| Terminal-Bench 2.1 | 61.94 | 53.93 | 50.98 | 44.38 | 37.22 |
| BrowseComp | 71.50 | 70.00 | 66.75 | 48.74 | 26.30 |
| GDPVal-AA-V2 | 34.48 | 31.13 | 28.47 | 25.75 | 15.33 |
| GPQA Diamond | 90.78 | 88.83 | 86.43 | 83.40 | 79.61 |
| HLE (no tools) | 31.84 | 28.73 | 25.30 | 19.56 | 17.42 |
| AA-Omniscience accuracy | 25.98 | 21.80 | 16.90 | 19.47 | 22.17 |
| IFBench (instruction, loose) | 73.04 | 68.71 | 66.17 | 63.71 | 77.25 |
The pattern is consistent. Pruning from 120B to 74B costs between 1.5 and 8 points depending on the benchmark; quantising to 2 bits costs a further 2.4 to 4.9 points. On Terminal-Bench 2.1 the full path from parent to 24 GB model loses about 11 points. Against the two smaller models NVIDIA picked, Bolt 24GB leads on the agentic rows shown, by a wide margin on BrowseComp.
It does not lead everywhere. On AA-Omniscience accuracy, a knowledge-recall measure, both Qwen 3.6 and Gemma 4 score higher than Bolt 24GB, and on instruction following Gemma 4 scores higher by about 11 points.[1] The ranking depends on what the agent needs to do.
Speed: the trade for the larger model
NVIDIA's performance table compares Bolt with a drafter against smaller models using other speed-up methods, all on the same RTX Spark laptop with llama.cpp:[1]
| Benchmark | Bolt 74B 24GB | Qwen 3.6 35B-A3B | Gemma 4 26B-A4B |
|---|---|---|---|
| Coding speed (tokens per second, higher is better) | 83.99 | 96.63 | 100.56 |
| Agent replay duration (seconds, lower is better) | 600.80 | 453.97 | 480.68 |
The competitors in this table run at 4-bit (UD-Q4_K_M), so they also carry more bits per weight than Bolt's experts. Bolt is slower on both measures. NVIDIA's text frames 84 tokens per second as "comparable to models with much lower active parameter counts" and sufficient for long-horizon agent work.[1] Whether 120 to 147 seconds of extra replay time per run matters depends on the workload. If accuracy on browsing and terminal tasks decides the outcome, the slower model may still win; if latency decides it, it will not.
Who should care, and what is still uncertain
- Local agent builders with a 24 GB class of memory now have a published recipe and measured numbers to test against their own tasks.
- Anyone comparing local models should treat NVIDIA's chosen rivals as a vendor's selection. NVIDIA's research page cites leaderboard placements (first on PinchBench, second on WebArena and BrowseComp among models of similar footprint), but these come from NVIDIA's pages.[3]
- Licence reviewers should read the licence itself. The model card names OpenMDW-1.1 and says the model is ready for commercial or non-commercial use.[4] NVIDIA's blog posts describe the release as open without naming a licence, and the licence page we opened lists the version but not its terms, so we cannot summarise the grant or conditions.[2][5][6]
- Reproducibility is partly open. NVIDIA's Super VL post says the family ships with open weights, datasets and training recipes, but that claim is about the parent, and we did not check each dataset.[5] The report notes one competitor score it could not reproduce (Qwen's MCP-Atlas, reported as 62.8 by its maker and 51.2 in NVIDIA's evaluation), a reminder that these tables come from NVIDIA's own harness.[1]
For how to weigh tables like these, see How to Read an AI Benchmark Claim Without Being Misled. For the licence question in general, see Open Weights vs Open Source AI. Meta's release the same day, the much larger model covered in Meta Muse Spark Open, sits at the opposite end of the self-hosting range, and the serving techniques mentioned here are explained in Inference Serving Explained. More on this theme is under open source.
Method and limits
We read the sources listed below on 2026-10-20, extracted the figures into the tables above, and checked each against the report text. We did not run Bolt, did not read the full licence text, and did not open the Hugging Face pages for the BF16 or GGUF files. The 2.0625-bit and 22.1 GB figures appear in the report's text; its memory table sums the weights to 22.01 GB, a small internal difference we have left as published. We will correct this page if NVIDIA revises the report.
