AI Primer · Running LLMs on your own hardware · August 2026

Two numbers decide
everything: capacity
and bandwidth.

Capacity decides whether a model runs at all. Bandwidth decides whether you can stand to watch it. Almost every disappointing local-AI purchase is someone who optimised one and forgot the other.

tokens/sec ≈ memory bandwidth (GB/s) ÷ bytes read per token (GB) × ~0.8
fits at all ⟺ weights + KV cache + overhead < memory capacity

In short

This is a short field guide to running large language models on hardware you own. It covers the four things that decide whether that works — memory capacity, memory bandwidth, model architecture and quantization — and then maps today's machines against them.

01Capacity, bandwidth, and why compute barely matters

The mental model

Capacity is how much memory the accelerator can address — VRAM on a GPU, unified memory on a Mac or an APU. It is a hard gate. If the weights plus the KV cache don't fit, the model either won't load or spills to system RAM and slows down by an order of magnitude.

Bandwidth is how fast that memory can be read. To emit one token, the machine reads every active weight once. So decode speed is a division problem, not a compute problem. A 70B model quantised to 4 bits is roughly 40 GB of weights: on 273 GB/s that's a ceiling of ~7 tok/s; on 1,792 GB/s it's ~45 tok/s. Same model, same code, 6× the difference — from memory speed alone.

Compute (TFLOPS, tensor cores) governs prefill — chewing through your prompt — and training. It has almost nothing to do with how fast words appear.

Capacity is the tank; bandwidth is the hose. A 128 GB unified-memory box holds a 70B model but drips it out. A 32 GB gaming card pours — but only models that fit in the tank.

Sizing a model in your head

Weights ≈ parameters × bytes-per-weight — FP16 = 2, FP8/Q8 ≈ 1, Q4 ≈ 0.55 (including quantisation metadata).
Add the KV cache: budget 10–20% of the weight size at ordinary context lengths. At 100K+ context it can rival or exceed the weights, and it grows as you talk.
Add 1–2 GB runtime overhead, and leave headroom on a Mac (the OS and display share the same pool).
Rule of thumb: usable model size ≈ capacity × 0.8.

One big caveat "Bytes read per token" is not the same as "size of the model" for Mixture-of-Experts architectures — which is what most current open frontier models are. That distinction is large enough to change which machine you should buy, so it gets its own section next.

02Dense vs. MoE: the same GB behave differently

Total parameters set capacity · active parameters set speed

A dense model reads every weight to produce every token. Total parameters and active parameters are the same number, so the two constraints move together: a 70B model needs 40 GB of memory and reads 40 GB per token.

A Mixture-of-Experts model splits its feed-forward layers into many experts and a router picks a handful per token. All experts must be resident in memory, but only the selected ones get read. Total and active parameters decouple — often by a factor of 10 or more.

The consequence, in one line Capacity scales with total parameters. Bandwidth cost scales with active parameters. A big MoE is memory-hungry and bandwidth-cheap; a dense 70B is the opposite kind of expensive.

What that does to real numbers

Model shape (4-bit)	Resident	Read per token	273 GB/s	819 GB/s	1,792 GB/s
Dense 70B	~40 GB	~40 GB	~5 tok/s	~16 tok/s	~36 tok/s
MoE 120B, ~5B active	~65 GB	~4 GB	~55 tok/s	fast	fast
MoE 235B, ~22B active	~130 GB	~13 GB	~17 tok/s	~50 tok/s	~110 tok/s

Ceilings, not measurements. Attention weights and any shared expert are read on every token, so expect roughly 50–70% of the MoE figures in practice — still several times the dense result on the same hardware.

How to use this when choosing hardware

A 128 GB / 256 GB/s box is an MoE machine. It will feel genuinely quick on a 120B MoE and close to unusable on a dense 70B. Benchmarking one and concluding things about the other is the most common local-AI mistake.
A 32 GB high-bandwidth card is a dense machine. Small dense models fly; large MoE models don't fit at all, because you must hold every expert you might route to.
Prefill is cheaper on MoE too — fewer FLOPs per token — so time-to-first-token improves alongside decode.
MoE tolerates offloading. Keeping attention and shared layers on the GPU while parking rarely-used experts in system RAM works far better than offloading a dense model, though routing is unpredictable and you'll see occasional stalls. On a dense model, any offload is a cliff.
Quality intuition: an MoE with A active and T total parameters behaves, very roughly, like a dense model somewhere near √(A×T) — clearly better than a dense A, clearly worse than a dense T. Don't read "235B" as competing with a dense 235B.
Multi-GPU note: MoE opens up expert parallelism — whole experts placed on different cards — which needs far less cross-card chatter than tensor parallelism on a dense model.

03Multi-GPU: two strategies, two very different links

Tensor parallel vs. pipeline parallel

Tensor parallelism splits every layer across cards, so all GPUs work on every token at once. It adds capacity and effective bandwidth — this is the only way to make a single conversation genuinely faster. The cost: cards synchronise several times per layer, so it needs a fat, low-latency link and works best with 2, 4, or 8 identical cards.

Pipeline parallelism splits the model by layers: card 1 holds layers 1–20, card 2 holds 21–40. Only small activation tensors cross the link, so it tolerates slow interconnects — even Ethernet. But for a single request only one card is busy at a time, so it buys you capacity, not speed. It pays off with batching (multiple users or parallel agents), where the pipeline stays full.

The link is the constraint

Interconnect	Practical throughput	Tensor parallel?	Pipeline parallel?
NVLink 5 (in-chassis)	~1,800 GB/s	Excellent	Yes
PCIe 5.0 ×16	~64 GB/s each way	Workable, sub-linear	Yes
PCIe 5.0 ×4 (the trap)	~16 GB/s	Bottleneck	Yes
ConnectX-7 200 GbE	~25 GB/s	Marginal	Yes
Thunderbolt 5 / USB4	~5–10 GB/s	No	Yes
10 GbE	~1.2 GB/s	No	Yes, with latency cost

What to look out for Consumer motherboards advertise "two ×16 slots" and then deliver ×8/×8, or ×16/×4 once an NVMe drive is populated. Check the chipset lane map, not the slot shape. Also budget for physical reality: two RTX-class cards need three slots of width, 1,200 W+ of clean power, and airflow. And note that PCIe bandwidth only matters for cross-card traffic — a single card that already holds the whole model barely touches the bus.

04Quantization: the highest-leverage decision you make

Fewer bits per weight = less memory and proportionally faster decode

Quantization stores each weight in fewer bits and expands it on the fly. Halving the bits halves the footprint and halves the bytes read per token — so it roughly doubles decode speed. That is why it is the first lever to pull, and the easiest one to over-pull.

Precision	Size of a 70B	Quality	Verdict
FP16 / BF16	~140 GB	Reference	Only if memory is free. Rarely worth it for inference.
FP8 / Q8	~70 GB	Near-identical	Differences are within benchmark noise. Safe default when it fits.
4-bit (Q4_K_M, AWQ, NVFP4, MXFP4)	~40 GB	Small, acceptable loss	The sweet spot. Where most local inference should live.
3-bit and below (Q3, Q2, IQ2)	~26 GB / ~20 GB	Steep degradation	Avoid. Fails first on code, maths, instruction-following, long context.

The capacity/quality trade-off, stated as a rule

Do trade precision for parameters down to 4 bits: a 70B at Q4 beats a 32B at FP16 in the same ~40 GB, nearly always.
Don't trade below 4 bits to squeeze in a bigger model. A 32B at Q4 beats a 70B at Q2. Below 4 bits you are not compressing the model, you are damaging it.
Small models are more fragile than large ones. An 8B at Q4 loses more, proportionally, than a 120B at Q4.
Quantize the KV cache separately. Q8 KV is close to free and buys a lot of context. Q4 KV starts to show at long context — test before trusting it.
Prefer quantizations the hardware understands natively — NVFP4/MXFP4 on Blackwell, MLX 4-bit on Apple — over generic formats that emulate.

05Tokens: where the time actually goes

Prefill · thinking · output

Every request has two distinct phases with different bottlenecks, and modern reasoning models add a third cost centre in between.

Prefill (input)

Your whole prompt is processed in parallel — compute-bound, thousands of tokens per second even on modest hardware. The metric is time to first token. Long prompts, RAG contexts and big code files land here, and this is where raw TFLOPS earn their keep. Prompt caching makes repeated prefixes nearly free; on agent loops it is the single biggest win available.

Thinking (reasoning tokens)

Reasoning models generate hundreds to thousands of hidden tokens before the first visible word. Those are output tokens paying full decode price. At 5 tok/s, 2,000 thinking tokens is a nearly seven-minute wait before you see anything. This is the phase that decides whether local AI feels usable, and it is the one buyers never budget for.

Output (decode)

One token at a time, each requiring a full pass over the active weights. Purely memory-bandwidth-bound. The metric is tokens per second, and it degrades as context grows because the KV cache grows with it.

Calibrate your expectations <5 tok/s — batch work only, not interactive. 10 tok/s — roughly human reading speed; tolerable for chat. 20–30 tok/s — comfortable. 50+ tok/s — the floor for agentic coding, where the model writes far more than you read. Benchmark your model at your context length; vendor figures are usually small models at short context with batching.

06Know your hardware limits

Six classes of machine, August 2026 street reality

Read this before the table The 2026 DRAM and GDDR7 shortage has rewritten this market. Prices are up 50–100% in a year and configurations have shrunk: Apple withdrew the 512 GB and then the 256 GB Mac Studio options, and the M3 Ultra now tops out at 96 GB, while the RTX PRO 6000 went from ~$8,500 at launch to $16,000. Treat every price below as a weekly snapshot.

Machine	Capacity	Bandwidth	Multi-unit	TP / PP	Price	70B Q4 ceiling
AMD Strix Halo Ryzen AI Max+ 395	128 GB unified	256 GB/s	10 GbE / USB4	no / yes	$2,000–3,100	~5 tok/s
NVIDIA DGX Spark GB10 Grace Blackwell	128 GB unified	273 GB/s	ConnectX-7 200 Gb/s, 2 units	marginal / yes	$4,699	~5 tok/s
Mac Studio M3 Ultra	96 GB unified	819 GB/s	Thunderbolt 5 only	no / niche	$5,299+	~16 tok/s
MacBook Pro M5 Max	128 GB unified	614 GB/s	Thunderbolt 5 only	no / niche	~$5,000	~12 tok/s
NVIDIA RTX 5090	32 GB VRAM	1,792 GB/s	PCIe 5.0, CUDA	yes / yes	~$4,200/card	doesn't fit (needs 2)
NVIDIA RTX PRO 6000 Blackwell, 96 GB	96 GB VRAM	1,792 GB/s	PCIe 5.0, CUDA	yes / yes	$13,000–16,000	~35–45 tok/s

What each class is actually for

Strix Halo — the cheapest entry to 128 GB, and unbeatable per gigabyte. ROCm/Vulkan has matured but still trails CUDA. Best for MoE models and experimentation. Multi-box means pipeline over Ethernet: capacity, not speed.
DGX Spark — you are buying the CUDA stack at 128 GB, not throughput. Correction worth internalising: NVLink-C2C is internal to the GB10 chip; box-to-box linking uses the onboard ConnectX-7 NIC at up to 200 Gb/s — about 25 GB/s, two units without a switch. Measured dense 70B decode has been reported around 2.7 tok/s, with bandwidth rather than compute as the binding constraint. Also watch sustained thermals.
Macs — highest bandwidth of any unified-memory box (819 GB/s on M3 Ultra), so "Macs are low bandwidth" is only true relative to discrete GPUs. MLX is excellent, single-machine, single-pool. The real limits: no CUDA, and capacity has gone backwards this year.
RTX 5090 — the tok/s-per-dollar champion for anything under ~28 GB. Two cards with tensor parallelism give 64 GB at full speed and beat every unified-memory box on interactive feel; you pay in power, heat, noise and PCIe planning.
RTX PRO 6000 — the only card that combines 96 GB with 1.8 TB/s. It is the honest answer for a single large model resident 24/7. At current prices it costs more than three or four 5090s, which would give you more total VRAM but far more complexity.

07The map

Every machine is a point in capacity × bandwidth — and price is the third axis

Capacity vs. bandwidth vs. price

Log axes. Up = bigger models fit. Right = faster words. Circle area scales with price, so the top-right corner is both empty and expensive. Faded points are two-unit configurations.

08Before you buy

Work down this list in order

Name the models you will actually run. Dense or MoE, and at what quantization. The hardware follows from this, never the reverse.
Compute the two numbers. Does it fit in 80% of capacity? Is bandwidth ÷ bytes-per-token above the tok/s you can live with?
Include the KV cache at your real context length, not at 4K. Long-context agents change the answer.
Price in the thinking tokens. If you use reasoning models, multiply your patience by 3–10×.
Check the software stack, not just the silicon. CUDA has the widest support; MLX is strong but Apple-only; ROCm works and still surprises you.
Verify the lane map before committing to multi-GPU, plus PSU headroom, case width and cooling.
Compare against renting. At current hardware prices, cloud GPUs win for bursty or uncertain workloads. Buy local for privacy, latency, air-gapped work, or constant utilisation.
Benchmark before you scale. One box, your models, your context length — then decide whether a second one helps.