How much RAM do you need to run AI locally in 2026: the honest guide for 8, 16, 32 and 64 GB

August 10, 202621 minlocal AI, LLM, llama.cpp, quantization, GGUF, Apple Silicon, RAM, self-hosting

Quick answer (60 seconds): the memory to run a local LLM is approximately parameters × bytes_per_parameter + KV cache + overhead. For a 7B Q4_K_M with 4K context you need ~6 GB; for a 70B Q4_K_M with 8K context you need ~45 GB. In practice: 8 GB of RAM runs a 3B Q4 with short context, 16 GB a 7B Q4 or 13B Q3, 32 GB a 30B Q4 or 70B Q2, 64 GB a 70B Q4 with comfortable context. If you have a GPU with little VRAM, use --n-gpu-layers to split. If it doesn't fit in RAM, mmap to SSD as a last resort (5-10x slower, not real-time).

Running an AI model on your own machine moved from experimental in 2024-2025 to commodity in 2026. The question is no longer "can you?" — it's "how much RAM do I need for the model I want to run?". The short answer is the formula above; the useful answer is the concrete configurations table for your hardware. This post is the operational guide: the formula, the real numbers per quantization, and what to pick for 8, 16, 32 and 64 GB.

Disclosure: I ran the tests on an M3 Max 128 GB and a PC with 64 GB DDR5 + RTX 4090. The throughput numbers are mine; the model sizes and base memory come from the llama.cpp docs, the GGUF publishers (TheBloke, bartowski, mradermacher) and the willitrunai.com benchmarks. Where I throw a speed number without exact source, it's measured by me on representative prompts.

The formula almost nobody shows you

The memory formula for LLM inference is surprisingly simple:

Total memory ≈ Weights + KV cache + Overhead
Weights      = parameters × bytes_per_parameter
KV cache     = 2 × layers × KV_heads × head_dim × context_length × bytes_per_KV
Overhead     ≈ 0.5-1.5 GB (runtime, buffers, scratch)

Three variables drive everything: model size (parameters), quantization (bytes per parameter), and context length (KV cache). The rest is runtime overhead (llama.cpp, vLLM, transformers) and stays noise unless you run multimodal models.

What the formula ignores: tokenizer, additional embeddings if you keep them separate, and OS process memory. For a setup dedicated to inference, leave 2 GB free for the OS; for running alongside your IDE, 4 GB.

Quantization: how much each model weighs

Quantization reduces bytes per parameter by compressing weights to lower precision. The table below summarizes the formats most common in GGUF (the llama.cpp format) and their approximate cost. Numbers are effective bytes per parameter, including block overhead and quantized scales:

Row by row:

• Q4_K_M is the recommended 2026 default. Almost 3x less memory than FP16 with under 3% quality loss on standard benchmarks. Most GGUF you download from HuggingFace ships in Q4_K_M.
• Q5_K_M or Q6_K if you want more quality. If your RAM allows, they give 1-2% more accuracy on reasoning and coding tasks. For a 7B the difference is marginal; for a 70B it's more visible.
• Q3_K_M and Q2_K are compromises. Useful when you want to run a larger model than Q4 fits. Don't use them for production: coherence on long sessions drops.
• IQ-series (importance-aware). Smarter quantization that prioritizes the most important weights. IQ4_XS performs comparable to Q4_K_M in similar files; IQ2_XXS is experimental. For modern production, prefer Q_K-M.

File size: from formula to GGUF

Applied to models of common use in 2026, the pure weights weigh approximately:

These are weights. To these numbers you must add KV cache (the next section) and overhead before calculating total RAM.

KV cache: the other variable that decides your hardware

The KV cache is the memory the model reserves to store attention for tokens already processed. It's what allows context not to be recomputed entirely on each step. It scales linearly with context length and model architecture.

Simplified formula per model:

KV cache (bytes) ≈ 2 × layers × KV_heads × head_dim × ctx_len × bytes_per_KV

With bytes_per_KV = 2 (FP16) or 1 (Q8) or 0.5 (Q4):

This means the KV cache for a typical context adds:

Quantizing the KV cache (Q8 or Q4) reduces consumption 2-4x with marginal quality loss — most runtimes in 2026 do it by default. If you're going to run long context, turn it on: --cache-type-k q8_0 --cache-type-v q8_0 in llama.cpp.

The GPU factor: VRAM is not RAM

Discrete GPU (NVIDIA RTX, AMD Radeon RX, Apple Silicon) has VRAM instead of RAM. VRAM is faster for tensors but scarcer and more expensive. If your model doesn't fit in VRAM, you have three paths:

1. Entire model in VRAM. The ideal case. Inference speed 5-20x higher than CPU. Limitation: the model has to fit. On RTX 4090 (24 GB) you comfortably fit a 13B Q4_K_M + moderate context.

2. Model split GPU + CPU with --n-gpu-layers. In llama.cpp you can say "put N layers on GPU, the rest on CPU". If N < total, the layers on CPU run slow but the total model works. Useful when the model is bigger than your VRAM but the difference isn't huge. Speed penalty: PCIe bandwidth (16-32 GB/s on PCIe 4.0 x16 vs 1000+ GB/s on modern VRAM).

3. Model split GPU + CPU + SSD with mmap. mmap lets you page the model from SSD when it doesn't fit in RAM. For inference it's usable but the first token can take 5-30 seconds. Useful for experimenting; not for production.

Quick table of available VRAM and models that fit comfortably:

Apple Silicon: unified memory. On M2/M3/M4 Max and Ultra, the GPU and CPU share the same memory pool. That "unified memory" behaves like RAM for the model: an M3 Max with 128 GB can run a 70B Q4_K_M without splitting. The downside: shared bandwidth (300-400 GB/s on M3 Max) is quite lower than an H100 (~3350 GB/s). For interactive inference the tradeoff pays off; for high-throughput serving, discrete GPU wins.

Real configurations for 8 GB of RAM

The 8 GB bracket is tight but possible. What you can expect:

• Operating system: 2-3 GB
• App overhead (Chrome, IDE, OS): 2-3 GB
• Free for LLM: 3-4 GB

Configurations that work on 8 GB:

What does NOT work on 8 GB: 7B models in Q4_K_M with context >2K (you run out of RAM and start swapping to SSD), anything >7B.

Operational recommendation: a Llama 3.2 3B Q4_K_M with 4K context is the most comfortable combo on 8 GB. Close the laptop for performance: if you need to close the IDE to free RAM, you lose more than you gain. If you have an integrated GPU (Intel Arc, AMD Radeon Graphics), ollama lifts layers until VRAM is full and completes with CPU.

Recommended software: llama.cpp with -t 4 (4 threads), ollama with OLLAMA_NUM_PARALLEL=1 (no parallel batches), or LM Studio with automatic n_gpu_layers.

Real configurations for 16 GB of RAM

16 GB is the average developer's sweet spot. The typical setup:

• OS + apps: 4-5 GB
• Free for LLM: 10-12 GB

Configurations that work on 16 GB:

What does NOT work on 16 GB: 70B in any quantization, 30B in Q4_K_M with context >8K, two models loaded in parallel.

Operational recommendation: a Llama 3.1 8B Q4_K_M with 8K context is the 16 GB default. For coding, Qwen 2.5 14B Q3_K_M with 4K context is better and fits comfortably. For Spanish, Llama 3.1 8B Instruct performs well. On M2 Pro or M3 Pro with 16 GB, unified memory gives you the same + the integrated GPU bonus (when not running with the screen on).

Recommended software: ollama with default config, or llama.cpp with -c 8192 (8K context). If you have an NVIDIA GPU with 8+ GB VRAM, move the whole model to GPU and leave CPU for batches.

Real configurations for 32 GB of RAM

32 GB is where it gets interesting. This is the bracket where you get into serious reasoning models without paying USD 3,000+ for GPUs.

• OS + apps: 5-6 GB
• Free for LLM: 25-27 GB

Configurations that work on 32 GB:

What does NOT work on 32 GB: 70B Q4_K_M complete (40 GB + KV cache + overhead > 32 GB), large MoE models.

Operational recommendation: a Qwen 2.5 32B Q4_K_M with 8K context is the 32 GB sweet spot. Performance comparable to Llama 3.1 70B in Q4 with half the memory. For coding, Qwen 2.5 Coder 32B is the best open-weight option in this bracket. If you have an NVIDIA GPU with 16-24 GB VRAM (RTX 4080, RTX 3090/4090), move the whole model to GPU and inference flies (40-100 tok/s in Q4_K_M).

Apple Silicon on 32 GB: an M3 Max 32 GB runs the 32B Q4_K_M comfortably. An M3 Pro 36 GB the same. An M2 Pro 32 GB is a bit tight.

Recommended software: vllm (best throughput on GPU), llama.cpp with -ctk q8_0 -ctv q8_0 (quantized KV cache), or ollama with OLLAMA_MAX_LOADED_MODELS=1 to avoid loading several in parallel.

Real configurations for 64 GB of RAM

64 GB is where you stop negotiating. You get into Llama 70B Q4_K_M complete with comfortable context.

• OS + apps: 6-8 GB
• Free for LLM: 55-58 GB

Configurations that work on 64 GB:

What starts to fit on 64 GB: Llama 70B Q5_K_M, Mixtral 8x22B (~85 GB Q4, doesn't fit), DeepSeek V3 quantized (~150 GB, doesn't fit).

Operational recommendation: a Llama 3.1 70B Q4_K_M with 16K context is the 64 GB setup. For coding, Qwen 2.5 Coder 72B Q4_K_M is top-tier. If you work with long documents, enable 32K context: the KV cache eats 2.5 GB extra but the model doesn't lose track.

M3 Max 64 GB vs M4 Pro 64 GB: recent benchmarks (April 2026) show that the M3 Max 64 GB runs Llama 3.3 70B Q4 at ~7.5 tok/s, while the M4 Pro 64 GB runs out of memory (doesn't load the 70B Q4_K_M complete). Technical reason: the M3 Max has 400 GB/s bandwidth vs 200 GB/s on M4 Pro, and this impacts load latency. If you're going to buy Apple Silicon for inference, prioritize Max over Pro.

Recommended software: llama.cpp with -ngl 999 (all to GPU if you have it), or vllm with --tensor-parallel-size 1 (single GPU). For Apple Silicon, mlx is giving the best throughputs in 2026.

When to use shared memory (unified memory)?

The unified memory of Apple Silicon (M1/M2/M3/M4) and the Shared Memory of CPUs with integrated GPU (Intel with iGPU, AMD with APU) offer the advantage of treating VRAM and RAM as a single pool. When it makes sense and when it doesn't:

When to USE shared memory:

• The model doesn't fit in dedicated VRAM. If your discrete GPU has 8 GB and you want to run a 13B Q4 (~7 GB), it fits; a 30B Q4 (~18 GB), no. With unified memory, an M3 Max 128 GB handles the 70B Q4 complete.
• You want interactive inference with a large model. The shared memory bandwidth of Apple Silicon (300-400 GB/s on Max) is enough for interactive generation at 7-10 tok/s on 70B Q4.
• You want simplicity. You don't have to think about --n-gpu-layers or splitting the model. Ollama lifts it alone.

When NOT to use shared memory:

• You need high throughput (>30 tok/s on large models). A dedicated RTX 4090 (1 TB/s bandwidth) generates 70B Q4 at ~30 tok/s. An M3 Max 128 GB generates the same model at ~7-8 tok/s. For serving, discrete GPU wins 4x.
• You want to run multiple models in parallel. The shared pool is a single resource; contention degrades all.
• You already have a cheap discrete GPU. If you already have an RTX 4090, you paid for 24 GB of VRAM that you can't use on M3 Max. Dedicated GPU wins on $/performance for inference.

The intermediate case: NVIDIA GPU + CPU + SSD. On a desktop with RTX 4090 (24 GB VRAM) + 64 GB DDR5 + 2 TB NVMe, you can run a 70B Q4 with parts in VRAM, parts in RAM, and part on SSD. It's not a "clean" setup but it works, and llama.cpp with --no-mmap and --n-gpu-layers 35 (wise layer) handles it automatically.

When to use storage as fallback (mmap)?

mmap lets the operating system page parts of the model from SSD as if they were memory. Useful when the model doesn't fit in RAM, but with a penalty:

• Modern NVMe SSD: 5-10 GB/s read. vs 50-100 GB/s DDR5.
• First token (prefill): 5-30 seconds for a 70B with parts on SSD.
• Generation: 2-5 tok/s instead of 10-20 tok/s in pure RAM.

When to USE mmap:

• You want to test a large model that doesn't fit in RAM. For experimenting and validating quality, mmap is sufficient. Not for production.
• You have NVMe and not RAM. If your only option is mmap, it's better than not running the model. UX is "slow first token, then fluid".
• Offline batch inference. If you don't need interactive latency, mmap runs the model at 3-5 tok/s and completes.

When NOT to use mmap:

• Production models. The variability of the first token (5-30s depending on what it pages) breaks UX.
• Slow SSD (SATA, HDD). If your SSD reads at 500 MB/s, mmap is practically unusable.
• Very long context. The KV cache also pages, and the compound cost becomes prohibitive.

Alternative to mmap: model split in layers. If you have 32 GB RAM and want to run a 70B Q4 (40 GB), llama.cpp with --n-gpu-layers 0 --no-mmap loads the first layers into RAM and swaps the rest. This is what ollama does by default when the model doesn't fit. Better than direct mmap if you have partial RAM.

Operational checklist: how much RAM you have and what model to load

Three questions to decide what to run today:

1. How much free RAM do you have? Open Activity Monitor (macOS), Task Manager (Windows), or htop (Linux). Leave 4 GB for OS + apps. The rest is your budget for the model.

2. Do you have a discrete GPU with VRAM? If you have NVIDIA RTX / AMD Radeon RX with VRAM ≥ model target, load the whole model into VRAM and leave CPU free. If not, use --n-gpu-layers to split.

3. What do you prioritize: quality, speed, or long context?

• Maximum quality: 70B Q5_K_M or Q6_K, 8K context. You need 64+ GB.
• Maximum speed: 7B Q4_K_M on dedicated GPU. 16 GB RAM + RTX 4060 = 50+ tok/s.
• Long context: 70B Q4_K_M with Q8 KV cache, 32K context. 64 GB minimum; 128 GB comfortable.
• Balance (recommended): 32B Q4_K_M with 8K context on 32 GB. 90% of 70B quality with half the memory.

Software per RAM bracket:

Conclusion

The formula is the same every time: memory = weights + KV cache + overhead. The weights depend on the model and quantization; the KV cache depends on context; the overhead is noise. Once you internalize the formula, you look at your free RAM and pick from the table.

Practical recommendation by bracket:

• 8 GB: Llama 3.2 3B Q4_K_M, 4K context. Coding: Qwen 2.5 Coder 3B. Philosophy: do what you can, not what you want.
• 16 GB: Llama 3.1 8B Q4_K_M, 8K context. Coding: Qwen 2.5 Coder 14B Q3. The developer sweet spot.
• 32 GB: Qwen 2.5 32B Q4_K_M, 8K context. Coding: Qwen 2.5 Coder 32B. The bracket where you get into serious reasoning.
• 64 GB: Llama 3.1 70B Q4_K_M, 16K context. Coding: Qwen 2.5 Coder 72B. No compromises.
• 128 GB (Apple Silicon Max): Llama 3.1 70B Q5_K_M, 32K context. A deal for interactive inference.

Note on prices and models: DDR5 RAM prices dropped to USD 4-5/GB in 2026 (vs USD 8-12/GB in 2023). If your workstation has 16 GB and you want to jump to 64 GB, that's USD 200-300 — the best investment for local inference. If you have to choose between more RAM and a discrete GPU, prioritize RAM: LLM inference is bandwidth-bound, not compute-bound on CPU.

If your startup is exploring self-hosting LLMs and you want to validate the setup with your real workloads before committing to hardware, book a free 30-minute call — in 20 minutes we can usually calculate the correct configuration for your task mix and budget.

---

Read also:

• MiniMax M3: the open-weight frontier model with 1M context and native multimodality — for when you need a frontier open-weight model and how to run it.
• Chinese open-weight models in SaaS: 30-46% of LLM gateway traffic in 2026 — which open-weight models are dominating the ecosystem and how to choose.
• Grok 4.5 vs Sonnet 5 vs Claude Opus 5: 2026 model menu — high-tier comparison for when managed API wins over self-hosting.
• AI-Driven Development Methodology — how I integrate local LLMs into the development workflow.
• Back to blog — all articles.

Frequently asked questions

How much RAM do I need to run an LLM locally?

It depends on the model and quantization. Rule of thumb: memory ≈ parameters × bytes_per_parameter + KV cache + overhead. For a 7B Q4_K_M, plan ~4-5 GB for weights + 1-2 GB per 8K of context + ~1 GB overhead. For a 70B Q4_K_M, plan ~40 GB for weights + 4-8 GB per 32K of context. On 8 GB, run a 3B Q4 with short context. On 16 GB, a 7B Q4 or 13B Q3. On 32 GB, a 30B Q4 or 70B Q2. On 64 GB, a 70B Q4 with comfortable context.

Which quantization should I use?

Q4_K_M is the recommended default in 2026: ~97% of FP16 quality with ~0.5 bytes per parameter. If you have RAM to spare, Q5_K_M (~0.6 bytes) or Q6_K (~0.75 bytes) get closer to FP16. If you're tight, Q3_K_M and Q2_K sacrifice coherence on long tasks but let you run larger models. IQ-series (IQ2, IQ3, IQ4) is experimental and for enthusiasts, not production.

Is it better to run the model on CPU or GPU?

Discrete GPU (NVIDIA, AMD, Apple Silicon) is 5-20x faster for inference because it has more memory bandwidth. Modern CPU (AVX2 / AVX-512, AMX) runs small models functionally but at 2-5 tokens per second. If you have a GPU with limited VRAM (8-12 GB), use --n-gpu-layers in llama.cpp to split the model. If you have no GPU, CPU with llama.cpp + Q4_K_M works, but don't expect more than 10-15 tok/s on a 7B.

Is Apple Silicon unified memory better than classic RAM?

It is different, not better in every way. The GPU and CPU share the same pool, so an M3 Max with 128 GB can run a 70B Q4 without splitting. The downside: shared bandwidth (300-400 GB/s on M3 Max) is lower than a discrete GPU (H100: 3350 GB/s), so generation speed is lower. For interactive inference the tradeoff usually pays off; for high-throughput serving, discrete GPU wins.

Can I use my SSD as RAM fallback?

Yes, with mmap: the model is paged from SSD as if it were memory. On a modern NVMe, the penalty is 5-10x in speed, not 100x. For a full 70B that doesn't fit in RAM, you can run part in RAM and part on SSD; speed drops but it works. Don't expect real-time — the first token can take 5-30 seconds. It's the last option, not the first.

How much RAM for long context (32K, 128K)?

KV cache scales linearly with context. For a 7B with 32K context, plan ~2-3 GB extra in Q4_K_M. For a 70B with 32K, plan ~6-10 GB extra. With 128K context on a 70B, you may need 20-30 GB just for KV cache. That's why large models with long context: 64 GB of RAM is the minimum for 70B + comfortable 32K; 128 GB for 70B + 128K. Quantizing the KV cache (Q8 or Q4) reduces the cost 2-4x with minor quality loss.