How much VRAM is needed for a 7B / 14B model?
For 7B models in Q4_K_M quantization, 4.8–5.8 GB VRAM is required with base context. For 14B models, 9.5–12 GB is needed. GPUs with 12–16 GB VRAM are optimal.
Calculate exact memory requirements for running local AI models (GGUF, AWQ, FP16), KV Cache allocation for long context, and verify hardware fit for NVIDIA RTX and Apple Silicon.
ollama run qwen2.5:7b-instruct-q4_K_M
./llama-server -m qwen2.5-7b-instruct-q4_k_m.gguf -c 8192 -ngl 99
For 7B models in Q4_K_M quantization, 4.8–5.8 GB VRAM is required with base context. For 14B models, 9.5–12 GB is needed. GPUs with 12–16 GB VRAM are optimal.
KV Cache stores attention keys and values for conversation history. In long context (32K-128K), it can consume 4-16 GB. Enabling FP8 or Q4 quantization reduces cache size 2-4x.
Yes, Apple M-series chips use unified memory accessible by both CPU and GPU. A Mac with 64-192 GB RAM runs 70B-120B models with long context via llama.cpp or MLX.
GGUF is tailored for CPU and hybrid CPU+GPU offloading (llama.cpp, Ollama). AWQ and EXL2 are designed for pure GPU inference (vLLM, ExLlamaV2) with highest tokens/sec.