In this article we will conduct a comprehensive comparative analysis of two dominant inference stacks - vLLM (PagedAttention architecture and continuous batching) and Ollama / llama.cpp (lightweight GGUF runtime). We will analyze the physics of VRAM consumption, measure the real tokenization speed (tokens/s) on NVIDIA RTX 3060 12GB, RTX 4060 Ti 16GB, RTX 4070 and RTX 4090 video cards, and also provide ready-made production manifests for Docker and custom Modelfiles.
1. Architectural basis: PagedAttention vs llama.cpp
The main limiter on the context length in modern transformers is the size of the Key-Value (KV) cache. With the classic memory allocation scheme in PyTorch, video memory for the KV cache is reserved in contiguous blocks for the maximum declared session length. This leads to two fundamental problems:
- Internal fragmentation: if a request takes 2,000 tokens and 32,768 are allocated, the rest of the memory is blocked for nothing.
- External fragmentation: alternating short and long generations makes it impossible to allocate new monolithic memory blocks without defragmentation and freezing the calculations.
vLLM approach: PagedAttention and Continuous Batching
Developed at UC Berkeley, the vLLM engine borrows the concepts of virtual memory and paging from operating systems. The KV cache space is divided into small fixed blocks (usually 16 tokens each). In the Block Table page table, physical blocks of memory are associated with a logical sequence of request tokens, even if they are physically scattered across different VRAM addresses.
Thanks to the Continuous Batching mechanism, new requests are mixed into the inference iteration immediately after the completion of generating a separate token in a parallel thread, without waiting for the complete output of other requests. This provides a 3-8x increase in throughput for multi-user workloads.
Ollama approach (llama.cpp): Zero-dependency and GGUF
Ollama is built on the basis of the llama.cpp library from Georgi Gerganov. Its key advantages:
- No Python environment and PyTorch: pure compiled C/C++ with direct operation through the CUDA driver and cuBLAS.
- GGUF quantization format: a single binary container with metadata and weights, allowing instant loading of the model via the
mmap()system call. - Flexible CPU/GPU offload: the ability to reset part of the neural network layers to the system RAM if the model does not fit entirely into the VRAM of the video card.
2. VRAM consumption and quantization benchmarks
For comparison, we tested different quantization options for the Qwen2.5-7B-Instruct and Qwen2.5-14B-Instruct models at different context lengths (4k, 8k and 32k tokens). Measurements were carried out using Ubuntu 24.04 LTS, CUDA 12.4 and the NVIDIA 550.120 driver.
| Model and quantization | Scale format | VRAM (Weights) | VRAM (4k Context) | VRAM (Context 32k) | Target GPU |
|---|---|---|---|---|---|
| Qwen 2.5 7B (BF16) | 16-bit unquantized | 14.8 GB | 17.2 GB | 33.5 GB | RTX 4090/A5000 |
| Qwen 2.5 7B (AWQ / GPTQ) | 4-bit GPU tensor | 4.9 GB | 6.8 GB | 11.4 GB | RTX 3060 12GB / RTX 4070 |
| Qwen 2.5 7B (GGUF Q4_K_M) | 4-bit llama.cpp | 4.7 GB | 6.2 GB | 10.8 GB | RTX 3060 12GB / 4060 Ti 16GB |
| Qwen 2.5 14B (AWQ) | 4-bit GPU tensor | 9.2 GB | 12.4 GB | 15.6 GB | RTX 4060 Ti 16GB / RTX 4080 |
| Qwen 2.5 14B (GGUF Q4_K_M) | 4-bit llama.cpp | 8.9 GB | 11.8 GB | 15.1 GB | RTX 4060 Ti 16GB / RTX 3090 |
| Qwen 2.5 14B (GGUF Q8_0) | 8-bit high-fidelity | 15.2 GB | 18.4 GB | 22.8 GB | RTX 3090 24GB / RTX 4090 24GB |
Engineering Note: On graphics cards with 12GB of memory (such as the super popular RTX 3060 12GB), the Qwen 2.5 7B-Instruct model in AWQ quantization fits entirely into memory along with the context of up to 32,768 tokens! However, the 14B model requires a minimum of 16 GB of VRAM for comfortable operation without offloading layers to slow PCIe/RAM.
Generation speed measurements (Tokens per Second)
We measured the tokenization speed on the RTX 4090 (24GB) and RTX 3060 (12GB) when generating responses with a length of 512 tokens.
| Script and stack | Qwen 2.5 7B (1 stream) | Qwen 2.5 7B (8 streams in total) | Qwen 2.5 14B (1 stream) | Qwen 2.5 14B (8 streams in total) |
|---|---|---|---|---|
| vLLM (AWQ, RTX 4090) | 112 t/s | 680 t/s 🔥 | 76 t/s | 435 t/s 🔥 |
| Ollama (Q4_K_M, RTX 4090) | 124 t/s | 185 t/s | 82 t/s | 120 t/s |
| vLLM (AWQ, RTX 3060 12GB) | 44 t/s | 142 t/s | —(OOM) | —(OOM) |
| Ollama (Q4_K_M, RTX 3060 12GB) | 49 t/s | 58 t/s | 18 t/s (with 20% offload in RAM) | 22 t/s |
Benchmark output: In single-threaded mode (single interactive stream), Ollama demonstrates slightly lower latency on the first token (Time To First Token - TTFT) due to the absence of Python overhead. But with even 4-8 concurrent requests, vLLM breaks away from the competition, delivering 3.5-4 times higher total throughput thanks to PagedAttention.
3. Practice: Production deployment of vLLM in Docker
For building an enterprise generative AI microservice, the most stable solution is a containerized vLLM launch with an official image and support for AWQ quantization. Below is the combat docker-compose.yml:
version: '3.8'
services:
vllm-qwen:
image: vllm/vllm-openai:v0.6.3.post1
container_name: vllm-qwen-service
runtime: nvidia
restart: unless-stopped
environment:
- HUGGING_FACE_HUB_TOKEN=${HF_TOKEN}
- VLLM_ATTENTION_BACKEND=FLASH_ATTN
ports:
- "8000:8000"
volumes:
- ~/.cache/huggingface:/root/.cache/huggingface
- /dev/shm:/dev/shm
ipc: host
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
command: >
--model Qwen/Qwen2.5-7B-Instruct-AWQ
--quantization awq
--dtype auto
--max-model-len 32768
--gpu-memory-utilization 0.92
--enforce-eager
--host 0.0.0.0
--port 8000
--api-key secret_proarena_tokenKey tuning flags:
--gpu-memory-utilization 0.92: Reserves 92% of VRAM for the model and dynamic KV cache, leaving 8% for CUDA kernel system buffers.--max-model-len 32768: Sets the upper bound of the context. If 8,192 tokens are enough for you, reducing this parameter will free up gigabytes of VRAM for additional parallel sessions.--enforce-eager: Disables pre-warming of CUDA Graphs, which saves up to 1.5 GB of VRAM on Ampere cards (RTX 30xx) and speeds up container cold starts.--kv-cache-dtype fp8: (for Ada Lovelace / RTX 40xx architecture cards) converts the KV cache to an 8-bit floating point format, reducing context memory consumption by exactly half with virtually no loss of accuracy!
Checking functionality via the standard OpenAI API interface:
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer secret_proarena_token" \
-d '{
"model": "Qwen/Qwen2.5-7B-Instruct-AWQ",
"messages": [
{"role": "system", "content": "Ты высококвалифицированный инженер по безопасности и сетям."},
{"role": "user", "content": "Объясни разницу между eBPF XDP и классическим iptables."}
],
"temperature": 0.4,
"max_tokens": 1024
}'4. Practice: Local launch in Ollama
For local development, integration with code editors (Cursor, Continue.dev, Zed) and personal assistants, Ollama remains the ideal tool thanks to its easy one-line deployment:
# Запуск 7B с квантованием Q4_K_M
ollama run qwen2.5:7b-instruct-q4_K_M
# Запуск 14B на видеокартах с 16GB+ VRAM
ollama run qwen2.5:14b-instruct-q4_K_MContext customization via Modelfile
By default, Ollama allocates a context window that is only 2,048 tokens. To unlock the full potential of Qwen 2.5 when analyzing large codebases, you need to create a custom Modelfile:
FROM qwen2.5:7b-instruct-q4_K_M
# Расширение контекстного окна до 32,768 токенов
PARAMETER num_ctx 32768
# Оптимальные параметры сэмплирования под Qwen 2.5
PARAMETER temperature 0.6
PARAMETER top_p 0.9
PARAMETER repeat_penalty 1.05
# Системный промпт инженера
SYSTEM "Ты эксперт по системному программированию, архитектуре HighLoad и кибербезопасности. Отвечай емко, технически аргументированно, с готовыми примерами конфигураций и тестами."Compiling and running the optimized model:
ollama create qwen-coder-32k -f Modelfile
ollama run qwen-coder-32k5. Summary selection matrix: vLLM vs Ollama
| Criterion | vLLM (PagedAttention) | Ollama (llama.cpp) |
|---|---|---|
| Main purpose | Enterprise APIs, backends, web services, agents | Local development, laptop, solo chats |
| Multi-threaded load | Exceptional (Continuous Batching) | Basic (queue with locks) |
| Long context (32k+) | Maximum VRAM savings (no fragmentation) | High risk of OOM with concurrent requests |
| Setting complexity | Medium (Docker, NVIDIA Container Toolkit, HuggingFace) | Zero (one binary or .exe) |
| Offload on CPU/RAM | No (requires full VRAM placement) | Yes (layered launch when there is insufficient video memory) |
| Optimal GPU | RTX 3060 12GB (under 7B AWQ) / RTX 4090 24GB | Any NVIDIA GPU / Apple Silicon / regular CPU |
Final conclusions
- For RTX 3060 12GB the best choice is Qwen 2.5 7B in AWQ quantization (vLLM) or Q4_K_M (Ollama) - this gives fair 45–50 t/s and free work with 32k tokens.
- For RTX 4060 Ti 16GB and RTX 4070 Ti Super 16GB the optimal becomes Qwen 2.5 14B AWQ - the model demonstrates the level of reasoning at the level of top commercial networks at speeds up to 30 t/s.
- If your task is to connect LLM to a website, Telegram bot or internal knowledge base search service (RAG), deploy vLLM: the speed increase with an influx of users will pay for any setup costs.
💬 Discussion 0
BBCode & Emoji supportedParticipate in discussions, ask authors questions, and share benchmarks by joining the ProArena engineering community.