llama.cpp vs Ollama: Which Should You Run in 2026?

Related on the site: Ollama Not Using GPU? Fix It on Linux, Windows and WSL.

Raw llama.cpp if you want maximum tokens/second and full control; Ollama if you want a one-command install and an API server that works out of the box. Ollama is not a competing engine — it is a Go service that bundles llama.cpp as its inference backend, adds model management (ollama pull llama3.1), and serves a REST API on port 11434. So the real question is not "which engine is faster" but "how much control do you want over that engine's knobs". Because Ollama ships conservative defaults (Q4_K_M quantization, modest context, no flash attention until recently), identical hardware can produce noticeably different numbers. Below: what actually differs, where the speed gap comes from, the same model running in both, and a decision table.

What is the difference between llama.cpp and Ollama?

llama.cpp is the inference engine: a single C/C++ project from GGUF creator Georgi Gerganov that runs quantized models on CPU, GPU, or a mix of both. It gives you llama-cli for one-shot prompts and llama-server — an OpenAI-compatible HTTP server — plus every tuning flag the engine supports: GPU layer offload, KV-cache quantization, speculative decoding, custom samplers.

Ollama is a product layered on top of that engine. It forks and vendors llama.cpp, then wraps it in:

• a model registry (ollama pull, ollama list) with automatic GGUF weight splitting,
• a Modelfile system (a Dockerfile-like spec for prompt templates and parameters),
• a background daemon that keeps models warm in VRAM and exposes its own REST API,
• automatic hardware detection with safe defaults.

The practical consequence: with Ollama you manage models; with llama.cpp you manage inference. If you have ever wanted to change the quantization format, quantize the KV cache, raise context past the default, or pin specific layers to the GPU, that is llama.cpp territory. Ollama hides most of those dials — deliberately.

Is llama.cpp faster than Ollama?

On the same GGUF file, same quantization, same context, and the same llama.cpp version — no, they are within noise of each other, because Ollama is llama.cpp doing the math. Every "Ollama is 30% slower" benchmark you see is really a comparison of defaults. The gap comes from three places:

1. Quantization choice. Ollama's registry defaults to Q4_K_M. Run the same model as Q5_K_M or Q6_K from llama.cpp and you get better quality per token at a similar speed — or choose Q4_0/IQ4 for raw speed.
2. Flash attention and KV-cache quantization. --flash-attn plus -ctk q8_0 -ctv q8_0 shrinks the KV cache dramatically, which raises tokens/second at long context and lets you fit bigger contexts in the same VRAM. Ollama only exposes some of this.
3. Version lag. llama.cpp lands kernel optimizations weekly; Ollama merges upstream on its own schedule. A fresh llama.cpp build can be measurably faster than a months-old Ollama binary on the same box — until Ollama catches up.

Quick benchmark command, engine-agnostic — it reports prompt evaluation and generation speed:

./build/bin/llama-bench -m Llama-3.1-8B-Instruct-Q4_K_M.gguf -ngl 99 -fa 1

Run it against Ollama's own model file (~/.ollama/models/blobs/..., renamed to .gguf) and you will usually match Ollama's numbers exactly — then beat them by adding -ctk q8_0 at 16k context.

How do you run the same model in both?

Both consume GGUF. Minimal end-to-end for each:

# --- Ollama path: install, pull, serve ---
curl -fsSL https://ollama.com/install.sh | sh
ollama run llama3.1:8b        # downloads Q4_K_M, loads into VRAM, opens a chat

# its API, OpenAI-compatible style:
curl -s http://localhost:11434/v1/chat/completions -d '{
  "model": "llama3.1:8b",
  "messages": [{"role": "user", "content": "Say hi in 5 words"}]
}'

# --- llama.cpp path: build, download GGUF, serve ---
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON    # or -DGGML_VULKAN=ON / -DGGML_HIP=ON
cmake --build build --config Release -j

huggingface-cli download bartowski/Meta-Llama-3.1-8B-Instruct-GGUF \
  Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf --local-dir models

./build/bin/llama-server -m models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
  -ngl 99 --ctx-size 16384 --flash-attn -ctk q8_0 -ctv q8_0 --port 8080

llama-server exposes the OpenAI Chat Completions schema, so the same curl against http://localhost:8080/v1/chat/completions works unchanged. Any tool built for the OpenAI API — scripts, editors, RAG pipelines — can point at either. The flags do the real work: -ngl 99 offloads every layer to the GPU, --flash-attn plus the -ctk/-ctv pair keeps a 16k context inside an 8 GB card that Ollama's defaults would refuse.

For picking the GGUF file itself and what the quant labels mean, see how to run GGUF models locally.

If you want the two engines running real GGUF files side by side — pulls, servers, flags — the how to run GGUF models locally guide walks Ollama, llama.cpp and vLLM command by command.

When does Ollama make more sense?

Most people should start with Ollama, and that is not a consolation prize:

• You want it working tonight. One command, model pulled, API up. llama.cpp means choosing a backend (CUDA/Vulkan/HIP/Metal), building, and fetching weights by hand.
• You juggle many models. The registry, automatic unloading, and Modelfiles beat hand-managing directories of GGUF files.
• Your machine is modest. Ollama's defaults are conservative for a reason — they almost always fit and run.
• You want a stable API surface. Ollama's daemon manages model lifecycles so a long-running service does not have to.

Choose raw llama.cpp when you are benchmarking, serving at any scale, running on a phone or a Raspberry Pi, need long context on small VRAM, or want a feature the day it merges upstream. Power users often run both: Ollama for daily driver models, a pinned llama.cpp build for the one workload that needs the last 20%.

If your comparison is really between desktop GUI apps, that is a different axis — see Ollama vs LM Studio — and for engine choice per task, the best local LLMs for coding covers the model side.

And if the GPU sits idle after you pick one, that is a separate bug — Ollama not using your GPU has the fix for Linux, Windows and WSL.

Which should you use?

Decide by control, not speed. The engines are the same; the defaults are not. Install Ollama if "it runs and serves an API" is the goal — you lose a few knobs you were not going to turn anyway. Build llama.cpp if tokens/second, context length, or quantization control is the goal — you get every knob, at the cost of managing models yourself. Either way you are running the same GGUF files, and switching later costs an afternoon, not a rewrite.

This ships daily at mrsaynothing.dev — the full archive, every piece in 19 languages, zero missed days. New posts land in the newsletter the moment they ship: join it here. Code at GitHub.