Hands-on Lab

llama.cpp server in Docker with an OpenAI-compatible API

October 21, 2026 4 min 1

0 / 5

Cheat sheet

Found it useful? Don't miss the next lab

The video has captions and a transcript. Everything it shows is written out, step by step, below. Cheat sheet with every command

0:00 / 0:00

We pull the llama.cpp server image from the GitHub registry with the pinned server-v0.5.0 tag. It holds only the llama-server binary and ships amd64 and arm64 variants; the unversioned server tag points at the nightly build and moves on its own.

docker pull ghcr.io/ggml-org/llama.cpp:server-v0.5.0

I did it

We check Docker picked the linux/arm64 variant of the image and ask the binary for its version: 0.5.0, build 11146, built for Linux aarch64. The -dev suffix shows up because the image is built with LLAMA_BUILD_IS_DEV turned on.

docker image inspect --format '{{.Os}}/{{.Architecture}}' ghcr.io/ggml-org/llama.cpp:server-v0.5.0 && docker run --rm ghcr.io/ggml-org/llama.cpp:server-v0.5.0 --version 2>&1 | grep -E '^(version|built)'

What this proves

linux/arm64 Docker pulled the arm64 variant of the multi-architecture image Goal · you know the same tag runs on your arm64 machine without building anything
version: 0.5.0-dev (build 11146 the binary in the image is version 0.5.0, build 11146 Goal · you confirm you run the stable release the guide pins, not a nightly

We create /srv/llama/llama_keys with a comment line and the throwaway key tu_clave_de_api, and set 600 permissions so only root can read it. The container will mount this file read-only; on a real server, use a long random key.

mkdir -p /srv/llama && printf '# clave de prueba del lab\ntu_clave_de_api\n' > /srv/llama/llama_keys && chmod 600 /srv/llama/llama_keys

I did it

We check the file belongs to root with 600 permissions and that, comments aside, it holds exactly one key.

stat -c '%a %U %n' /srv/llama/llama_keys && echo "keys: $(grep -vc '^#' /srv/llama/llama_keys)"

What this proves

600 root /srv/llama/llama_keys only root can read the key file Goal · the key stays out of reach of other users on the machine
keys: 1 there is a single key line, not counting the comment Goal · you know exactly which key the server will accept

We start llama-server in the background. With -hf it downloads the Q4_0 quantization of Qwen3.5 0.8B from Hugging Face into the llama-cache volume, which LLAMA_CACHE points to. With --alias we set the name clients will see, and with -t 4 it uses 4 of the 18 cores. The -c 4096 option caps the context, --api-key-file requires the key and the port is published on 127.0.0.1 only.

docker run -d --name llama-server -p 127.0.0.1:8080:8080 -v llama-cache:/cache -e LLAMA_CACHE=/cache -v /srv/llama/llama_keys:/run/secrets/llama_keys:ro ghcr.io/ggml-org/llama.cpp:server-v0.5.0 -hf ggml-org/Qwen3.5-0.8B-GGUF:Q4_0 --alias qwen3.5-0.8b -t 4 -c 4096 --port 8080 --api-key-file /run/secrets/llama_keys

I did it

We poll /health every 2 s, up to 200 attempts, instead of waiting blind with sleep. Downloading the GGUF takes up most of the wait. In the end /health returns {"status":"ok"} with a 200 without sending the key: llama-server leaves it open for healthchecks.

s=$(date +%s); for i in $(seq 1 200); do curl -sf -o /dev/null http://127.0.0.1:8080/health && break; sleep 2; done; echo "ready after $(( $(date +%s) - s )) s"; curl -sf -w '\nhealth without key: HTTP %{http_code}\n' http://127.0.0.1:8080/health

What this proves

{"status":"ok"} the server finished loading the model and accepts requests Goal · you know when you can start sending requests, without guessing a wait time
health without key: HTTP 200 /health answers 200 even though the request carries no key Goal · you can use /health as a Docker or proxy healthcheck without handing out the key

We read the startup log: the model it loaded, the 4 threads, the 4 slots sharing the 4096 tokens of context and the address it listens on. Then, inside the container, we measure the Qwen3.5-0.8B-Q4_0.gguf file left in the volume's cache.

docker logs llama-server 2>&1 | grep -oE 'loading model .*|n_threads = [0-9]+|n_slots = .*|listening on .*'; docker exec llama-server sh -c 'du -shL /cache/models--*/snapshots/*/*.gguf'

What this proves

n_threads = 4 llama-server uses the 4 threads we asked for with -t 4 Goal · you control how much CPU the server takes and leave cores free for everything else
Qwen3.5-0.8B-Q4_0.gguf the downloaded GGUF sits in the cache, inside the llama-cache volume Goal · you know where the model lives and that it is not part of the container layer

We ask for the model list without the Authorization header. The server refuses with a 401 and an error body shaped like OpenAI's: message, type authentication_error and code.

curl -s -w '\nHTTP %{http_code}\n' http://127.0.0.1:8080/v1/models

What this proves

Invalid API Key llama-server refuses the request because it carries no valid key Goal · you check nobody uses your model without the key, even if they reach the port
HTTP 401 the status code is 401 Unauthorized Goal · an OpenAI client will get the same authentication error it gets from the original API

We repeat the request with the Authorization: Bearer header and the key from the file. Now it answers 200, and jq filters the list in OpenAI's format: the id is the qwen3.5-0.8b alias, in Q4_0 and with 4096 tokens of context.

curl -s -o /dev/null -w 'with key: HTTP %{http_code}\n' -H 'Authorization: Bearer tu_clave_de_api' http://127.0.0.1:8080/v1/models; curl -s -H 'Authorization: Bearer tu_clave_de_api' http://127.0.0.1:8080/v1/models | jq '{object, data: [.data[] | {id, owned_by, ftype: .meta.ftype, n_ctx: .meta.n_ctx}]}'

What this proves

with key: HTTP 200 with the key from the file the request is accepted Goal · you confirm --api-key-file read the key and skipped the comment line
"id": "qwen3.5-0.8b" the model shows up under the name we set with --alias Goal · you know which value to put in your clients' model field
"ftype": "Q4_0" the server loaded the Q4_0 quantization we asked for with -hf Goal · you check the :Q4_0 tag picked the small file and not the Q8_0

We send the same request we would send to the OpenAI API, with temperature 0, a 64-token cap and reasoning_effort none so Qwen3.5 answers without reasoning first. jq pulls out the answer, the stop reason and the usage block with the prompt and completion tokens.

curl -s http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -H 'Authorization: Bearer tu_clave_de_api' -d '{"model": "qwen3.5-0.8b", "messages": [{"role": "user", "content": "Explica qué es un contenedor Docker en una frase."}], "max_tokens": 64, "temperature": 0, "reasoning_effort": "none"}' | jq '{model, content: .choices[0].message.content, finish_reason: .choices[0].finish_reason, usage: {prompt_tokens: .usage.prompt_tokens, completion_tokens: .usage.completion_tokens, total_tokens: .usage.total_tokens}}'

What this proves

"finish_reason": "stop" the model finished the answer on its own, before the 64-token cap Goal · you know the answer is complete and not cut off by max_tokens
"completion_tokens": the answer carries OpenAI's usage block with the tokens spent Goal · you can measure what each request costs just as with the OpenAI API
the whole answer came from the local model on CPU, without leaving the machine Goal · you have an OpenAI-compatible API that does not depend on an outside service

We delete the container with docker rm -f and create it again with exactly the same options and the same llama-cache volume. It is what you would do when changing the image tag to upgrade.

docker rm -f llama-server && docker run -d --name llama-server -p 127.0.0.1:8080:8080 -v llama-cache:/cache -e LLAMA_CACHE=/cache -v /srv/llama/llama_keys:/run/secrets/llama_keys:ro ghcr.io/ggml-org/llama.cpp:server-v0.5.0 -hf ggml-org/Qwen3.5-0.8B-GGUF:Q4_0 --alias qwen3.5-0.8b -t 4 -c 4096 --port 8080 --api-key-file /run/secrets/llama_keys

I did it

We repeat the wait on /health, now every second. Compare the ready after figure with the first startup: the server only loads the GGUF that was already in the llama-cache volume, without asking Hugging Face for it again.

s=$(date +%s); for i in $(seq 1 60); do curl -sf -o /dev/null http://127.0.0.1:8080/health && break; sleep 1; done; echo "ready after $(( $(date +%s) - s )) s"; curl -sf http://127.0.0.1:8080/health; echo; docker volume ls --filter name=llama-cache --format 'volume: {{.Name}}'

What this proves

{"status":"ok"} the new container is ready to serve requests Goal · you can recreate or upgrade the container without losing the model
volume: llama-cache the llama-cache volume survived docker rm Goal · you know the model download happens once per machine

We delete the llama-server container, the llama-cache volume with the downloaded GGUF and the /srv/llama directory with the throwaway key.

docker rm -f llama-server && docker volume rm llama-cache && rm -rf /srv/llama

I did it

We check no container named llama-server is left, no llama-cache volume and that the /srv/llama directory no longer exists.

docker ps -a --filter name=llama-server --format '{{.Names}}' | grep . || echo 'no llama-server container'; docker volume ls --filter name=llama-cache --format '{{.Name}}' | grep . || echo 'no llama-cache volume'; ls /srv/llama 2>/dev/null || echo 'no /srv/llama'

What this proves

no llama-cache volume the volume with the model no longer exists Goal · you get back the disk space the GGUF used
no llama-server container no lab container is left Goal · you leave the machine as it was before the lab

1 view

Was this lab useful? Rate it:

Click to rate this post!

[Total: 0 Average: 0]

Video transcript

0:00 A local model that answers like the OpenAI API, inside Docker.
0:06 We pull the llama.cpp server image with the pinned server-v0.5.0
0:12 tag, the same one the guide uses.
0:15 We serve a GGUF model with llama.cpp in Docker and talk to it like the OpenAI
0:21 API.
0:22 llama-server is the llama.cpp binary that loads a GGUF and serves it over
0:28 HTTP in the shape of the OpenAI API.
0:31 The path:
0:32 Hugging Face, the cache in a volume, llama-server and the answer in OpenAI
0:37 format.
0:38 We check it:
0:39 the image is linux/arm64 and the binary reports version
0:44 0.5.0, build 11146.
0:48 With --api-key-file, every request needs the Authorization header with one of the
0:53 keys in the file.
0:55 We store the API key in a file with 600 permissions, with a comment line and a
1:00 throwaway key.
1:01 We check it:
1:02 the file belongs to root, has 600 permissions and holds a single key.
1:10 With -hf, the container itself downloads the GGUF into the cache, loads it and
1:15 opens the port.
1:17 We start llama-server:
1:18 it downloads Qwen3.5 0.8B in Q4_0 into a volume, uses
1:24 4 threads and requires the key from the file.
1:28 We wait for /health to answer ok, with a cap on attempts.
1:32 While it downloads and loads the model, the server does not answer yet.
1:37 The log shows 4 threads and port 8080, and the volume holds the GGUF file it
1:43 fetched from Hugging Face.
1:46 Without a key, /v1/models answers 401 with an error in OpenAI's
1:52 format:
1:53 Invalid API Key.
1:55 With the key, the answer is 200 and /v1/models lists
2:00 qwen3.5-0.8b, in Q4_0 and with
2:05 4096 tokens of context.
2:08 The real test:
2:10 a request to /v1/chat/completions.
2:14 The model answers in one sentence and the usage block counts the tokens.
2:20 The volume lives outside the container, so the GGUF is still there after docker rm.
2:26 We delete the container and create another with the same volume, to see what happens
2:30 to the downloaded model.
2:32 This time /health answers ok much sooner:
2:35 the model was already in the volume and is not downloaded again.
2:41 We clean up:
2:42 we delete the container, the volume with the model and the key file.
2:47 We check it:
2:48 no container, no volume and no key are left.

This lab runs llama.cpp server in Docker and tests it as if it were the OpenAI API. It is the Docker part of our guide how to install llama.cpp on Linux, macOS and Docker, carried through to the API we cover in llama-server as an OpenAI-compatible API in Docker. Everything you see in the terminal really ran, on an arm64 machine with no GPU.

What this lab does

You start the ghcr.io/ggml-org/llama.cpp:server-v0.5.0 image, the same tag the API guide pins. With -hf ggml-org/Qwen3.5-0.8B-GGUF:Q4_0 the container downloads the model from Hugging Face into a volume mounted at /cache. You wait for /health to return {"status":"ok"} in a bounded loop, and then you check three things: without a key, /v1/models answers 401; with the key, it lists the model; and /v1/chat/completions returns an answer with its usage block.

Why this model and not the one in the API guide

The API guide uses Qwen3 1.7B in Q8_0, a 1.8 GB GGUF. To keep the lab repeatable with a small download we picked the install guide’s model, Qwen3.5 0.8B, from the ggml-org/Qwen3.5-0.8B-GGUF repository, in its Q4_0 quantization. That repository has no Q4_K_M, so without the :Q4_0 tag the server would fetch the heavier Q8_0. Qwen3.5 reasons before answering, which is why the request carries "reasoning_effort": "none" and a max_tokens of 64. If you want to pick another quantization, it is explained in model quantization and llama.cpp on your laptop.

The API key and the cache volume

The key lives in a file with 600 permissions that the container mounts read-only and reads with --api-key-file, one key per line with # comments, according to the llama-server options reference. It is the same principle as in our lab on managing secrets in Docker: the key stays out of the image and out of the command line. The value tu_clave_de_api is a throwaway; on your server use a long random key.

The model is stored in a named volume, as the Docker volumes documentation describes. That is why, when you delete the container and create another with the same volume, the server starts without downloading the GGUF again. If you want to see that mechanism in more detail, watch the lab on how data persists in Docker with volumes and bind mounts.

How to use this API from your code

The request body follows the format of the OpenAI chat API, so the official SDK works with base_url pointed at http://127.0.0.1:8080/v1 and your key. The port is published on 127.0.0.1 only; to expose it outside, put it behind a reverse proxy with TLS. The GPU variants of the image, such as server-cuda and server-vulkan, are listed in the llama.cpp Docker documentation; we did not test them here because the machine has no GPU.

FAQ

Which llama.cpp image tag should I use in Docker?

A pinned tag such as ghcr.io/ggml-org/llama.cpp:server-v0.5.0. The unversioned server tag points at the nightly build and moves on its own; with a pinned version you upgrade when you decide to, by changing the tag. In this lab the binary in that image reports build 11146 for Linux aarch64.

Do I need a GPU to run llama-server in Docker?

No. The server image runs on CPU only, and this lab was recorded on an arm64 machine with no GPU using 4 threads (-t 4). A 0.8 B model in Q4_0 answers a short question in under a second. For a GPU there are the server-cuda and server-vulkan variants of the same image.

Why does /health answer without a key if I started llama-server with –api-key-file?

Because llama-server leaves /health open on purpose, so a Docker or reverse-proxy healthcheck can probe the service without knowing the key. Everything else, /v1/models included, returns a 401 with the message Invalid API Key when the Authorization header is missing.

Where does the container keep the model it downloads with -hf?

In the path the LLAMA_CACHE variable points to, with the Hugging Face cache layout (models–user–repository). If that path is a Docker volume, the GGUF survives docker rm and the next container starts without downloading it again.

Cheat sheet

docker pull ghcr.io/ggml-org/llama.cpp:server-v0.5.0

mkdir -p /srv/llama && printf '# clave de prueba del lab\ntu_clave_de_api\n' > /srv/llama/llama_keys && chmod 600 /srv/llama/llama_keys

docker run -d --name llama-server -p 127.0.0.1:8080:8080 -v llama-cache:/cache -e LLAMA_CACHE=/cache -v /srv/llama/llama_keys:/run/secrets/llama_keys:ro ghcr.io/ggml-org/llama.cpp:server-v0.5.0 -hf ggml-org/Qwen3.5-0.8B-GGUF:Q4_0 --alias qwen3.5-0.8b -t 4 -c 4096 --port 8080 --api-key-file /run/secrets/llama_keys

docker rm -f llama-server && docker run -d --name llama-server -p 127.0.0.1:8080:8080 -v llama-cache:/cache -e LLAMA_CACHE=/cache -v /srv/llama/llama_keys:/run/secrets/llama_keys:ro ghcr.io/ggml-org/llama.cpp:server-v0.5.0 -hf ggml-org/Qwen3.5-0.8B-GGUF:Q4_0 --alias qwen3.5-0.8b -t 4 -c 4096 --port 8080 --api-key-file /run/secrets/llama_keys

docker rm -f llama-server && docker volume rm llama-cache && rm -rf /srv/llama

Found it useful? Don't miss the next lab

Subscribe to the newsletter

What this lab does

Why this model and not the one in the API guide

The API key and the cache volume

How to use this API from your code

FAQ

Which llama.cpp image tag should I use in Docker?

Do I need a GPU to run llama-server in Docker?

Why does /health answer without a key if I started llama-server with –api-key-file?

Where does the container keep the model it downloads with -hf?

Cheat sheet