Local reranking for your RAG with llama.cpp and pgvector, measured

Adding reranking to a pgvector RAG raised nDCG@10 from 0.805 to 0.850 on 300 real Spanish questions, with bge-reranker-v2-m3 served by llama.cpp on CPU. A reranker is a second model that rereads the passages vector search returns and puts them in a new order. I built the whole thing in Docker, with pgvector 0.8.6, llama-server b11206 and the MIRACL evaluation set, and this is how, how much it helps, how many candidates to hand it and which traps I hit on the way. There is also a Spanish version of this test.

Key takeaways

• Reranking pgvector’s top 20 candidates with bge-reranker-v2-m3 raised nDCG@10 from 0.805 to 0.850 and MRR@10 from 0.845 to 0.879, on 300 Spanish MIRACL questions.
• Handing the reranker 20 candidates was enough: with 50, nDCG@10 was 0.847 and the reranker processed 2.7 times more tokens. On CPU with 4 threads, reranking 20 passages took a median 3.04 s with BGE and 9.77 s with Qwen3, against 12.6 ms for the embedding and 14.8 ms for pgvector.
• Qwen3-Reranker-0.6B tied with BGE (0.853 against 0.847 on 162 questions; the gap is indistinguishable from zero).
• On top of PostgreSQL full-text search, the reranker took nDCG@10 from 0.306 to 0.709, but it did not catch up with BGE-M3 without a reranker.
• With pgvector’s default ef_search, a LIMIT 50 through HNSW returns 40 rows; raise it above your candidate count.
• llama-server does not validate top_n: a negative value returns an HTTP 500 (CVE-2026-52132), so do not publish its port.

What a reranker does and why it comes after pgvector

A reranker is a model that takes the question and one passage together and returns a relevance score for that pair. Vector search does something else: it turns the question and the passages into vectors separately and compares distances, so it never reads both texts at once. The bge-reranker-v2-m3 model card^[1] puts it this way: "Different from embedding model, reranker uses question and document as input and directly output similarity instead of embedding".

That joint reading costs one model pass per candidate passage, which is why the reranker does not replace pgvector. The flow has three steps:

1. pgvector returns the N passages closest to the question’s embedding, with N between 10 and 50
2. the reranker scores those N question and passage pairs, and reorders them
3. the top k, between 3 and 10, go into the prompt of the model that writes the answer

The theory and the comparison with services such as Cohere are in the article on re-ranking in RAG. Here I built it on my own machine and measured how much quality goes up with real Spanish questions.

What I built: pgvector 0.8.6 and three llama-server instances

The lab runs in Docker on an 18-core arm64 Linux VM with no GPU, shared with other jobs. Each container gets 4 CPUs and llama-server uses 4 threads (-t 4). These are the pieces, at their versions of 28 September 2026:

pgvector 0.8.6 came out on 29 July 2026, according to its changelog^[2]. The llama.cpp image reports itself with --version: version: 0.5.0-dev (build 11206, commit 2b129ccfa), built for aarch64. If you do not have llama.cpp running yet, the llama.cpp install guide covers Docker, Linux and macOS.

This is the reranker service in compose.yaml. The embeddings one is the same, with the BGE-M3 model and --embedding --pooling cls instead of --reranking:

  rerank:
    image: ghcr.io/ggml-org/llama.cpp:server-b11206
    command: >
      -m /models/bge-reranker-v2-m3-Q8_0.gguf --reranking
      -c 16384 -b 2048 -ub 2048 -np 8 -t 4
      --no-cache-prompt --cache-ram 0
      --host 0.0.0.0 --port 8080
    volumes:
      - ./models:/models:ro
    ports: ["127.0.0.1:24902:8080"]
    cpus: 4
    mem_limit: 4g

-c 16384 with -np 8 leaves 2048 tokens per slot, because llama-server splits the context across slots. In my first attempt, with -c 8192, each slot had 1024, and a 3004-token text came back with a 400: input (3004 tokens) is larger than the max context size (1024 tokens). skipping. The two flags on the next line turn off the prompt cache, which a reranker barely reuses and which, as I explain below, inflates memory. The port is published on 127.0.0.1 only, for the reason in the security section.

The –reranking flag already turns on rank pooling

The llama-server documentation^[3] says the reranking endpoint "Requires a reranker model (such as bge-reranker-v2-m3) and the --embedding --pooling rank options". In build b11206 you do not need them: the --reranking handler in common/arg.cpp^[4] sets embedding = true and pooling_type = LLAMA_POOLING_TYPE_RANK. The container log confirms it when started with --reranking alone: model default pooling_type is [-1], but [4] was specified.

How I built an honest evaluation set

Measuring a reranker takes questions whose answers people have labelled, and I did not make them up. I used MIRACL^[5], a retrieval set over Wikipedia in 18 languages with more than 726,000 relevance judgements by native speakers. Specifically, the Spanish dev split of MTEB’s MIRACLRetrievalHardNegatives^[6], which its card says was created "by pooling the 250 top documents per query from BM25, e5-multilingual-large and e5-mistral-instruct".

That split has 648 questions, 146,750 passages and 2,987 positive judgements. Embedding all 146,750 passages on CPU with 4 threads did not fit the test’s budget, so I built a 12,000-passage subset with this recipe:

• Judged: the 5,636 passages with a human judgement, 2,987 relevant and 2,679 marked not relevant
• Fill: 6,364 random passages (seed 20261028) from the rest of the corpus; each one was in the top 250 of BM25 or e5 for one of these questions, so they are hard distractors too
• Text: title, newline and passage, cut at 2,000 characters; this affects 120 of the 12,000

Each question has between 1 and 10 relevant passages, 4.6 on average. Unjudged passages count as not relevant, which is MIRACL’s convention and penalises every system equally. I measure three things:

• recall@k: the share of relevant passages among the top k
• nDCG@10: rewards having the relevant ones near the top
• MRR@10: the inverse of the rank of the first relevant one

For choosing metrics without fooling yourself, see the guide on evaluating a RAG system.

The pgvector query and the ef_search trap

The first stage is a nearest-neighbour query by cosine distance. The table stores the 1024-dimension vector next to the text, and the HNSW index uses pgvector’s defaults (m = 16, ef_construction = 64):

CREATE TABLE pasajes (
  id        text PRIMARY KEY,
  titulo    text NOT NULL,
  texto     text NOT NULL,
  embedding vector(1024)
);
CREATE INDEX ON pasajes USING hnsw (embedding vector_cosine_ops);

SET hnsw.ef_search = 100;
SELECT id FROM pasajes
ORDER  BY embedding <=> $1::vector
LIMIT  50;

The SET hnsw.ef_search = 100 is not decoration. The pgvector FAQ^[7] warns: "Results are limited by the size of the dynamic candidate list (hnsw.ef_search), which is 40 by default". I checked: with the default, LIMIT 50 returned 40 rows through the HNSW index, and with 100 it returned 50. If your reranker expects 50 candidates, raise ef_search above N or it will silently get fewer.

There was a second surprise. With 12,000 rows, the PostgreSQL planner chose an exact sequential scan over the index, even after an ANALYZE; to measure HNSW I had to force it with SET enable_seqscan = off. At this size the planner is right: the 50-neighbour query took a median 14.8 ms exact and 14.9 ms with HNSW.

With ef_search = 100, HNSW returned the same list of 50 as the exact search on 560 of the 648 questions. The mean overlap was 99.5%, and nDCG@10 dropped only from 0.811 to 0.809. With ef_search = 40 it returned 40 rows on every question but one, which got 41.

How to call /v1/rerank from your code

The endpoint takes the question and a list of texts, and returns indexes with their scores, sorted from highest to lowest. It also answers on /rerank, /reranking and /v1/reranking. This is a real request with three short sentences, saved as peticion.json:

{"query": "¿Cuál es el origen del rap?",
 "top_n": 2,
 "documents": [
  "El rap es un género musical que incorpora rima y habla rítmica.",
  "El precursor más temprano del rap moderno es la tradición griot del oeste de África.",
  "Las raíces del rap rock se remontan a bandas de la old school."]}

Send it with curl and you get the ranking with a score for each index:

curl -s http://127.0.0.1:24902/v1/rerank \
  -H 'Content-Type: application/json' -d @peticion.json

The question asks for the origin of rap. bge-reranker-v2-m3 put the griot passage first, at 2.117; griot is the West African tradition that preceded rap. The rap rock passage came second, at 0.301, and the definition of the genre fell outside top_n.

With the server idle, six repeats gave exactly those figures. While it was serving other requests, the same call returned 0.435 for rap rock and 2.105 for griot. The score shifts with how llama-server groups pairs into batches, although the order did not change here. Qwen3-Reranker-0.6B agreed on the first one, griot at 0.9969, but put the definition second, at 0.9709.

The two scales are not comparable. BGE returns the raw logit (its card says a sigmoid maps it to [0, 1]) and Qwen3 returns a probability already. If you set a cut-off threshold, set it per model.

The other figure in the response is usage.prompt_tokens: 86 tokens with BGE and 301 with Qwen3 for the same request. The gap comes from the Qwen3 GGUF wrapping each pair in a chat template with the instruction "Given a web search query, retrieve relevant passages that answer the query".

The whole flow in Python fits in one function. It takes the question vector, asks pgvector for 50 candidates and returns the best k according to the reranker:

def buscar(conn, pregunta, qvec, n=50, k=5):
    with conn.cursor() as cur:
        cur.execute("SET hnsw.ef_search = 100")
        cur.execute(
            "SELECT id, titulo || E'\\n' || texto FROM pasajes "
            "ORDER BY embedding <=> %s::vector LIMIT %s",
            (str(qvec), n))
        filas = cur.fetchall()
    r = requests.post(RERANK_URL, timeout=60, json={
        "query": pregunta,
        "documents": [t[:2000] for _, t in filas]})
    r.raise_for_status()
    orden = r.json()["results"][:k]
    return [filas[x["index"]][0] for x in orden]

How much reranking improves quality

With bge-reranker-v2-m3 reordering pgvector’s top 20 candidates, nDCG@10 rose from 0.805 to 0.850 on 300 questions picked at random from the 648. The mean gain is 0.045, with a 95% confidence interval from 0.028 to 0.062 (paired per-question bootstrap, 10,000 resamples):

The first result was relevant for 76% of the questions without a reranker and 80% with it. recall@5 never reaches 1 because some questions have up to 10 relevant passages.

The mean hides that the reranker also makes some answers worse. Out of 300 questions, nDCG@10 improved on 165, got worse on 87 and stayed the same on 48.

A clear improvement is "¿Por qué David Bowie usaba maquillaje?" (why did David Bowie wear make-up). BGE-M3 filled the top places with passages from Bowie’s biography that do not mention make-up. It left the Ziggy Stardust passage, which says Bowie drew on kabuki make-up, at rank 11, and the glam rock one at 16. The reranker moved them to second and first place.

Qwen3-Reranker-0.6B gave an equivalent result. I evaluated it on the first 162 questions of the same sample, because it processes more tokens per passage and did not reach 300 within the test’s time. On those 162, with 20 candidates:

Qwen3’s edge is 0.007 in nDCG@10, with a 95% interval from -0.015 to 0.029: indistinguishable from zero. Qwen3 won on 66 questions and BGE on 65. With this data, the choice between them comes down to other things, such as the memory and tokens each one costs, covered below.

How many candidates to hand the reranker

Handing the reranker 20 candidates is enough on this set. With 10, nDCG@10 stays at 0.835; with 20 it reaches 0.850, and with 30 or 50 it stays at 0.848 and 0.847.

The reason is the first stage. BGE-M3 already puts 91.3% of the relevant passages in its top 10 and 99.7% in its top 50. Below rank 20 there is almost nothing left to rescue.

The chart shows the three curves on the 151 questions that every configuration evaluated, so its figures differ slightly from the tables:

With 10 candidates, recall@10 cannot change, because the reranker only reorders the same 10 passages; all the gain comes from placing the relevant ones better. With 20 it can pull passages up from ranks 11 to 20, and recall@10 rises from 0.913 to 0.944.

This figure depends on your first stage. If your embeddings retrieve worse, as in a technical corpus full of jargon the model never saw, the relevant passages sit lower and handing over more candidates pays off. Measure your first stage’s recall@50 before picking N: if it is already close to 1, a small N is enough.

Is it worth more than hybrid search with BM25?

Not in this test. Hybrid search combines keyword search with vector search. I built it from what PostgreSQL already ships: a tsvector column with the spanish configuration, a GIN index and a query that ORs the question’s lexemes together, ordered by ts_rank_cd. That full-text search (FTS) on its own reached an nDCG@10 of 0.309 on the 648 questions.

Fusing both lists with Reciprocal Rank Fusion (RRF, with k = 60) gave 0.642, below the 0.811 of BGE-M3 alone. The FTS was weak enough to add noise to the good list. A proper BM25 with better-built queries would change this result, so read it as a warning about PostgreSQL FTS as it comes, not about every hybrid search.

The reranker does rescue a weak first stage. On 151 questions, reranking the FTS top 50 raised its nDCG@10 from 0.306 to 0.709 and its MRR@10 from 0.404 to 0.829. The first result went from relevant on 24.5% of the questions to 76.2%. For "¿En qué países es bitcoin una moneda oficial?" (in which countries is bitcoin legal tender), the right passage jumped from rank 43 to first.

Even so, FTS plus reranker stayed below BGE-M3 without a reranker, 0.709 against 0.796 on those same questions. The first stage sets the ceiling: the FTS only had 75.7% of the relevant passages in its top 50, and the reranker cannot lift what it never receives. Here depth does matter: with FTS, every step from 10 to 20, 30 and 50 candidates kept improving nDCG@10.

What reranking 50 passages costs in CPU

Reranking is the expensive part of the flow, and the usage.prompt_tokens field in each response lets you count it without a stopwatch. On a sample of 20 questions, embedding the question cost 14 tokens on average. Reranking its 20 candidates with bge-reranker-v2-m3 cost 3,420 tokens, and 50 candidates cost 9,066: about 240 and 640 times the work of the embedding.

Qwen3-Reranker-0.6B adds the chat template to every pair, about 72 extra tokens per passage in the sample request (301 against 86 with three sentences). With short passages, that template can double the tokens the reranker processes.

This is the practical case for picking N from the curve above. Going from 20 to 50 candidates multiplied the reranker’s tokens by 2.7 and did not improve nDCG@10.

In time, the reranker weighs more than a hundred times the other two stages. I timed 40 questions in sequence, three times, with each server capped at 4 threads and prompt caching turned off, and report the median of the three runs. The one-minute load average was between 2.55 and 2.95 when each run started:

On 4 CPU threads, reranking 20 passages with BGE added 3 s per question, about 110 times what the embedding and pgvector took together. Qwen3 was 3.2 times slower than BGE for the same quality. If that delay does not fit your budget, lower N, trim the passages or move the reranker to a GPU.

Security: negative top_n and a port you should not publish

llama-server does not validate top_n. The code in server-common.cpp^[8] sorts the results and calls resize with the minimum of top_n and the document count. It never checks the sign. CVE-2026-52132^[9], published on 1 September 2026, describes a denial of service "via a negative top_n value in a POST request to /rerank" up to commit 97f06e9.

On build b11206 I tried "top_n": -1. The response was an HTTP 500 with the message vector::_M_default_append, and the server kept answering 200 on /health. I did not reproduce a crash, but the input is still unvalidated, so do three things:

• validate top_n in your application before forwarding it
• publish the port on 127.0.0.1 only, or on an internal Docker network
• update the llama.cpp image: on 28 September I used b11206 and the release feed already had b11223, from the day before

Qwen3-Reranker did not fit in 4 GiB

I gave Qwen3-Reranker-0.6B the same settings as BGE (-c 16384, -ub 2048, 8 slots) and a 4 GiB limit. The container died a few minutes into the load, flagged by Docker as OOMKilled with exit code 137.

I restarted it with 8 GiB. Even then its memory kept growing until it neared the new limit, because llama-server keeps a prompt cache of up to 8192 MiB by default (--cache-ram). In a reranker every request is different and that cache adds little, so I turned it off with --cache-ram 0 and the memory stopped growing.

During the latency runs, with the cache off on all three servers, Qwen3 peaked at 4.96 GiB and settled at 4.5 GiB when idle. BGE peaked at 2.99 GiB and the embeddings server at 1.06 GiB.

Limits of this measurement

These results come from a single dataset, and they should be read with their conditions:

• Domain: factual questions over Spanish Wikipedia; your internal documents may behave differently
• Possible domain advantage: the BGE-M3 card thanks MIRACL among its open datasets, so the BGE family may have seen similar data; that is why the relative gain matters more than the absolute figure
• Reduced corpus: 12,000 passages, not the millions of a real deployment; with more passages the first stage fails more and the reranker has more to fix
• Incomplete labels: an unjudged relevant passage counts as a miss
• Quantisation: GGUF Q8_0, not the original weights
• Sample: reranking was evaluated on 300 of the 648 questions, picked at random, because of the CPU budget, and Qwen3 and FTS on the first 162 and 151 of that same sample
• Noisy scores: the scores were computed while the server handled parallel requests, and they vary slightly with the batch

Frequently asked questions

Do I need a GPU to run a local reranker?

Not to try it, although latency matters in production. This whole lab ran on arm64 CPU with 4 threads per server, and bge-reranker-v2-m3 took a median 3.04 s to rerank 20 passages. The three Q8_0 models take between 634 and 640 MB on disk each.

Which reranker should I pick for Spanish documents?

In this test with Spanish questions, bge-reranker-v2-m3 and Qwen3-Reranker-0.6B tied on quality. With 20 candidates they scored 0.847 and 0.853 nDCG@10, a gap within the margin of error. BGE was 3.2 times faster with 20 candidates and fitted in a 4 GiB container, so it is the cheaper starting point. Qwen3 lets you change the instruction, but in llama-server it is fixed inside the GGUF’s template.

Can I use the same llama-server for embeddings and reranking?

Not in the same instance: --reranking sets pooling to rank, and BGE-M3 embeddings need cls. Run two containers from the same image, each with its own model, as in the compose.yaml above.

Conclusion

A local reranker improves a pgvector RAG, and in this test the gain was measurable but moderate. It added 4.5 points of nDCG@10 on top of a first stage that was already good, with BGE-M3 retrieving 91% of the relevant passages in its top 10. Where it shines is when the first stage fails: on PostgreSQL FTS, the same reranker multiplied nDCG@10 by 2.3.

The recipe I take from this has four steps:

1. first measure your vector search’s recall@50 with your own questions; if it is close to 1, hand over 20 candidates and no more
2. start with bge-reranker-v2-m3 in llama-server, with --reranking and a container that does not publish its port
3. raise hnsw.ef_search above the number of candidates
4. validate top_n in your code and keep the llama.cpp image up to date

The next step is to build your own set of labelled questions, even just 50, and repeat the measurement on your documents. To take it to production, the guide to RAG with Postgres and pgvector in production covers the rest of the way.

Sources: [1] llama-server README^[3], [2] llama.cpp common/arg.cpp^[4], [3] llama.cpp server-common.cpp^[8], [4] CVE-2026-52132^[9], [5] bge-reranker-v2-m3 model card^[1], [6] BGE-M3 model card^[10], [7] Qwen3-Reranker-0.6B model card^[11], [8] pgvector^[7], [9] pgvector changelog^[2], [10] MIRACL^[5], [11] MTEB MIRACLRetrievalHardNegatives^[6].

Sources

1. bge-reranker-v2-m3 model card
2. changelog
3. llama-server documentation
4. common/arg.cpp
5. MIRACL
6. MTEB’s MIRACLRetrievalHardNegatives
7. pgvector FAQ
8. server-common.cpp
9. CVE-2026-52132
10. BGE-M3 model card
11. Qwen3-Reranker-0.6B model card