BREAD
The Benchmark for Retrieval Evaluation Across Domains. A representative view of how good any retriever is at straightforward relevance, from law to finance. A slot-in replacement for BEIR.
- Documents
- 27.4M
- Queries
- 8,925
- Datasets
- 21
- Categories
- 8
How scoring works
- nDCG@10
- All reported metrics are nDCG@10 (or RCP-nDCG@10 in the case of RCP-BREAD-SLICE), measuring how well each model retrieves and ranks its top ten documents.
- Equal categories
- Overall, the headline metric, is reported as the average of the 8 categories, so no model can dominate the board without meaningful generalisation.
- Rerankers
- All rerankers are evaluated in the same, fair setting, to avoid the "which BM25" effect: a pool of 100 candidates obtained by reciprocal rank fusion of 5 strong models (4 on obliq-congress, 2 on trec-tot) is provided.
Leaderboard
Every query over each dataset's full corpus, the most complete variant. All 21 datasets are processed with a comprehensive dense annotation pipeline.
| # | Model | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Mixedbread Wholembed v4 (preview)Mixedbread · · Proprietary | 79.16 | 78.3 | 92.5 | 71.8 | 75.1 | 87.9 | 45.8 | 88.0 | 94.0 |
| 2 | pplx-embed-v2-late-9bPerplexity · | 70.98 | 76.0 | 82.4 | 58.1 | 68.4 | 80.6 | 31.8 | 84.3 | 86.3 |
| 3 | gemini-embedding-2Google · · Proprietary | 69.44 | 79.1 | 85.6 | 59.6 | 69.3 | 73.8 | 25.7 | 76.6 | 85.9 |
| 4 | zembed-1ZeroEntropy · | 67.72 | 70.6 | 87.7 | 60.8 | 59.7 | 77.5 | 22.7 | 74.2 | 88.7 |
| 5 | Cohere Embed 5 ProCohere · · Proprietary | 66.83 | 70.9 | 85.3 | 52.1 | 65.3 | 79.4 | 18.8 | 75.9 | 87.1 |
| 6 | EVIE-8BTencent · | 66.79 | 73.7 | 74.0 | 50.1 | 58.8 | 80.2 | 34.0 | 82.1 | 81.5 |
| 7 | voyage-4-largeVoyage AI · · Proprietary | 65.43 | 72.6 | 82.4 | 49.8 | 62.0 | 77.0 | 16.6 | 75.0 | 87.9 |
| 8 | pplx-embed-v2-late-0.6bPerplexity · | 65.20 | 73.3 | 79.8 | 47.6 | 60.1 | 77.7 | 19.8 | 79.4 | 84.0 |
| 9 | Mixedbread Wholembed v3Mixedbread · · Proprietary | 64.41 | 68.2 | 79.4 | 49.2 | 60.4 | 77.7 | 20.6 | 74.5 | 85.3 |
| 10 | Octen-Embedding-8BOcten · | 61.81 | 70.3 | 83.1 | 46.3 | 56.4 | 67.5 | 15.6 | 69.4 | 86.0 |
| 11 | Nemotron-3.5-Embed-1BNVIDIA · | 61.61 | 73.1 | 76.0 | 49.9 | 54.7 | 72.4 | 14.5 | 71.2 | 81.1 |
| 12 | Qwen3-Embedding-8BQwen · | 61.45 | 70.6 | 83.3 | 46.7 | 56.9 | 65.6 | 16.7 | 66.9 | 85.0 |
| 13 | NV-Embed-v2NVIDIA · | 60.79 | 76.9 | 83.9 | 47.1 | 55.3 | 69.5 | 6.2 | 63.1 | 84.4 |
| 14 | LateOnLightOn · | 60.74 | 73.6 | 80.5 | 38.0 | 47.1 | 71.7 | 20.5 | 71.5 | 83.0 |
| 15 | Linq-Embed-MistralLinq · | 60.09 | 73.3 | 82.0 | 41.6 | 49.2 | 68.8 | 15.9 | 65.1 | 84.8 |
| 16 | Qwen3-Embedding-4BQwen · | 59.37 | 68.4 | 81.9 | 46.0 | 53.1 | 63.5 | 12.7 | 65.7 | 83.6 |
| 17 | pplx-embed-v1-4bPerplexity · | 59.16 | 67.8 | 80.9 | 47.0 | 51.4 | 68.0 | 14.1 | 64.5 | 79.7 |
| 18 | voyage-4-nanoVoyage AI · | 58.71 | 63.8 | 71.4 | 43.1 | 55.2 | 72.3 | 15.4 | 68.1 | 80.5 |
| 19 | gte-Qwen2-1.5B-instructAlibaba · | 57.58 | 66.3 | 82.2 | 38.1 | 49.0 | 66.4 | 12.4 | 64.3 | 82.0 |
| 20 | pplx-embed-v1-late-0.6bPerplexity · | 57.32 | 72.6 | 79.2 | 37.1 | 45.5 | 66.9 | 13.6 | 62.5 | 81.3 |
| 21 | ColBERT-ZeroLightOn · | 57.21 | 71.9 | 81.8 | 35.1 | 43.7 | 64.8 | 15.2 | 64.0 | 81.2 |
| 22 | F2LLM-v2-4BCodeFuse · | 57.20 | 69.0 | 83.7 | 41.5 | 51.2 | 65.8 | 10.2 | 64.6 | 71.7 |
| 23 | jina-embeddings-v5-text-smallJina AI · | 57.19 | 63.0 | 80.0 | 41.1 | 49.7 | 66.2 | 13.3 | 64.5 | 79.8 |
| 24 | text-embedding-3-largeOpenAI · · Proprietary | 56.66 | 69.6 | 80.7 | 40.1 | 49.7 | 65.7 | 7.8 | 58.6 | 81.0 |
| 25 | pplx-embed-v1-0.6bPerplexity · | 56.49 | 67.0 | 80.6 | 38.6 | 46.7 | 66.3 | 8.8 | 65.1 | 79.0 |
| 26 | e5-mistral-7b-instructMicrosoft · | 56.13 | 68.4 | 81.6 | 35.0 | 45.4 | 65.4 | 11.1 | 60.5 | 81.6 |
| 27 | EmbeddingGemma-300mGoogle · | 55.89 | 66.6 | 81.1 | 36.0 | 48.2 | 64.0 | 12.4 | 60.5 | 78.3 |
| 28 | harrier-oss-v1-0.6bMicrosoft · | 55.58 | 60.3 | 80.0 | 36.9 | 44.4 | 66.3 | 13.3 | 64.0 | 79.5 |
| 29 | snowflake-arctic-embed-m-v2.0Snowflake · | 54.99 | 65.1 | 78.0 | 35.0 | 45.9 | 64.8 | 11.7 | 61.7 | 77.6 |
| 30 | DenseOnLightOn · | 54.94 | 68.2 | 79.7 | 35.1 | 43.5 | 60.2 | 10.0 | 62.8 | 80.0 |
| 31 | linkup-sparseup-embed-v1Linkup · | 54.91 | 68.7 | 75.7 | 34.1 | 42.1 | 65.9 | 11.7 | 64.7 | 76.4 |
| 32 | snowflake-arctic-embed-l-v2.0Snowflake · | 54.70 | 64.7 | 79.8 | 36.2 | 45.2 | 64.1 | 9.7 | 61.5 | 76.4 |
| 33 | GTE-ModernColBERT-v1LightOn · | 54.38 | 71.8 | 82.4 | 31.3 | 43.0 | 58.2 | 8.7 | 58.7 | 81.0 |
| 34 | mxbai-edge-colbert-v0-32mMixedbread · | 53.72 | 66.5 | 79.7 | 32.8 | 37.5 | 66.6 | 10.2 | 60.2 | 76.3 |
| 35 | opensearch-neural-sparse-encoding-doc-v3-gteOpenSearch · | 52.74 | 68.3 | 77.2 | 36.2 | 32.8 | 63.8 | 8.0 | 63.1 | 72.4 |
| 36 | modernbert-embed-baseNomic · | 52.38 | 62.5 | 78.1 | 33.8 | 39.4 | 59.4 | 8.8 | 59.6 | 77.3 |
| 37 | SPLADE-v3Naver · | 51.81 | 68.2 | 80.5 | 35.6 | 33.6 | 60.5 | 7.3 | 54.9 | 73.9 |
| 38 | gte-large-en-v1.5Alibaba · | 51.61 | 62.3 | 78.0 | 31.8 | 38.0 | 58.7 | 10.5 | 57.5 | 76.1 |
| 39 | e5-large-v2Microsoft · | 51.19 | 69.3 | 80.4 | 31.0 | 36.5 | 57.1 | 7.7 | 50.5 | 77.0 |
| 40 | harrier-oss-v1-270mMicrosoft · | 51.05 | 58.1 | 76.8 | 30.5 | 37.0 | 61.2 | 9.3 | 58.3 | 77.3 |
| 41 | mxbai-embed-large-v1Mixedbread · | 50.93 | 64.6 | 78.2 | 32.0 | 36.8 | 56.1 | 10.5 | 52.1 | 77.1 |
| 42 | Qwen3-Embedding-0.6BQwen · | 50.88 | 59.0 | 74.7 | 32.9 | 42.8 | 56.0 | 8.0 | 57.4 | 76.1 |
| 43 | bge-large-en-v1.5BAAI · | 50.41 | 65.5 | 78.8 | 30.1 | 35.5 | 55.1 | 9.3 | 52.9 | 76.1 |
| 44 | nomic-embed-text-v1.5Nomic · | 50.34 | 65.3 | 78.5 | 28.9 | 34.2 | 57.2 | 7.8 | 56.8 | 74.0 |
| 45 | bge-m3BAAI · | 49.80 | 62.9 | 76.1 | 31.2 | 38.7 | 57.2 | 4.2 | 56.8 | 71.3 |
| 46 | gte-modernbert-baseAlibaba · | 49.74 | 65.2 | 75.8 | 27.7 | 36.6 | 56.1 | 6.1 | 55.4 | 74.9 |
| 47 | granite-embedding-english-r2IBM · | 49.65 | 63.2 | 67.9 | 27.5 | 37.8 | 57.8 | 7.4 | 60.1 | 75.4 |
| 48 | ColBERTv2Stanford · | 49.27 | 65.0 | 82.2 | 31.7 | 31.9 | 55.0 | 3.5 | 51.8 | 73.1 |
| 49 | e5-base-v2Microsoft · | 49.25 | 64.7 | 78.8 | 31.0 | 35.1 | 54.3 | 6.1 | 48.2 | 75.9 |
| 50 | multilingual-e5-large-instructMicrosoft · | 48.92 | 60.6 | 77.3 | 28.0 | 36.4 | 54.9 | 4.7 | 51.3 | 78.0 |
| 51 | bge-base-en-v1.5BAAI · | 47.94 | 63.1 | 77.0 | 28.8 | 32.1 | 52.2 | 6.0 | 48.6 | 75.7 |
| 52 | gte-baseAlibaba · | 47.43 | 58.6 | 75.9 | 29.2 | 33.3 | 53.4 | 7.3 | 46.2 | 75.4 |
| 53 | bge-small-en-v1.5BAAI · | 46.78 | 59.8 | 76.9 | 29.0 | 31.0 | 51.0 | 6.8 | 47.0 | 72.5 |
| 54 | granite-embedding-small-english-r2IBM · | 46.57 | 61.9 | 61.4 | 27.0 | 32.4 | 55.0 | 4.2 | 58.4 | 72.3 |
| 55 | RetroMAE (BEIR)BAAI · | 45.95 | 60.7 | 75.8 | 28.2 | 27.8 | 53.3 | 6.0 | 46.0 | 69.8 |
| 56 | Contriever (MS MARCO)Meta · | 45.22 | 60.3 | 75.6 | 28.8 | 26.7 | 53.7 | 5.8 | 43.8 | 67.2 |
| 57 | BM25 | 41.05 | 43.5 | 52.4 | 24.8 | 22.6 | 56.5 | 6.9 | 59.4 | 62.2 |
| 58 | all-MiniLM-L6-v2Sentence Transformers · | 40.35 | 53.3 | 71.0 | 27.1 | 28.9 | 43.7 | 3.6 | 36.7 | 58.5 |
Submit your model
Submissions are pull requests on mixedbread-ai/bread-benchmark. It's a very straightforward process, and our library is designed to be compatible with all models, both open and closed.
The submission guide here covers all you need to know.
- 1
Install bread
Python 3.11 or newer. The
denseextra runs sentence-transformers models; ColBERT, sparse, BM25 and hosted APIs have extras of their own.pip install "bread-benchmark[dense] @ git+https://github.com/mixedbread-ai/bread-benchmark@v1.0.0" - 2
Run your model
One bundle holds both benchmarks. Rerankers use
bread rerank, which reorders the same top 100 for every query. BREAD-SLICE alone runs on a laptop; all of BREAD is a GPU job.bread run -m my-org/my-model -t BREAD -t BREAD-SLICE - 3
Validate the bundle
The run writes it to
results/my-org__my-model. This is the check we run: it scores every run again from scratch.bread validate results/my-org__my-model - 4
Open a pull request
Add the bundle to
submissions/in your fork. It shows up here as pending review within ten minutes and is ranked once we merge it.gh repo fork mixedbread-ai/bread-benchmark --clonecp -r results/my-org__my-model bread-benchmark/submissions/cd bread-benchmarkgit add submissions/my-org__my-modelgit commit -m "Submit my-org/my-model"git pushgh pr create --fill