MixedbreadGitHub

BREAD

The Benchmark for Retrieval Evaluation Across Domains. A representative view of how good any retriever is at straightforward relevance, from law to finance. A slot-in replacement for BEIR.

Documents
27.4M
Queries
8,925
Datasets
21
Categories
8

How scoring works

nDCG@10
All reported metrics are nDCG@10 (or RCP-nDCG@10 in the case of RCP-BREAD-SLICE), measuring how well each model retrieves and ranks its top ten documents.
Equal categories
Overall, the headline metric, is reported as the average of the 8 categories, so no model can dominate the board without meaningful generalisation.
Rerankers
All rerankers are evaluated in the same, fair setting, to avoid the "which BM25" effect: a pool of 100 candidates obtained by reciprocal rank fusion of 5 strong models (4 on obliq-congress, 2 on trec-tot) is provided.

Leaderboard

Every query over each dataset's full corpus, the most complete variant. All 21 datasets are processed with a comprehensive dense annotation pipeline.

BREAD leaderboard
#Model
1Mixedbread Wholembed v4 (preview) · Proprietary79.1678.392.571.875.187.945.888.094.0
2pplx-embed-v2-late-9b70.9876.082.458.168.480.631.884.386.3
3gemini-embedding-2 · Proprietary69.4479.185.659.669.373.825.776.685.9
4zembed-167.7270.687.760.859.777.522.774.288.7
5Cohere Embed 5 Pro · Proprietary66.8370.985.352.165.379.418.875.987.1
6EVIE-8B66.7973.774.050.158.880.234.082.181.5
7voyage-4-large · Proprietary65.4372.682.449.862.077.016.675.087.9
8pplx-embed-v2-late-0.6b65.2073.379.847.660.177.719.879.484.0
9Mixedbread Wholembed v3 · Proprietary64.4168.279.449.260.477.720.674.585.3
10Octen-Embedding-8B61.8170.383.146.356.467.515.669.486.0
11Nemotron-3.5-Embed-1B61.6173.176.049.954.772.414.571.281.1
12Qwen3-Embedding-8B61.4570.683.346.756.965.616.766.985.0
13NV-Embed-v260.7976.983.947.155.369.56.263.184.4
14LateOn60.7473.680.538.047.171.720.571.583.0
15Linq-Embed-Mistral60.0973.382.041.649.268.815.965.184.8
16Qwen3-Embedding-4B59.3768.481.946.053.163.512.765.783.6
17pplx-embed-v1-4b59.1667.880.947.051.468.014.164.579.7
18voyage-4-nano58.7163.871.443.155.272.315.468.180.5
19gte-Qwen2-1.5B-instruct57.5866.382.238.149.066.412.464.382.0
20pplx-embed-v1-late-0.6b57.3272.679.237.145.566.913.662.581.3
21ColBERT-Zero57.2171.981.835.143.764.815.264.081.2
22F2LLM-v2-4B57.2069.083.741.551.265.810.264.671.7
23jina-embeddings-v5-text-small57.1963.080.041.149.766.213.364.579.8
24text-embedding-3-large · Proprietary56.6669.680.740.149.765.77.858.681.0
25pplx-embed-v1-0.6b56.4967.080.638.646.766.38.865.179.0
26e5-mistral-7b-instruct56.1368.481.635.045.465.411.160.581.6
27EmbeddingGemma-300m55.8966.681.136.048.264.012.460.578.3
28harrier-oss-v1-0.6b55.5860.380.036.944.466.313.364.079.5
29snowflake-arctic-embed-m-v2.054.9965.178.035.045.964.811.761.777.6
30DenseOn54.9468.279.735.143.560.210.062.880.0
31linkup-sparseup-embed-v154.9168.775.734.142.165.911.764.776.4
32snowflake-arctic-embed-l-v2.054.7064.779.836.245.264.19.761.576.4
33GTE-ModernColBERT-v154.3871.882.431.343.058.28.758.781.0
34mxbai-edge-colbert-v0-32m53.7266.579.732.837.566.610.260.276.3
35opensearch-neural-sparse-encoding-doc-v3-gte52.7468.377.236.232.863.88.063.172.4
36modernbert-embed-base52.3862.578.133.839.459.48.859.677.3
37SPLADE-v351.8168.280.535.633.660.57.354.973.9
38gte-large-en-v1.551.6162.378.031.838.058.710.557.576.1
39e5-large-v251.1969.380.431.036.557.17.750.577.0
40harrier-oss-v1-270m51.0558.176.830.537.061.29.358.377.3
41mxbai-embed-large-v150.9364.678.232.036.856.110.552.177.1
42Qwen3-Embedding-0.6B50.8859.074.732.942.856.08.057.476.1
43bge-large-en-v1.550.4165.578.830.135.555.19.352.976.1
44nomic-embed-text-v1.550.3465.378.528.934.257.27.856.874.0
45bge-m349.8062.976.131.238.757.24.256.871.3
46gte-modernbert-base49.7465.275.827.736.656.16.155.474.9
47granite-embedding-english-r249.6563.267.927.537.857.87.460.175.4
48ColBERTv249.2765.082.231.731.955.03.551.873.1
49e5-base-v249.2564.778.831.035.154.36.148.275.9
50multilingual-e5-large-instruct48.9260.677.328.036.454.94.751.378.0
51bge-base-en-v1.547.9463.177.028.832.152.26.048.675.7
52gte-base47.4358.675.929.233.353.47.346.275.4
53bge-small-en-v1.546.7859.876.929.031.051.06.847.072.5
54granite-embedding-small-english-r246.5761.961.427.032.455.04.258.472.3
55RetroMAE (BEIR)45.9560.775.828.227.853.36.046.069.8
56Contriever (MS MARCO)45.2260.375.628.826.753.75.843.867.2
57BM2541.0543.552.424.822.656.56.959.462.2
58all-MiniLM-L6-v240.3553.371.027.128.943.73.636.758.5

Submit your model

Submissions are pull requests on mixedbread-ai/bread-benchmark. It's a very straightforward process, and our library is designed to be compatible with all models, both open and closed.

The submission guide here covers all you need to know.

  1. 1

    Install bread

    Python 3.11 or newer. The dense extra runs sentence-transformers models; ColBERT, sparse, BM25 and hosted APIs have extras of their own.

    pip install "bread-benchmark[dense] @ git+https://github.com/mixedbread-ai/bread-benchmark@v1.0.0"
  2. 2

    Run your model

    One bundle holds both benchmarks. Rerankers use bread rerank, which reorders the same top 100 for every query. BREAD-SLICE alone runs on a laptop; all of BREAD is a GPU job.

    bread run -m my-org/my-model -t BREAD -t BREAD-SLICE
  3. 3

    Validate the bundle

    The run writes it to results/my-org__my-model. This is the check we run: it scores every run again from scratch.

    bread validate results/my-org__my-model
  4. 4

    Open a pull request

    Add the bundle to submissions/ in your fork. It shows up here as pending review within ten minutes and is ranked once we merge it.

    gh repo fork mixedbread-ai/bread-benchmark --clonecp -r results/my-org__my-model bread-benchmark/submissions/cd bread-benchmarkgit add submissions/my-org__my-modelgit commit -m "Submit my-org/my-model"git pushgh pr create --fill