LongEmbed

LongEmbed is a benchmark for testing how well embedding models retrieve from long documents. Most established retrieval benchmarks lean on short passages of at most 512 tokens (the small chunks of text a model reads in one pass), but LongEmbed's documents average more than 5,500 words. That lets it probe what happens when the answer to a query is buried deep inside a lengthy report, article, or contract instead of sitting in a tidy snippet.

It is a benchmark rather than a corpus you would serve in production, so you use it to measure, not to retrieve from. The materials and evaluation code live on GitHub under the MIT licence, which permits commercial use with no attribution requirement, so you can fold it into an internal evaluation harness without friction. It bundles several tasks, a mix of real long-document retrieval sets and synthetic probes that plant a target passage at controlled depths, which shows you whether a model degrades as the relevant text drifts further from the start.

It earns its keep when your RAG system genuinely works with long-form content and you need to know whether your embedding model and chunking strategy hold up: legal, financial, scientific, or documentation-heavy applications where naively truncating at 512 tokens would throw most of the document away. Running it tells you whether to invest in a long-context embedding model, a smarter chunker, or both.

The caveats are the ones common to all benchmarks. A good LongEmbed score measures retrieval on its particular tasks and document mix, which may not mirror your domain, so treat it as one signal beside your own evaluation set rather than a guarantee. It also tests the retrieval stage on its own, not the quality of the answer generated downstream.

Among its neighbours, BEIR remains the broad general-purpose retrieval benchmark and MTEB the wider embedding leaderboard, while reasoning-focused sets like BRIGHT probe a different weakness entirely. Reach for LongEmbed when document length, rather than breadth or reasoning, is the variable you need to pin down.