FreshStack
FreshStack is a reusable framework and benchmark for evaluating information retrieval over fast-moving technical documentation, built by Nandan Thakur and Jimmy Lin at the University of Waterloo with collaborators at Databricks, and presented at NeurIPS 2025 (Datasets and Benchmarks Track). It assembles evaluation sets automatically: it collects a corpus of chunked code snippets and technical documents from public GitHub repositories, then uses real, community-asked Stack Overflow questions and answers to generate nuggets and judge which document chunks support each answer. The October 2024 release covers five recent, niche domains where models are unlikely to have memorised the answers: LangChain, YOLO, Laravel, Angular and Godot.
You access it through the freshstack pip package (Python 3.10 and above) and the datasets on Hugging Face (the freshstack organisation, including the queries-oct-2024 and corpus-oct-2024 sets). Documents are distributed in BEIR format, so if you have run BEIR or similar suites the workflow will feel familiar: index the corpus, run your retriever over the queries, and score how often genuinely relevant chunks are returned. It supports dense and multi-vector evaluation and caps passages at 2048 tokens.
FreshStack is a measuring tape, not a knowledge base. It is best for choosing an embedding model or reranker before building RAG over code and evolving libraries, where keyword overlap is weak and recency matters. The authors report that off-the-shelf retrievers fall well short of oracle performance on all five domains, so it is a genuinely hard diagnostic. Alongside our sibling benchmarks, reach for it where BRIGHT tests reasoning-heavy retrieval and CoIR tests code retrieval broadly.
On licensing, the evaluation code is Apache 2.0 and the datasets are CC BY-SA 4.0, so commercial use is permitted but you must attribute and keep derivative corpora under the same share-alike licence. Note too that the underlying Stack Overflow content and GitHub repositories carry their own terms, so check those before redistributing the source material.
Related sources
BEIR
A collection of 18 information retrieval datasets spanning ad-hoc web search, question answering, fact verification, and duplicate question retrieval. Built by aggregating existing datasets, some originally created for other tasks and converted to retrieval format. Tests how well retrieval models generalise to unseen domains without fine-tuning, which matters for real-world RAG.
BRIGHT
A reasoning-intensive retrieval benchmark. It requires genuine reasoning to connect queries with relevant documents rather than keyword or semantic overlap, revealing model weaknesses that BEIR misses.
CoIR (Code Retrieval Benchmark)
A code retrieval benchmark spanning diverse programming tasks and languages. Built for evaluating how well models retrieve relevant source code, which matters if you are building RAG over codebases.
LegalBench-RAG
The first open benchmark for the retrieval step of legal RAG. It offers 6,858 human-annotated query-answer pairs over a corpus of roughly 79 million characters, with character-level ground-truth spans across contracts and privacy policies, assembled from CUAD, MAUD, ContractNLI and PrivacyQA.