Open datasets for fine-tuning LLMs
Fine-tuning needs example data, not a web corpus. A practical tour of open instruction, reasoning, and tool-use datasets for fine-tuning LLMs, with the commercial-licence catch that trips people up.
Fine-tuning is how you change the way a model behaves: its tone, its format, its skill at a specific task like following instructions, reasoning step by step, or calling tools. It is a different job from both pretraining (which builds the model’s general language ability from a huge web corpus) and from Retrieval-Augmented Generation (which leaves the model alone and feeds it documents at query time). Each needs a different kind of data, and the datasets for fine-tuning LLMs are the most specific of the three: not raw text, but worked examples of the behaviour you want. This post is a tour of the open ones, organised by what they teach.
If you are still deciding whether you even need fine-tuning, our What is RAG? guide has a short rule of thumb: reach for RAG to change what a model knows, and fine-tuning to change how it behaves. Plenty of production systems use both.
Instruction and supervised fine-tuning (SFT)
The workhorse of fine-tuning is supervised fine-tuning: a dataset of prompt-and-response pairs that shows the model how to answer. Give it tens of thousands of well-written instructions with good answers and it learns the shape of being a helpful assistant.
The strongest fully-open option in the directory is NVIDIA’s Nemotron Post-Training Dataset v2, a large synthetic SFT and RL set spanning maths, code, STEM, dialogue, and instruction-following across several languages. For a broad, provenance-traceable synthetic corpus with built-in RAG, extraction, and question-answering exercises, SYNTH from Pleias is purpose-built for training grounded, citeable models.
Reasoning and chain-of-thought
A newer and very active strand is teaching models to reason, by fine-tuning on worked solutions that show the steps, not just the final answer. Maths is the proving ground because answers are checkable.
OpenThoughts3 is the standout: over a million reasoning examples across maths, code, and science, each with a chain-of-thought answer, and it is Apache-licensed. NuminaMath pairs hundreds of thousands of competition problems with worked solutions, and Big-Math-RL-Verified filters maths problems to a single verifiable answer with a per-problem difficulty signal, which is useful when you want to grade the training mix. For reach into olympiad difficulty, Omni-MATH and MegaMath extend the range.
Tool-calling and agentic behaviour
If you want a model that uses tools (calling functions and APIs, not just chatting), you fine-tune it on examples of exactly that. This is one of the fastest-moving corners, and the directory now has a dedicated agentic and tool-use category.
Toucan-1.5M is the largest open set, 1.5 million tool-use trajectories synthesised from real Model Context Protocol servers. xLAM Function-Calling offers 60,000 verified function-calling examples, and Glaive Function Calling v2 is a popular, permissively-licensed set of multi-turn conversations that include tool calls. To evaluate the result rather than train on it, the Berkeley Function-Calling Leaderboard is the standard.
Teaching pedagogy (for tutors)
A specialised but growing case: fine-tuning a model to teach rather than just answer. MathDial is a set of one-to-one maths tutoring dialogues annotated with teacher moves, and MathTutorBench benchmarks a tutor’s scaffolding and Socratic questioning. Both are far more useful for a tutoring product than a generic instruction set.
The licence catch (read this before you train)
Here is the trap that catches people. Fine-tuning data is often released for research only, and training a commercial model on it is exactly the use a non-commercial licence forbids. A dataset being free to download tells you nothing about whether you can ship a model trained on it.
The commercially-clean options above, safe to train and deploy with attribution or share-alike, include OpenThoughts3 and NuminaMath (Apache 2.0), Toucan and Glaive Function Calling (Apache 2.0), Nemotron Post-Training v2 (mostly CC BY), and MathDial (CC BY-SA). Several otherwise-excellent sets, particularly real learner-interaction and some academic QA data, are CC BY-NC and cannot be used to train a shipped product without separate permission. Check the flag on each source page, and see choosing a dataset licence for a commercial product for how to think it through. These flags are a shortlist, not legal advice: confirm the terms at the source.
A note on format
Most instruction and chat fine-tuning data arrives as JSON or JSONL, one example per line, with fields for the prompt, the response, and often a system message or a list of tool calls. The exact schema varies by dataset and by the training framework you use, so the dataset card is the thing to read first.
Where to look next
The directory groups these and more by subject, each with its licence and formats. If your goal is the opposite problem, giving a model current knowledge rather than new behaviour, start with What is RAG? instead. And if you are assembling a pretraining mixture rather than a fine-tuning set, that is a third thing again, covered in our comparison of the big web corpora.