PixMo

PixMo (Pixels for Molmo) is a family of vision-language datasets built by Ai2 (the Allen Institute for AI) to train its open Molmo models. Its distinguishing feature is that the data is collected from people rather than distilled from closed vision-language models: the flagship captions in pixmo-cap were gathered by asking annotators to describe an image out loud for up to 60 to 90 seconds and then transcribing the speech, which produces unusually dense descriptions. The collection spans several subsets, including pixmo-cap (about 717,000 captions), pixmo-points (about 2.38 million 2D pointing annotations that teach a model to point at the objects it names), pixmo-docs, pixmo-cap-qa, pixmo-clocks, pixmo-ask-model-anything, pixmo-point-explanations and pixmo-count. You access it through the Hugging Face Hub as Parquet, loadable with the datasets library. One important caveat: most subsets ship image URLs rather than the images themselves, so the pictures must be downloaded separately, and each row carries a SHA-256 hash so you can confirm a fetched image matches the annotated one. For RAG and AI work PixMo is a supervised training and fine-tuning corpus for multimodal models rather than a retrieval index, and the pointing data in particular supports grounding and agentic use where a model must indicate where something is. It is best for teams building or fine-tuning open VLMs that read images, documents and charts. On licensing, the annotations are released under ODC-BY-1.0, which requires attribution but imposes no share-alike condition, and Ai2 states the data is intended for research and educational use under its Responsible Use Guidelines. The linked images keep their own, often unknown, rights, so this is annotations over external images rather than a cleanly redistributable image corpus, and the pixmo-cap-qa answers were generated with Claude and remain subject to Anthropic's terms. It sits alongside open multimodal corpora we catalogue such as FineVision, but is human-annotated and non-distilled.