MixtureVitae — Permissive-First Pretraining Dataset

Open Web-Scale Pretraining Dataset With High Quality Instruction and Reasoning Data Built from Permissive-First Text Sources

Huu Nguyen*, Victor May*, Harsh Raj*, Marianna Nezhurina, Yishan Wang, Yanqi Luo, Minh Chien Vu, Taishi Nakamura, Ken Tsui, Van Khue Nguyen, David Salinas, Aleksandra Krasnodębska, Christoph Schuhmann, Mats Leon Richter, Xuan-Son Vu, Jenia Jitsev * equal contribution

TMLR arXiv GitHub Dataset Model

MixtureVitae is a 422-billion-token permissive-first, risk-mitigated pretraining corpus with enhanced reasoning, math, and code capabilities, while being competitive with non-permissive baselines on standard language benchmarks

Numbers below are for a 1.7B base model trained for 300B tokens — no fine-tuning.

0.56

Language understanding
avg across 11 benchmarks — MMLU, BoolQ, ARC, HellaSwag, Winogrande, and others from the open-sci-ref suite

0.53 · 0.32 · 0.38

Math and code — GSM8K · HumanEval · MBPP
beats SmolLM2-1.7B-Instruct, which was fine-tuned on ≈11T tokens

Why MixtureVitae is different

Most permissive pretraining corpora try to answer a licensing question. MixtureVitae tries to answer a harder one: how much capability can you recover when the data is provenance-aware from the start. Instead of releasing one more "open" mixture and asking users to trust it, MixtureVitae tags sources by risk tier, keeps synthetic provenance explicit, and lets researchers ablate those choices rather than treating the corpus as a black box.

That is the main difference from The Common Pile / Comma-0.1. Comma-0.1 is a strong permissive baseline, but it is built mostly from traditional permissive sources and reaches its token budget with heavy repetition. MixtureVitae instead combines more unique content with a much denser reasoning and instruction block, so the comparison is not just "cleaner licenses," but a different answer to how permissive data should be composed.

The point is not to make a narrow capability-shaped dataset. The point is to build a permissive general-purpose corpus that keeps strong language-understanding performance while recovering much stronger reasoning, math, and code behavior during pretraining itself. In other words: not breadth or reasoning, but breadth with a reasoning-capable foundation.

Dataset

The corpus has three main parts: curated domain sources (scientific papers, code, encyclopedias, patents), a reasoning and instruction block that would usually show up only at fine-tuning, and a smaller web component. Each source is tagged with its synthetic status so the split is transparent.

Most of the corpus is curated or reasoning-focused, not raw web text. The right panel shows how much of each domain is synthetic, mixed, or fully human-written.

Provenance tiers

Every source is assigned to one of three tiers. This is risk-mitigated, not risk-free—copyright law varies by jurisdiction, and users in some regions should get legal advice before using Tier 2 or Tier 3 material.

Tier 1

Explicit Open Licenses & Public Domain

CC0, CC-BY, Apache 2.0, MIT, BSD, or statutory public domain. Synthetic data generated from permissively licensed models and seed data also lands here. The legal footing is clear.

315.8B
75.0%

Tier 2

Curated Permissive Repositories

2a: Sources like The Stack v1 and Wikipedia derivatives — released under permissive licenses, but provenance is only partially tracked at the file level.
2b: ≈4% of the corpus — synthetic data from generators or seeds whose provenance cannot be fully verified. Isolated so users can exclude it; removing it has no measurable effect on benchmark scores.

88.6B
21.0%

Tier 3

Civic / Governmental Works

US federal works and government websites — statutory public domain or created for broad public use. Further filtered with copyright keyword checks to reduce residual risk.

16.8B
4.0%

Results

All models use the same architecture and hyperparameters under the open-sci-ref protocol — only the dataset changes. That makes it straightforward to attribute differences in benchmark scores to the data.

General benchmarks

MixtureVitae clearly leads other permissive corpora and closes most of the gap to mixed-license baselines by 300B tokens. The MMLU gap is especially large — most baselines stay near random chance while MixtureVitae reaches 0.38.

Reasoning, math, and code

Training dataset	Tokens	IF-Eval	GSM8K	HumanEval	MBPP	Avg
open-sci-ref — 300B tokens
MixtureVitae permissive	300B	0.19	0.53	0.32	0.38	0.36
Comma-0.1 permissive	300B	0.19	0.06	0.13	0.22	0.15
CommonCorpus permissive	300B	0.13	0.02	0.05	0.05	0.06
C4 mixed	300B	0.20	0.02	0.00	0.00	0.06
SlimPajama mixed	300B	0.14	0.02	0.05	0.00	0.05
HPLT-2.0 mixed	300B	0.17	0.02	0.00	0.00	0.05
DCLM mixed	300B	0.13	0.02	0.01	0.01	0.04
Nemotron-CC-HQ mixed	300B	0.09	0.03	0.02	0.00	0.03
open-sci-ref — 1T tokens
FineWeb-Edu mixed	1T	0.20	0.03	0.00	0.00	0.06
Nemotron-CC-HQ mixed	1T	0.13	0.03	0.01	0.04	0.05
DCLM mixed	1T	0.15	0.03	0.00	0.01	0.05
Reference models
SmolLM2-1.7B	11T	0.18	0.31	0.01	0.35	0.21
SmolLM2-1.7B-Instruct	11T	0.28	0.37	0.28	0.37	0.33

MixtureVitae is a base model trained on 300B tokens. SmolLM2-1.7B-Instruct is fine-tuned on roughly 11T tokens. MixtureVitae matches or beats it on GSM8K, HumanEval, and MBPP.

Why trust these results?

Heavy instruction content in a pretraining corpus raises an obvious question: are these scores real, or is the model just memorizing test items? The paper addresses this directly.

Not a contamination artifact

A 13-gram decontamination sweep removed overlapping test items across all benchmarks. Scores were stable — GSM8K went from 0.53 to 0.54, MBPP stayed at 0.38. Retraining after removing the contaminated shards produced nearly identical results.

Instruction data is the main driver

Three models were trained: the full mix, without web data, and without instruction/reasoning data. Removing instructions caused GSM8K to drop from 0.47 to 0.03. Removing web data dropped it to 0.41. The instruction block is doing most of the work.

The uncertain slice can be removed

Tier 2(b) — synthetic data from restricted or opaque generators — makes up about 4% of the corpus. Removing it produces a training curve indistinguishable from the full model. Users who need stricter provenance can exclude it without giving up performance.

Instruction data is the main driver

Three variants were trained at 100B tokens: the full mix, without web data, and without the instruction and reasoning block.

Ablation: removing instruction data collapses math and code scores

Taking out the instruction and reasoning block (orange) causes a large drop. Taking out web data (blue) has a much smaller effect.

FAQ

What does "permissive-first" mean?

MixtureVitae starts with sources that are clearly reusable. Licensing comes first; quality is a secondary filter within that set.

Most web-scale datasets go the other direction — start from a broad crawl, filter out low-quality content, and treat licensing as someone else's problem. MixtureVitae inverts that: a source only gets in if its license is clear (CC-BY, Apache 2.0, public domain) or falls into a narrowly justified lower-risk category like US government works.

Is the dataset fully risk-free?

No. The goal is lower risk, not a universal legal guarantee. That is why the data is split into tiers and why users can exclude the stricter slices.

The tiering rests primarily on US fair-use doctrine and the EU Text and Data Mining exception. Copyright law varies by country — if you are deploying Tier 2 or Tier 3 material commercially outside the US or EU, it is worth talking to a lawyer. Users who want the most defensible subset can restrict to Tier 1 only.

How does MixtureVitae compare to Comma-0.1 and CommonCorpus?

Both are permissive baselines. The main difference is synthetic data. The Common Pile (which Comma-0.1 trains on) deliberately avoids synthetic content. MixtureVitae includes a large block of reasoning and instruction data generated from permissive seeds — and that turns out to matter a lot for math and code.

On general benchmarks at 1.7B / 300B tokens: MixtureVitae 0.56, Comma-0.1 0.50, CommonCorpus 0.47. On GSM8K specifically: 0.53 vs. 0.06 vs. 0.02. Comma-0.1 also repeats most of its sources up to 6× to reach 1T tokens — MixtureVitae contains 422B unique tokens without repetition.

Citation

If you use MixtureVitae in your work, please cite:

citation.bib

@article{nguyen2026mixturevitae,
  title   = {{MixtureVitae}: Open Web-Scale Pretraining Dataset With
             High Quality Instruction and Reasoning Data Built from
             Permissive-First Text Sources},
  author  = {Huu Nguyen and Victor May and Harsh Raj and Marianna Nezhurina and
             Yishan Wang and Yanqi Luo and Vu Minh Chien and Taishi Nakamura and
             Ken Tsui and Van Khue Nguyen and David Salinas and
             Aleksandra Krasnod{\k{e}}bska and Christoph Schuhmann and Mats Leon Richter and
             Xuan-Son Vu and Jenia Jitsev},
  journal = {Transactions on Machine Learning Research},
  issn    = {2835-8856},
  year    = {2026},
  url     = {https://openreview.net/forum?id=SyCcUNUUMf},
  note    = {Featured Certification}
}