Task-Seeded Synthetic Q&A Generation for Nemotron Pretraining

This article describes a task-seeded synthetic Q&A generation pipeline for LLM pretraining in Nemotron-family models.

• •The method uses about 70 public tasks from lm-eval-harness as seeds to generate synthetic examples with structured learning signals.
• •Generated examples include similar questions and answer-enriched samples with reasoning and task-relevant context.
• •The pipeline covers knowledge-intensive and reasoning-intensive tasks to enable broad skill transfer across task families.
• •A 100B-token continuation on Nemotron-3 Nano improved MMLU-Pro by +1.8, code by +1.9, commonsense by +1.6, and GPQA by +11.1.

This summary was automatically generated by AI based on the original article and may not be fully accurate.