Boost Training Goodput: How Continuous Checkpointing Optimizes Reliability in Orbax and MaxText

This post introduces continuous checkpointing in Orbax and MaxText, a feature designed to maximize training reliability and I/O utilization with minimal performance overhead.

• •Traditional fixed-interval checkpointing forces a tradeoff: infrequent saves risk large resource waste on failure, while frequent saves can block training on unstable networks
• •Continuous checkpointing initiates a new async checkpoint only after the previous save completes, avoiding bottlenecks while maximizing bandwidth usage
• •Enabling it in MaxText requires setting enable_continuous_checkpointing: True alongside async_checkpointing: True; it overrides the checkpoint_period setting
• •Benchmarks on llama-3.1-70B CPT on two v5p-128 slices show significantly smaller P50 checkpoint intervals with a modest increase in average training step time