Flatiron bringup, training parameters for 8xH100, improve training stability and efficiency - #491
Open
jpata wants to merge 50 commits into
Open
Flatiron bringup, training parameters for 8xH100, improve training stability and efficiency#491jpata wants to merge 50 commits into
jpata wants to merge 50 commits into
Conversation
Only collect step diagnostics (memory, batch stats, phase timings) and log to tensorboard every tensorboard_step_freq steps, avoiding the per-step overhead of collecting unused metrics.
ArrayRecordDataSource caches one file reader per distinct shard and never closes them, so worker processes accumulate file descriptors linearly as training touches more shards. This slowed down the psutil-based open-file monitoring inside tensorboard logging. Add a configurable max_open_readers cap (default 64): after each record is read, readers beyond the cap are closed. Reads are positional and stateless, so closing and reopening is safe. Verify reads return identical data after eviction.
…vice - _summarize_distributed_diagnostics: gather the union of numeric keys across ranks before all-reduce. Rank-0-only diagnostics keys (e.g. tensorboard_logging/tensorboard_flush) made ranks build differently-sized reduce tensors (20 vs 18 elements), hanging NCCL for 600s until the watchdog timeout. - init_process_group: pass device_id and set the device so barrier() and other collectives use the rank's GPU instead of falling back to the current-context device (0 on all ranks).
Use the full model learning rate for the learnable task-loss weights and keep LAMB layer-wise trust scaling (always_adapt) so the weights evolve as they did before the optimizer parameter-group split. Add a regression test covering the optimizer parameter groups.
jpata
changed the title
jpata
changed the title
jpata
changed the title
jpata
marked this pull request as ready for review
August 17, 2026 06:02
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
neighborhood features.
hit_vs_pf_comparison.slides.pdf