Parallelize AntColonyMTSP's cluster solving - #123
Merged
Conversation
Each cluster's ACO run is independent, but every solver draws from the seeded global RNG via core.random.spawn_streams()/rng(), whose spawn counter is documented as single-thread-only. Running clusters concurrently would have raced on that counter and broken the reproducibility guarantee the RNG-determinism work established. Adds spawn_stream_roots()/use_stream_root() to core/random.py: a nested-parallelism companion to spawn_streams()/use_stream() that lets a task which itself dispatches further parallel work (one multi-generation ACO run per cluster) draw from its own independent SeedSequence sub-tree instead of racing on the shared global one. Purely additive -- spawn_streams() behaves exactly as before when no such scope is active. AntColonyMTSP.solve() now spawns one stream root per cluster up front (single-threaded, before dispatch) and runs the clusters through joblib.Parallel, each wrapped in use_stream_root. Also splits the configured processor budget between cluster-level and per-cluster ant-level parallelism instead of giving every cluster the full budget (the old TODO here -- "handle the number of processors based upon parallel clusters" -- was asking for exactly this). Tests: RNG-isolation unit tests for spawn_stream_roots/use_stream_root (mirroring the existing spawn_streams/use_stream tests), plus an AntColonyMTSP-specific determinism test confirming a seeded run with concurrent clusters is reproducible and still seed-sensitive. flake8/mypy/black clean; full suite passing. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011r9W1JZRSoayjyLin4ZBPZ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
AntColonyMTSP.solve()previously solved each cluster's independent ACO tour sequentially, with a TODO noting it could be parallelized but wasn't (each solver draws from the seeded global RNG, andcore.random.spawn_streams()'s spawn counter is documented as single-thread-only, so naive concurrency would have raced on it and broken run reproducibility).spawn_stream_roots()/use_stream_root()tocore/random.py: a nested-parallelism companion to the existingspawn_streams()/use_stream(). A task that itself dispatches further parallel work (here: one full multi-generation ACO run per cluster) gets its own independentSeedSequencesub-tree to spawn from, instead of racing on the shared global root. Purely additive —spawn_streams()is unchanged when no such scope is active.AntColonyMTSP.solve()now spawns one stream root per cluster up front (single-threaded, before dispatch), then runs the clusters throughjoblib.Parallel, each wrapped inuse_stream_root.Test plan
spawn_stream_roots/use_stream_root(mirroring the existingspawn_streams/use_streamtests) intests/test_determinism.pyAntColonyMTSP-specific determinism test: a seeded run with concurrently-solving clusters is reproducible across repeated runs, and still seed-sensitiveflake8 ./src ./tests— cleanMYPYPATH=src mypy -p optimizers— cleanblack --check .— clean🤖 Generated with Claude Code
https://claude.ai/code/session_011r9W1JZRSoayjyLin4ZBPZ