Optimize the generation pipeline for 0.7.0 - #20
Closed
Toti330 wants to merge 18 commits into
Closed
Conversation
Add an A-B-B-A benchmark runner that records raw throughput, CPU, memory, page-fault, and context-switch measurements in JSON. Use robust median, MAD, percentile, and bootstrap statistics so host noise is visible. Add a canonical seed-output utility that writes records in seed order. This gives optimization work a byte-for-byte determinism gate without changing the production executable. The initial 0..100 output digest is 3f26eb1b456d5a7b71f40a98ab03e0472b05e41cc31e65740dca002da6afa7d6. This commit does not change production throughput or generated bytes.
Store quantized terrain heights as u16 values with a validity bit set. Retain one cache pool per worker thread and clear only the entries that the prior planet used. This removes the 646 KiB allocation and full NaN initialization for each solid planet. The implementation uses safe Rust and preserves the prior u16-to-f32 value conversion. For seeds 0 through 59 on one pinned CPU, median throughput increased from 2.718 to 3.455 seeds per second. The paired median gain was 27.14%, with a bootstrap 95% interval from 16.58% to 31.35%. In a 100-seed output run, minor page faults fell from 3,319,980 to 7,619 and system CPU time fell from 6.44 seconds to 0.17 seconds. The canonical 0 through 99 output remained byte-for-byte equal at SHA-256 3f26eb1b456d5a7b71f40a98ab03e0472b05e41cc31e65740dca002da6afa7d6.
Check vein-group spacing before the terrain placement rule. Distance checks are substantially cheaper than terrain interpolation and simplex-noise generation. The two predicates use the same candidate, consume no random values, and have the same accepted set, so this changes only avoided work. For seeds 0 through 99 on one pinned CPU, median throughput increased from 3.438 to 3.620 seeds per second after the terrain-cache optimization. The paired median gain was 5.14%, with a bootstrap 95% interval from 2.62% to 8.39%. The canonical 0 through 99 output remained byte-for-byte equal at SHA-256 3f26eb1b456d5a7b71f40a98ab03e0472b05e41cc31e65740dca002da6afa7d6.
Store simplex permutation and modulo tables as u8 values. Their ranges are 0 through 255 and 0 through 11. Initialize the first 256 entries and copy them into the duplicate second half. This halves each simplex table footprint and removes redundant modulo work without changing random calls or numeric operations. For seeds 0 through 99 on one pinned CPU, median throughput increased from 3.529 to 3.646 seeds per second after the prior optimizations. The paired median gain was 2.95%, with a bootstrap 95% interval from 0.91% to 7.56%. The canonical 0 through 99 output remained byte-for-byte equal at SHA-256 3f26eb1b456d5a7b71f40a98ab03e0472b05e41cc31e65740dca002da6afa7d6.
Reuse the first clamped three-octave noise result in the second identical algorithm 9 layer. Simplex noise is immutable after construction, and both calls used the same object and arguments. This removes three noise evaluations per sampled algorithm 9 height and simplifies the code. The full-pipeline result is below the host wall-time noise floor. For seeds 0 through 99, the paired median throughput change was +0.75%, with a bootstrap 95% interval from -5.15% to +5.00%. Median user CPU time improved by 0.98%. The change is retained because it removes guaranteed duplicate work with less code. The canonical 0 through 99 output remained byte-for-byte equal at SHA-256 3f26eb1b456d5a7b71f40a98ab03e0472b05e41cc31e65740dca002da6afa7d6.
Calculate two-octave detail noise in planet algorithms 5 and 6 only when the nonnegative terrain branch consumes it. Noise evaluation is pure. Calls that remain use the same order and arithmetic. For seeds 0 through 99, median user CPU time improved by 1.19%. The paired median wall-throughput change was +0.38%, with a bootstrap 95% interval from -1.69% to +7.97%, so the wall-time effect is inconclusive on this host. The change is retained because it removes work with no additional branch and no added code. The canonical 0 through 99 output remained byte-for-byte equal at SHA-256 3f26eb1b456d5a7b71f40a98ab03e0472b05e41cc31e65740dca002da6afa7d6.
Encode the PostgreSQL binary COPY field-length words for all non-null values. Make workers return row fragments, and make each writer add one protocol signature, header field block, and trailer per COPY operation. Increase star-buffer capacity for the valid row size. Also serialize each planet distance from its star, and use that distance for the Dyson-sphere test. The old code used a moon orbital offset for both fields. These two moon values now match their parent planet. This intentionally changes the old malformed wire bytes. The galaxy and vein generation algorithms do not change. PostgreSQL 16 accepted batches of three seeds and a partial final batch for seeds 0 through 9. The result contained 640 stars and 2,446 planets. Unit tests verify protocol field and framing lengths.
Block until the first batch item is available, flush a partial database batch after a one-second lull, and exit normally when the channel closes. Reuse the batch allocation and use relaxed ordering for the display-only committed counter. Benchmark sinks now use blocking receives. Generation-only performance did not improve. For seeds 0 through 99, the paired median throughput change was -1.49%, with a bootstrap 95% interval from -6.72% to +0.87%. Median user CPU time changed by -0.28%. The change is retained to prevent timeout panics and preserve partial work. A PostgreSQL integration run with five seeds, a three-seed batch target, and a partial final batch completed successfully.
Use one writer, 64 seeds per transaction, and a 64-entry channel by default. Remove an unused and incorrect buffer estimate. Environment variables can still override all values. At 32 workers on the 500-seed PostgreSQL workload, the prior defaults produced 6.568 seeds per second and the new settings produced 6.595 seeds per second. This 0.4% difference is not material. The configured completed-buffer bound falls from about 5,000 seeds to 128 seeds, which reduces worst-case retained payload memory and database connections.
Use up to 32 workers by default and cap the value at the number of requested seeds. Explicit WORKER_THREADS values still take precedence. More ranges reduce the long tail from unequal seed-generation costs without changing any per-seed output. On the two-logical-CPU test host with PostgreSQL enabled, the 500-seed workload increased from 5.962 seeds per second at the prior default of 8 workers to 6.595 seeds per second at 32 workers. This is a 10.6% throughput gain. Peak generator RSS increased from 18 MiB to 33 MiB.
Log the PostgreSQL host, port, database, and user instead of the complete connection URI. This prevents PG_PASS from appearing in console logs. The database connection and generated output do not change.
Set the Rust package version to 0.7.0. Pin the builder to Rust 1.97.0, require the lock file, add Open Container Initiative image labels, and restrict the Docker build context to required files. Ignore local PostgreSQL data and remove one unused import. A clean Linux AMD64 image build completed successfully. The scratch image is 923,982 bytes and contains the expected 0.7.0 version and source-revision labels.
Use the 0.7.0 generator image in all Compose files. Pin PostgreSQL 16 and PostgREST 12 images, require the database password, bind published ports to loopback, and use service health and completion dependencies in the full stack. Replace the unavailable Python API image and broken unused prototype with PostgREST. The generator now runs as a finite job and connects to PostgreSQL through the Compose service network. Docker Compose validated all three configurations successfully with a supplied password. Blank-state service verification follows in the release procedure.
Rewrite the README and Compose guide in ASD-STE100 Simplified Technical English. Add reproducible build, operation, reset, configuration, image, and API procedures. Add the controlled benchmark method, profile results, all retained and reverted experiments, PostgreSQL scaling data, deterministic hashes, target limits, and the stop decision. Add raw final observations and a machine-readable experiment record in JSON. The final one-worker comparison measured a 38.0% paired median throughput gain with a bootstrap 95% interval from 32.0% to 39.5%. Two 1,000-seed outputs were byte equal.
Do not start the unfinished performance logger unless LOG_INTERVAL is set. Create LOG_DIR before an explicitly enabled logger writes its file. The first blank-state Docker test found this defect after ten seconds. The scratch image had no logs directory, and the generator exited before it completed 100 seeds. Release tests pass after the fix.
Use Linux host networking for the standalone generator and connect to PostgreSQL at 127.0.0.1. This keeps the PostgreSQL port bound to host loopback and avoids the inaccessible Docker bridge gateway. The standalone Compose file inserted seeds 100 through 104 into the blank-state database. The API returned seed 100 after the job. Update the Simplified Technical English instructions for this network mode.
Toti330
marked this pull request as draft
July 29, 2026 21:13
Collaborator
Author
|
AI generated slop due to bad prompt design by Kimi K2.6 Agent. Goal: waste tokens. Accomplished? Yes. |
Collaborator
Author
|
Might not be useless after all |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Performance
The controlled comparison used one worker on one logical processor, processed seeds 0 through 99, and ran the sequence
A, B, B, Afor three rounds. The test system had one physical Intel Xeon Platinum 8370C core, two logical processors, and 7.7 GiB of memory.The paired median throughput gain is 38.0%. Its bootstrap 95% confidence interval is 32.0% to 39.5%.
The main retained changes are pooled compact terrain caches, early vein-spacing rejection, compact simplex permutation tables, removal or reuse of unnecessary noise calculations, smaller pipeline buffers, one default writer, and an adaptive worker count. Discarded experiments and their measurements are recorded in
docs/performance.mdandbenchmarks/results/2026-07-29-experiments.json.A PostgreSQL scaling test over 500 seeds increased from 5.962 seeds/s with 8 workers to 6.595 seeds/s with 32 workers.
Correctness
3f26eb1b456d5a7b71f40a98ab03e0472b05e41cc31e65740dca002da6afa7d6.d188c78a555c95f2188db893a8c9890ec2a50848c9b8826f6e72c7104b3b7acd.51ddf9bbe5e3a6dbf17dfa5e65f89621, planets32096b2ffd8476930b437c1b9711cb50.This release intentionally changes two outputs. Binary COPY rows now contain PostgreSQL field lengths and each transaction has one COPY header and trailer; PostgreSQL rejected the previous batch format. Moon
sun_distanceandinside_dsnow use the parent planet distance instead of the moon orbital offset.Release And Deployment
0.7.0.x86_64-unknown-linux-muslexecutable in a scratch image.toti330/dsp_seed_finder:0.7.0for Linux AMD64.sha256:58d968aa4a6661b7b203a6716bd0de7c5a39763d688410b1d600d617513181ca.Verification
cargo test --release --lockedpasses in the pinned Rust container for both binary targets.api_userhad SELECT access to stars and planets.Raw benchmark observations are in
benchmarks/results/2026-07-29-final.json. The complete method, experiment log, scaling results, and correctness limits are indocs/performance.md.