Add a spatial index#83
Open
boydjohnson wants to merge 16 commits into
Open
Conversation
This also adds a processing bench
In the original work, nodes and ways were modularized. This commit does the same for relations.
Author
|
This PR was talked about in #81 |
Author
|
I realized I messed up the --ids index. Before it was implicitly id ordered, as well as all the nodes, ways, and relations. So before Entity => Id was O(1), and then binary search for Id => Entity. With this PR, Entity => Id is still O(1) but there is no binary search for Id => Entity. I am going to add an index that will do the --reverse-ids. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The key insight was that the nodes, ways, and relations could be sorted in the curve's order and then found via binary search at query time. The schema is modified for relations to include the minimum bounding box, because that is often difficult to find at query time.
[i32; 4])way_curve()). Bbox stored in the archive.osmflatc now uses RocksDB to do the ordering based on the curve's index. There are osmflatc arguments for configuring RocksDB.
--scratch-dir <SCRATCH_DIR>
Scratch RocksDB directory; use a fast SSD with ample space [default: output's parent]
--block-cache-mb <BLOCK_CACHE_MB>
Scratch RocksDB block cache (MiB); raise (e.g. 8192) for planet-scale [default: 512]
--write-buffer-mb <WRITE_BUFFER_MB>
Scratch RocksDB write buffer (MiB); raise for higher write throughput [default: 128]
--max-open-files <MAX_OPEN_FILES>
Max open SST files RocksDB may cache (-1 = unlimited). Must stay below the process open-file limit (raise it with
ulimit -n); lower this if you hit "Too many open files" [default: -1]Note: Some of the code included in this PR was written by Claude Code.