Support proper numpy integration for ~100x performance boost#259
Merged
VeaaC merged 6 commits intoheremaps:masterfrom Apr 23, 2026
Merged
Support proper numpy integration for ~100x performance boost#259VeaaC merged 6 commits intoheremaps:masterfrom
VeaaC merged 6 commits intoheremaps:masterfrom
Conversation
Signed-off-by: Christian Vetter <christian.vetter@here.com>
Signed-off-by: Christian Vetter <christian.vetter@here.com>
Signed-off-by: Christian Vetter <christian.vetter@here.com>
Signed-off-by: Christian Vetter <christian.vetter@here.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
flatdata-py performance: vectorized access and scalar optimization
What
Adds NumPy-based vectorized field access to flatdata-py and optimizes the scalar (element-by-element) read path. Also fixes a pre-existing bug in
read_value()for unaligned 64-bit fields.Changes
Vectorized access (
data_access.py,resources.py)read_field_vectorized(): reads a bit-packed field from all vector elements at once via NumPy, returning anndarray. Zero-copy over the mmap'd buffer.Vector.__getattr__("field")returns a DataFrame column for the field.Vector.to_numpy()/to_data_frame()return all fields at once._VectorSlicegets the same vectorized methods._as_numpy_2d().Pre-computed field readers (
data_access.py,structure.py)make_field_reader(offset, width, signed)builds a specialized closure with all constants (byte offset, bit shift, mask, sign handling) pre-computed. Six variants cover the cross-product of field types.Structure.__init_subclass__builds a_READERSdict once per class.__getattr__,as_dict,as_list,as_tuple,as_nparrayall use_READERS.read_value()is preserved as a thin wrapper aroundmake_field_readerfor one-off reads.Bug fix (
data_access.py)read_value()for 64-bit fields at non-byte-aligned offsets could return values wider than 64 bits (Python arbitrary-precision ints). The bit mask was only applied whennum_bits < 64, missing the case whereoffset_extra_bits > 0. Fixed by masking whennum_bits < 64 or offset_extra_bits > 0.Other
__slots__ = ()added to generated Structure subclasses (generator template + 10 golden files). Reduces instance size from 72 to 48 bytes.Vector.__iter__uses local variable caching to avoid repeated attribute lookups.list()on dict keys inArchive.__getattr__.flatdata-py/README.md.py.yml).Performance
Measured on a vector from a test archive (5.8M elements, 20 fields, 32 bytes each):