Skip to content

Fix ItemStack hashing and classified ingredient lookups - #240

Merged
rubensworks merged 9 commits into
master-1.21-ltsfrom
fix/itemstack-hash-components-1.21
Sep 7, 2026
Merged

Fix ItemStack hashing and classified ingredient lookups#240
rubensworks merged 9 commits into
master-1.21-ltsfrom
fix/itemstack-hash-components-1.21

Conversation

@rubensworks

Copy link
Copy Markdown
Member

The 1.21 version of #238. Same three changes, measured on 1.21.1, where the effect is larger than on 26.1.

1. The ItemStack hash ignored data components

IItemStackHelpers.getItemStackHashCode hashed only the count and the item, while equality compares components in full. Every stack of the same item therefore hashed alike, so the hash-based ingredient collections collapsed into one bucket per item type and each lookup in such a bucket became a linear scan doing full component comparisons.

The method still carried the commented-out NBT hashing it inherited from before data components, with a note that tags were too expensive to hash. Component maps are not.

2. Classified lookups scanned everything when the classifier was empty

A single-classified collection partitions instances by a category type, and ItemStack is classified by its item. When a query's match condition covers that category, every match has to share the query's classifier, so an absent classifier means an empty result. contains and iterator already returned one directly. getAll, keySet, containsKey, countKey and count fell through to the unclassified path, which scans every instance to produce a result already known to be empty.

An IntegratedDynamics storage index keeps one classified map per priority level, so an item lookup visits every level, and every level not holding that item scanned all of its entries. This is a pre-existing bug, independent of change 1, and it is why item-only lookups were slow before any of this.

3. Plain stacks were hashing their item's default components

A component map hashes its prototype alongside its patch, and the prototype is the item's defaults, which the item already in the hash stands for. For a stack carrying no patch that walk was the entire cost of the hash, and plain stacks are what most of a storage network consists of.

This is safe on 1.21.1 for the same reason as on 26.1: PatchedDataComponentMap keeps its patch sanitized. I checked the 1.21.1 sources directly, both set and applyPatch drop an entry whose value equals the prototype's default rather than storing it, and hashCode is prototype.hashCode() + patch.hashCode() * 31. Two stacks of one item therefore have equal components exactly when they have equal patches. ItemStack.isComponentsPatchEmpty() exists in NeoForge 21.1.2, so the NeoForge override works here too, and vanilla getComponentsPatch().isEmpty() backs the common implementation.

Numbers

IntegratedDynamics index benchmarks on 1.21.1, PERFORMANCE_BENCHMARK_ENABLED=true ./gradlew runGameTestServer, ms per operation, medians of three whole runs each side. The shapes are added in CyclopsMC/IntegratedDynamics#PENDING_ID_BENCH.

benchmark before after
index_lookup_item 9.507505 0.001814 5241x faster
index_lookup_item_mixed 1.113108 0.001536 725x faster
index_lookup_item_plain 0.299095 0.001390 215x faster
index_lookup_exact_single_item 5.151762 0.001934 2664x faster
index_modification_single_item 4.780310 0.001982 2412x faster
index_modification_few_items 0.219300 0.001838 119x faster
index_lookup_exact_few_items 0.092090 0.001298 71x faster
index_modification 0.009933 0.001686 5.9x faster
index_lookup_exact 0.002876 0.001129 2.5x faster
index_modification_heavy_components 0.010344 0.004776 2.2x faster
index_modification_mixed 0.002748 0.001835 1.5x faster
index_lookup_exact_heavy_components 0.002968 0.002203 1.3x faster
index_lookup_exact_plain 0.001031 0.001043 same
index_lookup_nonempty_first 0.000242 0.000242 same
index_lookup_nonempty_all 0.024955 0.024100 same
index_lookup_item_single_item 0.717375 0.785046 1.09x slower
index_modification_plain 0.000451 0.000569 1.26x slower
index_lookup_exact_mixed 0.001296 0.001623 1.25x slower

How this differs from the 26.1 result, and what I cannot explain

On 26.1 the modification benchmarks regressed, roughly 2x on the spread shape and 4.9x with a heavy component payload. On 1.21 they improve instead, 5.9x and 2.2x. The reason is that the 1.21 baseline is far worse to begin with: index_modification starts at 0.009933 here against 0.000744 on 26.1, and index_lookup_item at 9.51 against 0.27. The added hashing cost is real on both branches, but on 1.21 it is swamped by how much worse the collisions were.

I have not isolated why the 1.21 baseline collides so much harder. The most likely explanation is that 1.21.1 registers fewer items, so spreading 5000 instances over them puts more component variants on each item and lengthens every collision chain. That is a hypothesis I did not test, not a measured finding.

Three rows still regress, all small and all with overlapping distributions at n=3:

  • index_modification_plain 1.26x (before 0.000438 to 0.000544, after 0.000463 to 0.000963)
  • index_lookup_exact_mixed 1.25x
  • index_lookup_item_single_item 1.09x, which is consistent across runs. That query returns every variant of an item that has 5000 of them, so classification cannot narrow it and each result costs a dearer hash.

Tests

  • TestItemStackHelpersHashCode: equal stacks hash equal, different items and counts hash differently, components affect the hash, 1000 component variants of one item produce over 99% distinct hashes, plain stacks still spread over items and counts, a component set back to its default hashes as plain again, and the hash agrees with ItemStack.isSameItemSameComponents plus count over 100 samples.
  • TestSingleClassifiedAbsentClassifier: the five affected methods return empty for an absent classifier and correct results for a present one, and a counting inner collection asserts that nothing is scanned in the absent case, so the optimization is pinned and not just its answer.

Both are written against this branch's JUnit 4 setup and bootstrap items through intrusive holders, matching the existing item tests here. A separate commit makes that adjustment, so the port is visible on its own.

./gradlew build passes on all loaders.

Relationship to the 26.1 branch

#238 is the same change against master-26-lts and stays open. The five ingredient collection files are byte identical between the two branches, so only ItemStackHelpersCommon and the tests needed adjusting. Merge whichever suits your upmerge direction; they are not meant to both land independently.

Notes for review

  • This changes the behaviour of every IntegratedDynamics storage lookup. It needs a CyclopsCore version bump, and dependent mods will need to require it.
  • Changes 2 and 3 stand on their own and could be split out. Change 2 in particular is a fix for a pre-existing bug and helps regardless of the hash.
  • No changelog entry, as requested.

What was not verified

  • No profiling was done on 1.21; the profiles that motivated changes 2 and 3 were taken on 26.1.
  • No terminal open measurements were taken on 1.21. Those exist only for 26.1.
  • Fabric and Forge keep the common hasComponentPatch implementation, which builds a patch instance for stacks that carry one. Only NeoForge was benchmarked.

Generated by Claude Code

getItemStackHashCode hashed only count and item. Equality compares data components,
so the hash was strictly less discriminating than equality: every stack of one item
landed in the same bucket of any hash-based ingredient collection.

Collapsed collections normalise the count to 1 before using a stack as a key, so the
count term is constant there and the hash degenerated to a function of the item alone.
With enough component variants per item the buckets treeify and every lookup turns into
a tree walk doing full data component comparisons, which is what made large Integrated
Dynamics storage networks scale quadratically.

Profiling a 50k stack storage terminal open showed 61% of server thread samples inside
treeified HashMap buckets and 65% inside IngredientInstanceWrapper.equals. Including
components takes that open from 4686 ms to 168 ms of server thread time.

The exclusion comment dated from NBT tags, which were expensive to hash. Component maps
are not, and vanilla hashes them the same way in ItemStack.hashItemAndComponents.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mxjin31W1Lmq5XK1CCe84v
(cherry picked from commit d710f4f)
ItemStackHelpersCommon is abstract on more than getItemStackHashCode, so an
anonymous subclass does not compile.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mxjin31W1Lmq5XK1CCe84v
(cherry picked from commit 8589bd8)
A single-classified collection partitions its instances by a category type. When
a query's match condition covers that category, every match has to share the
query's classifier, so an absent classifier means an empty result. contains and
iterator already returned one directly, but getAll, keySet, containsKey,
countKey and count fell through to the unclassified path, which scans every
instance to produce a result that was already known to be empty.

That fallback made the scan the common case rather than the exception. An
IntegratedDynamics storage index keeps one classified map per priority level, so
looking up an item touches every level, and every level that does not happen to
hold that item scanned all of its entries.

Measured on the IntegratedDynamics index benchmarks, item-only lookups over
5000 instances spread across 200 positions and 4 priority levels:

  index_lookup_item  0.240831 ms/op  before
                     0.001157 ms/op  after

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mxjin31W1Lmq5XK1CCe84v
(cherry picked from commit 855d78a)
Stacks carrying no component patch, and stacks whose only component was set
back to its default, are the cases most of a storage network consists of.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mxjin31W1Lmq5XK1CCe84v
(cherry picked from commit b052104)
A component map hashes its prototype alongside its patch, and the prototype is
the item's default components. The item is already in the hash, so hashing its
defaults again distinguishes nothing while walking the whole default map. For a
plain stack, which carries no patch at all, that walk was the entire cost of the
hash, and plain stacks are what most of a storage network consists of.

This stays consistent with equality because PatchedDataComponentMap keeps its
patch sanitized: setting a component to its default removes it from the patch
rather than storing it. Two stacks of one item therefore have equal components
exactly when they have equal patches.

Vanilla can only report an empty patch by building one, which is free for
exactly the stacks this catches, so the common implementation does that and
NeoForge overrides it with the direct check.

Measured on the IntegratedDynamics index benchmarks, over 5000 plain stacks
distinct by item and count:

  index_lookup_exact  0.002014 -> 0.000988 ms/op
  index_modification  0.001276 -> 0.000582 ms/op

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mxjin31W1Lmq5XK1CCe84v
(cherry picked from commit c676887)
This branch uses JUnit 4 and bootstraps items through intrusive holders rather
than binding components directly, so the two new test classes follow the
pattern the existing item tests here use. The assertions are unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mxjin31W1Lmq5XK1CCe84v

Copy link
Copy Markdown
Member Author

Correcting a placeholder in the description: the benchmark shapes referenced there are added in CyclopsMC/IntegratedDynamics#1727, which is the 1.21 counterpart of the benchmark PR.


Generated by Claude Code

@coveralls

coveralls commented Sep 7, 2026

Copy link
Copy Markdown

Coverage Status

coverage: 26.711% (+0.08%) from 26.63% — fix/itemstack-hash-components-1.21 into master-1.21-lts

@sonarqubecloud

sonarqubecloud Bot commented Sep 7, 2026

Copy link
Copy Markdown

@rubensworks
rubensworks merged commit 41f0074 into master-1.21-lts Sep 7, 2026
5 checks passed
@rubensworks
rubensworks deleted the fix/itemstack-hash-components-1.21 branch September 7, 2026 14:58
rubensworks pushed a commit that referenced this pull request Sep 7, 2026
Matches how this landed on master-1.21-lts in #240. The reasoning belongs in
the PR, not repeated in the file.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mxjin31W1Lmq5XK1CCe84v
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants