Skip to content

Fix thread safety in dissect processor by eliminating mutable shared state#6858

Open
shik-25 wants to merge 5 commits into
opensearch-project:mainfrom
shik-25:fix/dissect-thread-safety
Open

Fix thread safety in dissect processor by eliminating mutable shared state#6858
shik-25 wants to merge 5 commits into
opensearch-project:mainfrom
shik-25:fix/dissect-thread-safety

Conversation

@shik-25

@shik-25 shik-25 commented May 13, 2026

Copy link
Copy Markdown

Description

Fixes a thread-safety bug in the dissect processor by eliminating shared mutable state in Dissector. Previously, dissectText() mutated shared Delimiter.start/end and Field.value on every invocation, causing data races when multiple threads processed events concurrently. The @SingleThread annotation was a workaround that limited throughput unnecessarily.

Fix

  • dissectText() now uses local int[] arrays for delimiter positions and a local Map<Field, String> for field values instead of mutating shared objects.
  • Template field objects (normalFieldMap, indirectFieldMap, etc.) are read-only after construction and wrapped in Collections.unmodifiableMap().
  • dissectText() returns Map<String, String> directly, replacing the separate getDissectedFields() call.
  • @SingleThread is removed.

Issues Resolved

Resolves #5546

Check List

  • New functionality includes testing.
  • New functionality has a documentation issue. Please link to it in this PR.
    • New functionality has javadoc added
  • Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…state

Signed-off-by: Shikhar Dhawale <shikhar_dhawale@apple.com>
@shik-25

shik-25 commented May 13, 2026

Copy link
Copy Markdown
Author

Benchmark: GC allocation pressure before vs after thread-safety fix

Setup: JMH, @Threads(4) (shared Dissector instance across 4 concurrent threads), gc profiler, 2 forks × 5 iterations × 10s

Metric Before After Delta
Alloc/op (gc.alloc.rate.norm) 2336 B 2744 B +408 B (+17.5%)
Alloc rate (gc.alloc.rate) 4287 MB/s 5373 MB/s +25%
GC time/cycle 0.48ms 0.46ms -4%
Throughput 1,924,895 ops/s 2,053,606 ops/s +6.7%

What changed: The new code allocates local int[] arrays for delimiter positions and a Map<Field, String> for field values on every dissectText() call instead of mutating shared objects. These are short-lived allocations collected by the young gen immediately.

Verdict: The +408 bytes/op is the honest cost of the fix. The +25% allocation rate increase is proportional - 6.7% more throughput × 17.5% more per op. GC pause time per cycle is slightly lower (-4%), meaning the JVM handles the extra allocation efficiently. Throughput improves because shared-state contention is eliminated under concurrent execution

public List<Field> getDissectedFields(){
final List<Field> dissectedFields = new ArrayList<>();
Map<String, AppendField> appendFieldMap = getAppendedFields(unAppendedFieldsMap);
private Map<String, String> getDissectedFields(Map<Field, String> localValues) {

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Design note on Map<Field, String> localValues:

localValues uses Field objects as map keys without overriding
equals/hashCode, relying on identity semantics. This is intentional:

  • AppendField allows multiple instances with the same logical key (e.g., {+field1} appearing twice in a pattern). Implementing equals/hashCode by key would cause silent overwrites in localValues.
  • Field objects are never copied or rebuilt — the same instances from normalFieldMap, indirectFieldMap, etc. are used for both put and get, so identity-based lookup is always correct.

Identity semantics are the right tradeoff here.

@dlvenable dlvenable left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @shik-25 for this contribution! This looks like a good improvement, but I do see a possible increase in memory pressure from your benchmarks.

I left a few comments on the design and some ideas to try to reduce memory pressure as well.

Comment thread data-prepper-plugins/dissect-processor/build.gradle Outdated
@Setup
public void setUp() {
dissector = new Dissector("%{timestamp} %{level} %{message}");
input = "2024-01-15 ERROR service crashed";

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would probably be ideal to also have a benchmark that includes more scenarios like append fields, indirect fields, and strip-trailing. But this isn't necessary for the PR.

…xt, pre-size HashMaps, pre-sort AppendField lists, add ERROR logs, expand test coverage

Signed-off-by: Shikhar Dhawale <shikhar_dhawale@apple.com>
@shik-25 shik-25 force-pushed the fix/dissect-thread-safety branch from 89424c0 to 579b606 Compare June 3, 2026 05:54
Shikhar Dhawale added 2 commits June 3, 2026 12:11
Signed-off-by: Shikhar Dhawale <shikhar_dhawale@apple.com>
Signed-off-by: Shikhar Dhawale <shikhar_dhawale@apple.com>
@shik-25 shik-25 requested a review from dlvenable June 3, 2026 09:13
indirectField.setKey(appendFieldMap
.get(indirectField.getKey()).getValue());
String val = localValues.get(templateField);
if (resolvedKey != null && !resolvedKey.isEmpty() && val != null) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This condition may be a change of semantics somewhat. Using the test case from test_indirect_field_unresolved, the previous implementation would write this to field2. The new change is probably more correct.

Perhaps we should have an else statement for this condition that logs this?

LOG.debug("Indirect field reference %{{&{}}} could not be resolved; dropping value", templateKey);

@kkondaka , Do you have any concerns about this slight change in behavior?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am ok but It would impact the existing users, right?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kkondaka , It would impact existing users if they have this configuration. I think they may not have been getting the expect results in such situations anyway.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have added debug logging for both cases:

  • when the indirect field key can't be resolved
  • when the indirect field itself has no captured value

The log messages are distinct so users can tell them apart.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kkondaka , I think this is acceptable. I don't think the existing behavior is correct. What are your thoughts? Do we need to retain the old behavior or just move forward with breaking this incorrect behavior?

…ct processor

Signed-off-by: Shikhar Dhawale <shikhar_dhawale@apple.com>
@shik-25 shik-25 force-pushed the fix/dissect-thread-safety branch from 2a731f0 to 3ea5144 Compare June 8, 2026 15:22
@dlvenable dlvenable added the breaking change Any change which may break existing configurations and deployments label Jun 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

breaking change Any change which may break existing configurations and deployments

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make dissect processor thread-safe

3 participants