Fix thread safety in dissect processor by eliminating mutable shared state#6858
Fix thread safety in dissect processor by eliminating mutable shared state#6858shik-25 wants to merge 5 commits into
Conversation
…state Signed-off-by: Shikhar Dhawale <shikhar_dhawale@apple.com>
Benchmark: GC allocation pressure before vs after thread-safety fixSetup: JMH,
What changed: The new code allocates local Verdict: The +408 bytes/op is the honest cost of the fix. The +25% allocation rate increase is proportional - 6.7% more throughput × 17.5% more per op. GC pause time per cycle is slightly lower (-4%), meaning the JVM handles the extra allocation efficiently. Throughput improves because shared-state contention is eliminated under concurrent execution |
| public List<Field> getDissectedFields(){ | ||
| final List<Field> dissectedFields = new ArrayList<>(); | ||
| Map<String, AppendField> appendFieldMap = getAppendedFields(unAppendedFieldsMap); | ||
| private Map<String, String> getDissectedFields(Map<Field, String> localValues) { |
There was a problem hiding this comment.
Design note on Map<Field, String> localValues:
localValues uses Field objects as map keys without overriding
equals/hashCode, relying on identity semantics. This is intentional:
AppendFieldallows multiple instances with the same logical key (e.g.,{+field1}appearing twice in a pattern). Implementingequals/hashCodeby key would cause silent overwrites inlocalValues.- Field objects are never copied or rebuilt — the same instances from
normalFieldMap,indirectFieldMap, etc. are used for bothputandget, so identity-based lookup is always correct.
Identity semantics are the right tradeoff here.
dlvenable
left a comment
There was a problem hiding this comment.
Thank you @shik-25 for this contribution! This looks like a good improvement, but I do see a possible increase in memory pressure from your benchmarks.
I left a few comments on the design and some ideas to try to reduce memory pressure as well.
| @Setup | ||
| public void setUp() { | ||
| dissector = new Dissector("%{timestamp} %{level} %{message}"); | ||
| input = "2024-01-15 ERROR service crashed"; |
There was a problem hiding this comment.
It would probably be ideal to also have a benchmark that includes more scenarios like append fields, indirect fields, and strip-trailing. But this isn't necessary for the PR.
…xt, pre-size HashMaps, pre-sort AppendField lists, add ERROR logs, expand test coverage Signed-off-by: Shikhar Dhawale <shikhar_dhawale@apple.com>
89424c0 to
579b606
Compare
Signed-off-by: Shikhar Dhawale <shikhar_dhawale@apple.com>
Signed-off-by: Shikhar Dhawale <shikhar_dhawale@apple.com>
| indirectField.setKey(appendFieldMap | ||
| .get(indirectField.getKey()).getValue()); | ||
| String val = localValues.get(templateField); | ||
| if (resolvedKey != null && !resolvedKey.isEmpty() && val != null) { |
There was a problem hiding this comment.
This condition may be a change of semantics somewhat. Using the test case from test_indirect_field_unresolved, the previous implementation would write this to field2. The new change is probably more correct.
Perhaps we should have an else statement for this condition that logs this?
LOG.debug("Indirect field reference %{{&{}}} could not be resolved; dropping value", templateKey);
@kkondaka , Do you have any concerns about this slight change in behavior?
There was a problem hiding this comment.
I am ok but It would impact the existing users, right?
There was a problem hiding this comment.
@kkondaka , It would impact existing users if they have this configuration. I think they may not have been getting the expect results in such situations anyway.
There was a problem hiding this comment.
I have added debug logging for both cases:
- when the indirect field key can't be resolved
- when the indirect field itself has no captured value
The log messages are distinct so users can tell them apart.
There was a problem hiding this comment.
@kkondaka , I think this is acceptable. I don't think the existing behavior is correct. What are your thoughts? Do we need to retain the old behavior or just move forward with breaking this incorrect behavior?
…ct processor Signed-off-by: Shikhar Dhawale <shikhar_dhawale@apple.com>
2a731f0 to
3ea5144
Compare
Description
Fixes a thread-safety bug in the dissect processor by eliminating shared mutable state in
Dissector. Previously,dissectText()mutated sharedDelimiter.start/endandField.valueon every invocation, causing data races when multiple threads processed events concurrently. The@SingleThreadannotation was a workaround that limited throughput unnecessarily.Fix
dissectText()now uses localint[]arrays for delimiter positions and a localMap<Field, String>for field values instead of mutating shared objects.normalFieldMap,indirectFieldMap, etc.) are read-only after construction and wrapped inCollections.unmodifiableMap().dissectText()returnsMap<String, String>directly, replacing the separategetDissectedFields()call.@SingleThreadis removed.Issues Resolved
Resolves #5546
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.