Guard matched-status counters with the endpoint mutex - #6511
Open
PavelGuzenfeld wants to merge 3 commits into
Open
Guard matched-status counters with the endpoint mutex#6511PavelGuzenfeld wants to merge 3 commits into
PavelGuzenfeld wants to merge 3 commits into
Conversation
added 2 commits
August 22, 2026 00:34
get_publication_matched_status() reads publication_matched_status_ and resets its change counters while holding the writer mutex, but update_publication_matched_status() wrote the same struct holding only InnerDataWriterListener::matching_info_mutex_. A user thread polling the status therefore raced the discovery thread updating it. Take the writer mutex around the update as well. Use the RTPSWriter passed to the callback rather than DataWriterImpl::writer_, which is only assigned after create_writer() returns. Signed-off-by: Pavel Guzenfeld <me@pavelguzenfeld.com>
Same defect as the DataWriter side: get_subscription_matched_status() reads subscription_matched_status_ and resets its change counters while holding the reader mutex, but update_subscription_matched_status() wrote the same struct holding only InnerDataReaderListener::matching_info_mutex_. Scope the reader mutex to the counter update only. The rest of the function notifies user listeners through set_read_communication_status() and drops into try_notify_read_conditions(), which deliberately releases the reader mutex before taking the conditions mutex, so neither may run with the reader mutex held. Pass the RTPSReader from the callback rather than using DataReaderImpl::reader_, which is only assigned after create_reader() returns. Signed-off-by: Pavel Guzenfeld <me@pavelguzenfeld.com>
Each test keeps one long-lived endpoint, polls its matched status from a user thread, and churns batches of counterpart endpoints in a second participant so on_writer_matched() / on_reader_matched() keep running on the discovery thread concurrently with the polling. A warm-up match is confirmed before the churn starts, and the tests assert that matches were actually observed, so they cannot silently exercise nothing. The races are only visible under a sanitizer. Against master both tests report 14-18 TSan races per run; with the fixes they report none. Signed-off-by: Pavel Guzenfeld <me@pavelguzenfeld.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Both matched-status structs are read under the endpoint mutex but written under a
different one, so a user thread polling the status races the discovery thread updating
it.
DataWriterImpl::get_publication_matched_status()readspublication_matched_status_and resets its two change counters while holding the writer mutex
(DataWriterImpl.cpp:1603-1608).
update_publication_matched_status()writes the samestruct, but the only lock held on that path is
InnerDataWriterListener::matching_info_mutex_.DataReaderImplis the same shape:get_subscription_matched_status()reads under thereader mutex (DataReaderImpl.cpp:1198),
update_subscription_matched_status()writesunder
matching_info_mutex_only.TSan on a fully instrumented build, on master (2bad9bc):
M0 is the writer mutex the getter takes; it is not among the mutexes the discovery
thread holds.
Writer side
update_publication_matched_status()only touches the status struct, so the writermutex can simply wrap the call. The order is
matching_info_mutex_then writer mutex,which is the same order the existing
get_publication_matched_status()call furtherdown that callback already establishes.
Reader side
Here the lock has to be scoped to the counter update only.
update_subscription_matched_status()also callsset_read_communication_status(),which invokes user listeners, and
try_notify_read_conditions(), which deliberatelyreleases the reader mutex before taking the conditions mutex. Wrapping the whole
function would run user callbacks under the reader mutex and introduce a
reader-mutex-then-conditions-mutex order that does not exist today.
Both sides take the mutex from the endpoint passed to the callback rather than from
DataWriterImpl::writer_/DataReaderImpl::reader_, because those members are onlyassigned after
create_writer()/create_reader()returns.How this was verified
Fast-DDS, Fast-CDR and foonathan_memory all built with
-fsanitize=thread, Debug,GCC 13.3, Ubuntu 24.04.
Writer side — one process, two participants, data-sharing, a writer publishing flat out
while 60 late-joining readers are created and destroyed, main thread polling
get_publication_matched_status()to wait for each match:Reader side — a long-lived DataReader in one participant, a churn thread creating and
destroying DataWriters in a second participant so
on_reader_matched()firesrepeatedly, main thread polling
get_subscription_matched_status(). 30s per run, around3000 writers churned and 15M polls:
Every report in the reader baseline was on this struct — the counters at
DataReaderImpl.cpp:1170-1175 and 1200-1202, plus
InstanceHandleValue_t::operator unsigned char*()from copyinglast_publication_handle.Contributor Checklist
Commit messages follow the project guidelines.
The code follows the style guidelines of this project.
Tests that thoroughly check the new feature have been added/Regression tests checking the bug and its fix have been added; the added tests pass locally
DDSDataWriter.publication_matched_status_concurrent_with_discoveryandDDSDataReader.subscription_matched_status_concurrent_with_discovery. Each keeps onelong-lived endpoint, polls its matched status from a user thread, and churns batches of
counterpart endpoints in a second participant so the matched callbacks keep running on
the discovery thread. A warm-up match is confirmed before the churn starts and the
tests assert matches were observed, so they cannot silently exercise nothing. Around 5s
each.
Measured against master, fully instrumented: 14-18 TSan reports per run across 3 runs,
every one of them on these two structs. Run individually, 8 for the writer test and 7
for the reader test. With the fixes, 0 across 3 runs. Without a sanitizer they are
plain stress tests and pass either way, so the value is in your nightly TSan job.
Any new/modified methods have been properly documented using Doxygen.
N/A Any new configuration API has an equivalent XML API
Changes are backport compatible: they do NOT break ABI nor change library core behavior.
Changes are API compatible.
N/A New feature has been added to the
versions.mdfile.N/A New feature has been documented/Current behavior is correctly described in the documentation.
Applicable backports have been included in the description.