Skip to content

Guard matched-status counters with the endpoint mutex - #6511

Open
PavelGuzenfeld wants to merge 3 commits into
eProsima:masterfrom
PavelGuzenfeld:fix/publication-matched-status-race
Open

Guard matched-status counters with the endpoint mutex#6511
PavelGuzenfeld wants to merge 3 commits into
eProsima:masterfrom
PavelGuzenfeld:fix/publication-matched-status-race

Conversation

@PavelGuzenfeld

@PavelGuzenfeld PavelGuzenfeld commented Aug 21, 2026

Copy link
Copy Markdown

Description

Both matched-status structs are read under the endpoint mutex but written under a
different one, so a user thread polling the status races the discovery thread updating
it.

DataWriterImpl::get_publication_matched_status() reads publication_matched_status_
and resets its two change counters while holding the writer mutex
(DataWriterImpl.cpp:1603-1608). update_publication_matched_status() writes the same
struct, but the only lock held on that path is
InnerDataWriterListener::matching_info_mutex_.

DataReaderImpl is the same shape: get_subscription_matched_status() reads under the
reader mutex (DataReaderImpl.cpp:1198), update_subscription_matched_status() writes
under matching_info_mutex_ only.

TSan on a fully instrumented build, on master (2bad9bc):

WARNING: ThreadSanitizer: data race
  Read of size 4 at 0x726c00004484 by main thread (mutexes: write M0):
    #0 DataWriterImpl::get_publication_matched_status(...) src/cpp/fastdds/publisher/DataWriterImpl.cpp:1606
    #1 DataWriter::get_publication_matched_status(...) src/cpp/fastdds/publisher/DataWriter.cpp:270
  Previous write of size 4 at 0x726c00004484 by thread T4 (mutexes: write M1, write M2, write M3):
    #0 DataWriterImpl::update_publication_matched_status(...) src/cpp/fastdds/publisher/DataWriterImpl.cpp:1589
    #1 DataWriterImpl::InnerDataWriterListener::on_writer_matched(...) src/cpp/fastdds/publisher/DataWriterImpl.cpp:1408
    #2 EDP::pairing_reader_proxy_with_any_local_writer(...) src/cpp/rtps/builtin/discovery/endpoint/EDP.cpp:1312
    ...
    #31 UDPChannelResource::perform_listen_operation(...) src/cpp/rtps/transport/UDPChannelResource.cpp:78

M0 is the writer mutex the getter takes; it is not among the mutexes the discovery
thread holds.

Writer side

update_publication_matched_status() only touches the status struct, so the writer
mutex can simply wrap the call. The order is matching_info_mutex_ then writer mutex,
which is the same order the existing get_publication_matched_status() call further
down that callback already establishes.

Reader side

Here the lock has to be scoped to the counter update only.
update_subscription_matched_status() also calls set_read_communication_status(),
which invokes user listeners, and try_notify_read_conditions(), which deliberately
releases the reader mutex before taking the conditions mutex. Wrapping the whole
function would run user callbacks under the reader mutex and introduce a
reader-mutex-then-conditions-mutex order that does not exist today.

Both sides take the mutex from the endpoint passed to the callback rather than from
DataWriterImpl::writer_ / DataReaderImpl::reader_, because those members are only
assigned after create_writer() / create_reader() returns.

How this was verified

Fast-DDS, Fast-CDR and foonathan_memory all built with -fsanitize=thread, Debug,
GCC 13.3, Ubuntu 24.04.

Writer side — one process, two participants, data-sharing, a writer publishing flat out
while 60 late-joining readers are created and destroyed, main thread polling
get_publication_matched_status() to wait for each match:

build runs TSan reports
master 3 3 every run, all of them this race
with this change 3 0

Reader side — a long-lived DataReader in one participant, a churn thread creating and
destroying DataWriters in a second participant so on_reader_matched() fires
repeatedly, main thread polling get_subscription_matched_status(). 30s per run, around
3000 writers churned and 15M polls:

build runs TSan reports
master 3 18-19 every run
with this change 3 0

Every report in the reader baseline was on this struct — the counters at
DataReaderImpl.cpp:1170-1175 and 1200-1202, plus InstanceHandleValue_t::operator unsigned char*() from copying last_publication_handle.

Contributor Checklist

  • Commit messages follow the project guidelines.

  • The code follows the style guidelines of this project.

  • Tests that thoroughly check the new feature have been added/Regression tests checking the bug and its fix have been added; the added tests pass locally

    DDSDataWriter.publication_matched_status_concurrent_with_discovery and
    DDSDataReader.subscription_matched_status_concurrent_with_discovery. Each keeps one
    long-lived endpoint, polls its matched status from a user thread, and churns batches of
    counterpart endpoints in a second participant so the matched callbacks keep running on
    the discovery thread. A warm-up match is confirmed before the churn starts and the
    tests assert matches were observed, so they cannot silently exercise nothing. Around 5s
    each.

    Measured against master, fully instrumented: 14-18 TSan reports per run across 3 runs,
    every one of them on these two structs. Run individually, 8 for the writer test and 7
    for the reader test. With the fixes, 0 across 3 runs. Without a sanitizer they are
    plain stress tests and pass either way, so the value is in your nightly TSan job.

  • Any new/modified methods have been properly documented using Doxygen.

  • N/A Any new configuration API has an equivalent XML API

  • Changes are backport compatible: they do NOT break ABI nor change library core behavior.

  • Changes are API compatible.

  • N/A New feature has been added to the versions.md file.

  • N/A New feature has been documented/Current behavior is correctly described in the documentation.

  • Applicable backports have been included in the description.

Pavel Guzenfeld added 2 commits August 22, 2026 00:34
get_publication_matched_status() reads publication_matched_status_ and
resets its change counters while holding the writer mutex, but
update_publication_matched_status() wrote the same struct holding only
InnerDataWriterListener::matching_info_mutex_. A user thread polling the
status therefore raced the discovery thread updating it.

Take the writer mutex around the update as well. Use the RTPSWriter passed
to the callback rather than DataWriterImpl::writer_, which is only assigned
after create_writer() returns.

Signed-off-by: Pavel Guzenfeld <me@pavelguzenfeld.com>
Same defect as the DataWriter side: get_subscription_matched_status() reads
subscription_matched_status_ and resets its change counters while holding the
reader mutex, but update_subscription_matched_status() wrote the same struct
holding only InnerDataReaderListener::matching_info_mutex_.

Scope the reader mutex to the counter update only. The rest of the function
notifies user listeners through set_read_communication_status() and drops into
try_notify_read_conditions(), which deliberately releases the reader mutex
before taking the conditions mutex, so neither may run with the reader mutex
held.

Pass the RTPSReader from the callback rather than using DataReaderImpl::reader_,
which is only assigned after create_reader() returns.

Signed-off-by: Pavel Guzenfeld <me@pavelguzenfeld.com>
@PavelGuzenfeld PavelGuzenfeld changed the title Guard publication_matched_status_ with the writer mutex Guard matched-status counters with the endpoint mutex Aug 21, 2026
Each test keeps one long-lived endpoint, polls its matched status from a user
thread, and churns batches of counterpart endpoints in a second participant so
on_writer_matched() / on_reader_matched() keep running on the discovery thread
concurrently with the polling.

A warm-up match is confirmed before the churn starts, and the tests assert that
matches were actually observed, so they cannot silently exercise nothing.

The races are only visible under a sanitizer. Against master both tests report
14-18 TSan races per run; with the fixes they report none.

Signed-off-by: Pavel Guzenfeld <me@pavelguzenfeld.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant