firefly: preserve all IoVec segments in intra-node LoopReq (#2686) by deanchester · Pull Request #2687 · sstsimulator/sst-elements

deanchester · 2026-06-03T16:12:07Z

The intra-node loopback delivery path truncated multi-segment sends, causing ProcessQueuesState::copyIoVec() to abort on assert(copied == len) when an Allgather-class collective runs with multiple ranks bound to a single endpoint NIC.

Root cause

For on-node peers, processSendLoop() packs the MatchHdr plus every data segment of the sender's I/O vector into a single vector (vec[0] = MatchHdr, vec[1..N] = data segments). The LoopReq constructor, however, kept only the first segment (vec[1]) and discarded vec[2..N], while the MatchHdr it carries still reports the total byte count across all segments. At delivery, copyIoVec() is asked to copy the full multi-segment length but only has the first segment as source, so copied stalls below len and the assertion fires (or, under NDEBUG, a short/incorrect receive buffer is produced silently).

Multiple segments arise because Allgather::initIoVec() coalesces only physically contiguous chunks: a recursive-doubling stage whose send window wraps the modular buffer is split into two or more non-contiguous runs. This is why the failure reproduces reliably at high PPN (almost every neighbour is on-node) and even at 2 ranks/node with a large payload, but never for point-to-point or small single-segment collectives.

Fix

LoopReq (ctrlMsgProcessQueuesState.h): copy every data segment (vec[1..N]) instead of only vec[1], so multi-segment intra-node sends are delivered intact.
copyIoVec (ctrlMsgProcessQueuesState.cc): harden as defence-in-depth. Bound the copy loops on rV < dst.size() in the loop conditions (instead of an assert that is compiled out under NDEBUG), and replace the bare assert(copied == len) with a dbg().fatal() that reports copied, len, src/dst segment counts and total byte counts. A mismatch is an internal defect between sender and receiver of the same collective algorithm, so it is surfaced loudly rather than silently truncated.

Verified by reproducing the abort with the original code at numCores >= 8, then confirming clean completion after the fix across numCores = 2/8/16/56 and count = 1/131072 (Allgather).

Fixes #2686

Automatically Merged using SST Master Branch Merger

…tor#2686) The intra-node loopback delivery path truncated multi-segment sends, causing ProcessQueuesState::copyIoVec() to abort on assert(copied == len) when an Allgather-class collective runs with multiple ranks bound to a single endpoint NIC. Root cause ---------- For on-node peers, processSendLoop() packs the MatchHdr plus *every* data segment of the sender's I/O vector into a single vector (vec[0] = MatchHdr, vec[1..N] = data segments). The LoopReq constructor, however, kept only the first segment (vec[1]) and discarded vec[2..N], while the MatchHdr it carries still reports the *total* byte count across all segments. At delivery, copyIoVec() is asked to copy the full multi-segment length but only has the first segment as source, so copied stalls below len and the assertion fires (or, under NDEBUG, a short/incorrect receive buffer is produced silently). Multiple segments arise because Allgather::initIoVec() coalesces only physically contiguous chunks: a recursive-doubling stage whose send window wraps the modular buffer is split into two or more non-contiguous runs. This is why the failure reproduces reliably at high PPN (almost every neighbour is on-node) and even at 2 ranks/node with a large payload, but never for point-to-point or small single-segment collectives. Fix --- * LoopReq (ctrlMsgProcessQueuesState.h): copy every data segment (vec[1..N]) instead of only vec[1], so multi-segment intra-node sends are delivered intact. * copyIoVec (ctrlMsgProcessQueuesState.cc): harden as defence-in-depth. Bound the copy loops on `rV < dst.size()` in the loop conditions (instead of an assert that is compiled out under NDEBUG), and replace the bare assert(copied == len) with a dbg().fatal() that reports copied, len, src/dst segment counts and total byte counts. A mismatch is an internal defect between sender and receiver of the same collective algorithm, so it is surfaced loudly rather than silently truncated. Verified by reproducing the abort with the original code at numCores >= 8, then confirming clean completion after the fix across numCores = 2/8/16/56 and count = 1/131072 (Allgather). Fixes sstsimulator#2686

feldergast · 2026-06-16T19:24:44Z

Just noticed that this PR is agains the master branch. Can you move it over to target devel?

sst-autotester · 2026-06-17T08:53:24Z

Status Flag 'Pre-Test Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
NO INSPECTION HAS BEEN PERFORMED ON THIS PULL REQUEST! - This PR must be inspected by setting label 'AT: PRE-TEST INSPECTED'.

sst-autotester · 2026-06-17T18:20:10Z

Status Flag 'Pre-Test Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED by label AT: PRE-TEST INSPECTED! Autotester is Removing Label; this inspection will remain valid until a new commit to source branch is performed.

sst-autotester · 2026-06-17T18:23:22Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements

Build Num: 2801
Status: STARTED

Build Information

Test Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_Make-Dist

Build Num: 1560
Status: STARTED

Build Information

Test Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MT-2

Build Num: 2742
Status: STARTED

Build Information

Test Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MR-2

Build Num: 2744
Status: STARTED

Build Information

Test Name: SST__Autotest_OSX-15-XC16_OMPI-4.1.6_PY3.10_sst-elements

Build Num: 1278
Status: STARTED

Using Repos:

Repo: ELEMENTS (deanchester/sst-elements)

Repo: SQE (sstsimulator/sst-sqe)

Repo: CORE (sstsimulator/sst-core)

Repo: MACRO (sstsimulator/sst-macro)

Pull Request Author: deanchester

sst-autotester · 2026-06-17T19:05:44Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements

Build Num: 2801
Status: PASSED

Build Information

Test Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_Make-Dist

Build Num: 1560
Status: PASSED

Build Information

Test Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MT-2

Build Num: 2742
Status: PASSED

Build Information

Test Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MR-2

Build Num: 2744
Status: PASSED

Build Information

Test Name: SST__Autotest_OSX-15-XC16_OMPI-4.1.6_PY3.10_sst-elements

Build Num: 1278
Status: PASSED

sst-autotester · 2026-06-17T19:06:03Z

Status Flag 'Pre-Merge Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED AND APPROVED by [ feldergast ]!

sst-autotester · 2026-06-17T19:06:10Z