[LGR] flow: gather parallel LGR INIT transmissibilities by arturcastiel · Pull Request #7218 · OPM/opm-simulators

arturcastiel · 2026-07-08T15:01:43Z

This PR carries both the reusable parallel INIT compare infrastructure (the COMPARE_MODE init driver, first commit) and the parallel LGR INIT output feature. It supersedes and consolidates #7211, which is now closed — the infrastructure commit lives here as the branch base so the regression test lands together with the feature that makes it pass.

A parallel (MPI) flow run with LGRs cannot otherwise write a correct INIT file: the output path queries transmissibilities on the global refined grid, which the coarse per-rank globalTrans_ cannot answer. This PR makes the parallel LGR INIT output (TRANX/TRANY/TRANZ + NNC arrays) correct by reusing the transmissibilities the distributed simulator already computed.

Approach. Each rank walks its interior leaf cells and records every connection it owns from its own distributed transmissibilities, keyed by level-Cartesian index — same-level connections as (level, min, max), level-crossing ones as (smaller level, its index, larger level, its index). These keys are geometrically canonical: identical on the distributed grid and the I/O rank's global view. Each rank builds its (key, value) records directly and gathers them to the I/O rank with Opm::gatherv — one collective per kind. Every connection is recorded exactly once: a same-level connection by the owner of the smaller level-Cartesian index (a rank-independent comparison; the recording rank has its partner cell in the overlap layer), a level-crossing connection from the smaller-level side. Nothing is recomputed on the I/O rank, and in parallel LGR runs no whole-grid transmissibility object is stored. Serial runs and parallel runs without LGRs are unchanged.

Storage and lookup. On the I/O rank the gathered records are held in a flat open-addressing (linear-probe) table, LgrTransIndex, behind the findGatheredTrans_ seam (opm/simulators/flow/LgrOutputTransGather.hpp). The gather utility is templated on the value function so the same machinery can be reused per report step by summary and restart output, where the lookup runs on every step; a contiguous table is faster to build and query than a sorted vector + binary search and cache-friendlier than a node-based map, which also cannot be MPI-gathered. The sorted-vector lookup remains available behind the same seam. The gathered mode is an explicit std::optional, so an empty record set never silently falls back to the whole-grid object; a missing key during the output walk is a hard error agreed collectively before the NNC broadcast, so all ranks abort together.

Requirements and limitations.

At least one overlap layer is required (--num-overlap >= 1); runs with LGRs and zero overlap abort with a message.
A deck NNC whose endpoint cell is refined by an LGR is no longer a leaf connection and keeps its deck-specified transmissibility.
MPI's int counts/displacements bound the gathered record total across all ranks.

Testing.

compareParallelInitSim_flow+SPE1CASE1_CARFIN — serial vs 4-rank INIT/EGRID compare on the SPE1CASE1_CARFIN deck (two LGRs straddling rank boundaries). Passed.
compareParallelSim_flow+SPE1CASE2 — unchanged non-LGR path. Passed.
test_LgrTransIndex — unit test for the container: empty/non-I/O-rank state, exact value round-trip, absent-key termination, N=3 and N=4 key widths, and sizes straddling the capacity boundary. Passed.

Scope. INIT output only. Summary, restart write, restart read, and RFT for parallel LGR runs are separate work.

Add a serial-vs-parallel regression for the parallel LGR INIT output using existing infrastructure: - run-parallel-regressionTest.sh gains a backward-compatible "-m <mode>" flag (default "summary" keeps the current behaviour for every existing caller; "init" does a dry run and compares EGRID+INIT only, ignoring the parallel-only MPI_RANK keyword). - add_test_compare_parallel_simulation gains an optional COMPARE_MODE parameter that forwards "-m init" and names the test compareParallelInitSim_<sim>+<case>. - The test is registered against the existing SPE1CASE1_CARFIN deck (opm-tests/lgr), which has two 12-host-cell LGRs on a 10x10x3 grid. Under the default 4-rank partition the LGR host cells land on multiple ranks (rank 0 marks 9 of the 24 host cells during local refinement), so the compare exercises rank-boundary-straddling LGRs. The registration sits before the opm_set_test_driver switch to run-comparison.sh so it picks up the run-parallel-regressionTest.sh driver that understands "-m".

blattms · 2026-07-17T14:48:03Z

+        return this->globalTrans().transmissibility(c1, c2);
+    }
+
+    if (const double* value = this->findGatheredTrans_(key)) {


Not 100% sure but I can imagine that some tools might complain about this without "; value != nullptr

Done — changed to if (const double* value = findGatheredTrans_(key); value != nullptr).

blattms · 2026-07-17T14:49:43Z

+    std::string msg = "Gathered LGR transmissibilities: no value for connection key (";
+    for (std::size_t j = 0; j < N; ++j) {
+        msg += std::to_string(key[j]);
+        msg += (j + 1 < N) ? ", " : ")";
+    }
+    throw std::logic_error { msg };
+}


You should probably move this into findGatheredTrans_. Then you don't need the if above

Removed the redundant second lookup here: the branch now calls findGatheredTrans_(key) once and reuses the pointer, instead of testing it to guard the branch and then querying it again inside gatheredOrGlobalTrans_. I kept the has_value() guard, because at this deck-NNC site a null result has two distinct meanings that need different handling: no gathered records at all -> use the whole-grid transmissibility; records present but this key absent -> the NNC endpoint is refined by an LGR, so keep the deck-specified value (skip). Folding the guard away would merge those two outcomes. Behaviour is unchanged and the parallel INIT regression stays green.

blattms · 2026-07-17T14:56:50Z

+        }
+        records.emplace_back(key, values[i]);
+    }
+    std::sort(records.begin(), records.end());


You should not use the whole pair for sorting, but only the first entry. Please use an apropriate LessThan operator for this as an additional argument to std::sort

The sort is gone — this file no longer sorts the records at all. They are now placed in an open-addressing hash index (LgrTransIndex<N>) rather than a key-sorted vector, so there is no std::sort and no comparator. If we later adopt the pre-sort + std::inplace_merge route for the per-report-step reuse, we will sort by key only via a LessThan on .first (never on the double), as you suggest.

blattms · 2026-07-17T15:06:50Z

+    std::vector<int> sameKeys;      // flattened: level, minCart, maxCart per record
+    std::vector<double> sameValues;
+    std::vector<int> crossKeys;     // flattened: smallLevel, smallCart, largeLevel, largeCart
+    std::vector<double> crossValues;


Can we get rid off these intermediate containers and use the final containers only?

Later you use them to create new vectors with value_type std::pair<KeyType, ValueType>. that means we need twice the amount of memory and will do additional traversals and allocations.

There might be another advantage of using std::pair<KeyType, ValueType> here: You can already sort them here before gathering and then later use multiple std::inplace_merge of the gathered containers. Of course this might turn out as premature optimization and we will have to benchmark this.

Reworked: the separate flat key/value buffers and the zip pass are removed — each rank builds the final std::vector<std::pair<std::array<int,N>,double>> directly and gatherv moves the records in one collective per kind (the MPITraits specialisation composes the pair). The only remaining intermediate is the gathered vector, which gatherv necessarily materialises before the index is built. On inplace_merge: that optimises a sorted-vector build; here lookup dominates (the same gather is reused per report step by summary/restart), so the records go into an open-addressing hash — no sort, no merge. On a single container: same-level (N=3) and cross-level (N=4) records have different key arities; they can be unified into one sentinel-tagged N=4 table (one vector, one collective), which I would rather evaluate with the per-report-step summary output where the collective count actually matters, than in this one-shot INIT change.

blattms · 2026-07-17T15:13:00Z

+            const int levelOut = outside.level();
+
+            if (levelIn != levelOut) {
+                if (levelIn > levelOut) {


Not sure whether this helps, but we could use the DUNE global ids of the cells (grid.globalIdSet().id(inside)>grid.globalIdSet().id(outside) to decide which intersections to use.

Then the one "continue" would suffice and we could move that part before the if-statement

Nice idea for the structure. One caveat: the global-id comparison picks the recording side, but the key still has to be canonical (min/max level-Cartesian) on its own, because the output side looks connections up by {level, min, max}. Today the cartIn < cartOut check happens to do both at once; switching to global-ids means canonicalising the key explicitly (std::minmax for same-level, smaller-level-first for cross-level). I would like to take this with the summary/restart refactor so it lands with its own parallel regression, rather than re-touch the proven INIT path here.

blattms · 2026-07-17T15:42:19Z

+                outputTrans_->at(level).at("TRANX").template data<double>()[minLevelCartIdx] =
+                    gatheredOrGlobalTrans_(std::array{level, minLevelCartIdx, maxLevelCartIdx}, c1, c2);


I think for LGR this lookup is rather expensive. Hence we will need to improve this later (probably with specialized code for LGRs)

Agreed, this stays for later. The records now sit in an O(1) open-addressing index rather than a binary search, and I removed a redundant second lookup of the same key at the deck-NNC site, so the per-connection cost is already lower; the specialised LGR path is the natural next step on the summary/restart work.

arturcastiel · 2026-07-24T09:41:13Z

jenkins build this please

arturcastiel · 2026-07-24T10:37:59Z

jenkins build this please

…ties A parallel (MPI) flow run with LGRs cannot write a correct INIT file: the output path queries transmissibilities on the global refined grid, which the coarse per-rank globalTrans_ cannot answer. Reuse the transmissibilities the distributed simulator already computed instead. Each rank walks its interior leaf cells and records every connection it owns, keyed by level-Cartesian index: same-level connections as (level, min, max), level-crossing ones as (smaller level, its index, larger level, its index). The keys are geometrically canonical -- identical on the distributed grid and the I/O rank's global view -- so the existing output walk looks the values up directly (new gatherLgrOutputTrans in LgrOutputTransGather.hpp). Each rank builds its (key, value) records directly and gathers them with one collective per kind; the I/O rank indexes them in a flat open-addressing table (LgrTransIndex) behind findGatheredTrans_, holding every connection of the global grid without a node-based container. Every connection is recorded exactly once: a same-level connection by the rank owning the smaller level-Cartesian index (a rank-independent comparison; the recording rank has its partner cell in the overlap layer, so at least one overlap layer is required and runs with LGRs and --num-overlap=0 abort with a clear message), a level-crossing connection from the smaller-level side. A missing key during the output walk is a hard error, agreed collectively before the NNC broadcast so every rank aborts together. A deck NNC whose endpoint is refined by an LGR keeps its deck-specified transmissibility. Nothing is recomputed on the I/O rank and no whole-grid transmissibility object is stored in parallel LGR runs. Serial runs and parallel runs without LGRs are unchanged: with no gathered records set, the writer queries the whole-grid transmissibility object exactly as before.

Add a direct unit test for LgrTransIndex, the flat open-addressing table that holds the gathered LGR output transmissibilities on the I/O rank. The parallel INIT regression exercises it end to end; this pins it in isolation: a default-constructed index (the state on every non-I/O rank) misses every lookup; build-from-records returns each value exactly and reports absent keys as null, for both the same-level (N=3) and cross-level (N=4) key widths; an absent key terminates the probe sequence; negative and large key components round-trip through the hash; and sizes straddling the power-of-two capacity boundary find every key.

arturcastiel · 2026-07-24T11:37:04Z

jenkins build this please

arturcastiel added the manual:irrelevant This PR is a minor fix and should not appear in the manual label Jul 8, 2026

arturcastiel marked this pull request as draft July 8, 2026 15:03

blattms reviewed Jul 21, 2026

View reviewed changes

arturcastiel mentioned this pull request Jul 24, 2026

[LGR] flow: reuse parallel regression driver for INIT compare #7211

Closed

arturcastiel added 2 commits July 24, 2026 11:30

arturcastiel force-pushed the lgr-init-refined-trans-pr001a branch from bc8a4b2 to 9c10540 Compare July 24, 2026 11:32

arturcastiel marked this pull request as ready for review July 24, 2026 11:51

arturcastiel requested a review from blattms July 24, 2026 11:51

		outputTrans_->at(level).at("TRANX").template data<double>()[minLevelCartIdx] =
		gatheredOrGlobalTrans_(std::array{level, minLevelCartIdx, maxLevelCartIdx}, c1, c2);

Uh oh!

Conversation

arturcastiel commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arturcastiel commented Jul 24, 2026

Uh oh!

arturcastiel commented Jul 24, 2026

Uh oh!

arturcastiel commented Jul 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

arturcastiel commented Jul 8, 2026 •

edited

Loading