Enable using Kokkos with CUDA backend for MIRCO and in general by PhilipOesterlePekrun · Pull Request #2012 · 4C-multiphysics/4C

PhilipOesterlePekrun · 2026-05-04T10:31:46Z

Description and Context

This PR adds build-system support for compiling 4C with CUDA-enabled Kokkos. For this purpose, this PR introduces a compiler wrapper, clangcuda++, which should be used as the CMAKE_CXX_COMPILER (or, in the case of MPI, the OMPI_CXX backend), as well as some CMake additions which are optionally enabled.

This change is based on my attempts to get 4C to compile while using MIRCO (library) with Kokkos' CUDA backend for GPU offloading. Several 4C translation units fail to compile with the kokkos_launch_compiler or nvcc_wrapper provided by Kokkos because NVCC does not seem to be able to compile some more "advanced" C++ code, which we have a fair bit of in 4C since it is a large project. Clang happens to compile CUDA or CUDA-related code much better than Nvidia's own compiler (don't ask me why), but still has some issues by itself. Using the clangcuda++ wrapper as the CMAKE_CXX_COMPILER (or, in the case of MPI, the OMPI_CXX backend), along with the corresponding target properties and compile definitions by setting FOUR_C_CLANGCUDA=ON, allows compiling all of 4C with CUDA-enabled Kokkos.

The current implementation distinguishes between CUDA host-side compilation and CUDA device compilation. At the moment, 4C itself only requires CUDA host-side compilation, since it does not yet contain raw Kokkos device kernels such as Kokkos::parallel_for, KOKKOS_LAMBDA, etc. in its own sources. But, this is now possible by simply marking a target or source with FOUR_C_CLANGCUDA_DEVICE_COMPILE, allowing GPU offloading anywhere in 4C. As long as Trilinos is built with TPETRA_INST_CUDA=OFF, this also does not conflict with any existing MPI parallelism (NO will be KokkosSerial).

The changes were verified by compiling 4C successfully with the relevant Kokkos CUDA-enabled setup and I also tested small Kokkos kernel examples to confirm that host-side and device CUDA compilation are distinguished correctly. I have documented how Trilinos, MIRCO, and 4C should be compiled for this combination to work. I'll just attach that to this PR for now: DocumentKokkosCuda_4C_Trilinos_MIRCO.zip

Once imcs-compsim/MIRCO#146 is merged, FetchContent will work without issue for that MIRCO state.

My questions:

Should I make a build test?
Should I make a framework test (if we can test with GPUs)?
Should I make a small example or tutorial to show exactly how to enable using raw Kokkos device kernels?

Related Issues and Pull Requests

Blocked by imcs-compsim/MIRCO#146

Docker and Workflow tests

I have added a Dockerfile, Trilinos installation scripts, and the buildtest and docker workflows, which are similar to the main buildtest.yaml and docker.yaml.

Signed-off-by: Philip Oesterle-Pekrun <philipoesterlepekrun@gmail.com>

isteinbrecher · 2026-05-05T10:00:00Z

 #else

-    using ExecutionSpace = Kokkos::DefaultExecutionSpace;
+    using ExecutionSpace = Kokkos::DefaultHostExecutionSpace;


Why is this changed to host?

Sorry for the late reply @isteinbrecher.

This was causing an error related to ArborX when compiling (ArborX_compileFailure.log), but I just realized that this is because I let 4C fetch ArborX and then ArborX's CUDA support is off by default (ArborX also uses CUDA through Kokkos, but I think you need to enable it explictly in ArborX as well). You can build ArborX with CUDA and then it should work. However, I'm not sure whether this might lead to oversubscription of the GPU with our typical use case of MPI (multiple MPI ranks per compute node). I'll look into this further, but of course for my use case I can also turn ArborX off because it is not exactly related, so you're right, I'll amend that.

For now, I'd be fine with switching to host execution space here. This preserves prior behavior, so everything remains exactly as it was for ArborX.

I think nothing in the typical MIRCO execution path uses the geometric search, so I can actually just undo the change and turn off ArborX.

Yes, MIRCO-based simulations do not use ArborX at the moment. Yet, I actually support the switch to the host execution space just to be clear, that ArborX is working on host only so far.

@maxfirmbach Any thoughts on this?

We have decided to keep it as DefaultExecutionSpace and throw a warning when compiling with FOUR_C_CLANGCUDA and FOUR_C_WITH_ARBORX.

maxfirmbach · 2026-05-06T13:14:43Z

@PhilipOesterlePekrun Thanks for the work! I really like the efforts, yet I also have some concerns. The current PR comes with quite a few assumptions and restrictions we should clearly state and discuss (maybe something for the next community meeting?).

maxfirmbach · 2026-05-06T13:19:53Z

Concerning your questions:

I think a build test is necessary (currently this PR is clang related, do we have any clues about gcc?). Maybe with Mirco and ArborX activated, which can actually use GPUs under the hood.
Testing would be nice, yet I don't know if it makes too much sense. Only a handful of tests can make use of GPUs.

For my personal taste, the current state has too many constraints to get things to work. Are there parts we can work on separately to make the use of CUDA easier in the near future?

PhilipOesterlePekrun · 2026-05-07T08:23:50Z

@maxfirmbach Thanks for the feedback. I tried to keep it as isolated as possible, but yes it does change some global CMake files (though everything is conditional upon the global option I introduced). We can definitely discuss this in the community meeting (or even earlier).

Regarding your second comment, the constraint here is really that GCC cannot compile CUDA-related code but the kokkos_launch_compiler or nvcc_wrapper use NVCC which cannot compile some of our code--hence, Clang is required to compile this. My solution really does work, but I would be happy about any suggestions to integrate it more cleanly.

I do have a general question, @isteinbrecher and @maxfirmbach, to clarify the purpose of my PR.
It was my understanding that nobody ever really used device-enabled Kokkos (CUDA or HIP etc.) with 4C. But, that was I guess an assumption, so I should clearly ask: have you ever used device-enabled Kokkos with 4C, or do you know of any instance of that? I had to introduce the changes in this PR to make it work (for the reasons I stated), but obviously my solution came from a lot of trial and error and maybe someone else has done it better (but not pushed it to 4C?).

mayrmt · 2026-05-07T09:36:11Z

@PhilipOesterlePekrun To my knowledge, nobody ever ran 4C on device so far.

Copilot

Pull request overview

This PR adds build-system support to compile 4C with a CUDA-enabled Kokkos installation by introducing a clangcuda++ compiler wrapper and a FOUR_C_CLANGCUDA CMake option that adjusts compile definitions/launchers for clang-based CUDA host/device compilation. It also updates ArborX/Kokkos usage to explicitly run on the host when Kokkos’ default execution space may be CUDA, and refreshes MIRCO integration settings.

Changes:

Added utilities/clangcuda++ wrapper to drive clang CUDA host-only/device compilation based on compile definitions and file extensions.
Introduced FOUR_C_CLANGCUDA global option and applied related target/global launcher settings + FOUR_C_CLANGCUDA_HOST_ONLY compile definition in key build targets.
Switched geometric search ArborX execution/memory spaces to Kokkos::DefaultHostExecutionSpace to avoid unintended CUDA default execution space usage; updated MIRCO fetch configuration and git tag.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
utilities/clangcuda++	New clang CUDA compiler wrapper handling host-only/device compilation modes.
src/cut/4C_cut_pointgraph.cpp	Adds a clang-CUDA-related preprocessor workaround before including Boost graphviz.
src/core/geometric_search/src/4C_geometric_search_distributed_tree.cpp	Forces ArborX distributed tree build/query to use host execution space.
src/core/geometric_search/src/4C_geometric_search_bvh.hpp	Forces BVH execution/memory space selection to host execution space.
cmake/setup_global_options.cmake	Adds `FOUR_C_CLANGCUDA` global option.
cmake/functions/four_c_auto_define_module.cmake	Applies launcher overrides + host-only define to module object libraries when enabled.
cmake/configure/configure_Trilinos.cmake	Disables compiler/rule launchers when `FOUR_C_CLANGCUDA` is enabled.
cmake/configure/configure_MIRCO.cmake	Updates MIRCO fetch configuration and moves dependency registration outside the conditional.
apps/global_full/CMakeLists.txt	Applies launcher overrides + host-only define to the main executable when enabled.

PhilipOesterlePekrun · 2026-05-07T12:19:07Z

+# Sanity check
+if [[ "$cuda_host_only" == "1" && "$cuda_device" == "1" ]]; then
+  has_explicit_device=0
+  for arg in "$@"; do
+    if [[ "$arg" == "-DFOUR_C_CLANGCUDA_DEVICE_COMPILE" ]]; then
+      has_explicit_device=1
+      break
+    fi
+  done
+
+  if [[ "$has_explicit_device" == "1" ]]; then
+    echo "clangcuda++ wrapper error: both FOUR_C_CLANGCUDA_HOST_ONLY and FOUR_C_CLANGCUDA_DEVICE_COMPILE were set" >&2
+    exit 1
+  fi
+fi
+
+if [[ "$compile" == "1" && "$cuda_host_only" == "1" ]]; then
+  final=(
+    "$clang"
+    -x cuda
+    --cuda-host-only
+    --cuda-path="$cuda_path"
+    --cuda-gpu-arch="$arch"
+    -Wno-unknown-cuda-version
+    "${args[@]}"
+  )
+elif [[ "$compile" == "1" && "$cuda_device" == "1" ]]; then
+  final=(
+    "$clang"
+    -x cuda
+    --cuda-path="$cuda_path"
+    --cuda-gpu-arch="$arch"
+    -Wno-unknown-cuda-version
+    "${args[@]}"
+  )
+else


Fair point, I took the .cu idea from kokkos_launch_compiler, but I guess we really don't need it.

+#undef __noinline__
+#endif
 #include <boost/graph/graphviz.hpp>


+four_c_process_global_option(
+  FOUR_C_CLANGCUDA
+  DESCRIPTION
+  "Enable the relevant CMake compile definitions needed to use utilities/ClangCuda++ as the compiler. This is currently necessary to use the CUDA backend of Kokkos, e.g. along with MIRCO."


+
+  get_property(_global_rule GLOBAL PROPERTY RULE_LAUNCH_COMPILE)
+  get_property(_dir_rule DIRECTORY PROPERTY RULE_LAUNCH_COMPILE)


maxfirmbach · 2026-05-07T11:45:28Z

@maxfirmbach Thanks for the feedback. I tried to keep it as isolated as possible, but yes it does change some global CMake files (though everything is conditional upon the global option I introduced). We can definitely discuss this in the community meeting (or even earlier).

Regarding your second comment, the constraint here is really that GCC cannot compile CUDA-related code but the kokkos_launch_compiler or nvcc_wrapper use NVCC which cannot compile some of our code--hence, Clang is required to compile this. My solution really does work, but I would be happy about any suggestions to integrate it more cleanly.

I do have a general question, @isteinbrecher and @maxfirmbach, to clarify the purpose of my PR. It was my understanding that nobody ever really used device-enabled Kokkos (CUDA or HIP etc.) with 4C. But, that was I guess an assumption, so I should clearly ask: have you ever used device-enabled Kokkos with 4C, or do you know of any instance of that? I had to introduce the changes in this PR to make it work (for the reasons I stated), but obviously my solution came from a lot of trial and error and maybe someone else has done it better (but not pushed it to 4C?).

@PhilipOesterlePekrun Understood! I also don't think that someone has tried this so far, because there was frankly no reason for it ... all of the code in 4C itself is host only.

PhilipOesterlePekrun · 2026-05-15T14:52:54Z

@maxfirmbach A built test is possible, however this requires the following:

To change the core Dockerfile of 4C to apt-get the relevant cuda-related packages like nvcc, cusolver, cublas, etc. Then the docker image would have to be updated of course. I see the last update was in 2024, so I suspect this is something we don't want to do too often.
Add a second dependencies/current/trilinos/install.sh, which installs Trilinos with CUDA enabled, and potentially another installation if we want to test with Kokkos' OpenMP backend.

Again, the current changes are isolated with FOUR_C_CLANGCUDA, so I might suggest that we wait on introducing a build test until other parts of the code implement Kokkos device-side code. But, I'll make a build test if that's what we want. You can see an example of the changes to make such a build test in this MIRCO PR imcs-compsim/MIRCO#146.

By the way, an actual run test is of course not possible unless we have GPU test runners for the workflow/action.

Signed-off-by: Philip Oesterle-Pekrun <philipoesterlepekrun@gmail.com>

mayrmt · 2026-05-16T22:14:08Z

@PhilipOesterlePekrun Updating the Dockerfile should be fine. Especially due to the specialized nature of the Kokkos-Cuda integration of MIRCO into 4C, where the number of users and experts is limited to less than a handful, proper testing is really important. So, I'd support to change the docker base containers and provide Cuda with them.

ppraegla · 2026-05-27T13:16:45Z

@PhilipOesterlePekrun Updating the Dockerfile should be fine. Especially due to the specialized nature of the Kokkos-Cuda integration of MIRCO into 4C, where the number of users and experts is limited to less than a handful, proper testing is really important. So, I'd support to change the docker base containers and provide Cuda with them.

Just a heads up, it might not be possible to install the cuda packages into the normal Docker image we use for testing due to the size of cuda. In the past, we had problems that there was not enough space left on the runner (14 GB disk space) to build 4C because our docker image is so big. So, we need to be careful what we add to the dependency image.

Doing sudo apt install nvidia-cuda-toolkit in the docker container shows that the installation will be 7 GB, which is definitely too large for the runner. This would only leave 2 GB to clone and build 4C.

Maybe there is a way to install only a minimal version of cuda that is sufficient for the pipeline. Or you can try to create a separate docker image that only has the minimum dependencies to build 4C with Cuda, e.g., the 4c-minimal image is 5 GB smaller than the 4c image by removing doxygen, clang, ...

PhilipOesterlePekrun · 2026-05-27T14:51:16Z

@ppraegla Yes, a separate DockerFile like the one for trilinos_develop is the way to go. I do have it with the minimal cuda toolkit requirements, which includes cusolver, but it still adds a few GB so I don't want to affect the main 4C docker image.

davidrudlstorfer · 2026-05-27T14:59:38Z

We now also briefly discussed after the meeting, and I (in my opinion) still strongly oppose adding a lot of new functionality without testing it.

My main question would be: why is then the docker container even created in the first place if it's not used for anything afterwards?

If you use this workflow daily in the IMCS you will probably not download the docker container but install it on your local system. So why creating the docker container when it won't be tested at all?

@PhilipOesterlePekrun @4C-multiphysics/maintainer

PhilipOesterlePekrun · 2026-05-27T15:11:48Z

@davidrudlstorfer The compilation is tested, but it's true that this may not be necessary until we have Kokkos device code in some other parts of 4C, as the compilation of MIRCO itself is already tested in MIRCO's repository. So, if we agree that it is not needed currently, I can keep those changes in a branch of my fork in case we do want it later, that is totally fine too. The docker would have just been for the build test, yes.

PhilipOesterlePekrun · 2026-05-28T11:59:21Z

@mayrmt
I accidentally clicked close PR. Regarding nvcc and the need for clangcuda++, I've compiled with both nvcc_wrapper and kokkos_launch_compiler. The latter automatically uses nvcc on everything with -DKOKKOS_DEPENDENCE, which ends up being all or almost all 4C targets because they transitively link to Kokkos due to Xpetra being used in core/linalg. I think nvcc_wrapper and kokkos_launch_compiler are functionally the same then, and everything is compiled with nvcc as device code (even though it isn't). And nvcc has some issues

I've attached the full build.log which I made with ninja -j64 -k 0 2>&1 | tee build.log, as well as a filtered version which is an attempt to show the errors uniquely (there are 15 different headers which are root issues for many TUs/targets). It is identical whether nvcc_wrapper or kokkos_launch_compiler is used. Basically, fixing this for nvcc might require changing multiple different parts of the code, whereas clangcuda++ is at least isolated (in this PR, I did have to conditionally undefine one macro for an #include, but that is a CUDA thing in general). Clang is simply more compliant with C++ than nvcc.

build.log.gz (compressed due to GitHub file size limit)
unique-error-files.txt

Signed-off-by: Philip Oesterle-Pekrun <philipoesterlepekrun@gmail.com>

sebproell · 2026-06-23T18:37:57Z

+                   RULE_LAUNCH_COMPILE ""
+                   RULE_LAUNCH_LINK ""
+        )
+      target_compile_definitions(${_target}_objs PRIVATE CLANGCUDA_MODE_HOST)


Ah, I see. I think what you are trying to say is that set_target_properties does not propagate to users of that target via target_link_libraries(). Too bad.

In this case, I would create a helper function four_c_set_cuda_properties(target) that performs this setup and call it in the four places that currently have the code copy-pasted.

Co-authored-by: Sebastian Proell <63345705+sebproell@users.noreply.github.com> Signed-off-by: Philip Oesterle-Pekrun <philipoesterlepekrun@gmail.com>

Signed-off-by: Philip Oesterle-Pekrun <philipoesterlepekrun@gmail.com>

PhilipOesterlePekrun · 2026-06-24T10:02:36Z

Ok the kokkosparallel buildtests take super long for some reason. Let's not merge this until I fix that.

Signed-off-by: PhilipOesterlePekrun <philipoesterlepekrun@gmail.com>

jeremylt

I'm getting myself oriented here, so I have few questions as a 4C/Trilinos/Kokkos newcomer

jeremylt · 2026-07-01T09:13:36Z

+
+|FOURC| is primarily developed around MPI parallelism, but also offers the ability to use shared memory parallelism through `Kokkos <https://kokkos.org/> _`, enabling hybrid parallelism on the CPU through OpenMP and GPU acceleration through CUDA.
+
+Kokkos (and Kokkos-Kernels) can be built within Trilinos or specified as an external TPL in Trilinos, and its configuration follows the usual procedure for the desired backend (see the `Kokkos configuration guide <https://kokkos.org/kokkos-core-wiki/get-started/configuration-guide.html> _`). Trilinos then requires `Trilinos_ENABLE_<Backend>=ON` and, specifically for CUDA, `Trilinos_ENABLE_TPL_CUDA=ON`. To prevent oversubscription and unwanted shared memory parallelism in 4C, one should disable these backends for Tpetra with `TPETRA_INST_<BACKEND>=OFF` and explicitly set `TPETRA_INST_SERIAL=ON`.


Question: Does this potentially induce a bottleneck that could noticeably slow end to end simulation time? I don't know where/how Tpetra is used or how much of the end to end time it typically consumes, so this is for my curiosity and trying to think ahead.

For now, this is not a bottleneck, since we cannot use Tpetra anyways. The need set TPETRA_INST_<BACKEND>=OFF and explicitly set TPETRA_INST_SERIAL=ON is just, that we are still using Epetra. This will hopefully change, when we move to Tpetra at some point in the future.

jeremylt · 2026-07-01T09:17:12Z

+
+Kokkos (and Kokkos-Kernels) can be built within Trilinos or specified as an external TPL in Trilinos, and its configuration follows the usual procedure for the desired backend (see the `Kokkos configuration guide <https://kokkos.org/kokkos-core-wiki/get-started/configuration-guide.html> _`). Trilinos then requires `Trilinos_ENABLE_<Backend>=ON` and, specifically for CUDA, `Trilinos_ENABLE_TPL_CUDA=ON`. To prevent oversubscription and unwanted shared memory parallelism in 4C, one should disable these backends for Tpetra with `TPETRA_INST_<BACKEND>=OFF` and explicitly set `TPETRA_INST_SERIAL=ON`.
+
+To build 4C with this configuration, a compiler wrapper, `utilities/clangcuda++` must be used as the `CMAKE_CXX_COMPILER`, while clang should be used as the `CMAKE_C_COMPILER`. When using MPI, these should instead be set as the `OMPI_CXX` and `OMPI_CC` environment variables respectively. To change the GPU architecture or default clang++ and CUDA paths, one should set the corresponding environment variables listed at the start of the `utilities/clangcuda++` compiler wrapper. Additionally, the `FOUR_C_CLANGCUDA` compile option must be enabled in 4C. Due to incompatibility with the serial version of ArborX, it is recommended to disable `FOUR_C_WITH_ARBORX`.


Question: 'should' or 'must' for using Clang as the C compiler? Can one use GCC for C while using Clang for CXX? Has this combination been tested with 4C?

Also, 'clang' or 'Clang' here?

Additional note for this section - it might be worth mentioning to users that given the large number of dependencies, failure late in the building process is normal so setup for 4C with CUDA enabled Kokkos may be very time consuming

jeremylt · 2026-07-01T09:30:37Z

+          image-name: ${{ env.REGISTRY }}/${{ env.PROJECT_NAMESPACE }}/4c-${{ env.IMAGE_SUFFIX }}
+          dependencies-hash: ${{ needs.check-if-build-dependencies-is-required.outputs.dependencies_hash
+            }}
+          base-image: "nvidia/cuda:12.8.1-devel-ubuntu24.04"


Are these containers used widely outside of CI? If so, would it be beneficial for some users to have a separate lighter container with only OpenMP and not CUDA?

jeremylt · 2026-07-01T09:32:40Z

  )
 if(FOUR_C_MIRCO_FIND_INSTALLED)
-
+  # Note that MIRCO and 4C must point to the same Kokkos and Kokkos-Kernels installation. Otherwise, there will be errors.


Question for understanding - exactly the same installation or is merely the same version sufficient? (I'd still generally default to advising exactly the same installation)

As far as I know, it must be the same installation, since the Kokkos backends need to match. Same software version is not sufficient.

jeremylt · 2026-07-01T09:36:05Z

 See the `MIRCO repository <https://github.com/imcs-compsim/MIRCO>`_ for details and downloads.

-Building |FOURC| with MIRCO enabled automatically fetches the repository during the configure stage and later builds the library as dependency.
+Building |FOURC| with MIRCO enabled automatically fetches the repository during the configure stage and later builds the library as dependency. Alternatively, one can specify an external MIRCO installation. In either case, MIRCO can make use of shared memory parallelism through Kokkos :ref:`when enabled <build4Cwithkokkoscuda>` in |FOURC|. Note that 4C and MIRCO must depend on the same Kokkos installation. In case using Kokkos with CUDA enabled, MIRCO must be built with `CMAKE_POSITION_INDEPENDENT_CODE=ON`.


Question for understanding: is the position independent code flag needed on the MICRO or 4C side or both?

jeremylt · 2026-07-01T09:39:25Z

 #include "4C_cut_pointgraph_simple.hpp"
 #include "4C_cut_side.hpp"

+#if defined(CLANGCUDA_MODE_HOST) || defined(CLANGCUDA_MODE_DEVICE)


Is this a known documented issue for Boost with an issue/bug tracker entry somewhere that we can point future developers at in case we don't need this in the future?

Signed-off-by: PhilipOesterlePekrun <philipoesterlepekrun@gmail.com>

PhilipOesterlePekrun · 2026-07-02T06:38:40Z

So, I changed it to a minimal ctest for two reasons. Firstly there were a couple edge cases where the tests ran into errors for KokkosOpenMP, due to some Epetra/Tpetra backend stuff or just timeouts. The thing is, this PR does not add anything directly OpenMP related other than the workflow--it is about CUDA. There are many ways to change the Trilinos configuration such that it breaks certain parts of 4C. Working with OpenMP enabled is not something most people should or will do, so I don't think a full test run is that useful.

Secondly, I don't want to add another 50% to the time every PR needs to wait for the checks. In fact, if it is ok, @sebproell @mayrmt , I might suggest making the ctest not required for merging, for future safety--that is, removing it from the buildtest_kokkos_parallel / ensure_all_tests_pass (and then renaming that to ensure_all_builds_pass).

jeremylt · 2026-07-03T14:01:12Z

The concern about build time makes sense. Though, I was under the impression that we do not currently have CUDA hardware that we can run CI on and OpenMP could be used as a surrogate to ensure shared memory parallelism is covered by CI once it is added in follow up development work?

mayrmt · 2026-07-03T14:17:14Z

@jeremylt, that is true! Though at least, we can test the CUDA build (without execution). That is the current plan.

jeremylt · 2026-07-09T07:35:11Z

Hmm, I successfully built locally for OpenMP, but not for CUDA. When building Mirco, I get errors about

clang++: error: unknown argument: '-extended-lambda'
clang++: error: unknown argument: '-expt-relaxed-constexpr'
clang++: error: unknown argument '-arch=sm_90'; did you mean '-march=sm_90'?

so it looks like the NVCC flags aren't quite being correctly converted to the Clang equivalent? If I am understanding correctly? I'm on your branch and using the scripts/configurations for Docker. Am I missing something that I need to set?

PhilipOesterlePekrun · 2026-07-09T08:17:25Z

@jeremylt Yes, I know this error, but I forgot what causes it--happens when one of the necessary steps is missing. Did you enable the compile definition FOUR_C_CLANGCUDA? The following compile definitions have to be set as in the docker preset

"cacheVariables": {
  "MPI_C_COMPILER": "<path/to/mpicc>",
  "MPI_CXX_COMPILER": "<path/to/mpicxx>",
  "FOUR_C_CLANGCUDA": "ON",
  "FOUR_C_WITH_ARBORX": "OFF",
  "FOUR_C_WITH_DEAL_II": "OFF",
  "FOUR_C_WITH_VTK": "OFF",
  "FOUR_C_ENABLE_METADATA_GENERATION": "OFF"
}

Also, you should set this in environment

export OMPI_CC=<path/to/clang>
export OMPI_CXX=<path/to/4C>/utilities/clangcuda++

Lastly, you can change things like the GPU architecture through the environment variables as described in the clangcuda++ wrapper.

Thanks for trying to compile it. It's useful to make sure that others can compile it.

PhilipOesterlePekrun · 2026-07-09T09:11:29Z

Should we merge this? We can add additional documentation in a new PR I think.

jeremylt · 2026-07-09T09:30:37Z

Ah, this was a missing piece

Also, you should set this in environment
export OMPI_CC=<path/to/clang>
export OMPI_CXX=<path/to/4C>/utilities/clangcuda++

I have been setting "FOUR_C_CLANGCUDA": "ON", but unfortunately no dice yet. /usr/bin/clang++ is still being used over <path/to/4C>/utilities/clangcuda++.

mayrmt

@PhilipOesterlePekrun Thanks for this effort! I am fine with the current changes, though there are still some question by @sebproell and @jeremylt, where I do not know, if they are still relevant.

mayrmt · 2026-07-09T09:45:48Z

  )
 if(FOUR_C_MIRCO_FIND_INSTALLED)
-
+  # Note that MIRCO and 4C must point to the same Kokkos and Kokkos-Kernels installation. Otherwise, there will be errors.


As far as I know, it must be the same installation, since the Kokkos backends need to match. Same software version is not sufficient.

mayrmt · 2026-07-09T09:49:53Z

+
+|FOURC| is primarily developed around MPI parallelism, but also offers the ability to use shared memory parallelism through `Kokkos <https://kokkos.org/> _`, enabling hybrid parallelism on the CPU through OpenMP and GPU acceleration through CUDA.
+
+Kokkos (and Kokkos-Kernels) can be built within Trilinos or specified as an external TPL in Trilinos, and its configuration follows the usual procedure for the desired backend (see the `Kokkos configuration guide <https://kokkos.org/kokkos-core-wiki/get-started/configuration-guide.html> _`). Trilinos then requires `Trilinos_ENABLE_<Backend>=ON` and, specifically for CUDA, `Trilinos_ENABLE_TPL_CUDA=ON`. To prevent oversubscription and unwanted shared memory parallelism in 4C, one should disable these backends for Tpetra with `TPETRA_INST_<BACKEND>=OFF` and explicitly set `TPETRA_INST_SERIAL=ON`.


For now, this is not a bottleneck, since we cannot use Tpetra anyways. The need set TPETRA_INST_<BACKEND>=OFF and explicitly set TPETRA_INST_SERIAL=ON is just, that we are still using Epetra. This will hopefully change, when we move to Tpetra at some point in the future.

PhilipOesterlePekrun · 2026-07-09T09:58:36Z

@mayrmt I think I answered all of the reviews/questions by Sebastian and Jeremy. Regarding your last two questions:

Yes, the same installation is needed, because the symbols and such are defined globally--that is just how ldd works.
There are certain places where we use Xpetra or Tpetra, such as in the MueLu adapter. Also, I think that not all Trilinos packages are completely clean and will sometimes use the Kokkos::DefaultExectionSpace directly, which would lead to oversubscription with MPI. I have noticed this, so it is best to explicitly do TPETRA_INST_<BACKEND>=OFF in Trilinos. There is one Trilinos bug which means that such an installation with OpenMP will give an error in 4C, which I have fixed with a hack (in 4C), but this PR is about CUDA so I will keep that for later.

sebproell

CMake looks reasonable now, thanks

sebproell

CMake looks reasonable now, thanks

mayrmt

Looks good!

jeremylt · 2026-07-10T07:06:13Z

+  -D Trilinos_ENABLE_CUDA:BOOL=ON \
+  -D TPL_ENABLE_CUDA:BOOL=ON \
+  -D Kokkos_ENABLE_CUDA:BOOL=ON \
+  -D Kokkos_ARCH_HOPPER90:BOOL=ON \


hmm, is there a way to comment in these since the are commonly used as examples - people will need to update this part to their arch

@jeremylt This particular install script is for CUDA, so one can use it as a CUDA example. There is another install script for OpenMP, that should have different settings. Does this make sense to you?

yes, I know that part, but not every CUDA device will have the arch of Hopper 90 - my development machine is an Ampere 86 for example

Ah, I see. So, you proposal is to define the architecture as a variable in the script and then insert it here into the cmake command, right?

That would work. I'm not familiar with the standard user workflows when it comes to using these scripts as examples, so I am not sure what users would find normal/expected for that sort of adjustment to match their hardware

I do not use these scripts. Personally, I'd invoke the Trilinos cmake command from a script, but do everything else such as cloning Trilinos or later building/installing Trilinos manually.

I have to take back my earlier recommendation. Since this is a boolean option, exposing it as a variable does not help, since I have to choose the right variable for each architecture. Right now, I do not see a good option. We could link to the Kokkos documentation (https://kokkos.org/kokkos-core-wiki/API/core/Macros.html#architectures).

I am always a fan of directing people to external documentation so they get the most accurate information.

I mentioned the scripts as examples because I seem to remember the 4C documentation calling out these scripts as examples of how to install 4C and the dependencies, unless I'm misremembering (totally possible, honestly). If that's the case in fact, then my worry is that a new user would look here and incorrectly follow the script because they don't know about CUDA arches.

I agree!

I am not sure how to refer to the Kokkos documentation. We cannot comment in the cmake command. We could add a regular comment at the beginning of the script. @PhilipOesterlePekrun has added some remarks to our installation guide. That might also be a better place to refer to external documentation.

Signed-off-by: PhilipOesterlePekrun <philipoesterlepekrun@gmail.com>

jeremylt · 2026-07-13T09:29:49Z

+
+Kokkos (and Kokkos-Kernels) can be built within Trilinos or specified as an external TPL in Trilinos, and its configuration follows the usual procedure for the desired backend (see the `Kokkos configuration guide <https://kokkos.org/kokkos-core-wiki/get-started/configuration-guide.html> _`). Trilinos then requires `Trilinos_ENABLE_<Backend>=ON` and, specifically for CUDA, `Trilinos_ENABLE_TPL_CUDA=ON`. To prevent oversubscription and unwanted shared memory parallelism in 4C, one should disable these backends for Tpetra with `TPETRA_INST_<BACKEND>=OFF` and explicitly set `TPETRA_INST_SERIAL=ON`.
+
+To build 4C with this configuration, a compiler wrapper, `utilities/clangcuda++` must be used as the `CMAKE_CXX_COMPILER`, while clang must be used as the `CMAKE_C_COMPILER`. When using MPI, these must instead be set as the `OMPI_CXX` and `OMPI_CC` environment variables respectively. To change the GPU architecture or default clang++ and CUDA paths, one should set the corresponding environment variables listed at the start of the `utilities/clangcuda++` compiler wrapper. Additionally, the `FOUR_C_CLANGCUDA` compile option must be enabled in 4C. Due to incompatibility with the serial version of ArborX, it is recommended to disable `FOUR_C_WITH_ARBORX`.


This says

When using MPI, these must instead be set as the OMPI_CXX and OMPI_CC environment variables respectively.

which implies that CMAKE_CXX_COMPILER and CMAKE_C_COMPILER should not be set as in the previous sentence - but that leads to a compilation error for me.

I still cannot get this to build after a week of trying, so either I am missing something in these directions or these directions are missing something, or both. I think at the very least I am misunderstanding something here.

@PhilipOesterlePekrun, I do not know where the problem comes from. Though, it is detrimental that others can build this, before we merge.

hmm, I think sometimes CLANGCUDA_MODE_DEVICE is being set when I would not expect that to happen:

FAILED: src/core/fem/benchmark_tests/CMakeFiles/benchmarktests_core.dir/__/__/__/__/tests/benchmark_tests/4C_benchmark_tests_main.cpp.o /home/thompson/Dev/4C/utilities/clangcuda++ -DBOOST_GRAPH_DYN_LINK -DBOOST_GRAPH_NO_LIB -DBOOST_MAJOR_VERSION=1 -DBOOST_MINOR_VERSION=83 -DBOOST_REGEX_DYN_LINK -DBOOST_REGEX_NO_LIB -DCLANGCUDA_MODE_DEVICE -DKOKKOS_DEPENDENCE -DMIRCO_ENABLE_VISUALIZATIONEXPORT=0 -DMPICH_SKIP_MPICXX -DOMPI_SKIP_MPICXX -D_MPICC_H -I/home/thompson/Dev/4C/src -I/home/thompson/Dev/4C/src/adapter ..... -O3 -DNDEBUG -std=c++20 -fPIE -Wall -Wextra -Wvla -Wno-unused-parameter -Wno-overloaded-virtual -Werror -ftrapping-math -O3 -funroll-loops -Wno-unknown-pragmas -extended-lambda -Wext-lambda-captures-this -expt-relaxed-constexpr -arch=sm_86 -MD -MT src/core/fem/benchmark_tests/CMakeFiles/benchmarktests_core.dir/__/__/__/__/tests/benchmark_tests/4C_benchmark_tests_main.cpp.o -MF src/core/fem/benchmark_tests/CMakeFiles/benchmarktests_core.dir/__/__/__/__/tests/benchmark_tests/4C_benchmark_tests_main.cpp.o.d -o src/core/fem/benchmark_tests/CMakeFiles/benchmarktests_core.dir/__/__/__/__/tests/benchmark_tests/4C_benchmark_tests_main.cpp.o -c /home/thompson/Dev/4C/tests/benchmark_tests/4C_benchmark_tests_main.cpp

I'm not sure what I'm doing that is making the build system think that /home/thompson/Dev/4C/tests/benchmark_tests/4C_benchmark_tests_main.cpp should have any reason to be compiled for the device 🤔

Enable the usage of Kokkos with CUDA backend

d15acc8

Signed-off-by: Philip Oesterle-Pekrun <philipoesterlepekrun@gmail.com>

PhilipOesterlePekrun marked this pull request as draft May 4, 2026 10:32

PhilipOesterlePekrun force-pushed the mirco/KokkosCudaCleanedUp branch from da5207d to 575c3b3 Compare May 4, 2026 12:53

isteinbrecher reviewed May 5, 2026

View reviewed changes

amgebauer requested a review from Copilot May 7, 2026 09:41

Copilot started reviewing on behalf of amgebauer May 7, 2026 09:42 View session

Copilot AI reviewed May 7, 2026

View reviewed changes

PhilipOesterlePekrun force-pushed the mirco/KokkosCudaCleanedUp branch 4 times, most recently from 5332d8c to 3d2f036 Compare May 15, 2026 12:59

This was referenced May 15, 2026

Meeting on 27.05.2026 4C-multiphysics/community-meeting#21

Open

Enable compilation with clangcuda++ and enable kokkoscuda build tests [Related to 4C] imcs-compsim/MIRCO#146

Merged

Add global option and ArborX conflict warning

701569c

Signed-off-by: Philip Oesterle-Pekrun <philipoesterlepekrun@gmail.com>

PhilipOesterlePekrun closed this May 28, 2026

PhilipOesterlePekrun reopened this May 28, 2026

PhilipOesterlePekrun force-pushed the mirco/KokkosCudaCleanedUp branch from 3d2f036 to 6da5580 Compare May 28, 2026 14:37

Add documentation for using Kokkos with CUDA in 4C

b261b1c

Signed-off-by: Philip Oesterle-Pekrun <philipoesterlepekrun@gmail.com>

sebproell reviewed Jun 23, 2026

View reviewed changes

Throw error for VTK and ClangCuda

0690f82

Co-authored-by: Sebastian Proell <63345705+sebproell@users.noreply.github.com> Signed-off-by: Philip Oesterle-Pekrun <philipoesterlepekrun@gmail.com>

PhilipOesterlePekrun marked this pull request as ready for review June 24, 2026 08:31

Add a function for setting clangcuda mode

3051cc0

Signed-off-by: Philip Oesterle-Pekrun <philipoesterlepekrun@gmail.com>

PhilipOesterlePekrun force-pushed the mirco/KokkosCudaCleanedUp branch from b447897 to 3051cc0 Compare June 24, 2026 08:45

Fix kokkosparallel docker

6d5cfe2

Signed-off-by: PhilipOesterlePekrun <philipoesterlepekrun@gmail.com>

jeremylt reviewed Jul 1, 2026

View reviewed changes

Add test_minimal to kokkosparallel workflow

a97aab8

Signed-off-by: PhilipOesterlePekrun <philipoesterlepekrun@gmail.com>

PhilipOesterlePekrun requested a review from mayrmt July 9, 2026 09:10

mayrmt approved these changes Jul 9, 2026

View reviewed changes

PhilipOesterlePekrun requested a review from sebproell July 9, 2026 12:41

sebproell reviewed Jul 9, 2026

View reviewed changes

sebproell previously approved these changes Jul 9, 2026

View reviewed changes

mayrmt previously approved these changes Jul 9, 2026

View reviewed changes

jeremylt reviewed Jul 10, 2026

View reviewed changes

Add CUDA arch explaination to documentation

e2cee2a

Signed-off-by: PhilipOesterlePekrun <philipoesterlepekrun@gmail.com>

PhilipOesterlePekrun dismissed stale reviews from mayrmt and sebproell via e2cee2a July 11, 2026 08:34

PhilipOesterlePekrun force-pushed the mirco/KokkosCudaCleanedUp branch from b1fb1eb to e2cee2a Compare July 11, 2026 08:34

mayrmt approved these changes Jul 11, 2026

View reviewed changes

jeremylt reviewed Jul 13, 2026

View reviewed changes


		get_property(_global_rule GLOBAL PROPERTY RULE_LAUNCH_COMPILE)
		get_property(_dir_rule DIRECTORY PROPERTY RULE_LAUNCH_COMPILE)


		\|FOURC\| is primarily developed around MPI parallelism, but also offers the ability to use shared memory parallelism through `Kokkos <https://kokkos.org/> _`, enabling hybrid parallelism on the CPU through OpenMP and GPU acceleration through CUDA.

		Kokkos (and Kokkos-Kernels) can be built within Trilinos or specified as an external TPL in Trilinos, and its configuration follows the usual procedure for the desired backend (see the `Kokkos configuration guide <https://kokkos.org/kokkos-core-wiki/get-started/configuration-guide.html> _`). Trilinos then requires `Trilinos_ENABLE_<Backend>=ON` and, specifically for CUDA, `Trilinos_ENABLE_TPL_CUDA=ON`. To prevent oversubscription and unwanted shared memory parallelism in 4C, one should disable these backends for Tpetra with `TPETRA_INST_<BACKEND>=OFF` and explicitly set `TPETRA_INST_SERIAL=ON`.


		Kokkos (and Kokkos-Kernels) can be built within Trilinos or specified as an external TPL in Trilinos, and its configuration follows the usual procedure for the desired backend (see the `Kokkos configuration guide <https://kokkos.org/kokkos-core-wiki/get-started/configuration-guide.html> _`). Trilinos then requires `Trilinos_ENABLE_<Backend>=ON` and, specifically for CUDA, `Trilinos_ENABLE_TPL_CUDA=ON`. To prevent oversubscription and unwanted shared memory parallelism in 4C, one should disable these backends for Tpetra with `TPETRA_INST_<BACKEND>=OFF` and explicitly set `TPETRA_INST_SERIAL=ON`.

		To build 4C with this configuration, a compiler wrapper, `utilities/clangcuda++` must be used as the `CMAKE_CXX_COMPILER`, while clang should be used as the `CMAKE_C_COMPILER`. When using MPI, these should instead be set as the `OMPI_CXX` and `OMPI_CC` environment variables respectively. To change the GPU architecture or default clang++ and CUDA paths, one should set the corresponding environment variables listed at the start of the `utilities/clangcuda++` compiler wrapper. Additionally, the `FOUR_C_CLANGCUDA` compile option must be enabled in 4C. Due to incompatibility with the serial version of ArborX, it is recommended to disable `FOUR_C_WITH_ARBORX`.

Uh oh!

Conversation

PhilipOesterlePekrun commented May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description and Context

Related Issues and Pull Requests

Docker and Workflow tests

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maxfirmbach commented May 6, 2026

Uh oh!

maxfirmbach commented May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

PhilipOesterlePekrun commented May 7, 2026

Uh oh!

mayrmt commented May 7, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maxfirmbach commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

PhilipOesterlePekrun commented May 15, 2026

Uh oh!

mayrmt commented May 16, 2026

Uh oh!

ppraegla commented May 27, 2026

Uh oh!

PhilipOesterlePekrun commented May 27, 2026

Uh oh!

davidrudlstorfer commented May 27, 2026

Uh oh!

PhilipOesterlePekrun commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

PhilipOesterlePekrun commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PhilipOesterlePekrun commented Jun 24, 2026

Uh oh!

jeremylt left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

PhilipOesterlePekrun commented May 4, 2026 •

edited

Loading

maxfirmbach commented May 6, 2026 •

edited

Loading

maxfirmbach commented May 7, 2026 •

edited

Loading

PhilipOesterlePekrun commented May 27, 2026 •

edited

Loading

PhilipOesterlePekrun commented May 28, 2026 •

edited

Loading

PhilipOesterlePekrun commented Jul 2, 2026 •

edited

Loading

PhilipOesterlePekrun commented Jul 9, 2026 •

edited

Loading