Add support for generating coverage reports w/ gcov - #4489
Conversation
|
I suppose we need to install gcovr first? Where should we put that info? I don't want (or can't) install using pip globally... |
|
Here is one set of instructions (from chatgpt). Sound reasonable? (I've never used pipx before). Do the following once (using homebrew, on mac) Restart a terminal, do to upgrade or remove: |
|
It's available from most package managers ( |
|
Yeah |
|
The info about pipx suggested by chatGPT is too complicated and not needed on a mac: I believe just doing |
|
|
||
| if(GCOV) | ||
| find_program(GCOVR NAMES gcovr) | ||
| set(_coverage_dir ${CMAKE_BINARY_DIR}/coverage) |
There was a problem hiding this comment.
Could we add a small CI smoke test with coverage enabled, especially for the CMake path? The current checks all use the default GCOV=OFF, so they don't exercise the instrumentation flags or either coverage target; the PR also notes that the CMake support hasn't been tested. Even a minimal configure/build/run/coverage-report job would catch toolchain and gcovr invocation problems.
AI-assisted review comment made by GPT 4.6-Sol under Andrew's direction in order to observe deeper code reaches.
There was a problem hiding this comment.
That sounds like a great idea, but I think it's out of scope for this PR. In particular, I don't think we want a distinct coverage-enabled build running as part of our usual test_build.yml workflow that runs during every PR.
But maybe like a weekly build that runs the report for the engine unit tests and uploads it to GitHub pages or something.
A copy-and-paste slip in the (currently dormant) Clang source-based coverage module: LLVM_COV was searched for under the name clang-format. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Build with --coverage -O0 (and without stripping), modeled on
--enable-profile. The flags are applied after the Fortran/BLAS/LAPACK
detection, since gfortran expands --coverage to -lgcov, which would
otherwise be baked into FCLIBS/BLAS_LIBS and collide with the compiler's
own coverage runtime at link time.
Coverage data (.gcda) is generated automatically whenever a
coverage-built binary exits, so no target runs the tests for you: just
run the unit tests, "make check", or the M2 binary on your own code.
Two new top-level targets clear and report on the accumulated data:
coverage-reset delete accumulated .gcda (tree-wide)
coverage-report produce an HTML report via gcovr, whose path is
printed as a clickable OSC 8 terminal hyperlink
The report covers the whole tree by default; narrow it by overriding
COVERAGE_ROOT/COVERAGE_OBJDIR or with GCOVR_OPTIONS. The object tree is
handed to gcovr as a positional search path rather than via
--object-directory, which would pin gcov to a single directory and fail
to resolve sources for objects built in deeper VPATH subdirectories.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The cmake counterpart of --enable-gcov: -DGCOV=ON builds with --coverage -O0, mirroring the PROFILING flag block, and the new cmake/gcov.cmake provides coverage-reset and coverage-report targets matching the autotools ones. Extra gcovr options may be passed with -DGCOVR_OPTIONS="...". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Finally got around to testing the cmake version. It was broken lol. Claude fixed it up, cleaned up the code, and squashed everything down to 3 logical commits. For review, this is how it works. First install gcovr ( autotools./configure --enable-gcov # + whatever other options
make -jN # N = number of cores
# C-c once you have a working binary (we really need "make M2-core" lol)
make coverage-reset
# run whatever M2 code you want to look at
make coverage-reportcmakecmake -DGCOV=ON # + whatever other options
ninja M2-core
ninja coverage-reset
# run whatever M2 code you want to look at
ninja coverage-report |
andrew-tawfeek
left a comment
andrew-tawfeek
left a comment
There was a problem hiding this comment.
This looks good. I verified both generated Autotools and CMake coverage paths with GCC and gcovr: instrumentation produced counter data, coverage-report wrote a nonempty HTML report, and coverage-reset removed the counters. The Autotools configuration also applied --coverage -O0 after Fortran and BLAS detection and disabled stripping as intended.
AI-assisted PR review alongside GPT 5.6-Sol
3 participants
This replaces MichaelABurr#65
As discussed in the C++ refactoring group Zoom meeting yesterday: If we build M2 using
./configure --enable-gcov, then we can runmake coverage-reportto get a gcovr-generated coverage report of the engine and interpreter. In particular, everything that was covered since we built M2 (or ranmake coverage-reset).This will be useful to figuring out which features still need unit testing (e.g., by running
make coverage-reset && make -C Macaulay2/e check && make coverage-report, we can see what all the unit tests hit.) There's also CMake support that I haven't tested.AI Disclosure
🤖 This was Claude through and through 🤖