XS coverage for Perl
Most of what I write for CPAN these days is XS. Ex::Op, HAMT, Object::Proto, Hyperman, the File::Raw family: the Perl side of each is a package, a $VERSION and a call to XSLoader, and everything that matters is in a .xs file and a handful of headers. That is a problem for coverage, because the tool every Perl project reaches for, Devel::Cover, covers Perl. Run it on one of those modules and it tells you, with a straight face, that the module is 100% covered. It is reporting on the three lines of Perl.
Devel::Cover does know about C. It has a gcov mode, and it is worth seeing exactly what that gives you before explaining why I wrote something on top of it.
What Devel::Cover gives you for C
The compiler can instrument C for coverage itself. With gcc or clang, -fprofile-arcs -ftest-coverage makes every object write a counter file when the process exits, and gcov turns those counters into a text listing per source file. Devel::Cover ships gcov2perl, which reads that listing into its own database, so the C ends up in the same report as the Perl. The whole thing, by hand, is this:
cover -delete
make test "OPTIMIZE=-O0 -fprofile-arcs -ftest-coverage" \
"OTHERLDFLAGS=-fprofile-arcs -ftest-coverage" \
HARNESS_PERL_SWITCHES=-MDevel::Cover
gcov -abc -o . Op.xs
gcov2perl -db cover_db Op.xs.gcov include/*.gcov
cover
Or in one go, since cover -test turns the gcov path on by itself when your perl was built with gcc or clang:
cover -test
Here it is on Ex::Op, a fresh checkout, nothing built yet:
cover: running make test "OPTIMIZE=-O0 -fprofile-arcs -ftest-coverage" "OTHERLDFLAGS=-fprofile-arcs -ftest-coverage"
...
cover: running gcov -abc -o . Op.c
cover: running gcov -abc -o . Op.xs
cover: running gcov2perl -db cover_db Op.xs.gcov ex_op_call.h.gcov ex_op_reg.h.gcov sv_inline.h.gcov ex_op_kw.h.gcov inline.h.gcov
gcov2perl: Warning: ignoring branch with 6 targets at Op.xs:325 switch (type) {
gcov2perl: Warning: ignoring branch with 5 targets at Op.xs:346 switch (type) {
gcov2perl: Warning: ignoring branch with 6 targets at Op.xs:373 switch (type) {
gcov2perl: Warning: ignoring branch with 16 targets at include/ex_op_reg.h:195 switch (o->op_type) {
gcov2perl: Warning: ignoring branch with 7 targets at .../CORE/sv_inline.h:385 switch (type) {
...
---------------------------- ------ ------ ------ ------ ------ ------ ------
File stmt bran cond sub pod time total
---------------------------- ------ ------ ------ ------ ------ ------ ------
...lib/perl5/JSON/MaybeXS.pm 0.0 0.0 0.0 0.0 n/a n/a 0.0
...ulti-2level/CORE/inline.h 90.0 66.6 n/a n/a n/a n/a 84.6
...i-2level/CORE/sv_inline.h 65.3 51.3 n/a n/a n/a n/a 60.7
...0sc0000gn/T/a1Isp5XId2.pl n/a n/a n/a n/a n/a n/a n/a
Op.xs 99.6 76.5 n/a n/a n/a n/a 88.7
blib/lib/Ex/Op.pm 100.0 n/a n/a 100.0 n/a 100.0 100.0
include/ex_op_call.h 100.0 84.7 n/a n/a n/a n/a 93.4
include/ex_op_kw.h 100.0 92.3 n/a n/a n/a n/a 96.6
include/ex_op_reg.h 92.9 77.3 n/a n/a n/a n/a 85.4
Total 92.7 76.0 0.0 40.0 n/a 100.0 84.4
---------------------------- ------ ------ ------ ------ ------ ------ ------
So it works, and it is real data. I used it for a long time. But look at what is in that table and what is not.
Every switch is gone. gcov reports a jump with N arms as N branches on one line. gcov2perl only knows two-arm branches, so it prints a warning and drops the line. The walker at the heart of Ex::Op is a switch over op types with sixteen arms, and it is simply absent from the branch column. The same happens to any if (a && b), because the compiler lays that out as two jumps on one line.
The sub column is empty for C. gcov prints function eo_register called 2320 for every C function, and gcov2perl ignores those lines. There is no way to see that an XSUB was never entered.
Perl's own headers are in my report. inline.h and sv_inline.h come from the perl install, and their inline functions are instantiated in my object, so gcov lists them and they drag my totals around. They are not mine to cover.
The generated C is thrown away. xsubpp writes Op.c from Op.xs, and the wrappers it generates are the only record that import or export_to_level was called at all. cover drops Op.c.gcov whenever Op.xs.gcov exists.
Branches in macros count. A line like if (!stash || !HvNAME_get(stash)) has two decisions in the source and seven in the compiled code, because HvNAME_get expands to a chain of ternaries. gcov reports all seven. Five of them belong to perl's macro and nothing in my test suite can ever take their other arm, so the branch column is red for reasons no test can fix.
And one more, which is not a shortcoming so much as a trap. If the module was already built when you ran cover -test, make has nothing to do, the objects are the optimised ones with no instrumentation, no counters are written, and the table has no Op.xs row at all. It does not say so. You get the Perl side at 100% and a warm feeling.
What xs_cover does instead
Devel::XSCover is a driver over Devel::Cover, gcov and the compiler. It does not replace any of them. One command, run in the distribution directory:
xs_cover
On the same checkout of Ex::Op:
xs_cover: touched 8 source files
xs_cover: running make OPTIMIZE=-O0 -g -fprofile-arcs -ftest-coverage -fprofile-update=atomic OTHERLDFLAGS=-fprofile-arcs -ftest-coverage
xs_cover: t/31-registry-lifetime.t use threads, which Devel::Cover does not work with, so it runs without it
xs_cover: running prove -b -j1 t/00-load.t ... t/73-across-keyword.t under Devel::Cover
xs_cover: running prove -b -j1 t/31-registry-lifetime.t without Devel::Cover
xs_cover: running /usr/bin/gcov -abct -o Op.gcda Op
xs_cover: running /usr/bin/gcov --dump -n -o Op.gcda Op
xs_cover: read 1 object
xs_cover: writing 4 files, 1444 statements, 333 branches, 94 subroutines to cover_db/runs/1793725310.85048.52249
xs_cover: running cover -nogcov -report XSCover cover_db
-------------------- ------ ------ ------ ------ ------ ------ ------
File stmt bran cond sub pod time total
-------------------- ------ ------ ------ ------ ------ ------ ------
Op.xs 100.0 100.0 n/a 100.0 n/a n/a 100.0
blib/lib/Ex/Op.pm 100.0 n/a n/a 100.0 n/a 100.0 100.0
include/ex_op_call.h 100.0 100.0 n/a 100.0 n/a n/a 100.0
include/ex_op_kw.h 100.0 100.0 n/a 100.0 n/a n/a 100.0
include/ex_op_reg.h 100.0 100.0 n/a 100.0 n/a n/a 100.0
Total 100.0 100.0 n/a 100.0 n/a 100.0 100.0
-------------------- ------ ------ ------ ------ ------ ------ ------
xs_cover: removed 2 counter files and touched 8 sources, so the next make rebuilds without coverage
Those are not the same numbers as above because the branch criterion is not measuring the same thing, and that is the point. Step by step, what differs:
It builds in place, every time. The sources are touched and make runs with the coverage flags on the command line, so the objects you test are the instrumented ones whatever state the tree was in. -fprofile-update=atomic is on as well, because without it a threaded test loses about half its counts. At the end the counters are removed and the sources touched again, so your next plain make rebuilds optimised and you do not ship an instrumented bundle by mistake.
Threads are handled. Devel::Cover refuses to load under threads, so a test file that uses them fails outright under cover -test. xs_cover finds those files and runs them by themselves without Devel::Cover. Their C side still counts, since the counters are the compiler's, not Devel::Cover's.
Every function is a subroutine, under its Perl name. The generated C is read, not thrown away. Each XS_Ex__Op_import wrapper is matched back to the import(...) declaration in Op.xs and written as the subroutine Ex::Op::import on that line, with the count of the wrapper. An ALIAS gives it one entry per name. Plain C functions appear under their C names on their first line. That is the sub column at 100 above, where Devel::Cover had n/a.
A switch is N branches, never dropped. A jump with more than two arms becomes one branch entry per arm, arm 3 of 16, each one against the others, so an arm no test reaches is a red mark on the line that has it.
Branches are the decisions of your source. This is the one I am most pleased with. gcov's text tells you how many times each arm of each block was taken, but not where an arm goes. The compiler's flow graph does: llvm-cov gcov --dump prints every block with its arcs and counts, and gcov --json-format on gcc 9 or later gives the same. A decision in the source is the set of arcs that leave the lines its parentheses span. The arcs that stay on those lines are the macro's own inner tests, and they vanish. if (!stash || !HvNAME_get(stash)) has two arms again, the one into the body and the one past it, which is what Devel::Cover's branch criterion means for Perl. On Ex::Op this took Op.xs from 73.5 to 89.4 branch coverage with no change to the source, and every remaining red mark was something a test could reach or a guard that nothing can.
Perl's headers are out, your headers are in. Anything outside the distribution directory is dropped, along with ppport.h. A header of your own that is inlined into several objects is merged across them, lines unioned and counts summed, rather than written as conflicting runs.
With gcc 14 the C gets a condition column. gcc 14 added -fcondition-coverage, and gcov --conditions reports, for every term of a decision, whether its true and its false outcome each decided the result. xs_cover turns that on when it sees gcc 14 and writes it as Devel::Cover conditions. clang has nothing equivalent in its gcov output, so on a Mac the column stays n/a for C, and I run the gcc pass in a container. Any gcc 14 image with cpanminus on it will do:
docker run --rm -v "$PWD:/work" -w /work gcc:14 sh -c '
apt-get update >/dev/null && apt-get install -y cpanminus >/dev/null
cpanm -q Devel::XSCover && xs_cover'
The report says what is missing. The default report is Devel::Cover::Report::XSCover, a self-contained HTML page per file with the counts drawn beside each line: a chip per branch arm with how often it was taken, one chip per arm of a switch, the subroutine name on its declaration line, and a sentence for every line with something missing, such as arm B of the branch if (!svp) was never taken, arm A taken 9993 times. It works on any Devel::Cover database, so cover -report XSCover is fine on a pure Perl distribution too.
Marks still work. Devel::Cover's # uncoverable comments are honoured on C lines, so a guard that nothing can reach takes /* # uncoverable branch true */ at the end of its line and stops being an error. Which is how Ex::Op got to the table above: tests for everything a test could reach, and a mark with a reason for every NULL check that the code around it already makes impossible.
The things it will not do
It is gcov underneath, so a child process that ends with POSIX::_exit or a signal writes no counters. Module::Build distributions and Windows are not currently supported. And the branch arms of a C decision are labelled A and B rather than true and false, because gcov cannot say which arm is which, only which comes first in the compiler's layout.
If you have XS on CPAN and have only ever looked at