We can do coverage testing of ASCEND using the 'gcov' tool. To do this, build ASCEND using 'scons GCOV=1' (see building ASCEND for more details) and the coverage 'instrumentation' will be added.
To get coverage information in plaintext form, for a single code file, you can use the gcov command, eg
# run all this from the top-level working directory cd ~/ascend # compiler with coverage instrumentation enabled scons -j4 DEBUG=1 GCOV=1 test solvers models ascend # run the test test/test solver_qrslv.bug564 # process the covereage data gcov -p ascend/compiler/mergeinst.c --object-directory ascend/compiler # view the results vim ascend\#compiler\#merginst.c.gcov
- You don't get complete gcov data if a program exits or aborts prematurely. You can get a trace though by calling __gcov_flush() at some point close to where the error occurs (maybe you can even call it from GDB, haven't tried that). See this page.
- gcov shows results one file at a time. Tools like ggcov, lcov, gcovr can give some aggregated coveraged data, create GUI/HTML output, and convert to 'cobertura' output format for use with other tools.
- See also gprof if you want information about time spent in different functions.
The lcov tool can be used to produce nice, easy-to-read coverage reports from our CUnit test suite (and probably more).
Note: we found that the version of LCOV available in the Ubuntu repositories was not functional, and the newer code from git clone https://github.com/linux-test-project/lcov.git needed to be used. It can be built and installed using sudo make install, which will install it to /usr/local/bin/lcov.
To run lcov with ASCEND, try the following:
#!/bin/bash LCOV=/usr/local/bin/lcov GENHTML=/usr/local/bin/genhtml scons MALLOC_DEBUG=1 GCOV=1 CC=gcc CXX=g++ -j4 -c test ascend models solvers scons -j4 test ascend models solvers $LCOV -c -i -d . mycov-1.info ASCENDLIBRARY=models:solvers/qrslv LD_LIBRARY_PATH=. test/test general_color general_dstring general_hashpjw general_list general_listio \ general_mem general_pool general_pretty general_stack general_table \ general_tm_time general_ospath general_env general_ltmatrix general_ascMalloc \ utilities_ascEnvVar utilities_ascPrint \ utilities_ascSignal utilities_readln utilities_set linear_qrrank linear_mtx \ compiler_basics compiler_expr compiler_fixfree compiler_fixassign $LCOV -c -d . mycov-2.info $LCOV -a mycov-1.info -a mycov-2.info -o mycov-tot.info $LCOV -r mycov-tot.info "*stdout*" -o mycov-tot2.info $GENHTML mycov-tot2.info
Note that the command $LCOV -r mycov-tot.info "*stdout*" -o mycov-tot2.info is required, because of the way the ascend/compiler/scanner.l file is compiled. In the gcov output it is shown with a source-code filename of <stdout>, which is incorrect/false.
The above approach has been implemented in a Python script relerrorlist:test/test-cov.py which be run to produce full coverage reports for the CUnit test suite in ASCEND. So far we don't have any coverage testing from the Python or C++ parts.
- download ggcov 0.8.4 from here (note that the GIT code linked from the ggcov SF.net page is currently buggy)
- install the Glade and GNOME UI developer libraries for your distro. On Ubuntu, use apt-get install libglade2-dev libgnomeui-dev.
- ./configure --prefix=/usr/local
- make -j4
Currently, ggcov gets all confused if there aren't *.o files to match the *.gcno files produced by the --coverage instrumentation in GCC. You can quickly create at least the need .o files by copying all the .os files that SCons produces to corresponding .o files:
find . -name "*.os" | xargs rename 's/\.os$/\.o/'
If you do that, then ggcov gives bunch of warning due to symbol conflicts (static functions are used in ASCEND with the same name in more that one location, probably we can fix these with some renaming and/or merging):
Callgraph name collision: /home/john/ascend/ascend/compiler/logrelation.c:DoBreakPoint and /home/john/ascend/ascend/compiler/instance_io.c:DoBreakPoint Callgraph name collision: /home/john/ascend/ascend/compiler/rel_common.c:CmpP and /home/john/ascend/ascend/compiler/exprsym.c:CmpP Callgraph name collision: /home/john/ascend/ascend/compiler/relation.c:realloc_term_stack and /home/john/ascend/ascend/compiler/logrelation.c:realloc_term_stack
Other than that, ggcov seems to work OK now.
To get fresh tallies of covered code you might need to remove the coverage data, eg
find . -name "*.gcda" | xargs rm