Regression Testing

From ASCEND
Jump to: navigation, search

Historically, ASCEND was developed by a small group of tightly-coordinated members of the Art Westerberg research group. Individualized ad hoc testing of modules and successful running of test models was adequate under this development model.


As the ASCEND project grows, we anticipate that more of our developers will be unfamiliar with parts of the code, and for this reason we want to move towards providing a regression/unit testing suite for ASCEND. This is aim to prevent bugs from re-emerging by testing for expected behaviour following an applied fix (regression tests) and thoroughly testing the behaviour of a component currently under development (unit testing, test-driven development).


Preliminary test suite

Jerry has implemented testing of parts of the ASCEND codebase. To run the tests, read about BuildingAscend then try the following commands:


cd ~/src/ascend/trunk;
scons WITH_CUNIT_TESTS=1
./base/generic/test/test

The following tests are currently implemented:

[root@cruncher2 generic]# ls '''/test/test_'''.c
general/test/test_dstring.c                       utilities/test/test_asc[[DynaLoad]].c
general/test/test_hashpjw.c                       utilities/test/test_asc[[DynaLoad]]_shlib.c
general/test/test_list.c                                  utilities/test/test_asc[[EnvVar]].c
general/test/test_listio.c                              utilities/test/test_ascMalloc.c
general/test/test_pool.c                                  utilities/test/test_ascPanic.c
general/test/test_pretty.c                              utilities/test/test_ascPrint.c
general/test/test_register_general.c  utilities/test/test_ascSignal.c
general/test/test_stack.c                                utilities/test/test_mem.c
general/test/test_table.c                                utilities/test/test_readln.c
general/test/test_tm_time.c                       utilities/test/test_register_utilities.c
solver/test/test_register_solver.c       utilities/test/test_set.c
solver/test/test_slv_common.c

We're looking at how best to integrate tests like this into an automatic testing process. The C-based tests don't provide easy cross-system reporting as Python-based tests suites do.


Strategy for Regression/Unit Testing

Unfortunately, the size and complexity of ASCEND makes manually retrofitting a test suite an enormous task. That's not to say that it isn't worth doing, just that it will consume considerable resources. The availability of an automated testing tool would help a lot, but may not be easily available.

A possible approach to implementing a test suite would be


  1. identify the best testing protocol(s) to use for ASCEND
  2. prototype the protocol(s) for a limited subset of ASCEND
  3. implement test-driven development of new code
  4. implement tests for legacy code

Here are some general testing approaches that could be used. Please add to the list and edit the topics to discuss ideas, preferences, experiences, etc.


  • Static code analysis
  • Manual testing frameworks
  • Automated Testing Tools: we currently have BuildBot installed.

ASCEND Model Self-Testing

Some work is also underway to implement a tool which will find models containing their own 'self-testing' methods, and run those tests. This will be a higher-level complement to the low-level tests offered here, and will be used to verify the language features are working correctly.

See ModelSelfTesting.


Python Test Suite

A python test suite is being developed using the unittest framework, and is growing quickly to cover parts of the code not currently covered by the CUnit tests. The Python test suite will ultimately sweep in the above ModelSelfTesting as well.

To run the Python test suite, build the Python bindings for ASCEND (type scons) then run python pygtk/test.py.


Buildbot

A buildbot has been set up which automatically compiles ASCEND and runs the CUnit and Python test suites each time changes are committed to the Subversion repository. See BuildBot.

A missing attachment: Pycunit-0.2-1.src.rpm.