Developer's Manual

From ASCEND
Jump to: navigation, search
This article is incomplete or needs expanding. Please help out by adding your comments.

This document contains some partial high-level documentation of the ASCEND system for use by developers. It attempts to describe all the major functionality and how it all fits together, because that is the one thing that it is hard to comprehend just from reading the source code.

In addition to these notes, there is a lot of descriptive material embedded in the source code, accessible via doxygen at http://www.ascend4.org/doxy/.

Some very good, although outdated, information is also available in the document Implementation of an ASCEND interpreter by Tom Epperly. See also more information in the ASCEND bibliography.

See also the user's manual (1 MB .pdf) for user-level information including description of the modelling language and syntax.

Getting started

ASCEND is primarily a program for solving systems of equations. These could be systems of non-linear equations, or systems of differential equations, or mixtures of both. Equations and variables are specified using a bespoke modelling language, then ASCEND compiles them into a form that it can solve. It exposes that form to a number of different solvers that the user can run depending on the type of problem they are solving: solution, optimisation, dynamic simulation, etc. All of the 'core' of ASCEND is written in C code, but there are also bits of Python (for the PyGTK GUI and Python language bindings), as well as Tcl/Tk (for the Tcl/Tk GUI and language bindings), as well as C++ (part of the Python language bindings).

See Building ASCEND for information on building from source code (either our subversion repository or a source code tarball)

See Debugging ASCEND for some information on ways of tracking down errors in the ASCEND code, including how to run a 'local copy' of ASCEND using the 'ascdev' script.

Once you have built the code, you can test bits of the 'core' code using our CUnit test suite: scons test && test/test.

There are also many model files that include self-test methods. Open those files with ASCEND, run the 'on_load' method, then solve, then run 'self_test'.

If writing new code for ASCEND itself, please learn and respect our coding style.

libascend

libascend is the 'core' ASCEND engine. It is written entirely in C, although some of this code is auto-generated using Flex and Bison. It contains all of the routines that load a model file and any dependent files that it calls, allows METHODs to be run on the model tree, and creates a 'flat' version of the model that can be passed to solvers. Several kinds of external libraries can be plugged in to libascend; this is how solver functionality is provided, for example.

So, libascend includes two main parts, the 'compiler' (ascend/compiler and ascend/system) and the 'solver' (ascend/solver and ascend/integrator). These are organised in directories in the ascend subdirectory of the source distribution. There are also 'utilities', 'general' and 'packages' subdirectories.

The ascend/utilities directory is intended to contain utility routines that only really of use to ASCEND but which are linked specifically neither to the compiler nor the solver. The ascend/general directory contains general purpose routines which would be of use in any similar software project, with no specific dependencies on the other ASCEND code.

The ascend/packages directory contains a few so called 'internal packages'. These provide functionality that is outside the compiler and solver, but which is important for basic standard-case use of ASCEND. As example is the code which performs the 'ClearAll' function to reset the values the 'fixed' flag.

Finally, there is the ASCEND GUI. There is currently a Tcl/Tk GUI as well as a PyGTK GUI, which are detailed below. These can be used independently; there is not interdependency.

Diagram showing the architecture of ASCEND with some 'cuts' into modules (some cuts of which are yet to be completed)

Compiler

The compiler includes a parser which constructs a types hierarchy which is stored in a library. When a model (or instance) of a specific type is instantiated, it creates an Instance tree, the top node of which is a 'simulation' instance, and underneath which is a simulation root corresponding in structure to the type that was instantiated.

The Library, and TypeDescriptions

As files are loaded into ASCEND, they are added as 'modules' to the in-memory 'library' structure. It is possible to have multiple models active and instantiated from a single in-memory library, although the current PyGTK interface does not yet support this.

Each of the MODELs and ATOM descriptions in the modules resides within the library in the form of a TypeDescription object, which in turn is composed of Statements objects.

Parser, and how to add to the language

ASCEND uses Flex and Bison to implement a scanner/parser for the ASCEND modelling language (see also documentation on). The files that define the language are ascend/compiler/ascParse.y and ascend/compiler/scanner.l.

On machines that don't have Flex and/or Bison, the source code contains some reasonably up-to-date copies of the compiled output from these files (the files *_no_yacc.* and *_no_flex.*, in particular).

These parsers basically 'read' the input file and for each statement, they add a Statement object within a TypeDescription object. This effectively reassembles a machine-readable form of the input file. Parsing is only a small part of the job of making a working model; instantiation is probably the bigger part.

Adding new statements to the language requires that the Statement struct be expanded to contain the necessary extra fields, and then requires functions to be implemented that can store parsed statements into the TypeDescription, and then more functions that actually run the statements. See ascend/compiler/statement.h, eg CreateFREE.

Note the 'families' of functions around each statement type, eg, 'run', 'do', 'create'.

Instantiation

When the user requests to create a model from a given TypeDescription, this is when ASCEND starts the process of MODEL instantiation. This is a very complicated process, because of the process of creating sub-models, and then handling all of the relationships between instances such as ALIASES, ARE_THE_SAME as well as the rigours of parameterised models. There is also very sophisticated processing called 'anonymous merging' that helps large repetitive models (eg arrays of similar equations) to not use more memory than they have to. Start by reading ascend/compiler/instantiate.h.

The phases of instantiation are as follows (text from User:DanteStroe):

  1. Models (instantiate models and disregard any other type of statements)
  2. Relations (instantiate "relations" and disregard any other type of statements)
  3. Logical Statements (instantiate logicals and disregard any other type of statements)
  4. When Statements (instantiate "when"s and disregard any other type of statements)
  5. LINK Statements (instantiate LINKS and disregard any other type of statements) - Added
  6. default (visit instance tree once more and if it is the case warn about any un-executed statements).

More info (from Ben Allan:

Pass does proof items
1 models/non-rel arrays/selection of switches/atoms existence of sets, values of sets, values of constants
2 relations existence of names appearing in relations subject to limits that possible future refinement induces
3,4,5 whens,bounds,logical relations more or less like 2
6 default statements in model instance(deprecated) existence of atom instance names, s.t. refinement later

The definition of ASCEND (unordered declaration and 'infinite pass compiler') is not such that a failure in any instance is allowed to stop work on all the other instances and thus on later passes. In pass 2, we don’t loop over individual relations -- each gets tried exactly once, as is the case for passes 3,4,5.

So, references to unknown dot-qualified/subscripted names and incompletely-instantiated objects (null) in a relation (blackbox or otherwise) are generally unresolved, but not provably bad. When a name is provably bad is when its interpretation requires looking up an array element that does not exist in an array that is fully instantiated; relations with this problem are/should be marked as permanently 'finished' statements, the user warned about their impossibility, and henceforth ignored. Arrays, once completed in pass 1, are fixed forever because we do not allow 'dynamic sets' to define arrays.

A simple name, like foo.x, without array content may become valid in a later refinement of foo. Restart of the instantiator (usually only useful after an interactive refinement or merge or assignment of a constant/set value) may make something valid that was invalid before.

In the case of relations, we can limit this impact by checking (as much as possible, but not perfectly based on typedesc) that all names in an equation or arg list are plausible. We have wild-dimensioned atoms as a work-around for not having forward declarations and templates in some circumstances, like the numerical methods models.

This article is incomplete or needs expanding. Please help out by adding your comments.

Bintokens

ASCEND includes the ability to compile machine-language representation of equations in the model you have written. This functionality has been found to be only marginally faster than the current expression evaluation engine, and is currently not operational, although it would not take a huge amount of work to reinstate it.

Bintokens only function when 'relation sharing' is turned on.

The code in ascend/compiler/bintoken.c does the job of writing a C source code file to /tmp/asc_bintoken.c and then compiling it and loading it using ascDynaLoad, and linking the evaluation routines into the instance tree. The code in ascend/bintokens/btprolog.h is a special header file designed to provide the minimum necessary data structures to allow ASCEND to communicate with the binary-compiled relations. Currently, this header file still depends on a couple of other ASCEND header files, which will impact on how we package this functionality (for Debian, RPM, Windows etc).

For the PyGTK GUI, pygtk/compiler.cpp defines some default pathnames for passing to the bintoken routines. The PyGTK GUI also currently fails to warn the user about the need to reload a model in order to force bintokens to be built.

Instance hierarchy

An instantiated MODEL (i.e. one which is loaded into memory but which has not yet been sent to a solver) in comprised of a tree of Instance objects (see doxy:Compiler Instance Hierarchy) arranged in a parent-child hierarchy. There are many different Instance types, eg doxy:RelationInstance and doxy:RealAtomInstance. More generally, the key instance types include

  • relations
  • variables
  • constants
  • arrays
  • sets
  • MODELs
  • simulation (the 'root' instance object)

These Instance objects are all structs and all start with an initial enum inst_t t which identifies the structure of the following data in memory. Given a 'vanilla' struct Instance * pointer, it is therefore necessary to examine that enum inst_t t then cast to the appropriate class before reading any other fields.

The various Instance structs have not been formed into a union as that would have the result that they would then all consume an equal amount of memory. The current approach is therefore adopted to reduce memory consumption.

It should be noted that the Instance hierarchy provdes for parent-child relationship between instances (such as a relation being part of a model, or a variable being part of an array), but other relationships between variables, relations, etc, are also possible, such as ALIASES, which transcend the parent-child structure.

The various Instance structs are defined in ascend/compiler/instance_types.h. See also doxy:Compiler Instance Hierarchy.

As an example of an Instance data type, here is the struct Relation data type:

Relation data structures in the ASCEND compiler (for clickable version, see the doxy:corresponding doxygen page).

METHODs

METHODs run at the level of the ASCEND compiler, not at the level of the solver, so functions relating to the execution of METHODs revolve around manipulations of the instance tree. Start by reading the function Initialize in the file ascend/compiler/initialize.h.

Relationships

The following explicit relationships are tracked by the Compiler

  • Part-whole (model members), including relations in models.
  • structural isomorphism (are-alike clique), stored in the alike_ptr of Instance structs, see ascend/compiler/instance_types.h
  • structural identity (are-the-same clique, parameter passing, aliasing), can be revisited using the AllPaths and WriteAliases functions (see ascend/compiler/instance_io.h.
  • continuous variable referenced by equation or boundary.
  • discrete variable referenced by logical equations and WHENs.
  • discrete constants determining compilation alternatives (SWITCH).
  • satisfaction of a nonlinear equation/inequality to a tolerance.
  • units and dimensionality of variables, equations.
  • a variety of stuff about sets of discrete values.
  • variable values.
  • interpretation of names in methods in one or more object scopes.

The following implicit relations are tracked:

  • bounds, scaling, ode-classification on variables, scaling on relations.
  • semi-continuous variable treatment for mixed-IP (solver_semi).
  • relation inclusion in a model to be solved (.included, WHEN).
  • model inclusion (WHEN).
  • time dependent/event state-transition would be if we had the solver to manage it.
  • objective functions.
  • output file logging.

Note that the implicit (soft) nature of the relationships is precisely what we intended. It serves to separate model definition (which is simply listing a set of equations we assert to be true) from the model solution process. As soon as one gets overly aggressive, the compiler turns into a solver and this is a bad idea.

Analyser

The analyze.c code lies in the ascend/system directory and is responsible for 'visiting' the instance hierarchy maintained in memory by the Compiler and turning it into a 'flat' system that can be passed on the the Solvers.

First, the GUI calls the routine system_build which allocates space for slv_system_t object and then calls analyze_make_problem.

analyze_make_problem

This is where the main work begins.

Space is allocated for a temporary object of type problem_t. First the instance hierarchy is visited by the CountStuffInTree method, which accumulates counts of the various types of variables.

analyze_make_master_lists

Next there is a called to analyze_make_master_lists, which allocates memory for lists of each type of variable (according the counts made), then calls PushInterfacePtrs, which seems to somehow fish out the correct links to items in the instance hierarchy and create the interim items in the lists inside the problem_t object. This is where variables are flagged as FIXED and FREE, etc, and where derivatives are identified as such?

(a bit vague here) After that CollectRelsAndWhens. This seems to do the job of organising the lists of WHEN statements, including the possibility of nested and conditional parts of the moment.

Next all variables in the found relations are marked incident. Variables found in objective relations are also marked incident, as are real variables in CONDITIONAL relations, and boolean variables in logical relations and CONDITIONAL logical relations. WHENs are numbered.

Variables that did not get marked incident in the above process are moved off to separate 'unattached' lists. This completes the call to analyze_make_master_lists.

Note that objective relations (MINIMIZE an MAXIMIZE) are kept in a separate list. Lists of 'normal' relations, objective relations and conditional relations are mutually exclusive.

analyze_make_solvers_lists

Next, analyze_make_solvers_lists is called. (This seems to be a bit muddy: stuff seems to be done here to both the master list as well as the solver's list.)

system_generate_diffvars

Next system_generate_diffvars is called, which if any derivatives were found, creates a set of derivative chains. This gives the relationship between variables declared in the model, to identify which vars are supposed to be derivatives of which other vars.

analyze_configure_system

The routine analyze_configure_system does the job of passing off the lists from the problem_t and giving them to the slv_system_t. NULLs are assigned in the problem_t in order to make it clear that a list has been 'disowned' by the temporary problem_t structure.

PopInterfacePtrs?

analyze_free_lists

Finally analyze_free_lists cleans up any not-disowned stuff still in the problem_t structure. The result is that the slv_system_t contains everything you need to solve your problem, but in such a way that you don't need to 'talk' directly to the ASCEND instance hierarch. This way your solver only needs to speak the reduced vocabulary of the slv_system_t and can safely ignore the implementation details of the Compiler.

System

The system is the 'flat' representation of the instance tree, generated by the analyser (above). It is (in essence) a set of lists of relations and variables (there are various types of variables; the main one is the real-valued solver_var, but boolean, integer, 'symbol' variables are possible, and conditional modelling constructs are provided for also). The system also keeps lists of boolean variables, logical relations, conditionals and when disjunctions.

The system code is intended to cover all aspects of creating and retrieving data from the 'flat' system representation created by the analysis steps above. It is intended that solvers will 'talk' only to the system representation of a model, and never to the instance tree directly. In principle, it should be possible to use ASCEND in a different way, by create a new 'system provider' to replace the current instance tree implementation, this could be for example some kind of CFD code, or something like that.

In user interface terms, it is desirable to be able to solve different sub-models independently. For example, in a whole flowsheet of a power station, it may be of interest to be able to, in isolation, solve what is happening just in the turbine, for a given set of turbine inputs and parameters. This is especially useful in hard-to-solve (hard-to-converge) models. In this use case, ASCEND would create a number of different 'system' objects all linked back to a common instance tree. Then solvers would in turn be able to 'visit' these system and run their algorithms upon them.

Some key 'system' functions include

  • relman_eval (residual of a relation)
  • relman_diff (derivative of a residual with respect to the variables in the relation)
  • relman_diff2, relman_hess (second derivatives, hessian matrix elements)
  • bndman_real_eval, bndman_log_eval (for evaluation of boundary crossings)

Flags and Filters

Each of the relations and variables present in the system have some memory associated with them in which a set of flags can be stored. These are used to pass information from the solver back to the user interface, and vice versa. Some of these flags include:


  • REL_INCLUDED: set to true when a relation is 'included' in the current system. This is a user-controllable thing. It can be set using the GUI, or in a METHOD with code like myrel.included := TRUE;.
  • REL_EQUALITY: set to true when a relation is an equality relation, as opposed to an inequality.
  • VAR_ACTIVE: set to true when a variable is in the currently-being-solved 'block'.
  • VAR_FIXED: set to true when a variable was set FIXed by the user (either via a METHOD, or manually using the GUI).

When implementing a new solver, it is critical that you make use of these flags and the associated filter methods (see rel_apply_filter in ascend/system/rel.h and var_apply_filter in ascend/system/var.h), because in complex ASCEND models, there will often be variables and relations that you should not include in your solver calculations.

Solver

Add:

  • solver interface: what the solver sees, and what the ASCEND compiler sees.
  • mtx routines

The system module hands off the slv_system_t to the slv server and allows it to be queried by one of the slv clients that actually does the solving work.

The slv server also implements a general-purpose way of getting and setting solver parameters for the various solvers. See also doxy:Solver Parameters.

The solver performs blocking of equations. This means dividing the overall system into sequences of subsystems that can be solved without knowledge of any other as-yet-unsolved variables. The allows the Newton iteration to work on smaller matrices, for great speed and improved convergence.


Solver API

Communication between solvers and ASCEND is a two-way process. When the solver is first loaded, ASCEND looks for a 'register' function that tells ASCEND what functions its provides. A struct slv_registration_data is passed to ASCEND containing various function points. Then, when ASCEND wants to solve its problem, it calls these various functions in sequence. The essential functions are:

  • ccreate Create solver-specific data structures (invoked via slv_select_solver and slv_switch_solver)
  • cdestroy Destroy solver-specific data structures (invoked via slv_destroy and slv_destroy_client)
  • celigible Determines whether or not the solver is capable of solving the given slv_system_t as it is currently set up (e.g. some solvers cannot do optimization, or inequalities, etc). (invoked via slv_elegible_solver)
  • getdefparam Copies the solver's default parameters values into the given structure (invoked via slv_get_default_parameters)
  • get_parameters Copies the current solver parameters to the given structure (invoked via slv_get_parameters)
  • set_param Sets the current system parameters to the values contained in the given structure. It is recommended that one gets the parameters first, before one modifies them and sets them, especially if not all of the parameters are to be modified (and you never know when that might suddenly become true because a new parameter was added to the structure). Parameters will only be accepted by an engine if they came from that engine, so fetching before setting is not only a good idea, it's the law (gas engines don't run on diesel very well...). @par (invoked via slv_set_parameters)
  • getstatus Copies the current system status into the given structure (invoked via slv_get_status)
  • solve Attempts to solve the entire system in one shot (i.e. performs as many iterations as needed or allowed). For some solvers, slv_iterate() and slv_solve() may mean the same thing. Returns 0 on success (invoked via slv_solve)

See ascend/solver/solver.h for the full function prototypes above. For sample implementations, see the solvers directory, perhaps starting with QRSlv, IPOPT and CONOPT.

See doxy:Solver API page from our code docs, too.

The solve function above (as well as the other methods) make various calls back to the system API (above), which allows list of equations to be obtained, residuals of equations to be calculated, etc.

Integrator

The integrator sits aside the solver and coordinates a time-stepping process that may be imeplemented using whatever additional logic you want. We currently have the LSODE integrator as well as a early implementations of IDA and DOPRI5. The IDA solver is capable of doing conditional model switching, but the ASCEND wrapper doesn't yet fully implement that.

The Integrator API permits you to implement your own blocking/solving algorithms, or to re-use the main solvers, as desired. The LSODE implementation currently uses the ASCEND solver to perform block solution. The IDA implementation isn't quite as clever; it uses IDA's built-in dense matrix solver at this stage, but some early code to hook into ASCEND's solver has been partly written.

See Integrator API for more details.

Utility and General Routines

There are various list and string and collection 'classes' in these directories that support the solver and compiler functionality, as well as some of the GUI functions.

Error, warning, and message reporting

Error, warning and message reporting in ASCEND is achieved via ascend/utilities/error.h. Calls to this function work like calls to 'printf', except that error output is channeled to an appropriate place in the PyGTK GUI, via Python, or else goes to the console in the case of the Tcl/Tk GUI.

Some typical usage would be:

if(something){
    ERROR_REPORTER_HERE(ASC_USER_ERROR,"You did something wrong");

    return 1;
}
ERROR_REPORTER_HERE(ASC_PROG_NOTE,"Value myvar = %d",myvar);

if(somethingelse){
    ERROR_REPORTER_HERE(ASC_USER_WARNING,"Something is going on, and I don't what it is");
}

Sometimes it is necessary to compact several 'fprintf' style calls into a single error message, sometimes because of the way old ASCEND output functions were written. For example, from statio.c,

error_reporter_start(sev,filename,line,NULL);
  FPRINTF(ASCERR,"%s\n",message);

  if(stat!=NULL){
    /* write some more detail */
    g_show_statement_detail = ((noisy!=0) ? 1 : 0);

    WriteStatement(ASCERR,stat,2);
    g_show_statement_detail = 1;

    if (GetEvaluationForTable()!=NULL) {
      WriteForTable(ASCERR,GetEvaluationForTable());

    }
  }else{
    FPRINTF(f,"NULL STATEMENT!");
  }

  error_reporter_end_flush();

In addition, we have some ways to cache many different error messages, and only to output them if we decide we want them later. In this case, we use the 'error_reporter_tree' functionality:

bool has_error = 0;
error_reporter_tree_start();
do_subordinate_tasks();
if(error_reporter_tree_has_error()){
	has_error = 1;
}else{
	has_error = 0
}
if(has_error){
	error_reporter_tree_end();
	ERROR_REPORTER_NOLINE(ASC_USER_ERROR,"failed");
}else{
	ERROR_REPORTER_NOLINE(ASC_USER_SUCCESS,"success");
}

This allows us to make use of the error reporter mechanism to track errors occurring in deeply nested code without needing to use an exception-like mechanism. This is particularly useful in code that transcends python and C and C++, such as when solvers are integrating models containing external relations.

Note that in ASCEND, we usually use FPRINTF instead of fprintf. FPRINTF is a macro which effectively wraps the above error_reporter mechanism, ensuring that we can capture all console output and display it in the GUI.

For messages which are genuinely only of interest to developers, please use the CONSOLE_DEBUG macro. This includes automatic output of the file, line, and function in which the statement is located, aiding debugging.

You should also note that some external solvers make use of error reporting callback functions. An example of how to hook these functions into ASCEND is given in solvers/ida/ida.c with the function integrator_ida_error.

File pathname handling: ospath.h

In order to address the demands of locating and opening files in a cross-platform code environment, we have implemented a small set of routines that can deal with both Windows/DOS file paths as well as Linux/Unix style paths. This was inspired somewhat by the Python 'import os.path' module, but is a completely different implementation. The code is outlined in ascend/general/ospath.h. Note that correct use of this module requires that returned paths be carefully freed using ospath_free, because these functions use dynamically allocated memory.

External Libraries

External libraries are supported. You can use both 'black box' external relations (for computation) or 'external methods' for doing special things to the model hierarchy (model setup and reporting).

The architecture is there for dlopening of whatever module you need at runtime. And example of how to build and link an external package is given in the models/johnpye/extfn directory.

Python wrapper

The Python interface to ASCEND is implemented as a thin C++ layer on top of the core C code (directory ascxx). This C++ layer is then wrapped with SWIG to provide an object-oriented interface to the ASCEND library, types, simulation, instance tree, variables and relations, and so on. There is only mimimal code required to interface the C++ layer with Python, since SWIG does most of this work (and also permits ASCEND to be linked to other languages if desired).

The use of C++ in ASCEND is actually rather unfortunate, as it results in a much larger binary than would be required if only C had been used. Use of C++ was helpful in the wrapping however, because it makes it easier to create object-oriented wrappings, and also provides some useful memory management via the STL libraries of C++ (map, list, vector, etc). But we would like to get rid of the C++ wrapper one day and wrap libascend for Python using just plain C.

The key file for understanding how libascend functionality is exposed in Python is pygtk/ascpy.i, which in turn makes use of many pygtk/*.h C++ header files. You will need to understand basic usage of SWIG first.

The Python wrapper is binary dependent on Python; if you want to use ASCEND from a new version of Python, you need to recompile the Python wrapper.

GUI interfaces

GUIs talk to the solver and the compiler depending on what they're doing. All GUI function should be designed such that the internals of the data structures used by the solver and or compiler are not depended on: a proper API.

PyGTK interface

The PyGTK interface (see PyGTK Screenshots) implements a fully object-oriented interface on top of the Python wrapper ('import ascpy') of libascend.

An important distinction in the PyGTK code is that the 'library', which holds details of the currently open models and external packages, etc, is considered to be subordinate to the model which is being worked on. In the Tcl/Tk GUI, the library is an entity that stands aside regardless of the currently instantiated model.

Another distinction is that the 'solver' in the PyGTK interface is kept as a somewhat distributed concept. Unlike the Tcl/Tk interface, there is not a dedicated 'solver' window. You can edit the parameters for the solver via the menus, and you can examine its blocking and solving via the 'diagnose' window. It was thought that by tying the solver more tightly with the model browser it would lead to a more intuitive interface.

Because the PyGTK GUI is totally implemented as Python code (directory pygtk, starting point pygtk/gtkbrowser.py), with no compiled code associated with GUI elements, there is no need to have PyGTK installed on your system at the time of building ASCEND. This means that you can upgrade PyGTK on your machine and there should be no need to recompile ASCEND.

Tcl/Tk interface

The Tcl/Tk interface is implemented via directly-coded access to the Tcl/Tk interpreter using the C-language bindings provided by Tcl. The Tk graphical interface is defined via a series of Tcl scripts. This implements full access to the ASCEND C functionality, but expanding the interface is rather laborious due to leg-work of transferring string and numerical values to and from Tcl/Tk.

Extending ASCEND


Testing ASCEND

We have a number of techniques for testing ASCEND, from the low-level 'libascend' code, right up through to the C++ and Python bindings.

CUnit test suites

In the directories ascend/compiler/test, ascend/utilities/test, ascend/general/test, etc, you will find test code written in pure C that interacts directly with libascend. This code depends on the use of CUnit for managing the test suite and assertions, and it is necessary (Mar 2012) that you use the latest CUnit code from the mingw64 branch of the CUnit subversion repository (details here). We're also involved with CUnit and will try to get a release out with the new changes.

To build a copy of CUnit that works with ASCEND, we suggest you use

sudo apt-get install automake autoconf libtool
svn co https://cunit.svn.sourceforge.net/svnroot/cunit/branches/mingw64 cunit 
cd cunit
./bootstrap
./configure --prefix=/usr/local --enable-examples --enable-debug
make -j2
sudo make install

Current versions of CUnit available from Ubuntu and other sources contain unpatched errors so can't be used.

Once you have got CUnit installed on your system, you should have a file /usr/local/lib/libcunit.so (check that!). Then, you should be able to build and run the test suites using

# enter your ascend 'working directory':
cd ~/ascend
# compile the tests suite (check 'config.log' if it says CUnit is not detected) 
scons --config=force test
# set up your environment for the test suite
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:~/ascend:/usr/local/lib
export ASCENDLIBRARY=~/ascend/models
# also need to locate any of the solvers that you are needing...
export ASCENDSOLVERS=~/ascend/solvers/ida:~/ascend/solvers/qrslv
# run the test program to display command-line options available for it
test/test -?
# for example, to run the 'compiler_basics' suite
test/test compiler_basics
# to run a specific test, use the suite name then a dot, then the test name
test/test compiler_basics.stop

We aim to grow these test suites over time, to increasingly provide tests of all the low-level 'libascend' functionality, and to ensure that there are no (significant) memory leaks arising from ASCEND.

See also regression testing which to some extent duplicates this information.

We have been working quite a bit to reduce/eliminate memory leaks with these CUnit-based tests, together with Valgrind. If you find any memory leaks, please let us know!

Python test suite

As well as the low-level CUnit test suite, some additional tests are implemented in Python. These test both the C++ interface layer as well as the Python wrappers, that are compiled using SWIG. This test suite can be run from ./test.py in the top-level of the ASCEND distribution. Open the source file to see the available tests.

Build system

These are some implementation notes that can be read alongside the instructions at Building ASCEND.

ASCEND uses SCons as its build tool, which is a powerful and cross-platform Python-based alternative to Autotools, CMake and others. It is readily installable on all the major platforms. Having said these things, ASCEND is a large and complex software package with many dependencies. Our SCons script aims to support builds on all platforms, but in practice supports Linux, Windows and Mac. This section will outline some key site-specific aspects of the way we use SCons.

  • Use of 'tools' to detect external libraries. It might not be standard practise, but the scons directory of our code contains various 'tools' that perform the job of detecting external libraries (Graphviz, CUnit, GSL, etc) as required. These tools do platform-specific stuff like looking in the registry in Windows, or running 'gsl-config' or similar scripts and parsing the results. These tools then add 'env' variables to SCons with things like 'GSL_LIBS', the list of libraries that must be linked to if GSL is being used.
  • Python detection. We assume that SCons is run via the same version of Python as that against which ASCEND should be linked. So be careful that you install SCons correctly.
  • Tcl/Tk detection. We currently use a bunch of platform-specific kludges to detect Tcl/Tk. We have a script scons/tcl-config.py which we would like to use instead, but we have not yet implemented that.
  • Documentation. We use LyX, which runs on top of pdflatex, to generate a nice big PDF documentation file ('book.pdf', see Category:Documentation). If you don't have LyX and pdflatex, then you can use scons WITH_DOC=1 to disable attempting to buildi the documentation file.
  • Tarball preparation. We use the SCons DistTar tool to create a source code tarball release of ASCEND, via the command scons dist. The main SConstruct file contains a big list of include/exclude pathname patterns that aim to ensure that no big nasty binary files get included in our tarball. Our generated PDF user-documentation is the exception there.
  • PACKAGE metafiles. We aim to make use of some files with the name 'PACKAGE' within our model library folders. This would be a means of flagging mature, tested model files that we would like to package. This is also only partly implemented. In the absence of such a file, all models in the current directory would be included. In the presence of such a file, only files named therein would be included. We need to write a SCons tool that makes use of these files when preparing the DistTar include/exclude lists.
  • Python 'pyc' files. SCons is able to and does prepare compiled python bytecode. Whether this is the optimal given the presence of tools like python-support in Ubuntu is another question. One could also run install-time compilation directly from the NSIS-based Windows installer.
  • libascend. This is the 'core ASCEND engine', comprising the parser/compiler, expression evaluation engine, equation/variable sorting and blocking algorithms, system/solver/integrator APIs, and various shared utility routines including memory management, error reporting, and so on. All this code is in the ascend subdirectory of our source code tree. SCons builds this code by building object code for all the C files in the subdirectory of ascend then linking them together into a bug DLL/SO. Previously, libascend had a dependency on Graphviz but that has been recently converted to a run-time 'dlopen' dependency, avoiding the requirement to have Graphviz before ASCEND can run.
  • Solvers. All solvers are compiled as standalone DLL/SOs, which are linked against libascend. The WITH_SOLVERS flag for SCons allows control over which solvers are attempted.
  • External methods/relations etc. External objects for ASCEND are all located within the models tree, because that way they are where they need to be when the IMPORT command is used to load them. SCons compiles these objects and links them against libascend.
  • PyGTK GUI. The GUI code for Python has a binary dependency on libpython (python27.dll etc) but there is no binary dependency on GTK or PyGTK, because that is handled entirely at the Python end of things. That means that with Python (python-dev on Linux) installed on your machine, you can build all the necessary code for the PyGTK GUI. You can install GTK+ and PyGTK after compiling ASCEND if you wish to run that GUI.

Our development server

Things to know about our development environment:

  • There are lots of templates that can make writing in our wiki easier. Check out {{src}}, {{bug}} and others. In general, please avoid using explicit URLs to ASCEND pages. See Help:Editing.
  • Our buildbot is triggered by any code commits in the trunk of the repository. If you would like a buildbot slave to monitor your branch as well, just ask.
  • We have a Twitter feed generated from all code commit messages. Watch that feed if you want to see the recent code changes.
  • If you commit code with a 'bug XXX' (eg 'bug 513') in the commit message, your commit message will automatically be added as a note to that bug in our bug tracker.
  • URLs like http://ascend4.org/r3000 will redirect you to our ViewVC subversion repository browser, showing the content of a particular changeset/revision.
  • URLs like http://ascend4.org/b500 will redirect you to our bug tracker, showing you that particular number bug.