Controlling the solvers from within METHODS

This page documents an experimental feature. Please tell us if you experience any problems.

Support is added for three new statements relating to 'solver requests' in the ASCEND language:

• SOLVER to specify the solver that should be used.
• OPTION to specify values for options to be used during solving.
• SOLVE to actually trigger the user interface to go ahead and solve the MODEL.

WARNING. There appears to be a bug with the the SOLVER statement, which is that it must be the last statement executed before solution of the model is attempted. Otherwise FIXed and FREEd variables may not be correctly updated (bug 472).

MODEL mymodel;
    x,y IS_A solver_var;

    y = x^2 - x*4 + 4;

METHODS;
METHOD on_load;
    FIX y;
    y := 0;

    SOLVER QRSlv;
    OPTION convopt 'RELNOM_SCALE';
    SOLVE;
END on_load;

END mymodel;

Syntax

SOLVER solvername;
OPTION optionname valuexpression;
SOLVE;

It is proposed to additionally set the syntax for solving particular instances, perhaps as

SOLVER solvername WITH myinstance

or else perhaps as

SOLVE myinstance

Both of these have implications for the GUI layer and how it organises itself.

Motivation

Up until now, the ASCEND language has been carefully designed to avoid polluting the model description with anything relating to actually solving models; solving was left as something that had to be externally directed, either by .a4s scripts, Python scripts, GUI clicks, or low-level C or C++ code. This has been frequently frustrating when using the PyGTK GUI, because that GUI promotes the idea of being able to just load a model and immediately solve it.

However, many models won't just solve with a simple click of the 'solve' button. Some models need specific solvers to be selected first, and other models will only solve if suitable values of the solver parameters are defined. Neither of these things can be done from within the modelling language, so the result is that the write-test-debug loop for model-making has required the user to either do a lot of spurious clicking to set options each time, or else to write a script to 'drive' ASCEND with the correct settings. Neither of these are particularly user-friendly; John Pye thinks it should be possible for the user to do 'everything they want' from within a single modelling file.

Ben Allan has pointed out that such a goal (doing 'everything you want' from within an .a4c file) is unreasonable, because the syntax available within METHODs is too limited to allow anything very sophisticated, and we don't want to develop a full-blown language there. That's a fair point, but the addition of these commands is seen as being so advantageous to user experience that John Pye thought it was worth attempting, regardless.

Implementation

Implementation of this was complicated, because of the deliberate separation of concerns in ASCEND. We have the instance tree which is manipulated by METHODs, then we have a slv_system_t which is built (by system_build), and then a solver essentially just visits that slv_system_t object in order to solve the model. The solver and its associated parameters exist outside the instance tree, making it hard for a METHOD, which works within the instance tree, to access those things.

A new pointer was added to the SimulationInstance object, called slvreq_hooks, allowing an ASCEND 'driver' (GUI, command-line interface, test suite, etc) to provide hook functions (via a call to slvreq_assign_hooks) to which the different SOLVER, OPTION and SOLVE commands could be passed. What those commands then do is left entirely up to the 'driver'.

Whenever the SOLVER command is encountered, it eventually finds its way to a call to slvreq_set_solver. This function locates the slvreq_hooks object in the SimulationInstance, and then, if available, runs the set_solver_fn hook function (hopefully) earlier provided by the 'driver'.

The OPTION allows an expression to be provided as its argument. This expression is first evaluated, returning a 'value_t' structure. This value is then passed to slvreq_set_option, which then runs the set_option_fn hook function earlier assigned, if available.

When the SOLVE command is executed, it calls slvreq_do_solve and then do_solve_fn.

There is currently no support for solving any MODEL other than that returned from GetSimulationRoot. Support for this seems to be a little difficult to design.

As mentioned above, how these functions work exactly is up to the 'driver' to define. Some standard error codes are defined, so that the Initialize function in ascend/compiler/initialize.c is able to make sensible error messages for the user. But otherwise, the 'driver' gets to decide.

In order to allow the 'driver' to store implementation-dependent data along with the slvreq_hooks, a user_data pointer can be provided when the hooks are assigned. This user_data parameter is then provided back the driver whenever any of those hook functions are actually called.

Sample driver: test_slvreq.c

A sample 'driver' implementation with support for the slvreq functions is slvreq:ascend/compiler/test/test_slvreq.c. This implementation works as follows

• SimulationInstance->slvreq_hooks points to a structure containing the simulation instance pointer together with (optionally) a slv_system_t object, sys. If the system has been built this 'sys' is non-NULL. Otherwise it is NULL. The system is built only once, when the first SOLVER statement is encountered.
• SOLVER command identifies the requested solver, calls 'system_build' if necessary, then selects the found solver on the sys.
• OPTION command checks that system has been built, and searches for the requested option. If found, it sets the value, after first checking that the value provided in the METHOD matches the value type declared in the slv_parameter structure.
• SOLVE runs slv_solve on the sys, and returns error flags back as needed.
• The main driver is the function test_slvreq_c. This essentially just loads a model, instantiates the instance tree, assigned the slvreq hooks, then runs the 'on_load' method.

Connecting 'slvreq' to the C++ layer

Connecting the 'slvreq' mechanism to the C++ layer was a bit more complicated. A SolverHooksManager singleton was created which can be assigned with a SolverHooks object. The SolverHooks object has virtual methods for setSolver, setOption, and doSolve. This can be sub-classed according to the user interface. A C++-only test was made (slvreq:pygtk/testslvreq.cpp to demonstrate this code in use:

Invalid language.

You need to specify a language like this: <source lang="html">...</source>

Supported languages for syntax highlighting:

a4c, abap, abc, abnf, actionscript, ada, agda, alan, algol, ampl, amtrix, applescript, arc, arm, as400cl, ascend, asciidoc, asp, aspect, assembler, ats, autohotkey, autoit, avenue, awk, ballerina, bat, bbcode, bcpl, bibtex, biferno, bison, blitzbasic, bms, bnf, boo, c, carbon, ceylon, charmm, chill, chpl, clean, clearbasic, clipper, clojure, clp, cmake, cobol, coffeescript, coldfusion, conf, cpp2, critic, crk, crystal, cs_block_regex, csharp, css, d, dart, delphi, diff, dockerfile, dts, dylan, ebnf, ebnf2, eiffel, elixir, elm, email, erb, erlang, euphoria, exapunks, excel, express, factor, fame, fasm, felix, fish, fortran77, fortran90, frink, fsharp, fstab, fx, gambas, gdb, gdscript, go, graphviz, haml, hare, haskell, haxe, hcl, html, httpd, hugo, icon, idl, idlang, inc_luatex, informix, ini, innosetup, interlis, io, jam, jasmin, java, javascript, js_regex, json, jsp, jsx, julia, kotlin, ldif, less, lhs, lilypond, limbo, lindenscript, lisp, logtalk, lotos, lotus, lua, luban, makefile, maple, markdown, matlab, maya, mercury, meson, miranda, mod2, mod3, modelica, moon, ms, msl, mssql, mxml, n3, nasal, nbc, nemerle, netrexx, nginx, nice, nim, nix, nsis, nxc, oberon, objc, ocaml, octave, oorexx, org, os, oz, paradox, pas, pdf, perl, php, pike, pl1, plperl, plpython, pltcl, po, polygen, pony, pov, powershell, pro, progress, ps, psl, pure, purebasic, purescript, pyrex, python, q, qmake, qml, qu, r, rebol, rego, rexx, rnc, rpg, rpl, rst, ruby, rust, s, sam, sas, scad, scala, scilab, scss, shellscript, slim, small, smalltalk, sml, snmp, snobol, solidity, spec, spn, sql, squirrel, styl, svg, swift, sybase, tcl, tcsh, terraform, tex, toml, tsql, tsx, ttcn3, txt, typescript, upc, vala, vb, verilog, vhd, vimscript, vue, wat, whiley, wren, xml, xpp, yaiff, yaml, yaml_ansible, yang, zig, znn

In this implementation, the SolverHooks can be instantiated with a SolverReporter, so that this SolverReporter can be passed by the doSolve when it makes its call to Simulation::solve(solver,solverreporter). If no SolverReporter or SolverHooks are set, then the SolverHooksManager instantiates a default fallback implementation. To get different behaviour, one sub-classes SolverHooks and uses the setHooks call to register the solver hooks that you want, see slvreq:pygtk/solverhooks.h

Connecting 'slvreq' to the Python layer

The 'slvreq' functionality is connected to ASCEND's Python API using the SWIG 'director' functionality. This allows the class SolverHooks from the C++ layer to be sub-classed in the Python layer, and the additional of Python-specific or GUI-specific stuff to be done as required.

Here is a first sketch of how this functionality would look from Python:

Invalid language.

You need to specify a language like this: <source lang="html">...</source>

Supported languages for syntax highlighting:

a4c, abap, abc, abnf, actionscript, ada, agda, alan, algol, ampl, amtrix, applescript, arc, arm, as400cl, ascend, asciidoc, asp, aspect, assembler, ats, autohotkey, autoit, avenue, awk, ballerina, bat, bbcode, bcpl, bibtex, biferno, bison, blitzbasic, bms, bnf, boo, c, carbon, ceylon, charmm, chill, chpl, clean, clearbasic, clipper, clojure, clp, cmake, cobol, coffeescript, coldfusion, conf, cpp2, critic, crk, crystal, cs_block_regex, csharp, css, d, dart, delphi, diff, dockerfile, dts, dylan, ebnf, ebnf2, eiffel, elixir, elm, email, erb, erlang, euphoria, exapunks, excel, express, factor, fame, fasm, felix, fish, fortran77, fortran90, frink, fsharp, fstab, fx, gambas, gdb, gdscript, go, graphviz, haml, hare, haskell, haxe, hcl, html, httpd, hugo, icon, idl, idlang, inc_luatex, informix, ini, innosetup, interlis, io, jam, jasmin, java, javascript, js_regex, json, jsp, jsx, julia, kotlin, ldif, less, lhs, lilypond, limbo, lindenscript, lisp, logtalk, lotos, lotus, lua, luban, makefile, maple, markdown, matlab, maya, mercury, meson, miranda, mod2, mod3, modelica, moon, ms, msl, mssql, mxml, n3, nasal, nbc, nemerle, netrexx, nginx, nice, nim, nix, nsis, nxc, oberon, objc, ocaml, octave, oorexx, org, os, oz, paradox, pas, pdf, perl, php, pike, pl1, plperl, plpython, pltcl, po, polygen, pony, pov, powershell, pro, progress, ps, psl, pure, purebasic, purescript, pyrex, python, q, qmake, qml, qu, r, rebol, rego, rexx, rnc, rpg, rpl, rst, ruby, rust, s, sam, sas, scad, scala, scilab, scss, shellscript, slim, small, smalltalk, sml, snmp, snobol, solidity, spec, spn, sql, squirrel, styl, svg, swift, sybase, tcl, tcsh, terraform, tex, toml, tsql, tsx, ttcn3, txt, typescript, upc, vala, vb, verilog, vhd, vimscript, vue, wat, whiley, wren, xml, xpp, yaiff, yaml, yaml_ansible, yang, zig, znn

This functionality still needs a little re-thinking. In fact, it should not be necessary to pass solver and reporter parameters during the 'sim.solve' call, because the SolverParameters are actually stored in the 'sys' within the 'Simulation', so passing in a new solver is just ambiguous and strange.

However, this is now tested; you can see the test code in slvreq:test.py.

Problems and questions

Bug: there is a bug if the SOLVER and other commands are placed before all the METHODS that specify values etc. bug 472.

Problem: some effort was made (simlist.c) to implement support for multiple SimulationInstances per instance tree. This has never been fully realised (and the merits of this idea are not yet completely clear to me). But the the current slvreq depends crucially on the fact that, within a running METHOD, it's possible to determine what is the current SimulationInstance.

Question: in some modelling environments, it is possible to define 'subsolvers'. For example, a conditional solver might be able to make use of either CONOPT or IPOPT for optimisation, so it might be necessary to provide syntax that would work for this case. Or an optimisation solver might be wrapper around a dynamic solver. Can we support that? What changes would be required?

Question: is it OK to associate the hook functions with the SimulationInstance? Is there any case where this data should be stored somewhere else?

See also Solver NOTES, another proposed solution to this problem.

GUI integration

This syntax has not yet been incorporated into the ASCEND PyGTK or Tcl/Tk GUI. As a result it's not yet generally usable. The proposed GUI integration flow is something like this:

• GUI is started
• GUI loads a model
• GUI instantiates model
• GUI runs slvreq_register_hooks
• libascend stores register hooks in SimulationInstance
• GUI requests 'on_load' method to be run.
• libascend starts executing method statements
• libascend encounters 'USE' statement.
• libascend locates SimulationInstance and find slvreq_hooks object.
• libascend runs slvreq_hooks->set_solver_fn.
• GUI received request to set solver
• GUI checks that solver is elegible for this problem
• GUI sets solver as requests (or writes an error message and returns error code)
• libascend proceeds with more method statements
• libascend find 'SET' statment
• libascend runs slvreq_hooks->set_param_fn (passes string value of requested solver parameter to GUI)
• GUI receives reqest
• GUI either
  • finds that solvername.parametername is valid, and sets the parameter (after casting it to the appropriate type)
  • finds that solvername.parametername is not valid, and returns an error code
• libascend proceeds with more METHOD statments...