User:Arash

Arash Sadrieh is working on developing GPU-based solvers for ASCEND. He is a PhD student at Murdoch University in Western Australia.

Development branch: arash:

Goals

GSOC-2011 Goals

• Complete the current prototype.
• Implement the batch multi-vector residual evaluator
• Integrate the approach to QRCUDA
• Integrate the QRCUDA into the ASCEND GUI.
• Test the project with different hardware and software platforms.

Project Plan

• Complete the current prototype.
  • Clear step-by-step instructions allowing a new user to setup and test/use your solver
  • General architecture improvement
  • Move the initialization and shutdown tasks from the unit test to the “QRCUDA.c”.
  • Fix the distillation case study, the current model is unsolvable.
  • Optimise the CUDA code
    • Change kernels memory access pattern to coalesced access
    • Store the mapping information into fast texture constant memory
    • Change Memory management model from standard model to PINNED memory management. This makes the memory transfer between host and device faster.
  • Implement hybrid CPU/GPU based evaluation instead of GPU-based evaluation. By doing this, the CPU can be used for the small equation groups while the GPU is busy evaluating the large groups.
    • support for models containing 'external relations'
  • Prepare a multi platform Makefile to compile and build BinCUDAs
  • Complete the external functions in “btcudapl.cu”

• Implement the batch multi-vector residual evaluator
  • Define the heuristic formula for multi-vector residual evaluator
  • Research all of the variations of Armijo's rule (Grippo et al., 1986)
  • Convert current kernels from 2D kernels into 3D; the extra dimension is used for each input vector.
  • Implement the heuristic formula in the kernels
  • Implement a separate kernel that finds the lowest residuals normal and returns the index of the lowest residual normal

• Integrate the approach to QRCUDA
  • Add block evaluation feature to batch single-vector evaluator.
  • Modify standard residual/gradient evaluator to use new single-vector evaluator.
  • Integrate batch multi-vector evaluator into QRCUDA line search.
  • Modify current line search algorithm to use the batch multi-vector evaluator.
  • Benchmark the results.

• Integrate the QRCUDA into the ASCEND GUI.
  • Fix the Bintoken unloading bug
  • Fix Bintoken auto rebuild sensing feature in the PyGTK
  • Add GUI menus and dialogs
    • ensuring all required user-configurable parameters are exposed though the solver API
    • implement testing of CUDA hardware availability when the solver is first loaded; only make QRCUDA available if the tests succeed, give user feedback if fails.

• Test the project with different hardware and software platforms.
  • testing of memory leakage and stability.

Progress

• After 23-May
  • The GPU memory management model was changed from standard to PINNED. This makes data transferrer between host and device two times faster.
  • Batch evaluator can now perform hybrid CPU/GPU evaluations so that the CPU can be used for small equation groups while the GPU is busy evaluating the large groups.
  • The benchmark model was modified slightly so it is now solvable.
• After 6-June
  • Cleanup in the prototype
  • The GPU init and shutdown methods are moved to the QRCUDA.
  • The dependency to the common makefile and headers (located in sdk samples) was removed.
  • The linux version of BinCUDA's makefile was created (windows and mac/os versions are coming soon).
  • A testcase for QRCUDA was implemented
  • A new form added to the main GUI that shows some information about current CUDA enable devices in the system (speed, number of cores, max memory, number of multiprocessors ....).
  • The Bincuda unload bug was fixed in the clean ups.
• After 16-June
  • More clean-ups in the BinCUDAs.
  • The active block evaluation mechanism was added to the batch evaluator.
  • QRCUDA is now using GPU-based model evaluation for the residual evaluation in large blocks (the code was tested on arash:models/test/bintok/test2.a4c, more testing is required).
  • QRCUDA was tested with arash:models/test/bintok/larg_distil.a4c and after some bug fixes, the GPU evaluator results are now identical to the same results achieved from standard calc_residuals method.

Ideas and Issues

A list of ideas and issues with the current implementation is provided as follows (comments and critiques are greatly appreciated):

1. In the batch evaluator (relman.c:relman_batch_eval), 60 % of the total time is consumed in the rel_set_residual() calls. How can we optimize this function?.
2. Can the solver provide cheap feedback to the user showing the degree of parallelism that was achieved during a particular model solution?
3. Sometimes QRSlv makes use of a Brent solver for blocks with a single equation. Is that the best approach when a GPU is available?
4. More large demonstration models are needed. Let's go and find some.

Installation

To run BinCUDA objects, the host machine should be supplied with a NVIDIA CUDA enabled GPU card (preferably Fermi or more recent architecture). The card should have the ability to perform 'double' floating point calculations (compute_13+). In addition to the GPU hardware, the CUDA SDK and developer driver should be installed on the host machine and it is necessary to link the BinCUDA's Makefile to the SDK directory.

Installing CUDA SDK on Linux

The following explains step by step instructions for installing CUDA SDK on an Ubuntu (10.04) 32bit machine. It should be noted that the installation process on other flavors of Linux is quiet similar, however, the equivalent file distribution from NVIDIA website should be replaced with the current Ubuntu (10.04) 32bit file addresses.

1) In the terminal window issue

Invalid language.

You need to specify a language like this: <source lang="html">...</source>

Supported languages for syntax highlighting:

a4c, abap, abc, abnf, actionscript, ada, agda, alan, algol, ampl, amtrix, applescript, arc, arm, as400cl, ascend, asciidoc, asp, aspect, assembler, ats, autohotkey, autoit, avenue, awk, ballerina, bat, bbcode, bcpl, bibtex, biferno, bison, blitzbasic, bms, bnf, boo, c, carbon, ceylon, charmm, chill, chpl, clean, clearbasic, clipper, clojure, clp, cmake, cobol, coffeescript, coldfusion, conf, cpp2, critic, crk, crystal, cs_block_regex, csharp, css, d, dart, delphi, diff, dockerfile, dts, dylan, ebnf, ebnf2, eiffel, elixir, elm, email, erb, erlang, euphoria, exapunks, excel, express, factor, fame, fasm, felix, fish, fortran77, fortran90, frink, fsharp, fstab, fx, gambas, gdb, gdscript, go, graphviz, haml, hare, haskell, haxe, hcl, html, httpd, hugo, icon, idl, idlang, inc_luatex, informix, ini, innosetup, interlis, io, jam, jasmin, java, javascript, js_regex, json, jsp, jsx, julia, kotlin, ldif, less, lhs, lilypond, limbo, lindenscript, lisp, logtalk, lotos, lotus, lua, luban, makefile, maple, markdown, matlab, maya, mercury, meson, miranda, mod2, mod3, modelica, moon, ms, msl, mssql, mxml, n3, nasal, nbc, nemerle, netrexx, nginx, nice, nim, nix, nsis, nxc, oberon, objc, ocaml, octave, oorexx, org, os, oz, paradox, pas, pdf, perl, php, pike, pl1, plperl, plpython, pltcl, po, polygen, pony, pov, powershell, pro, progress, ps, psl, pure, purebasic, purescript, pyrex, python, q, qmake, qml, qu, r, rebol, rego, rexx, rnc, rpg, rpl, rst, ruby, rust, s, sam, sas, scad, scala, scilab, scss, shellscript, slim, small, smalltalk, sml, snmp, snobol, solidity, spec, spn, sql, squirrel, styl, svg, swift, sybase, tcl, tcsh, terraform, tex, toml, tsql, tsx, ttcn3, txt, typescript, upc, vala, vb, verilog, vhd, vimscript, vue, wat, whiley, wren, xml, xpp, yaiff, yaml, yaml_ansible, yang, zig, znn

2) Stop the X Windows by pressing CTRL+ALT+F1 and then issue

Invalid language.

You need to specify a language like this: <source lang="html">...</source>

Supported languages for syntax highlighting:

a4c, abap, abc, abnf, actionscript, ada, agda, alan, algol, ampl, amtrix, applescript, arc, arm, as400cl, ascend, asciidoc, asp, aspect, assembler, ats, autohotkey, autoit, avenue, awk, ballerina, bat, bbcode, bcpl, bibtex, biferno, bison, blitzbasic, bms, bnf, boo, c, carbon, ceylon, charmm, chill, chpl, clean, clearbasic, clipper, clojure, clp, cmake, cobol, coffeescript, coldfusion, conf, cpp2, critic, crk, crystal, cs_block_regex, csharp, css, d, dart, delphi, diff, dockerfile, dts, dylan, ebnf, ebnf2, eiffel, elixir, elm, email, erb, erlang, euphoria, exapunks, excel, express, factor, fame, fasm, felix, fish, fortran77, fortran90, frink, fsharp, fstab, fx, gambas, gdb, gdscript, go, graphviz, haml, hare, haskell, haxe, hcl, html, httpd, hugo, icon, idl, idlang, inc_luatex, informix, ini, innosetup, interlis, io, jam, jasmin, java, javascript, js_regex, json, jsp, jsx, julia, kotlin, ldif, less, lhs, lilypond, limbo, lindenscript, lisp, logtalk, lotos, lotus, lua, luban, makefile, maple, markdown, matlab, maya, mercury, meson, miranda, mod2, mod3, modelica, moon, ms, msl, mssql, mxml, n3, nasal, nbc, nemerle, netrexx, nginx, nice, nim, nix, nsis, nxc, oberon, objc, ocaml, octave, oorexx, org, os, oz, paradox, pas, pdf, perl, php, pike, pl1, plperl, plpython, pltcl, po, polygen, pony, pov, powershell, pro, progress, ps, psl, pure, purebasic, purescript, pyrex, python, q, qmake, qml, qu, r, rebol, rego, rexx, rnc, rpg, rpl, rst, ruby, rust, s, sam, sas, scad, scala, scilab, scss, shellscript, slim, small, smalltalk, sml, snmp, snobol, solidity, spec, spn, sql, squirrel, styl, svg, swift, sybase, tcl, tcsh, terraform, tex, toml, tsql, tsx, ttcn3, txt, typescript, upc, vala, vb, verilog, vhd, vimscript, vue, wat, whiley, wren, xml, xpp, yaiff, yaml, yaml_ansible, yang, zig, znn

3) The X Windows should be restarted with the new NVIDIA driver and then you should be able to install the CUDA 3.2 toolkit and samples (it is recommended to use default directories - i.e. /usr/local/cuda and ~/NVIDIA_GPU_Computing_SDK).

Invalid language.

You need to specify a language like this: <source lang="html">...</source>

Supported languages for syntax highlighting:

a4c, abap, abc, abnf, actionscript, ada, agda, alan, algol, ampl, amtrix, applescript, arc, arm, as400cl, ascend, asciidoc, asp, aspect, assembler, ats, autohotkey, autoit, avenue, awk, ballerina, bat, bbcode, bcpl, bibtex, biferno, bison, blitzbasic, bms, bnf, boo, c, carbon, ceylon, charmm, chill, chpl, clean, clearbasic, clipper, clojure, clp, cmake, cobol, coffeescript, coldfusion, conf, cpp2, critic, crk, crystal, cs_block_regex, csharp, css, d, dart, delphi, diff, dockerfile, dts, dylan, ebnf, ebnf2, eiffel, elixir, elm, email, erb, erlang, euphoria, exapunks, excel, express, factor, fame, fasm, felix, fish, fortran77, fortran90, frink, fsharp, fstab, fx, gambas, gdb, gdscript, go, graphviz, haml, hare, haskell, haxe, hcl, html, httpd, hugo, icon, idl, idlang, inc_luatex, informix, ini, innosetup, interlis, io, jam, jasmin, java, javascript, js_regex, json, jsp, jsx, julia, kotlin, ldif, less, lhs, lilypond, limbo, lindenscript, lisp, logtalk, lotos, lotus, lua, luban, makefile, maple, markdown, matlab, maya, mercury, meson, miranda, mod2, mod3, modelica, moon, ms, msl, mssql, mxml, n3, nasal, nbc, nemerle, netrexx, nginx, nice, nim, nix, nsis, nxc, oberon, objc, ocaml, octave, oorexx, org, os, oz, paradox, pas, pdf, perl, php, pike, pl1, plperl, plpython, pltcl, po, polygen, pony, pov, powershell, pro, progress, ps, psl, pure, purebasic, purescript, pyrex, python, q, qmake, qml, qu, r, rebol, rego, rexx, rnc, rpg, rpl, rst, ruby, rust, s, sam, sas, scad, scala, scilab, scss, shellscript, slim, small, smalltalk, sml, snmp, snobol, solidity, spec, spn, sql, squirrel, styl, svg, swift, sybase, tcl, tcsh, terraform, tex, toml, tsql, tsx, ttcn3, txt, typescript, upc, vala, vb, verilog, vhd, vimscript, vue, wat, whiley, wren, xml, xpp, yaiff, yaml, yaml_ansible, yang, zig, znn

4) Add /usr/local/cuda/bin to PATH and /user/local/cuda/lib to LD_LIBRARY_PATH by appending this text to ~/.bashrc file:

Invalid language.

You need to specify a language like this: <source lang="html">...</source>

Supported languages for syntax highlighting:

a4c, abap, abc, abnf, actionscript, ada, agda, alan, algol, ampl, amtrix, applescript, arc, arm, as400cl, ascend, asciidoc, asp, aspect, assembler, ats, autohotkey, autoit, avenue, awk, ballerina, bat, bbcode, bcpl, bibtex, biferno, bison, blitzbasic, bms, bnf, boo, c, carbon, ceylon, charmm, chill, chpl, clean, clearbasic, clipper, clojure, clp, cmake, cobol, coffeescript, coldfusion, conf, cpp2, critic, crk, crystal, cs_block_regex, csharp, css, d, dart, delphi, diff, dockerfile, dts, dylan, ebnf, ebnf2, eiffel, elixir, elm, email, erb, erlang, euphoria, exapunks, excel, express, factor, fame, fasm, felix, fish, fortran77, fortran90, frink, fsharp, fstab, fx, gambas, gdb, gdscript, go, graphviz, haml, hare, haskell, haxe, hcl, html, httpd, hugo, icon, idl, idlang, inc_luatex, informix, ini, innosetup, interlis, io, jam, jasmin, java, javascript, js_regex, json, jsp, jsx, julia, kotlin, ldif, less, lhs, lilypond, limbo, lindenscript, lisp, logtalk, lotos, lotus, lua, luban, makefile, maple, markdown, matlab, maya, mercury, meson, miranda, mod2, mod3, modelica, moon, ms, msl, mssql, mxml, n3, nasal, nbc, nemerle, netrexx, nginx, nice, nim, nix, nsis, nxc, oberon, objc, ocaml, octave, oorexx, org, os, oz, paradox, pas, pdf, perl, php, pike, pl1, plperl, plpython, pltcl, po, polygen, pony, pov, powershell, pro, progress, ps, psl, pure, purebasic, purescript, pyrex, python, q, qmake, qml, qu, r, rebol, rego, rexx, rnc, rpg, rpl, rst, ruby, rust, s, sam, sas, scad, scala, scilab, scss, shellscript, slim, small, smalltalk, sml, snmp, snobol, solidity, spec, spn, sql, squirrel, styl, svg, swift, sybase, tcl, tcsh, terraform, tex, toml, tsql, tsx, ttcn3, txt, typescript, upc, vala, vb, verilog, vhd, vimscript, vue, wat, whiley, wren, xml, xpp, yaiff, yaml, yaml_ansible, yang, zig, znn

5) Now you should be able to compile the SDK samples,

Invalid language.

You need to specify a language like this: <source lang="html">...</source>

Supported languages for syntax highlighting:

a4c, abap, abc, abnf, actionscript, ada, agda, alan, algol, ampl, amtrix, applescript, arc, arm, as400cl, ascend, asciidoc, asp, aspect, assembler, ats, autohotkey, autoit, avenue, awk, ballerina, bat, bbcode, bcpl, bibtex, biferno, bison, blitzbasic, bms, bnf, boo, c, carbon, ceylon, charmm, chill, chpl, clean, clearbasic, clipper, clojure, clp, cmake, cobol, coffeescript, coldfusion, conf, cpp2, critic, crk, crystal, cs_block_regex, csharp, css, d, dart, delphi, diff, dockerfile, dts, dylan, ebnf, ebnf2, eiffel, elixir, elm, email, erb, erlang, euphoria, exapunks, excel, express, factor, fame, fasm, felix, fish, fortran77, fortran90, frink, fsharp, fstab, fx, gambas, gdb, gdscript, go, graphviz, haml, hare, haskell, haxe, hcl, html, httpd, hugo, icon, idl, idlang, inc_luatex, informix, ini, innosetup, interlis, io, jam, jasmin, java, javascript, js_regex, json, jsp, jsx, julia, kotlin, ldif, less, lhs, lilypond, limbo, lindenscript, lisp, logtalk, lotos, lotus, lua, luban, makefile, maple, markdown, matlab, maya, mercury, meson, miranda, mod2, mod3, modelica, moon, ms, msl, mssql, mxml, n3, nasal, nbc, nemerle, netrexx, nginx, nice, nim, nix, nsis, nxc, oberon, objc, ocaml, octave, oorexx, org, os, oz, paradox, pas, pdf, perl, php, pike, pl1, plperl, plpython, pltcl, po, polygen, pony, pov, powershell, pro, progress, ps, psl, pure, purebasic, purescript, pyrex, python, q, qmake, qml, qu, r, rebol, rego, rexx, rnc, rpg, rpl, rst, ruby, rust, s, sam, sas, scad, scala, scilab, scss, shellscript, slim, small, smalltalk, sml, snmp, snobol, solidity, spec, spn, sql, squirrel, styl, svg, swift, sybase, tcl, tcsh, terraform, tex, toml, tsql, tsx, ttcn3, txt, typescript, upc, vala, vb, verilog, vhd, vimscript, vue, wat, whiley, wren, xml, xpp, yaiff, yaml, yaml_ansible, yang, zig, znn

6) The executable binaries for the samples will be created in the "bin" directory. For example you should be able to run N-Body simulation sample:

Invalid language.

You need to specify a language like this: <source lang="html">...</source>

Supported languages for syntax highlighting:

a4c, abap, abc, abnf, actionscript, ada, agda, alan, algol, ampl, amtrix, applescript, arc, arm, as400cl, ascend, asciidoc, asp, aspect, assembler, ats, autohotkey, autoit, avenue, awk, ballerina, bat, bbcode, bcpl, bibtex, biferno, bison, blitzbasic, bms, bnf, boo, c, carbon, ceylon, charmm, chill, chpl, clean, clearbasic, clipper, clojure, clp, cmake, cobol, coffeescript, coldfusion, conf, cpp2, critic, crk, crystal, cs_block_regex, csharp, css, d, dart, delphi, diff, dockerfile, dts, dylan, ebnf, ebnf2, eiffel, elixir, elm, email, erb, erlang, euphoria, exapunks, excel, express, factor, fame, fasm, felix, fish, fortran77, fortran90, frink, fsharp, fstab, fx, gambas, gdb, gdscript, go, graphviz, haml, hare, haskell, haxe, hcl, html, httpd, hugo, icon, idl, idlang, inc_luatex, informix, ini, innosetup, interlis, io, jam, jasmin, java, javascript, js_regex, json, jsp, jsx, julia, kotlin, ldif, less, lhs, lilypond, limbo, lindenscript, lisp, logtalk, lotos, lotus, lua, luban, makefile, maple, markdown, matlab, maya, mercury, meson, miranda, mod2, mod3, modelica, moon, ms, msl, mssql, mxml, n3, nasal, nbc, nemerle, netrexx, nginx, nice, nim, nix, nsis, nxc, oberon, objc, ocaml, octave, oorexx, org, os, oz, paradox, pas, pdf, perl, php, pike, pl1, plperl, plpython, pltcl, po, polygen, pony, pov, powershell, pro, progress, ps, psl, pure, purebasic, purescript, pyrex, python, q, qmake, qml, qu, r, rebol, rego, rexx, rnc, rpg, rpl, rst, ruby, rust, s, sam, sas, scad, scala, scilab, scss, shellscript, slim, small, smalltalk, sml, snmp, snobol, solidity, spec, spn, sql, squirrel, styl, svg, swift, sybase, tcl, tcsh, terraform, tex, toml, tsql, tsx, ttcn3, txt, typescript, upc, vala, vb, verilog, vhd, vimscript, vue, wat, whiley, wren, xml, xpp, yaiff, yaml, yaml_ansible, yang, zig, znn

BinCUDA Makefile settings

After installing CUDA SDK, the CUDA_INSTALL_PATH variable in the makefile (arash:ascend/bintokens/bincuda/Makefile) should be pointed to the SDK directory.

Test models

A distillation column model was created to test the GPU-based bintokens, the model was proposed by Ben Allan.

Distillation Column Model

REQUIRE "column.a4l";
MODEL larg_distil() REFINES test_demo_column();
        demo IS_A
        demo_column(['n_butane','n_pentane','n_hexane','n_heptane','n_octane','n_nonane','n_decane'],'n_decane',100,51);
METHODS
END larg_distil;

Number of Equations

The model originally has 128 unique equation symbolic forms and 19959 equation instances.The number of relations in the model can be adjusted by changing two parameters, 100 and 51, by a multiplicative factor. For example in arash:models/test/bintok/larg_distil.a4c,

REQUIRE "column.a4l";
MODEL larg_distil() REFINES test_demo_column();
        demo IS_A
        demo_column(['n_butane','n_pentane','n_hexane','n_heptane','n_octane','n_nonane','n_decane'],'n_decane',500,255);
METHODS
END larg_distil;

Alternatively, multiple columns can be used instead of single column (arash:models/test/bintok/larg_distil_2.a4c),

REQUIRE "column.a4l";
MODEL c5_10_demo_column() REFINES test_demo_column();
        demo,demo2,demo3,demo4 IS_A
        demo_column(['n_butane','n_pentane','n_hexane','n_heptane','n_octane','n_nonane','n_decane'],'n_decane',100,51);
METHODS
END c5_10_demo_column;

Running the test

A CUnit test case was prepared to test BinCUDA generation and execution. The code is located in test_bincuda.c(arash:ascend/compiler/test/test_bincuda.c). You can run the test by executing "test/test compiler_bincuda.gen" at top level ASCEND directory.

The test case outputs the CPU-based evaluation time, GPU-based evaluation time and the number of equations in the model. It generates the code in the "/tmp" directory and the Makefile located in the same directory is responsible for building the shared binary object for BinCUDAs. The CUDA build and compile commands are provided in the Makefile(arash:ascend/bintokens/bincuda/Makefile).

To change the current benchmark model, you can change the macro FILENAMESTEM in the code. [Please note that if your model includes any specific ASCEND function (e.g. asc_ipow) the function should be defined in the btcudapl.cu (arash:ascend/bintokens/bincuda/btcudapl.cu) file.]