OcamlParser

For GSOC2016, Hugo implemented an experimental parser for the ASCEND modeling language that allows a faster rate of experimentation for new syntactic features compared to the existing ASCEND parser.

The new parser was designed to make it easy to parse ASCEND syntax into a syntax tree, perform transformation passes over this syntax tree and then serialize the result back into a textual format, which can be fed into the existing ASCEND compiler. This lets us implement new language features in terms of existing ones and experiment without needing to modify the core ASCEND compiler.

One advantage of having this new parser and syntax-tree tooling is that working with syntax trees is hard to do in the existing parsing architecture because ASCEND's C compiler parses and interprets the code at the same time, and doesn't give us a chance to apply syntax transformers before the code is interpreted. Another advantage of the new parser is that it is written in Ocaml, which as a functional programming language is specially suited for operating on tree-like data structures.

Compiling and Using the New Parser

The code for the new experimental parser as well as the infrastructure for writing syntax transformers can currently be found in the hugo: branch of the ASCEND Subversion repository. It has not been merged into the trunk yet.

svn co http://svn.ascend4.org/branches/hugo ascend-hugo

To run the new parser and to write syntax transformers, you will need to install the Ocaml compiler, and some Ocaml libraries. The recommended way to do this is to use your distro's package manager to install the Ocaml compiler and the Opam package manager and then use Opam to download and install the required Ocaml libraries.

Invalid language.

You need to specify a language like this: <source lang="html">...</source>

Supported languages for syntax highlighting:

a4c, abap, abc, abnf, actionscript, ada, agda, alan, algol, ampl, amtrix, applescript, arc, arm, as400cl, ascend, asciidoc, asp, aspect, assembler, ats, autohotkey, autoit, avenue, awk, ballerina, bat, bbcode, bcpl, bibtex, biferno, bison, blitzbasic, bms, bnf, boo, c, carbon, ceylon, charmm, chill, chpl, clean, clearbasic, clipper, clojure, clp, cmake, cobol, coffeescript, coldfusion, conf, cpp2, critic, crk, crystal, cs_block_regex, csharp, css, d, dart, delphi, diff, dockerfile, dts, dylan, ebnf, ebnf2, eiffel, elixir, elm, email, erb, erlang, euphoria, exapunks, excel, express, factor, fame, fasm, felix, fish, fortran77, fortran90, frink, fsharp, fstab, fx, gambas, gdb, gdscript, go, graphviz, haml, hare, haskell, haxe, hcl, html, httpd, hugo, icon, idl, idlang, inc_luatex, informix, ini, innosetup, interlis, io, jam, jasmin, java, javascript, js_regex, json, jsp, jsx, julia, kotlin, ldif, less, lhs, lilypond, limbo, lindenscript, lisp, logtalk, lotos, lotus, lua, luban, makefile, maple, markdown, matlab, maya, mercury, meson, miranda, mod2, mod3, modelica, moon, ms, msl, mssql, mxml, n3, nasal, nbc, nemerle, netrexx, nginx, nice, nim, nix, nsis, nxc, oberon, objc, ocaml, octave, oorexx, org, os, oz, paradox, pas, pdf, perl, php, pike, pl1, plperl, plpython, pltcl, po, polygen, pony, pov, powershell, pro, progress, ps, psl, pure, purebasic, purescript, pyrex, python, q, qmake, qml, qu, r, rebol, rego, rexx, rnc, rpg, rpl, rst, ruby, rust, s, sam, sas, scad, scala, scilab, scss, shellscript, slim, small, smalltalk, sml, snmp, snobol, solidity, spec, spn, sql, squirrel, styl, svg, swift, sybase, tcl, tcsh, terraform, tex, toml, tsql, tsx, ttcn3, txt, typescript, upc, vala, vb, verilog, vhd, vimscript, vue, wat, whiley, wren, xml, xpp, yaiff, yaml, yaml_ansible, yang, zig, znn

Then, go to the "ocaml" folder in the ASCEND source directory and build things with the makefile. Now you can start playing with the new parser.

Invalid language.

You need to specify a language like this: <source lang="html">...</source>

Supported languages for syntax highlighting:

a4c, abap, abc, abnf, actionscript, ada, agda, alan, algol, ampl, amtrix, applescript, arc, arm, as400cl, ascend, asciidoc, asp, aspect, assembler, ats, autohotkey, autoit, avenue, awk, ballerina, bat, bbcode, bcpl, bibtex, biferno, bison, blitzbasic, bms, bnf, boo, c, carbon, ceylon, charmm, chill, chpl, clean, clearbasic, clipper, clojure, clp, cmake, cobol, coffeescript, coldfusion, conf, cpp2, critic, crk, crystal, cs_block_regex, csharp, css, d, dart, delphi, diff, dockerfile, dts, dylan, ebnf, ebnf2, eiffel, elixir, elm, email, erb, erlang, euphoria, exapunks, excel, express, factor, fame, fasm, felix, fish, fortran77, fortran90, frink, fsharp, fstab, fx, gambas, gdb, gdscript, go, graphviz, haml, hare, haskell, haxe, hcl, html, httpd, hugo, icon, idl, idlang, inc_luatex, informix, ini, innosetup, interlis, io, jam, jasmin, java, javascript, js_regex, json, jsp, jsx, julia, kotlin, ldif, less, lhs, lilypond, limbo, lindenscript, lisp, logtalk, lotos, lotus, lua, luban, makefile, maple, markdown, matlab, maya, mercury, meson, miranda, mod2, mod3, modelica, moon, ms, msl, mssql, mxml, n3, nasal, nbc, nemerle, netrexx, nginx, nice, nim, nix, nsis, nxc, oberon, objc, ocaml, octave, oorexx, org, os, oz, paradox, pas, pdf, perl, php, pike, pl1, plperl, plpython, pltcl, po, polygen, pony, pov, powershell, pro, progress, ps, psl, pure, purebasic, purescript, pyrex, python, q, qmake, qml, qu, r, rebol, rego, rexx, rnc, rpg, rpl, rst, ruby, rust, s, sam, sas, scad, scala, scilab, scss, shellscript, slim, small, smalltalk, sml, snmp, snobol, solidity, spec, spn, sql, squirrel, styl, svg, swift, sybase, tcl, tcsh, terraform, tex, toml, tsql, tsx, ttcn3, txt, typescript, upc, vala, vb, verilog, vhd, vimscript, vue, wat, whiley, wren, xml, xpp, yaiff, yaml, yaml_ansible, yang, zig, znn

The makefile, hugo:ocaml/makefile, shows how to compile the code and the tests filder has some examples of how to run it. For more details see hugo:ocaml/README.txt.

Writing Syntax Transformers

The goal of this new Ocaml parser is to support syntax experiments that work by mapping ASCEND syntax trees (possibly with some extensions) into a vanilla ASCEND syntax tree that can be fed into the existing C compiler. However, writing these syntax tree transformers by hand can be a very laborious process because you need to write a lot of code just to traverse the syntax tree and get to the interesting leaves that we might want to modify. (Have a look at hugo:ocaml/tokenize_tree.ml to see how traversing a tree by hand looks like).

One of the ways to avoid boilerplate traversal code is to use the "Visitor Pattern" and write traversals by extending a default traversal that doesn't do anything. To support this pattern, there is an interface for syntax tree iterators in hugo:ocaml/ast_iterator.ml as well as a default_iterator that doesn't do anything but that can be easily extended into an iterator that does something interesting in relevant nodes and just walks through everything else. There is also a similar system for transformers (functions that convert one AST into another) in hugo:ocaml/ast_mapper.ml.

A good example of how to use the tree iteration and transformation modules can be found in hugo:ocaml/infer_derivatives.ml. This syntax transformer receives makes DERIVATIVE OF declarations optional by inferring them from the uses of the der() operator throughout the program.