Matei

Easy intro to writing a BASIC interpreter (with ANTLR)

2017-11-26T00:00:00+00:00

Getting started with parsing can be a bit daunting in the beginning. There’s a lot of information to be digested before even a simple parser can be written. This is yet another attempt to present the things in a simple and straightforward manner. Since I very much believe in learning by doing let’s just try to write a simple BASIC interpreter.

The plan

Basically we need two things for our very simple interpreter:

The parser. It’s job is to consume a stream of character and create a higher level representation of our program.
The interpreter. It takes the higher representation produced by the parser and executes it.

The tools

A grammar. This defines the structure of our language.
A parser generator. We’ll use ANTLR. It’s a popular tool and has good documentation and support. We’ll use it to generate a parser that can ingest a program with our own defined syntax.
ANTLR plugin for Maven to make our life easy.

Source code

You can find the Git repo for this article at https://github.com/mateiw/littlebasic.

The language

BASIC comes in many variants. Since the goal here is just to play with ANTLR we’ll just define our own very simple BASIC dialect.

An simple program could be like this:

REM Greatest common divisor
INPUT "A=" ain
INPUT "B=" bin
a = VAL(ain)
b = VAL(bin)

WHILE b > 0
    t = a MOD b
    a = b
    b = t
END

PRINT "GCD=" + a

Although quite simple it should be enough to cover the basic (no pun intended) concepts.

The grammar

To get started we have to to defined the grammar of our toy language. Like a human grammar this will define what words and sentences are acceptable in our language.

The very first thing we need is a the so called lexer grammar. This is responsible for breaking up the continuous stream of character that constitutes our program into recognizable tokens like literals, strings, operators, etc.

lexer grammar LBTokens; // note "lexer grammar"

// operators
MUL : '*' ;
DIV : '/' ;
ADD : '+' ;
SUB : '-' ;
EXP : '^' ;
MOD : 'MOD' ;

// logical
NEQ : '<>' ;
GTE : '>=' ;
LTE : '<=' ;
GT  : '>' ;
LT  : '<' ;
EQ  : '=' ;

// relational
AND : 'AND' | 'and' ;
OR  : 'OR' | 'or' ;
NOT : 'NOT' | 'not' ;

// other
COMMA  : ',' ;
LPAREN : '(' ;
RPAREN : ')' ;

// functions
LEN : 'LEN' | 'len' ;
VAL : 'VAL' | 'val' ;
ISNAN   : 'ISNAN' | 'isnan' ;

// keywords
PRINT   : 'PRINT' | 'print' ;
INPUT   : 'INPUT' | 'input' ;
LET     : 'LET' | 'let' ;
REM     : 'REM' | 'rem' ;
IF      : 'IF' | 'if' ;
THEN    : 'THEN' | 'then' ;
ELSE    : 'ELSE' | 'else' ;
END     : 'END' | 'end';
FOR     : 'FOR' | 'for' ;
WHILE   : 'WHILE' | 'while' ;
REPEAT  : 'REPEAT' | 'repeat' ;
UNTIL   : 'UNTIL' | 'until' ;
STEP    : 'STEP' | 'step' ;
NEXT    : 'NEXT' | 'next' ;
TO      : 'TO' | 'to' ;
CONTINUE    : 'CONTINUE' | 'continue' ;
EXIT    : 'EXIT' | 'EXIT' ;

// comments
COMMENT : REM ~[\r\n]* ;

// literals
ID              : [a-zA-Z]+ ;  // match identifiers
NUMBER          : [0-9]+ ('.' [0-9]+)?;   // match integers
STRINGLITERAL   : '"' ~ ["\r\n]* '"' ;
DOLLAR          : '$' ;
NEWLINE         :'\r'? '\n' ;  // return newlines to parser (end-statement signal)
WS              : [ \t]+ -> skip ; // toss out whitespace

For the most part this is pretty self explanatory.

The last part, the comments and literals, uses some more interesting tricks.

In case of the comments we want to keep things simple and only allow one lined comments. So we match any line starting with REM and any character up to newline. ~[\r\n] means any character except \r or \n.

The numbers should be clear for those familiar with regular expressions. [0-9]+ ('.' [0-9]+)? means one ore more digits followed optionally by . and one or more decimals.

Since in our language the spaces and tabs are not significant we just want to ignore them, therefore [ \t]+ -> skip. The -> skip statement is a lexer command that instructs Antlr to, well, skip to the next token.

Now, with the lexer grammar in place we can proceed to writing the parser grammar. The parser grammar will take the stream of tokens produced by the lexer grammar and will try to match them using the following rules.

grammar LittleBasic;

// imports the other tokens and the grammar for expressions
import LBExpression, LBTokens;

// a programm is a block followed by an end-of-file
prog: block EOF; 

// the list of accepted statements
statement
    : letstmt
    | printstmt
    | inputstmt
    | ifstmt
    | forstmt
    | whilestmt
    | repeatstmt
    | continuestmt
    | exitstmt
    | COMMENT;

// a block is a list of statementes where each statement is followed by
// either a newline or the end of file
block
    : (statement (NEWLINE+ | EOF))*
    ;

// variable assignation. The LET keyword is optional here
letstmt
    : LET? vardecl EQ expression
    ;

// the name of the variable with an optional suffix
vardecl
    : varname varsuffix?
    ;

// the name of the variable must be a valid identifier
varname
    : ID
    ;

// the only acceptable suffix is the $ sign
varsuffix
    : DOLLAR
    ;

// PRINT expr
printstmt
    : PRINT expression;

// INPUT "something=" var
inputstmt
    : INPUT string vardecl
    ;

// IF expr
// THEN
// ... do smth
// ELSE
// ... something else
// END
ifstmt
    : IF expression NEWLINE* THEN NEWLINE+ block elifstmt* elsestmt? END
    ;

elifstmt
    : ELSE IF expression NEWLINE* THEN NEWLINE+ block
    ;

elsestmt
    : ELSE NEWLINE+ block
    ;

// FOR i = 1 TO 10 
//   .. do smth 
// NEXT
forstmt
    : FOR vardecl EQ expression TO expression (STEP expression)? NEWLINE+ block NEXT
    ;

// WHILE expr
//   .. do smth
// END
whilestmt
    : WHILE expression NEWLINE+ block END
    ;

// REPEAT
//  .. do smth
// UNTIL expr
repeatstmt
    : REPEAT NEWLINE+ block NEWLINE* UNTIL expression
    ;

// CONTINUE in FOR/WHILE
continuestmt
    : CONTINUE
    ;

// EXIT
exitstmt
    : EXIT
    ;

Again this should be self explanatory for the most part.

Generating the parser

Since we’re using Maven it’s just a matter of using the ANTLR plugin. By default the plugin binds to the generate-sources phase of Maven lifecycle. It reads the grammar files from the src/main/antlr4/basic directory and then produces .java and .tokens files in the output directory target/generated-sources/antlr4. The generated Java files then become available in the build path.

    <plugin>
        <groupId>org.antlr</groupId>
        <artifactId>antlr4-maven-plugin</artifactId>
        <version>4.6</version>
        <configuration>
            <libDirectory>${basedir}/src/main/antlr4/basic</libDirectory>
        </configuration>
        <executions>
            <execution>
                <id>antlr</id>
                <goals>
                    <goal>antlr4</goal>
                </goals>
            </execution>
        </executions>
    </plugin>

The interpreter

With the parsing part now done it’s time to execute our program. Generally there are two ways to approach the problem of executing a program. Either compile it into some executable form or use another program to intepret it.

For the sake of simplicity we’ll use interpreter. Fortunately ANTLR already provides an API to help us with this task.

After the parsing is done, ANTLR produces a parse tree. As the name suggests it’s a tree representation of the syntax of our program. In case of the example above it looks something like this:

Interpreting the code is a matter of walking the parse tree and executing some action for each statement that’s encountered.

We’re gonna use the LittleBasicBaseVisitor<Value> class generated by ANTLR. This class implements a visit(tree) method that pretty much does what the name suggests, namely visits each node of the given tree.

Additionally this class has callback methods that will be invoked for each node. By default these methods don’t do anything but we’re free to override them and plug in our own custom logic. To do this we’ll simply extends this class:

public class LittleBasicVisitor extends LittleBasicBaseVisitor<Value> {
    
    @Override
    public Value visitProg(LittleBasicParser.ProgContext ctx) {
        // ....
    }
        
    @Override
    public Value visitString(LittleBasicParser.StringContext ctx) {
        // ...
    }
    
    // ...
}

Now we can add here all our interpreter logic. Calling LittleBasicVisitor.visit(tree) will invoke our custom logic for each statement that is encountered by the visitor.

We won’t go into the details of how the interpreter implementation. Just take a look at the LittleBasicVisitor class. Since it’s just a toy interpreter the code should be quite self explanatory.

Putting it all together

Now that we have the grammars and the interpreter let’s see how to make the various parts work together.

In the Interpreter class you’ll find this code:

    public Value run(InputStream progrIn) throws IOException {
        // wrap the input stream
        ANTLRInputStream input = new ANTLRInputStream(progrIn);
        // initialize the lexer with the input stream
        LittleBasicLexer lexer = new LittleBasicLexer(input);
        // initialize the tokens stream with the lexer
        CommonTokenStream tokens = new CommonTokenStream(lexer);
        // initialize the parser with the tokens stream
        LittleBasicParser parser = new LittleBasicParser(tokens);
        // set the error handling strategy
        parser.setErrorHandler(new BailErrorStrategy());
        // remove default error listeners
        parser.removeErrorListeners();
        // set our custom error listener
        parser.addErrorListener(new ErrorListener(stderrPrint));
        try {
            // parse the input into a parse tree
            ParseTree tree = parser.prog();
            memory = new Memory();
            // init our custom visitor
            LittleBasicVisitor eval = new LittleBasicVisitor(memory, stdin, stdoutPrint, stderrPrint);
            // visit the parse tree and interpret the program
            eval.visit(tree);
        } catch (InterpreterException e) {
            // handle interpreter errors
            stderrPrint.println(e.getMessage());
        } catch (ParseCancellationException e) {
            // handle parser errors
            if (e.getCause() instanceof InputMismatchException) {
                InputMismatchException inputEx = (InputMismatchException)e.getCause();
                String msg = Utils.formatErrorMessage(
                        inputEx.getOffendingToken().getLine(),
                        inputEx.getOffendingToken().getCharPositionInLine(),
                        "Syntax error");
                stderrPrint.println(msg);
            }
        }
        return null;
    }

Now we have a simple but fully functional Basic interpreter. Go ahead and build the project:

mvn install

and run it:

java -jar LittleBasic.jar /path/to/MyAwesomeProgram.bas

Enjoy!

Suse Open Build Service tutorial

2016-06-30T00:00:00+00:00

Suse’s Open Build Service (OBS) is a tool for building and distributing software packages for various Linux distros. The main target is of course OpenSUSE and SLES but there’s also support for RedHat, Debian, Ubuntu and Arch.

It consists of a web service that takes care of building the packages and publish them to repositories and of a command line tool (osc).

OpenSuse uses RPM as its packaging format. Building RPMs is possible using the rpmbuild command but the osc command line tool makes it much more convenient to build the package locally or remotely in OBS.

If building locally osc can pull the build dependencies of the package and prepare an isolated environment where the build will take place. Under the hood it (usually) creates an chroot with all the needed dependencies installed and it executes the rpmbuild in that environment. So no need to fiddle with the packages installed on your machine to make the build work.

Now let’s see how it works in practice. Let’s build our first package with OBS. Altough some of the steps outlined below can be performed using the web GUI I’ll focus on the command line client utility osc.

The first obvious step is to get an account on Open Build Service. Let’s asume for the rest of the tutorial that your OBS username is obsuser.

You will also need the client tools installed. On OpenSuse this is only a matter of running:

myuser@linux:~> zypper in osc

If you use another distro you may also get osc from OpenSUSE Tools.

2. Checkout your home project

Once you log in you can go to your home project. This is created by default by OBS and it has the name home:obsuser.

To check it out run:

myuser@linux:~> osc checkout home:obsuser

3. Create the package in OBS

Next is of course the package. For the sake of simplicity let’s package jupp, a portable version of JOE’s Own Editor. To create the package execute the following in the directory where you’ve checked out your home project:

myuser@linux:~/home:obsuser> osc meta pkg -e home:obsuser jupp

This will open your text default editor with an empty xml template. Fill in the details of the package like this:

 
<package name="jupp">
  <title>A text editor</title> <!-- Title of package -->
  <description>
    This is the portable version of JOE’s Own Editor, which is currently developed at sourceforge
  </description> <!-- for long description --> 
. . .
  <url>https://www.mirbsd.org/jupp</url>
. . . 
</package>

After saving and exiting your editor osc will send the details to OBS.

Go to the directory that contains home:obsuser and update the project:

myuser@linux:~/> osc up

This will create an empty directory home:obsuser/jupp.

4. Define repositories

You need at least one repository for your project. Usually here you specify a distro version against which you want you package built. In my case I want to build packages for OpenSUSE Leap 42.1.

myuser@linux:~/home:obsuser/jupp> osc meta prj -e home:obsuser

Define the repository inside the XML. The element points to another OBS project, in this case the project for Leap 42.1.

 
<project name="home:mateialbu">
  <title/>
  <description/>
  <person userid="obsuser" role="maintainer"/>
  <repository name="openSUSE_Leap_42.1">
    <path project="openSUSE:Leap:42.1" repository="standard"/>
    <arch>x86_64</arch>
    <arch>i586</arch>
  </repository>
</project>

5. Create the spec file

Now it’s time for the real work. The .spec file contains both metadata and build instructions. This is the central piece of an RPM package.

So let’s create one for our package. Use you favourite editor to create a file named jupp.spec. At least on OpenSUSE you get a template .spec file where you can fill the details. Paste the following into your file:

#
# spec file for package jupp
#
# Copyright (c) 2016 SUSE LINUX Products GmbH, Nuernberg, Germany.
#
# All modifications and additions to the file contributed by third parties
# remain the property of their copyright owners, unless otherwise agreed
# upon. The license for this file, and modifications and additions to the
# file, is the same license as for the pristine package itself (unless the
# license for the pristine package is not an Open Source License, in which
# case the license is the MIT License). An "Open Source License" is a
# license that conforms to the Open Source Definition (Version 1.9)
# published by the Open Source Initiative.

# Please submit bugfixes or comments via http://bugs.opensuse.org/
#

Name:           jupp
Version:        3.1jupp28
Release:        0
License:        GPL-1.0
Summary:        A Text Editor
Url:            https://www.mirbsd.org/jupp
Group:          Productivity/Editors/Other
Source:         https://www.mirbsd.org/MirOS/dist/jupp/joe-3.1jupp28.tgz
Conflicts:      joe
BuildRequires:  automake
BuildRequires:  ncurses-devel
BuildRoot:      %{_tmppath}/%{name}-%{version}-build

%description
This is the portable version of JOE’s Own Editor, which is currently developed at sourceforge, licenced under the GNU General Public License, Version 1, using autoconf/automake. This version has been enhanced by several functions intended for programmers or other professional users, and has a lot of bugs fixed. It is based upon an older version of joe because these behave better overall.

%prep
%setup -q -n jupp
chmod +x configure

%build
%configure \
  --prefix=%{_prefix}

make %{?_smp_mflags}

%install
make install DESTDIR=%{buildroot} %{?_smp_mflags}
for i in jmacs jpico jstar rjoe; do
  ln -s joe.1.gz %{buildroot}%{_mandir}/man1/$i.1.gz
done
rm -rf %{buildroot}/%{_datadir}/%{name}/lang

%post

%postun

%files
%defattr(-,root,root)
%{_mandir}/man1/*
%config %{_sysconfdir}/joe/*
%dir %{_sysconfdir}/joe
%{_bindir}/*

%changelog

A detailed explanation of the .spec file is out of scope of this tutorial but the file is pretty self explanatory.

6. Test the build locally

To test the spec file, trigger the build locally. In the package directory run:

myuser@linux:~/home:obsuser/jupp> osc build

If everything goes well osc will inform you that the resulted rpm is in:

/var/tmp/build-root/openSUSE_Leap_42.1-x86_64/home/abuild/rpmbuild/RPMS/x86_64/jupp-3.1jupp28-0.x86_64.rpm

7. Add the files to the project

osc works similarly to a VCS like git. Firs you have to add the files:

myuser@linux:~/home:obsuser/jupp> osc add <file>

or simply

myuser@linux:~/home:obsuser/jupp> osc addremove

to add/remove all files in the package dir.

8. Add the changelog

myuser@linux:~/home:obsuser/jupp> osc vc

This will open your default editor with a template. Fill the initial message:

------------------------------------------------------------------- 
Wed Jun 29 13:40:29 UTC 2016 - <user.email>
- Initial version of the package

A new file jupp.changes will be written in the package directory.

To check the status before committing run:

osc status

To add jupp.changes do a:

myuser@linux:~/home:obsuser/jupp> osc add jupp.changes 
A    jupp.changes 

Now finally everything ready to commit:

myuser@linux:~/home:obsuser/jupp> osc commit 
Sending    joe-3.1jupp28.tgz 
Sending    jupp.spec 
Sending    jupp.changes 
Transmitting file data .. 
Committed revision 1.

9. Check the build status

After you commit OBS will schedule the build. To check the status you can either go to the web interface or use osc:

myuser@linux:~/home:obsuser/jupp> osc results
openSUSE_Leap_42.1   x86_64     finished*
openSUSE_Leap_42.1   i586       succeeded

10. Download and install the package

You can download and install the package from the web interface. Go to the package page and look for the aptly named “Download package” link. On OpenSuse you’ll get the option to use One Click Install.

If you prefer the command line another option is to use add the repository and install the package. First get the repo url:

myuser@linux:~/home:obsuser/jupp> osc repourls 
http://download.opensuse.org/repositories/home:/obsuser/openSUSE_Leap_42.1/home:obsuser.repo

Then simply add the repository using zypper and install the package:

myuser@linux:~/home:obsuser/jupp> sudo zypper ar http://download.opensuse.org/repositories/home:/obsuser/openSUSE_Leap_42.1/home:obsuser.repo
Repository 'home:obsuser (openSUSE_Leap_42.1)' successfully added 
Enabled     : Yes
Autorefresh : No
GPG Check   : Yes
Priority    : 99
URI         : http://download.opensuse.org/repositories/home:/obsuser/openSUSE_Leap_42.1/ 

myuser@linux:~/home:obsuser/jupp> sudo zypper ref
. . . 
myuser@linux:~/home:obsuser/jupp> sudo zypper in jupp

And voila! You have jupp installed on your machine.