Antlr

ANTLR is a parser generator tool.

BITPlan has been using ANTLR in projects for a few years now and Wolfgang Fahl has been active in improving ANTLR see e.g. Antlr#Motivation

Library with helpers for ANTLR Language development com.bitplan.antlr

To simplify Parser development with ANTLR BITPlan has created a library with some helper code for ANTLR Language Development and published it as Open Source at:

• https://github.com/BITPlan/com.bitplan.antlr

Base Class Language Parser

The abstract base class LanguageParser has some helper code that makes language development and debugging easier.

Example

Exp grammar

/**
 * Copyright 2016-2017 BITPlan GmbH
 * Author: Wolfgang Fahl
 *
 * this is an Example Antlr Grammar
 *
 *
 * it is specified using antlr syntax and uses the ANTLR V4 parser generator
 * see http://www.antlr.org
 * 
 * for Eclipse you might want to install the IDE support:
 * https://github.com/jknack/antlr4ide
 * 
 */
grammar Exp;
 /* This will be the entry point of our parser. */
eval returns [double value]
    :    exp=additionExp {$value = $exp.value;}
    ;

/* Addition and subtraction have the lowest precedence. */
additionExp returns [double value]
    :    m1=multiplyExp       {$value =  $m1.value;} 
         ( '+' m2=multiplyExp {$value += $m2.value;} 
         | '-' m2=multiplyExp {$value -= $m2.value;}
         )* 
    ;

/* Multiplication and division have a higher precedence. */
multiplyExp returns [double value]
    :    a1=atomExp       {$value =  $a1.value;}
         ( '*' a2=atomExp {$value *= $a2.value;} 
// ...
         )* 
    ;
    
/* An expression atom is the smallest part of an expression: a number. Or 
   when we encounter parenthesis, we are making a recursive call back to the
   rule 'additionExp'. As you can see, an 'atomExp' has the highest precedence. */

atomExp returns [double value]
    :    n=Number                {$value = Double.parseDouble($n.text);}
    |    '(' exp=additionExp ')' {$value = $exp.value;}
    ;

/* A number: can be an integer value, or a decimal value */
Number
    :    ('0'..'9')+ ('.' ('0'..'9')+)?
    ;

/* We're going to ignore all white space characters */
WS  
    :   (' ' | '\t' | '\r'| '\n') {
    	
    }
    ;

Java Source for ExpLanguage Parser

This code wraps the generated ExpParser into a class derived from LanguageParser

/**
 * example parser
 * @author wf
 *
 */
public class ExpLanguageParser extends LanguageParser {
  private ExpParser parser;
  ExpLexer lexer;
  
  public ExpParser getParser() {
    return parser;
  }
  
  @Override
  protected ParseTree getRootContext(Parser parser) {
    if (!(parser instanceof ExpParser)) {
      throw new RuntimeException("wrong parser type for getRootContext, expected Rule but got "+parser.getClass().getName());
    } else {
      ExpParser expParser=(ExpParser) parser;
      return expParser.eval();
    }
  }

  @Override
  protected ParseTree parse(ANTLRInputStream in, String inputText)
      throws Exception {
    lexer = new ExpLexer(in);
    parser=new ExpParser(getTokens(lexer));
    ParseTree result=super.parse(lexer,getParser());
    return result;
  }

  @Override
  public void showParseTree() {
    super.showParseTree(getParser());
  }

}

JUnit Test

With the BaseTest abstract base Junit Test class testing and debugging gets easier. Take the following JUnit Test:

JUnit Test Source Code

  @Test
  public void testExpressionParser() throws Exception {
    String expressions[] = { "2*3", "4+5", "(2+3)*(4+5)" };
    ExpLanguageParser exprParser = new ExpLanguageParser();
    for (String expression : expressions) {
      super.runParser(exprParser, expression, 0);
    }
  }

if you add some invalid Expression to the expressions:

JUnit Test Source Code

 
   String expressions[] = { "2*3", "4+5", "(2+3)*(4+5)","(4+5)--(6-7)" };

a graphical parse Tree will show showing you what's wrong:

Motivation

While struggling with an Issue in ANTLR 4.4 see

• https://github.com/antlr/antlr4/issues/994

there was a need to be able whether a parser would "timeout". The support was this was implemented by adding a timed call

/**
   * test the given rule with the given timeout
   * 
   * @param inputText
   * @param expectedErrors
   * @param timeOutMSecs
   * @throws Exception
   * @return the parser
   */
  public LanguageParser doTestParser(final String inputText, final int expectedErrors, int timeOutMSecs)
      throws Exception {
        if (debug) {
          System.out.println(inputText);
        }
        LanguageParser result = timedCall(new Callable<LanguageParser>() {
          public LanguageParser call() throws Exception {
            return runParser(inputText, expectedErrors);
          }
        }, timeOutMSecs, TimeUnit.MILLISECONDS);
        return result;
      }

Another need was to be able to test hundreds of files in a list of directories. For this the SourceDirectory class was added.

This base class can then be used with the testParseFilesInDirectories() function

public List<LanguageParser> testParseFilesInDirectories(File rootDir, List<SourceDirectory> sourceDirectories,
      String[] extensions, String[] ignorePrefixes, int limit, int progressStep) throws Exception {

Prerequisites

• https://github.com/bkiers/rrd-antlr4

This is another useful function that is linked with this library.

Installation

# get the Railroad Diagrams for ANTLR 4 grammar rules. 
git clone https://github.com/bkiers/rrd-antlr4
# make it locally available
cd rrd-antlr4
mvn clean install 
# then install the local com.bitplan.antlr 
cd ..
git clone https://github.com/BITPlan/com.bitplan.antlr
cd com.bitplan.antlr
mvn install