Class STLexer

  • All Implemented Interfaces:
    org.antlr.runtime.TokenSource

    public class STLexer
    extends java.lang.Object
    implements org.antlr.runtime.TokenSource
    This class represents the tokenizer for templates. It operates in two modes: inside and outside of expressions. It implements the TokenSource interface so it can be used with ANTLR parsers. Outside of expressions, we can return these token types: TEXT, INDENT, LDELIM (start of expression), RCURLY (end of subtemplate), and NEWLINE. Inside of an expression, this lexer returns all of the tokens needed by STParser. From the parser's point of view, it can treat a template as a simple stream of elements.

    This class defines the token types and communicates these values to STParser.g via STLexer.tokens file (which must remain consistent).

    • Nested Class Summary

      Nested Classes 
      Modifier and Type Class Description
      static class  STLexer.STToken
      We build STToken tokens instead of relying on CommonToken so we can override STLexer.STToken.toString().
    • Field Summary

      Fields 
      Modifier and Type Field Description
      static int AND  
      static int AT  
      static int BANG  
      (package private) char c
      current character
      static int COLON  
      static int COMMA  
      static int COMMENT  
      (package private) char delimiterStartChar
      The char which delimits the start of an expression.
      (package private) char delimiterStopChar
      The char which delimits the end of an expression.
      static int DOT  
      static int ELLIPSIS  
      static int ELSE  
      static int ELSEIF  
      static int ENDIF  
      static char EOF  
      static int EOF_TYPE  
      static int EQUALS  
      (package private) ErrorManager errMgr  
      static int FALSE  
      static int ID  
      static int IF  
      static int INDENT  
      (package private) org.antlr.runtime.CharStream input  
      static int LBRACK  
      static int LCURLY  
      static int LDELIM  
      static int LPAREN  
      static int NEWLINE  
      static int OR  
      static int PIPE  
      static int RBRACK  
      static int RCURLY  
      static int RDELIM  
      static int REGION_END  
      static int RPAREN  
      (package private) boolean scanningInsideExpr
      This keeps track of the current mode of the lexer.
      static int SEMI  
      static org.antlr.runtime.Token SKIP  
      static int SLASH  
      (package private) int startCharIndex
      When we started token, track initial coordinates so we can properly build token objects.
      (package private) int startCharPositionInLine  
      (package private) int startLine  
      static int STRING  
      int subtemplateDepth
      To be able to properly track the inside/outside mode, we need to track how deeply nested we are in some templates.
      static int SUPER  
      (package private) org.antlr.runtime.Token templateToken
      template embedded in a group file? this is the template
      static int TEXT  
      (package private) java.util.List<org.antlr.runtime.Token> tokens
      Our lexer routines might have to emit more than a single token.
      static int TRUE  
    • Constructor Summary

      Constructors 
      Constructor Description
      STLexer​(org.antlr.runtime.CharStream input)  
      STLexer​(ErrorManager errMgr, org.antlr.runtime.CharStream input, org.antlr.runtime.Token templateToken)  
      STLexer​(ErrorManager errMgr, org.antlr.runtime.CharStream input, org.antlr.runtime.Token templateToken, char delimiterStartChar, char delimiterStopChar)  
    • Method Summary

      All Methods Static Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      org.antlr.runtime.Token _nextToken()  
      (package private) org.antlr.runtime.Token COMMENT()  
      protected void consume()  
      void emit​(org.antlr.runtime.Token token)  
      (package private) org.antlr.runtime.Token ESCAPE()  
      java.lang.String getSourceName()  
      protected org.antlr.runtime.Token inside()  
      static boolean isIDLetter​(char c)  
      static boolean isIDStartLetter​(char c)  
      static boolean isUnicodeLetter​(char c)  
      static boolean isWS​(char c)  
      (package private) void LINEBREAK()  
      void match​(char x)
      Consume if x is next character on the input stream.
      (package private) org.antlr.runtime.Token mID()
      ID : ('a'..'z'|'A'..'Z'|'_'|'/') ('a'..'z'|'A'..'Z'|'0'..'9'|'_'|'/')* ;
      (package private) org.antlr.runtime.Token mSTRING()
      STRING : '"' ( '\\' '"' | '\\' ~'"' | ~('\\'|'"') )* '"' ;
      (package private) org.antlr.runtime.Token mTEXT()  
      org.antlr.runtime.Token newToken​(int ttype)  
      org.antlr.runtime.Token newToken​(int ttype, java.lang.String text)  
      org.antlr.runtime.Token newToken​(int ttype, java.lang.String text, int pos)  
      org.antlr.runtime.Token newTokenFromPreviousChar​(int ttype)  
      org.antlr.runtime.Token nextToken()  
      protected org.antlr.runtime.Token outside()  
      static java.lang.String str​(int c)  
      (package private) org.antlr.runtime.Token subTemplate()  
      (package private) org.antlr.runtime.Token UNICODE()  
      (package private) void WS()  
      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
    • Constructor Detail

      • STLexer

        public STLexer​(org.antlr.runtime.CharStream input)
      • STLexer

        public STLexer​(ErrorManager errMgr,
                       org.antlr.runtime.CharStream input,
                       org.antlr.runtime.Token templateToken)
      • STLexer

        public STLexer​(ErrorManager errMgr,
                       org.antlr.runtime.CharStream input,
                       org.antlr.runtime.Token templateToken,
                       char delimiterStartChar,
                       char delimiterStopChar)
    • Method Detail

      • nextToken

        public org.antlr.runtime.Token nextToken()
        Specified by:
        nextToken in interface org.antlr.runtime.TokenSource
      • match

        public void match​(char x)
        Consume if x is next character on the input stream.
      • consume

        protected void consume()
      • emit

        public void emit​(org.antlr.runtime.Token token)
      • _nextToken

        public org.antlr.runtime.Token _nextToken()
      • outside

        protected org.antlr.runtime.Token outside()
      • inside

        protected org.antlr.runtime.Token inside()
      • subTemplate

        org.antlr.runtime.Token subTemplate()
      • ESCAPE

        org.antlr.runtime.Token ESCAPE()
      • UNICODE

        org.antlr.runtime.Token UNICODE()
      • mTEXT

        org.antlr.runtime.Token mTEXT()
      • mID

        org.antlr.runtime.Token mID()
          ID  : ('a'..'z'|'A'..'Z'|'_'|'/')
                ('a'..'z'|'A'..'Z'|'0'..'9'|'_'|'/')*
              ;
          
      • mSTRING

        org.antlr.runtime.Token mSTRING()
          STRING : '"'
                   (   '\\' '"'
                   |   '\\' ~'"'
                   |   ~('\\'|'"')
                   )*
                   '"'
                 ;
         
      • WS

        void WS()
      • COMMENT

        org.antlr.runtime.Token COMMENT()
      • LINEBREAK

        void LINEBREAK()
      • isIDStartLetter

        public static boolean isIDStartLetter​(char c)
      • isIDLetter

        public static boolean isIDLetter​(char c)
      • isWS

        public static boolean isWS​(char c)
      • isUnicodeLetter

        public static boolean isUnicodeLetter​(char c)
      • newToken

        public org.antlr.runtime.Token newToken​(int ttype)
      • newTokenFromPreviousChar

        public org.antlr.runtime.Token newTokenFromPreviousChar​(int ttype)
      • newToken

        public org.antlr.runtime.Token newToken​(int ttype,
                                                java.lang.String text,
                                                int pos)
      • newToken

        public org.antlr.runtime.Token newToken​(int ttype,
                                                java.lang.String text)
      • getSourceName

        public java.lang.String getSourceName()
        Specified by:
        getSourceName in interface org.antlr.runtime.TokenSource
      • str

        public static java.lang.String str​(int c)