The Token class in Java

Maker ECC200 in Java The Token class

Let s look more closely at each part First, each instance of Tokenizer has its own Matcher, which is referred to via mat This means that two or more Tokenizers can be used within the same program, each operating independently Next, notice the various patterns The regular expressions all begin with \G, which means that a matching sequence must begin at the end of the previous match As explained, this enables the regular expression engine to advance through the string (If you don t include the \G boundary matcher, then the regular expression engine will find a matching sequence anywhere in the string, possibly skipping several tokens in the process) If you add additional patterns, then you must be sure to begin each with \G Notice the patterns that tokenize a subset of Java The kwOrIdent pattern matches either keywords or identifiers (It s not often practical to distinguish between the two during tokenization Other parts of a compiler or interpreter usually handle that chore) Other patterns match the various separators and operators Notice that four different patterns are required to handle the operators, each matching a different type of operator, including those composed of single characters, double characters, or triple characters The assignment operators could have been handled as double-character operators, but for clarity, they are given a pattern of their own Patterns that match comments and string literals are also provided

2:

After the patterns have been compiled, the TokType enumeration is created It defines the token types that correspond to the patterns and links each pattern to its type This linkage of pattern with type shows the power of Java enumerations, which are class types rather than simple lists of named integers (as they are in several other computer languages) The patterns array holds a list of TokType objects that specify what types of tokens will be obtained by the tokenizer The order of the types in the array specifies the order in which Tokenizer searches for a token Because the order in which the search is conducted can matter (for example, you must look for <= before you look for <), you must take care to order this array properly In all cases, the last two entries in the array must be END and UNKNOWN Next, the Token class is defined This nested class is not technically necessary, but it allows a token and its type to be encapsulated within a single object The curTok variable, which is an instance of Token, holds the most recently obtained token A reference to it is returned by getToken( ) Technically, curTok is not necessary because it would be possible to simply construct a new Token object for each token obtained, and have getToken( ) return the new object However, when tokenizing a very large string, a large number of objects would be created and then discarded This would result in additional garbage collection cycles By using only one Token object, this potential inefficiency is eliminated The one-parameter Tokenizer constructor creates a tokenizer that handles regular English text by reducing it to words, punctuation, and numbers The string to be tokenized is passed to the constructor It assigns to patterns an array that handles these simple text elements The two-parameter constructor allows you to specify an array of TokType that will be assigned to patterns This enables you to configure a tokenizer that will handle different types of input In TokenizerDemo, the progToks array is constructed in such a way that a subset of the Java language can be tokenized Other types of tokenizers can be created by specifying a different TokType array One other point: Notice that this constructor always adds END and UNKNOWN on to the list of token types This is necessary to ensure that getToken( ) finds the end of the string and that it returns UNKNOWN when it can t find any of the patterns The getToken( ) method obtains the next token in the string It will always succeed because the token will be END when the end of the string has been reached and UNKNOWN if no matching token is found Notice that getToken( ) skips any leading whitespace Normally, tokens don t include whitespace, so getToken( ) simply discards it To use the tokenizer, first create a Tokenizer that searches for the tokens that you desire Next, set up a loop that calls getToken( ) on the tokenizer until the end of the string is reached When this occurs, the type of the token returned will be EOS In the program, this procedure is demonstrated by the TokenizerDemo class An unknown token can either be ignored (as it is in the example) or treated as an error condition