He rb Sc h i ldt s Java P rog ra m m i n g Cookbook in Java

Paint Data Matrix 2d barcode in Java He rb Sc h i ldt s Java P rog ra m m i n g Cookbook

Instead of tokenizing based on delimiters, the second approach extracts each token based on the pattern that it matches For example, when tokenizing a program, a typical identifier will match a pattern that starts with a letter or underscore and is followed by other letters, digits, or underscores A comment will match a pattern that starts with a // and terminates with the end of the line, or that starts with a /* and ends with a */ An operator will match an operator pattern, which can be defined to include single-character operators (such as +) and multicharacter operators (such as +=) The advantage to this technique is that the tokens do not need to occur in any predefined order or be separated by a fixed set of delimiters Instead, the tokens are identified by the pattern that they match This is the approach used by this recipe As you will see, it is both flexible and readily adaptable Tokenizing is such an important task that Java provides extensive built-in support for it For example, it supplies three classes specifically designed for this purpose: StreamTokenizer, Scanner, and the obsolete StringTokenizer Furthermore (as mentioned), the String class contains the split( ) method, which can also be used to tokenize in certain simple situations As helpful as these classes are, they are most useful for tokenizing a string that is defined in terms of delimiters To tokenize based on patterns, you will usually find Java s regular expression API a better choice This is the approach used by the recipe in this section By using the regular expression API, you gain direct and detailed control over the tokenization processes Also, implementing a tokenizer by using the Pattern and Matcher classes provides an elegant solution that is clear and easy to understand

To tokenize a string based on patterns by using the regular expression API requires the following steps: 1 Create the set of regular expressions that define the patterns that you will be matching Each type of token that you want to obtain must be represented by a pattern To enable the regular expression engine to move progressively through the string, begin each pattern with the \G boundary specifier This requires the next match to begin at the point at which the previous match ended 2 Compile the patterns into Pattern objects by using Patterncompile( ) 3 Create a Matcher that contains the string to be tokenized 4 Obtain the next token by adapting the following algorithm: while(still patterns to try) { Specify pattern to match by calling usePattern( ) on the Matcher Search for the pattern by calling nd( ) on the Matcher If a match is found, obtain token Otherwise try next pattern } Often, the patterns must be tried in a speci c order Therefore, when you implement this algorithm, you must try each pattern in the correct order 5 Once a token has been found, obtain the token by calling group( ) on the Matcher 6 Repeat Steps 4 and 5 until the end of the string is reached

2:

The basic procedure used to create a Pattern and a Matcher and to use find( ) and group( ) to search a string is described in the preceding recipe (Match and Extract Substrings Using the Regular Expression API), and those discussions are not repeated here Instead, we will focus on the two key elements that enable a string to be tokenized The first is the definition of the set of patterns that describe the tokens The second is usePattern( ), which changes the pattern being matched The key to using the regular expression API to tokenize a string is the \G boundary matcher By starting each pattern with \G, each match must begin precisely where the previous match left off (In the first match, the \G will match the start of the string) Using this mechanism, the find( ) method can be called repeatedly on a Matcher Each subsequent match will begin where the previous one ended This enables the regular expression engine to move through the string, but not skip any tokens in the process Here are some examples of regular expressions that match words, punctuation, whitespace, and numbers:

Matches Words Punctuation Whitespace Numbers Pattern \G\p{Alpha}+ \G\p{Punct} \G\s+ \G\d+\ \d*

In each case, the token being matched must begin immediately after the previous match For example, given this string: Jump one, not 2, times first, Jump is matched by the word pattern The next pattern that will match is whitespace because it is the only pattern that can follow the previous match Next, one is matched by the word pattern, again because it is the only pattern that can follow the previous match The punctuation pattern will then match the comma, the word pattern will match not , the number pattern will match 2 , and so on A key point is that because each pattern must start at the end of the previous match, no tokens are skipped when the regular expression engine attempts to find a match For example, after Jump is found, an attempt to find a number will fail because 2 does not immediately follow Jump To enable one Matcher to use different patterns, you will use the usePattern( ) method This method changes the pattern without resetting the entire Matcher It is shown here: Matcher usePattern(Pattern regExpr) The pattern to use is specified by regExpr When attempting to obtain the next token, the order in which the patterns are tried is important For example, consider these two patterns: \G[<>=!] \G((<=)|(>=)|(==)|(!=))