LEXING AND PARSING in VB.NET

Encoder Data Matrix 2d barcode in VB.NET LEXING AND PARSING

Here is an example use: > parseEmployee line;; val it : string * string * System.DateTime * string = ("Smith", "John", 20/01/1986 00:00:00 { ... }, "Software Developer")

You can turn a file into an on-demand sequence of results using a sequence builder: open System.IO let readEmployees (fileName : string) = seq { use reader = File.OpenText fileName while not reader.EndOfStream do yield reader.ReadLine() |> parseEmployee } The following example takes the first three entries from an artificially generated file containing 10,000 copies of the same employee: > File.WriteAllLines("employees.txt", Array.create 10000 line);; val it : unit > let firstThree = readEmployees("employees.txt") |> Seq.take 3;; val firstThree : (string * string * System.DateTime * string) list > for (last,first,startDate,title) in firstThree do printfn "%s %s started on %A" first last startDate;; John Smith started on 20/01/1986 00:00:00 John Smith started on 20/01/1986 00:00:00 John Smith started on 20/01/1986 00:00:00 This technique is often used to do exploratory analysis of large data files. After the algorithm is refined using a prefix of the data, the analysis can then easily be run directly over the full data file.

Another technique that s frequently used to extract information from strings is to use regular expressions. The System.Text.RegularExpressions namespace provides convenient string-matching and -replacement functions. For example, let s say you have a log file containing a record of HTML GET requests. Here is a sample request: GET /favicon.ico HTTP/1.1

The following code captures the name of the requested resource (favicon.ico) and the lower version number of the HTML protocol (1) used: open System.Text.RegularExpressions let parseHttpRequest line = let result = Regex.Match(line, @"GET (.* ) HTTP/1\.([01])$") let file = result.Groups.[1].Value let version = result.Groups.[2].Value file, version The relevant fields are extracted by using the Groups attribute of the regular expression match to access the matched strings for each parenthesized group in the regular expression.

Although it s possible to hand-code lexers by using a range of ad hoc techniques such as those discussed in the previous section or by writing functions that explicitly manipulate lists of characters, doing so can be boring and time consuming. Instead, it s often easier to rely on a lexer generator to do this job for you. This section looks at how to use the fslex tool that comes with the F# Power Pack to perform lexical analysis. The F# Power Pack also includes fsyacc, which is also available as a separate download from MSDN and CodePlex. When you use fslex or fsyacc generated code in your projects, you must include a library reference to FSharp.PowerPack.dll from the F# Power Pack. Let s start with a simple example. Listing 16-1 shows a lexer that replaces all < and > characters in an input stream with their HTML equivalents, < and >. Listing 16-2 shows a small program that uses this generated lexer. Listing 16-1. Replacing Characters with Their HTML Equivalents: text2htmllex.fsl { module Text2HtmlLex let lexeme = Lexing.LexBuffer<_>.LexemeString (* You can add your helper functions here *) } rule convertHtml chan = parse | '<' { fprintf chan "<" convertHtml chan lexbuf } | '>' { fprintf chan ">" convertHtml chan lexbuf } | eof { () } | _ { fprintf chan "%s" (lexeme lexbuf) convertHtml chan lexbuf } Listing 16-2. Replacing Characters with Their HTML Equivalents: text2html.fs open System.IO open System.Text let main() =

let args = System.Environment.GetCommandLineArgs() if args.Length <= 2 then let exe = Path.GetFileName(args.[0]) eprintfn "Usage: %s dir pattern" exe exit 1 let directory = args.[1] let pattern = args.[2] for fileName in Directory.GetFiles(directory, pattern) do // Open a file stream for the file name use inputReader = File.OpenText fileName // Create a lex buffer for use with the generated lexer. The lex buffer // reads the inputReader stream. let lexBuffer = Lexing.LexBuffer<_>.FromTextReader inputReader // Open an output channel let outputFile = Path.ChangeExtension(fileName,"html") use outputWriter = (new StreamWriter(outputFile) :> TextWriter) // Write the header fprintfn outputWriter "<html>\n<head></head>\n<pre>" // Run the generated lexer Text2htmllex.convertHtml outputWriter lexBuffer // Write the footer fprintfn outputWriter "</pre>\n</html>\n" main() You can produce an F# source file from the previous lexer definition by running the following command at the Windows command prompt: fslex text2htmllex.fsl --unicode This produces text2htmllex.fs, which contains the implementation of the lexer convertHtml. This lexer is imperative, in that it prints to an output stream instead of returning tokens. The signature of the entry point to the generated lexer is as follows: val Text2HtmlLex.convertHtml: System.IO.TextWriter -> Lexing.Lexbuffer<char> -> unit You can now compile the driver and the lexer together: fsc text2htmllex.fs text2html.fs r FSharp.PowerPack.dll You can run the resulting program as follows, giving a source directory and a file pattern and producing an .html version of each file that matches by applying the HTML conversion: text2html . *.txt