In-Memory Queries and LINQ in Visual Basic .NET

Make Data Matrix 2d barcode in Visual Basic .NET In-Memory Queries and LINQ

In this section, you ve seen how you can use aggregate sequence operators and sequence expressions to query and manipulate in-memory data structures using a SQL-like syntax. This is essentially the idea behind Microsoft s LINQ technology. You apply the same techniques to queries over relational data (LINQ to SQL) later in this chapter. In 3, you saw that sequences can be used to wrap collections and that a number of collection data types implement the sequence interface. It s possible to provide parallel implementations for the iteration, aggregation, and transformation operations of these data types for instance, a parallel list implementation that uses worker threads and assume independence between the transformation steps. This idea forms the basis of Parallel LINQ, a parallel version of LINQ. This library can be accessed using the F# Power Pack library FSharp.PowerPack.Parallel.Seq.dll, for parallel sequence operations.

Storing data in various files, reading them into memory, and querying and manipulating the resulting in-memory data collection is a sufficient approach for many applications working with data, but there comes a point where more persistent data handling is required. For example, consider a business that has two applications that need to access the same set of data about employees. One way to do this is to store data in a text file and work with this data file from both applications. But this approach is vulnerable; you d quickly realize how redundant text files can get when adding new pieces of data (say you wanted to add an address next to the name, department, and age of each employee) or if you wanted to group your employee data by introducing various relationships (say, multiple addresses per employee). Even if you managed to find a good storage alternative, you d still have to modify both applications to read and write this new data format. Databases make scenarios like these much easier to cope with by enforcing centralized control of persistent data (data that is more persistent than what appears in a single application) and by giving you the freedom to define, manipulate, and query this data in an application-neutral way.

Databases provide many benefits. Some of the more important ones are listed here: Data security: When you have centralized control of your data, you can erect a full security system around the data, implementing specific access rules for each type of access or parts of the database. Sharing data: Any number of applications with the appropriate access rights can connect to your database and read the data stored within without needing to worry about containing the logic to extract this data. As you see shortly, applications use various query languages (most notably SQL) to communicate with databases. A logical organization of data: You can write new applications that work with the same data without having to worry about how the data is physically represented and stored. On the basic level, this logical structure is provided by a set of entities (data tables) and their relationships. Avoiding data redundancy: Having all requirements from each consuming application up front helps to identify a logical organization for your data that minimizes possible redundancy. For instance, you can use foreign keys instead of duplicating pieces of data. Data normalization is the process of systematically eliminating data redundancy, a large but essential topic that we don t consider in this book. Transactions: Reading from and writing to databases occurs atomically, and as a result, two concurrent transactions can never leave data in an inconsistent, inaccurate state. Isolation levels refer to specific measures taken to ensure transaction isolation by locking various parts of the database (fields, records, tables). Higher isolation levels increase locking overhead and can lead to a loss of parallelism by rendering concurrent transactions sequential; on the other hand, no isolation can lead to inconsistent data. Maintaining data integrity: Databases make sure the data stored within is accurate. Having no redundancy is one way to maintain data integrity (if a piece of data is changed, it s changed in the only place it occurs; thus, it remains accurate); on the other hand, data security and transaction isolation are needed to ensure that the data stored is modified in a controlled manner.