# Thursday, March 10, 2011

A Little History – Getting to a decent data pattern

Years ago, I learned how to write N-Tier applications. In the apps I wrote in those days, a data tier would retrieve data from a database - probably by calling a stored procedure. ADO (and later ADO.Net) packaged the retrieved data into a DataSet, which contained one or more DataTables. A DataTable looks a lot like a table in a database. It presents the data as rows and columns (i.e., in rectangular format). So if a DataTable contains 5 rows, each row will have the same columns.

In my early N-Tier applications, I would pass the DataSet to other application tiers - such as a business tier to apply business rules - until it ultimately arrived at the user interface tier, where the user could consume this data. Microsoft encouraged the practice of sharing a dataset among all application tiers by supplying controls that bound easily to a DataSet.

These DataSets and DataTables are still widely used today. But the practice of passing them from the Data Access layer up to the User Interface Layer presents problems. DataSets tend to look very much like the databases from which they were retrieved. And databases are typically not built with the user interface or with business rules in mind. A database has different goals, such as to optimize storage or reporting. By contrast, a user interface should present data in way that it is easily understood, consumed and manipulated by the end user.

So, a few years ago, I changed my strategy. In my next architecture pattern, the Data Layer of my application continued to retrieve data from a database in the same way and to produce DataSets and DataTables as before. But I also created entity classes that more closely resembled objects as my business saw them. Rather than holding data in rows and columns, I stored it in objects and properties. Some of these properties pointed to another object or to a collection of objects, allowing my data to to take on a hierarchical format, rather than the rectangular shape encouraged by database tables. For example, a customer object contained properties for FirstName and LastName; but it also contained a property of a set of orders.  I could drill into that set and examine an individual order and I could drill into that order and examine individual lines of that order. I found these business entities easier to understand, easier to work with, and easier to explain to non-technical people. The hierarchical nature that exists between objects is more obvious in the entity representation. Entity representations of data are also more easily consumable in an object-oriented language, such as C#, because entity properties are strongly-typed.

I bought into the pattern described above for all enterprise software that I write. The only problem with this pattern is that it required me to write a lot of code. If I add a new table to my database, I need to write the following code

  • Stored procedures to add, update, read and delete data in the table.
  • Data Access methods to add, update, read and delete the data in the table.
  • An entity to represent the table
  • Code to map a set of rows in a DataTable into a set of entities.

I needed to write the code described above over and over – for each of my entities. The time spent writing this “plumbing” code could be spent addressing the customer’s business problems. Worse, any time I added a column to a table, I needed to modify code in each of the modules listed above to take this change into account.

Object Relational Mappers

An Object Relational Mapping (ORM) tool can alleviate this problem by writing part of this code for you and eliminating the need for other parts.

An ORM takes care of mapping between Entity objects and database objects. By taking much of this work away from the developer, he can focus on writing business logic.

An ORM typically consists of three parts: a data layer, a mapping layer, and an entity layer. The data layer retrieves data from the database, storing it in traditional data objects, such as DataSets and DataTables. The entity layer defines the objects that are exposed to the object-oriented system. The mapping layer (as you might guess) maps between these systems. The figure below shows these layers.


Depending on the particular ORM framework, these layers might be implemented in code, in configuration files, or using a defined naming convention. In any event, there is typically much less code involved than in

Advantages of an ORM

As noted above, an ORM makes it easier transform data into entities that more naturally fit an object-oriented system.

An ORM can reduce the amount of code you write by providing conventions and configurations (and sometimes, a user interface).

The abstraction layer between the database and the application makes it easier to switch databases. The developer needs to worry only about the data layer and most ORM frameworks help with this. It’s not common to replace one database with another, but it is common to produce a commercial application that supports more than one data platform.

Many ORM frameworks include lazy loading (the ability to defer query execution until an entity is used) and LINQ support, which makes them more flexible.

ORMs make it simple to update data. Calling a save method on an entity object takes care of all the database commands required by ADO.Net.

Transactions are typically handled automatically by an ORM framework. If an entity is saved, all associated entities are saved at the same time and the database update occurs within an atomic transaction.

ORM Frameworks

Hibernate is an ORM framework created ten years ago for Java. A few years later, Hibernate was ported to .Net as NHibernate. This remains the most mature ORM on the .Net platform and it continues to be enhanced. Many more .Net ORMs have appeared over the past few years, such as Subsonic, LLBLGen, and LINQ to SQL. Microsoft is currently investing heavily in the Entity Framework which still lags NHibernate in features but shows a lot of promise.


These days, it’s rare for me to write any ADO.Net code. My last three projects involving data access have used NHibernate or Entity Framework. As a result, I spend less time writing code and more time writing business logic.

Thursday, March 10, 2011 4:07:00 PM (GMT Standard Time, UTC+00:00)
Thursday, March 10, 2011 2:03:00 AM (GMT Standard Time, UTC+00:00)
# Monday, March 7, 2011
Monday, March 7, 2011 8:58:00 PM (GMT Standard Time, UTC+00:00)
# Saturday, March 5, 2011

Last year, I was awarded a Microsoft C# MVP Award and this week, I attended the Microsoft MVP Summit for the first time.

Before attending, I signed a non-disclosure agreement, so I am restricted from talking about much of the content of the conference. However, I can talk about my experiences and impressions.

Geek Give

I arrived a couple days before the conference in order to attend GeekGive – a charity event organized by Steve Andrews and others. Dozens of MVPs and their friends worked at the Northwest Harvest food bank, packing food into boxes for distribution to the needy in the Seattle area. It was a great way for people to give back to the community and to connect with one another.

Sunday was the Summit registration and several groups organized mixers in the evening. I attended a gathering put on by the Microsoft Developer Platform Evangelism team, at which MVPs had the opportunity to meet with evangelists across the region. I know the evangelists in the Heartland very well because they do a great job connecting with the local community. But I also got to know some of their counterparts in other parts of the Central Region. Afterward, the summit held a social gathering for new MVPs, welcoming us to the program. Finally, the good folks at Telerik invited me to an excellent dinner, where I was able to talk with many people smarter than me. I didn’t know in advance that Bill Evjen would be there. Bill is editing a .Net 4 book to which I am contributing and we were able to talk about that project. It was the first time we had met in person. Although we had exchanged e-mails, it was the first time we had met in person (a theme that would repeat itself with many other people throughout the week). The evening concluded with Party With Palermo, which was hosted at a local Red Robin restaurant by Jeff Palermo and is apparently an MVP Summit tradition.

Me and the Gu

Meeting people smarter than me was the theme of this summit. On Monday morning, I tweeted “Day 1 of #MVP11. Looking forward to being the dumbest guy in every room.”. This was quoted by Mary Foley of ZDNet. Microsoft speakers delivered content in several rooms, but I stayed in the largest room that day and watched some very good presentations, including an interesting talk by Scott Guthrie on some things coming up from Microsoft.

Monday evening, I gave a presentation on Visual Studio Database Tools at the Redmond .Net User Group on the Microsoft Campus. It was a great audience and I received a lot of positive feedback. There was an attendee party Monday night, but I arrived at the very end. I did enjoy two nearby parties hosted by Pluralsight and by the DeepFriedBytes podcast. At the Pluralsight event, I had a chance to meet K Scott Allen, who is scheduled to speak at the Great Lakes Area .Net User Group in August. The DeepFriedBytes party was still going strong when I left to get some sleep. I stuck around long enough to hear Keith Elder lead the crowd in singing Happy Birthday to Chris Woodruff and to me.

Steve Ballmer

Tuesday morning featured keynote speeches from Microsoft management. I heard Steve Ballmer speak for the first time. Mr. Ballmer is a very energetic and engaging speaker and I recommend you see him, if you get a chance. In the afternoon, I heard a series of talks from both the product team and from other MVPs. I had dinner Tuesday with a friend I met last year at Codestock. After dinner, I received a call from a colleague I worked with 4 years ago, so we had a chance to get together. Tuesday evening, I met up with all the Sogeti MVPs before heading over to Ted Neward’s house for a party filled with even more people smarter than me.

Wednesday, we met with leads from the product teams who showed us what they were working on in future versions and solicited our feedback. In the evening, Microsoft rented 47,000-seat Safeco Field for an attendee party. Before I left, I had a chance to meet Scott Hanselman, who invited me to the ASP.Net Insiders Summit the following day.

Me and Hanselman

The ASP.Net Insiders Summit Thursday was excellent. The content (also covered under the NDA I signed) was more concentrated than at the summit. Exciting things are coming from Microsoft in their web tools and we saw a glimpse into this. Many thanks to Mr. Hanselman for the invitation as this was the most educational day of my trip. At the ASP.Net summit, I also had a chance to meet Glenn Block in person. Glenn was the main driving force behind the Managed Extensibility Framework and now leads the WCF team and we have been corresponding for over a year, but never met face-to-face. I’m working to get him to come out to my user group as well.

The content of the sessions was very good, but the best thing I got out of it was the connections with other MVPs. I had a chance to reconnect with old friends and to meet meet many new people. Often, the people I met were people that I already knew by reputation. After a week in Washington, my brain and my belly are both full and I am ready to go home and digest what I consumed.

And in the all the rooms at the MVP Summit, I don’t think I ever found anyone dumber than me.

More photos

Saturday, March 5, 2011 7:26:39 PM (GMT Standard Time, UTC+00:00)
# Wednesday, March 2, 2011
Wednesday, March 2, 2011 9:03:00 PM (GMT Standard Time, UTC+00:00)
# Monday, February 28, 2011
Monday, February 28, 2011 9:00:00 PM (GMT Standard Time, UTC+00:00)
Monday, February 28, 2011 6:15:00 PM (GMT Standard Time, UTC+00:00)
# Wednesday, February 23, 2011

Last week, I delivered a new presentation on Microsoft Windows Work Foundation at the Microsoft Developers of Southwest Michigan user group in Kalamazoo, MI. The slide deck is below. The presentation was heavy on demos and light on slides, so I had fun with PowerPoint – I included a photo of Mike Eaton on almost every slide. Mike is the president and host of the user group and the man who invited me to speak there. He was impressed enough to suggest a throatpunch was appropriate.

I have retitled this presentation “Real World Lessons with Windows Workflow Foundation” and I am scheduled to deliver it again in ann arbor, East Lansing, Toledo, and Flint over the next few months. In addition, I will be presenting on Visual Studio database tools at the Redmond .Net User Group next week.

Below is my upcoming schedule.

Date Event Location Topic
Mar 28 Redmond .Net User Group Redmond, WA Migrating Database Schemas with Visual Studio 2010
Apr 13 Ann Arbor .Net User Group ann arbor, MI Real World Lessons with Windows Workflow Foundation
Apr 21 Greater Lansing .Net User Group East Lansing, MI Real World Lessons with Windows Workflow Foundation
May 17 Northwest Ohio .Net User Group Toledo, OH Real World Lessons with Windows Workflow Foundation
Jun 22 West Michigan .Net User Group Grand Rapids, MI Real World Lessons with Windows Workflow Foundation
Wednesday, February 23, 2011 2:00:00 PM (GMT Standard Time, UTC+00:00)
# Tuesday, February 22, 2011

A workflow is a set of related activities. The relationship between these activities can take many forms.

Activity 3 may take place immediately after Activity 2.

Or Activity 6 some condition that is tested in Activity 7 and determines whether Activity 8a or 8b executes next.

Or Activity 9 may test a condition and repeat Activities 6-8 until the condition is false.

Below is a diagram representing a simple workflow that you may have written in a general purpose programming language, such as C# or Visual Basic. You may even have drawn a flowchart similar to this, either to assist in designing the program or to help document the program’s logic after it was written.

Figure 1

General purpose languages work very well for these simple workflows, as you can use constructs such as if statements and while loops to construct the program and assemble the activities.

But workflows can contain activities with more complex relationships and this complexity can make it difficult to write these workflows in a general purpose language.

Figure 2

In the workflow of Figure 2, four tasks are executed in parallel. For our purposes, parallel simply means that it doesn’t matter in which order each task begins or ends – it only matters that all 4 of them must complete before the workflow continues to the next activity.

This relationship can be complicated further if the workflow demands that only one of these activities complete before proceeding or if any 3 of the 4 activities complete. These scenarios would be difficult to write in procedural code. Doing so requires tracking global variables and a way to cancel activities that started and no longer need to complete.

Another complication occurs when a workflow requires external input at a step in he middle. Imagine a scenario in which an e-mail is sent to a manager. The e-mail contains a link to a web page. The manager clicks the link, reviews information on the web page and clicks a button to allow the workflow to continue to the next activity. The manager may click this button shortly after the e-mail is sent; or he may wait hours, days, or even weeks before clicking it. He may never click the button and our workflow should be able to handle this possibility (possibly by invoking a different activity after a specified timeout period). It is not scalable for a workflow to remain in memory for days or weeks, waiting for input. The practical solution is to serialize the state of the workflow and save it to a persistent data store; then de-serialize it and reload it into memory when the input finally arrives. If multiple copies of the same workflow are in progress, it’s also important that the correct instance is loaded into memory. When the manager clicks the button, some piece of data must tell the workflow which instance to retrieve.

A workflow framework can help to manage these complex relationships between workflow activities. Allowing a framework to manage these relationships also tends to keep the business activity code (the code executed by each activity) separate from the workflow code (the code that manages relationships between the activities.) This makes for a simpler and more maintainable code base.

Microsoft Windows Workflow Foundation (WF) is a framework designed to help you create and manage workflows.

You could write such a framework yourself, but why do that if this would be a lot of work and frameworks already exist to manage this for you?

WF provides a graphical user interface to design workflows by dragging a dozens of built-in activities from the Visual Studio toolbox onto a design surface. WF provides a rich API for building workflows and custom activities.

By hosting a workflow in IIS, you can accept incoming requests as web services or other WCF services; by utilizing WF’s integration with SQL Server, you can persist a workflow to a database and allow it to be retrieved at a later time; by using the built-in activities that ship with WF, you can simplify some of the programming tasks you need; and by creating a workflow with the graphical Workflow Designer in Visual Studio, you can more easily visualize the relationship between each step.

WF takes care of much of the workflow infrastructure for you and makes it easier to separate the internal activity logic from the workflow logic.

Tuesday, February 22, 2011 2:00:00 PM (GMT Standard Time, UTC+00:00)
# Monday, February 21, 2011
Monday, February 21, 2011 7:14:00 PM (GMT Standard Time, UTC+00:00)