Lily Rides a bike

A momentous occasion occurred at my house tonight. After dinner, my six year old daughter, Lily, finally figured out how to ride her bike.

The joy of having a large amount of pavement and a slight hill (my house is on the top of a hill, and there used to be a tennis court below the house). I got her to coast down the hill with her feet down, then when the pavement flattened out she just started pedaling.

Best part: no having daddy trying to run behind her on his bad knees.

LINQ to SQL: Extension Methods

I've talked about Extension Methods (I probably called them partial methods) before. Now that I actually have Linq to SQL in my hands I thought I would take a look at what extension methods are created when you start with LINQ to SQL.

In my previous example, I showed a sample database that looked like this:


My designer is named "LinqTests.dbml", which is an xml file that is used to generate LinqTests.designer.cs. That is where we find all of the code.

First, we will look at the classes that map to the tables. We have a Customer, Product, and CustomerProduct tables. In each class there is a region named "Extensibility Method Definitions", that is where we are looking right now.

For the Customer class, this is what we see:

Basically, that accounts for two extension methods for every column in a table, plus three more: OnLoaded(), OnValidate(), and OnCreated().

If you can't tell from the name: OnCreated is called right as a new Customer object is created, the OnLoaded() is called after data is loaded into it by Linq to Sql, and OnValidate is called before the code is saved (I think -- since I can't find the code that actually calls it).

The others are for before and after a particular property is set.

Looking into the code further you can see how the code is being called. Here is the constructor:

public Customer()

{

    OnCreated();

    this._CustomerProducts = new EntitySet<CustomerProduct>(
          new Action<CustomerProduct>(this.attach_CustomerProducts),
          new Action<CustomerProduct>(this.detach_CustomerProducts));

}

The line OnCreated(); calls the OnCreated method (duh!), but if you search the code you wont find anywhere that OnCreated does anything. That is for you to do in a partial class.

Now, to add some business logic to my Customer class for when the class is created or loaded, the code will look like this:

public partial class Customer

{

    partial void OnCreated()

    {

        this.Name = "Billy-Mac";

    }

    partial void OnLoaded()

    {

        if (string.Empty( this.Name ))

            this.Name = "Jim-Bob";

    }

}

Back to basics here. The generated DataContext class also has some Extension Methods



So now you have three extension methods for each table that the Data Context has to manage. One for Insert, Update, and Delete; plus one more for when the Data Context class is created.


Lets take a step back now. Does this really get us anywhere? Before, with generated domain objects (those are the objects that are created to hold data) you could still accomplish all of this using inheritance. The problem with using inheritance is that now you have to use the inherited class and shouldn't user the base class. Worse, there really isn't a good way to tell people not to use the base classes.

Now with extension methods you will have less need to inherit your base domain objects. Now you have the hooks you need to extend your domain objects to fit your business needs. Your developers (or just yourself) now have one set of objects to remember to use. Kind of like one stop shopping.

I've still heard a bit of moaning and groaning about extension methods on the web -- I don't agree with them. I think extension methods are a wonderful addition to C# that will really help fix a need. And what Linq for SQL has done has only helped to illustrate that need for everyone.

LINQ to SQL: Many-To-Many Tables and Joins

Still having fun with LINQ to SQL over here. One quick note, I've found that the best way to test LINQ queries is to have a unit test class ready to go. Makes things much easier.

So what I've been playing with lately is using LINQ to SQL for querying against Many-To-Many tables. For an illustration, I'll use a table structure like this: (click on it to see a bigger version)


Once imported into your Linq to Sql file type in visual studio, and viewed through a class diagram, you get this:



In my Customer table I have 5 customers, I also have 5 products (Product ID=1 is "Computer"). The CustomerProduct table has about 20 records.

Now we can start having fun with some queries. The question I wanted to answer was: "Which customers have a particular product?". Simple enough.

I found, as with many things, there are multiple ways of getting the same answer. My first query looked like this:

int Computer = 1;
LinqTestsDataContext cx = new LinqTestsDataContext();
var customersWithProduct = from c in cx.Customers
    from cp in c.CustomerProducts
    where cp.ProductID == Computer
    select c;

OK, background note. This code is using the LinqTestsDataContext object to query the Customer table and the CustomerProduct table to find all of the customers with a Computer (product id = 1). What is returned is a IQueryiable object. If I want to parse through each Customer object indivitually, can call customersWithProduct.ToList() and get a List object returned.

This lovely piece of Linq generates the following SQL code:

{SELECT [t0].[ID], [t0].[Name]
FROM [dbo].[Customer] AS [t0], [dbo].[CustomerProduct] AS [t1]
WHERE ([t1].[ProductID] = @p0) AND ([t1].[CustomerID] = [t0].[ID])}

How do I know that is the SQL that is generated? After I run that line I can mouse over the variable (customersWithProduct) and the tooltip displays the generated SQL. I cant change it (that I know of), but at least I can look at it.

Anyway, that is not what I would call the best SQL I have ever seen -- and it is slow.

Next came attempt two at Linq to SQL. I wrote this:

int Computer = 1;
LinqTestsDataContext cx = new LinqTestsDataContext();
var customersWithProduct = from c in cx.Customers
    join cp in cx.CustomerProducts on c.ID equals cp.CustomerID
    where cp.ProductID == Computer
    select c;

In the previous example, I used that Customer.CustomerProducts object to filter the products. This time I am explicitly joining the Customers and the CustomerProducts tables together in Linq. The only odd part of the query was the "equal" keyword that you have to use in the join.

The SQL generated was much better:

{SELECT [t0].[ID], [t0].[Name]
FROM [dbo].[Customer] AS [t0]
INNER JOIN [dbo].[CustomerProduct] AS [t1] ON [t0].[ID] = [t1].[CustomerID]
WHERE [t1].[ProductID] = @p0}

Look: an actual join. Trust me, this works much faster. The first query took 3.24 seconds, the second took 0.06 seconds. I call that significant. Especially considering the amount of data I am querying (not much). You add some real data (thousand and millions of records) and you could be talking about some significant downtime.

Lots more to discover here. All a matter of time.

LINQ to SQL: SQL IN clause

Here is a problem I recently had to figure out with LINQ: how do you do an IN with LINQ for SQL?

In standard SQL your query would look like this:

SELECT myColumn FROM myTable
WHERE myColumn IN (myVal1, myVal2, myVal3)

But lets put another kink in, shall we. I'm not using LINQ, I'm using Lambda.

It turns out my savior was Contains.

Every generic List ( List ) with System.Linq available has a Contains< > extension method (this is .Net 3.5 we are talking about here), and that is what you use.

So, when creating a new query via LINQ for SQL in Lambda, you get a IQueryable interface with a Where extension method ...

Oh crap, here is the code, this will take to long to fully explain:

MyDataDataContext cx = new MyDataDataContext(); // from the LINQ to SQL dbml
IQueryable q = cx.MyTables.AsQueryable(); // creates the query object

List listOfData = {1,2,3,4};

q.Where( x => listOfData.Contains(x.MyIntValue));

var result = q2.Select(x => x.MyIntValue);

So what have we, and what was created. Well, we just used LINQ to SQL to generate a query that will look like this:

SELECT [t0].[MyIntValue]
FROM [dbo.MyTable] AS [t0]
WHERE [t0.][MyIntValue] in ( @p1, @p2, @p3, @p4 )

There is a @p[number] for every value in the list 'listOfData' above. And to me, the generated SQL is pretty good. That is probably what I would write.

LINQ: XElement and Namespaces

I had an opportunity to play with LINQ for XML and Google Maps recently. Quite a bit of fun actually. Anyway, I ran into a problem in that Google Maps encodes all of their XML in Namespaces. I hate Namespaces. I understand that somewhere there is a reason for namespaces to live -- but I still hate them. More than that, I hate having to query for them. XPath with Namespaces is just a pain beyond belief.

So one of the cool new objects you get in .NET 3.5 is XElement for reading xml and being able to use LINQ on it. But...what about querying with namespaces?

Here is how you do it:

Declare a XNamespace object with the namespace. In my case it looked like this:
XNamespace ns = "http://earth.google.com/kml/2.0";

Grab the XML, via a url, into an XElement node:
XElement resultNode = XElement.Load( url );

Now, I just wanted one node buried deap in the structure called "coordinates". This is how I grabbed it:
XElement coordinateNode = resultNode.Descendants( ns + "coordinates" ).First();

So that last line of code. I'm using the Descendants method to return all of the nodes in a flat list. I could have used resultNode.Descendants().Last() and gotten the node I wanted (since the coordinates node is the last node in the document -- but there isn't much of a guarantee that will always be the case.

So I'm asking the Descendants method for the "coordinates" node by name, and I'm prefixing that name with the namespace. Descendants returns an IEnumerable (a list), so I use the First() method to just give me the first instance.

Compared to XPath, this was a dream. But I still hate namespaces.

Here is all the code again in list:

XElement resultNode = XElement.Load( url );
XNamespace ns = "http://earth.google.com/kml/2.0";
XElement coordinateNode = resultNode.Descendants( ns + "coordinates" ).First();

C# Parital Methods

Update: Orcas Beta 2 is out, and I found a problem in my syntax. Now fixed.

I was just looking through the Orcas Beta 1, and I saw something I have been waiting to talk about...Partial Methods!

This allows you to create a method sub in a partial class so you can provide the implementation in another partial class.

A case where I would use this: in code generation. You generate a data object as a partial class. Now say you want to customize the constructor a bit more. Current you have to use inheritance to do that -- but then you have to remember to use the inherited class and not the generated calss. But with partial methods you can use the main class and provide that logic. Also, if the partial method is not implemented in the other partial class, it is taken out by the compiler. Nice.

A second use: logging. You can stub out methods where you would like logging to occur, but you don't want the partial class to have to deal with the implementation of logging, making it easier to switch out later.

I hope you see why I am excited about this enhancement.

A couple of caveats though,

1. the method must return void (or a sub for you VB folks).
2. out parameters are not allowed -- but parameters are allowed in general.
3. You are not allowed access to internal/private members
4. They can only be defined and implemented in a partial class.
   

Below is a stub implementation:

// my generated class in file1.cs
public partial class Class1
{
// a partial method
partial void CalledByInit();

public void Init()
{
CalledByInit();
}

}

// my implementation class in file2.cs
public partial class Class1
{
public void CalledByInit()
{
// do my stuff here
}
}

Now why is this better than:
1. just declaring a delegate in the middle of your class?
2. passing in the code as a predicate parameter?

The answer:
1. A delegate can be assigned by anyone, and potentially have multiple implementations. That isn't always good. The partial method can only have one implementation and only by other partial class.
2. aesthetically that makes a really ugly call, plus it requires that the caller have intimate knowledge of the workings of the method that it shouldn't have to have. Now you can localize that knowledge in one place.

Again, thing about the main use case for the partial method: it should be used in conjunction with code generators. If any of you have used SubSonic, Typed Datasets, NHibernate, or other technologies, right now you are using a mixture of inheritance and partial classes. With this you might be able to drop the inherited class all together. And frankly, I'm for anything that reduces the number of classes lying around a project.

If you are interested in other new features in C# 3.0 I found a nice Power Point from Raj that you can check out.

Partial Methods are mentioned on LukeH's blog.
And they are talked about on the VB Team blog.

Some days your better off going home

Yesterday I spent 4 hours trying to solve an "Ambiguous match found" error in one of my ASP.NET applications that I had just converted from a Web Site to a Web Application

For any who don't know the difference, ASP.NET Web Site lets IIS compile the code as needed, but not until then. Makes debugging a nightmare and you have to deploy all of your C# code to the server. Web Applications compile to a dll and you only have to deploy the aspx and dll files.

Anyway, I spent 4 hours beating my head against this problem. I found MANY hits on google, none really helped. Including an often quoted article by Eran Sandler.

Finally I went home. I had to drive 200 miles to pick up my kids from grandma anyway yesterday night (that is round trip mileage). Four hours in a car either by yourself or with sleepy kids does give you time to think tho. I devised a plan.

My problem was confined to one custom control, I had already figured that part out. I would recreate the control piece by piece until I found the problem (don't ask why I didn't think of this earlier).

Next day, nice and fresh, I did just that. I had the problem fixed in 15 minutes. On this particular control I had a Validator control and an event with the same name. That was causing the problem (it wasn't a problem in the ASP.NET Web Site mind you -- just in the Web application).

15 minutes. That was it. About the same amount of time it took me to write this up!

Just goes to prove, when you are really beating your head against the wall over and over again, sometimes it is just better to go home.

3 months, 3 LINQ presentations

That is right, in the past 3 months I have given the same basic presentation three times. I don't know if that counts as a groove or a rut. But there are some nice things about doing that: don't need new Power Point for one. But more importantly: more questions that make you question things more.

Starting off, I'm no LINQ expert. Yet I have to distill it to the group. Luckily LINQ is an easy sell. There is something there for everyone in LINQ. But when it gets right down to it, what is LINQ about? I put it like this: FOR loops are evil and LINQ is the cure.

FOR loops are not run for cover and grab a Bible evil, more of a general GOTO type evil. It isn't as if GOTO is evil in itself, in some languages the GOTO is a required statement. But like all inherently benign language constructs, in the wrong hands it can go really badly.

For myself, I've seen some of the worst code in my life nestled in for loops. And even worse, most people don't even know it. How would they? You could say that they just don't know any better. But in reality, there often isn't a better way.

What is the FOR loop but a structured GOTO. Really, that is it. And it isn't a very thick abstraction. If you don't believe me, go check out assembly language. Same with WHILE.

Next, what are you doing in the FOR loop (looping through a list -- DUH)? Sorry, I need a better question: what are you trying to accomplish with the loop? Now, look at the loop, and how easy is it to figure that out after the fact?

There is a reason people don't like assembly anymore, it is too hard to understand after the fact AND it is to hard to write in the first place. There are too many moving parts. Even adding two number (registers) is a multi-step operation. Things you do in loops have many of the same qualities.

Here is an example: find the largest value in an array of integers.

First the array:
int[] i = int[]{1, 2, 3, 4, 5, 6, 7 };

Here is what you write in C#:

int iMax = i[0];
foreach(int j in i)
{
if (iMax < j)
iMax = j;
}

Here is what you would write thanks to LINQ (and Extension Methods):

int iMax = i.Max();

How many ways are there for the first code to go wrong? There are 6 lines, 4 of them have code. There is one obvious bug in the code anyway...what if the list has no items? You will get an index out of bounds error right there. But there are many ways to incorrectly write this code. This is also a simplistic example, so imagine how bad this can get when doing real code.

In the second example: I can't find one. Plus, there is very little chance that you, or someone else, will not understand what the code is doing.

Now this is simplistic, which is bad because it hides the true power that is hiding underneath. There is more to link than grab bag of small statistic functions (e.g. Sum, Min, Max, Count). Add in a complex object and the Where method and we begin to see.

Visualize a customer object. It will have properties like FirstName, LastName, Address, City, etc. This in in a CSV file that is coming from Sales and Marketing.

First part: load the CSV into your program. No problem, we have all done that from time to time. Now find me all of the people in Idaho. Crap.

Not in LINQ. If you loaded your data into a list (List list) you would write code like this:

var idahoCustomers = from c in list
where c.State = "ID"
select c;

Want that in Lambda:

var idahoCustomers = list.Where(c => c.State == "ID");

Something you should know about now, there are two ways of doing the same thing, and you should probably know both. First is LINQ. If you see "from blah blah where blah blah select blah blah" -- you are looking at LINQ. If you see a "=>" you are looking at Lambda.

Personally, I love Lambda more than LINQ. Lambda can do everything LINQ can do, plus everything else. Another way of saying that is "LINQ is a subset of Lambda."

Anyway, this post could run on and on about the wonders of LINQ and Lambda -- but there are plenty of other people doing that. Hopefully, you have already read some of that. Where I want to finish off with is a few suggestions for anyone looking to get a grip on all of this.

First, there are a lot of new things to learn these days. WPF, WF, WCF, LINQ, Lambda, etc. Is this different? Yes it is. You need to learn LINQ. Personally I will be asking interview questions based on link in the future.

Second, considering you have limited time, what should you concentrate on? My answer is Lambda and Extension Methods. The more you learn about Extension Methods the more you will be able to do with Lambda. (Warning, if you are going to learn Extension Methods, you should probably learn about Predicates as well).

And some words of warning. Watch your return value types. You will see a lot of IQueryable, IEnumerable, and other strange interfaces as return types. These will often be hidden in 'var's. Be warned, each has its own capabilities, and you should know how to convert between them.

For example. In a List object, you get the ForEach extension method. You don't get that with IEnumerable or IQueryable. But you can get there by calling ToList() on either of those object types.

Finally: measure. Grab a profiler and run with it. Just like FOR saved you from the uglyness of GOTO, LINQ saves you from the complexity of FOR. But it doesn't get you away from the costs. There will still be times when it is better to write the loop yourself. A good profiler will tell you when.



Oh, one final note: I'm using Visual Studio.NET 2008 Beta 1 like everyone else. All code samples are subject to change when Beta 2 releases this week.

Where are the DSLs?

DSL is a Domain Specific Language. It took me a couple of years to really wrap my head around what the term means. Now I have a bad understanding -- but where are the languages?

First off, in my understanding of a DSL, it might not be a proper definition. Frankly, I don't care if the language is a full on language, with it's own syntax and control structures, or just a set of very specific libraries that add classes and methods for easily reading what is going on inside of an existing language -- like C# or VB.NET.

Personally, I like the second one. I would take the inspiration from SubSonic and NHiberate. Heck, throw in NUnit if you want. The nice thing about using these as your examples is that you know things are easily extendable.

There are a couple that I am thinking of right now. I've been playing around with MSBuild and NANT lately. Those two tools have convinced me that XML sucks as a language format. How do you define a loop in XML? You can't do it in any terse format I can tell you that. XML is for data, not logic. That is why JavaScript doesn't look like HTML.

So no, I dont buy XML as a language. Personally, I have a hard time seeing through all of the angle bracket -- really, I do. They just seem to create a lot of unnecessary noise to me. Readability stinks, which hurts the overall expressiveness of the language -- not to mention aesthetics -- and maintainability is also terrible. Not a whole lot of syntax highlighters for xml these days. Debuggability: None. (is debuggability a word?) Either it works or it doesn't. If it doesn't...may the force be with you.

Anyway, back to the build scripts example. Basically, even if you have sample build scripts to work off of, it will easily waist one day of a developer's life to get a build script off the ground. Just to get started. Then countless more hours keeping the stupid thing up to date.

Why don't we have a build script DSL? Some already do exist, Ruby Rake is one. I'm considering learning it. I would like a C# based one, but I'll take Ruby if I have to.

Another use that I have thought of often has to do with ETL. Now I find I'm not alone. Ayande has recently been bitten by the bug. Which is good. Because with his given track record he might just be able to do it. Would you rather deal with Integration Services GUI or a language specifically designed for that purpose.

Bear in mind, these examples break down at one crucial point: threading. Both of the examples mentioned, ETL and Build Scripts, can benefit heavily from working in parallel. And that is one area that our current languages don't help us with a whole lot yet. It is easy to tell Integration Services to load a bunch of tables at the same time in the GUI -- that is a lot more work in C#.

Now it can all be mocked with the Unit of Work pattern so everything can be batched together. That would hide the complexity from the user of the language at the very least. But that still leaves a lot of complexity behind the scenes.

Which leads me to one of the new great hopes coming from Microsoft sometime in the future: PLINQ. Aka: Parallel LINQ. LINQ is all about building a better FOR loop. But it is still iterating over a list one item at a time. PLINQ takes things that next crucial step: multi-threading the iteration. And all without changing the syntax of LINQ. Now that is flippen cool. Unfortunately, I have heard no release date for PLINQ, and it probably wont come out with .NET 3.5 at all.

Anyway, those are my thoughts. I could go on, but I need to get back to work.

I (hart) Fiddler

I fell in love with Fiddler today. I was trying to duplicate a SOAP call from one client to my own. Both applications were using the same api to perform the SOAP calls, but one worked, and the other did not (mine was the one that didn't work).

Enter Fiddler. I was able to intercept the call, check what was being sent and returned, and with the help of the file compare in Context I was able to figure out the problem.

Now I've seen Fiddler before, but that was always as a that-could-be-useful-later. Today it was useful. I don't know how I would have tracked down the problem otherwise.