{STATIC} hippo

Reading config file when called through COM interop

Noah — Wed, 18 Aug 2010 21:25:44 GMT

Seems obvious in retrospect, but gave me some trouble yesterday. I’m using hMailServer (a nifty, open source SMTP server) which supports scripting via VBScript event handlers. Now I don’t want to code in VBScript so I wrote a .NET library and was using some COM interop to pass everything through. However, I wanted my library to read configuration values via the ConfigurationManager class and didn’t know what config file it was looking for!

In a .NET app you typically have an App.Config that gets copied to a new name on build – YourApp.exe.config. I was building a library though, not an executable, and my library was being called through COM, so what should the config file be named?

Well after some research (running procmon – man that’s a useful utility!) I found that the config file that ConfigurationManager was looking for used the same naming conventions as if it was called from a .NET app: “calling-executable.exe.config” – in my case “hMailServer.exe.config” in the \hMailServer\bin directory.

Good to know!

Visual Studio 2010 and the .NET 4 Client Profile (or issues with class libraries in VS2010)

Noah — Thu, 15 Jul 2010 18:04:02 GMT

If you create a new class library in VS2010 using .NET 4, the default target framework is a new package called .NET 4 Client Profile which is a subset of the full .NET 4 (more info at http://msdn.microsoft.com/en-us/library/cc656912.aspx). The Client Profile is useful for for streamlining client application installations, but it causes some odd behavior in Visual Studio 2010 with regards to referenced assemblies.

Basically, if you reference another assembly that makes use of references outside the .NET 4 Client Profile, you’ll get full intellisense before you build. However, your build will fail. And afterwards, intellisense will disappear for those unsupported assemblies and the IDE will mark them with squiggly lines indicating that they’re unknown classes. At first it seems as though the dll was deleted or something, but it wasn’t. It’s just that the default target framework for Class Libraries in Visual Studio 2010 is the .NET 4 Client Profile which doesn’t support something you’re doing. I assume that the reason for the weird behavior has something to do with the build-as-you-type functionality of VS2010 not paying full attention to this.

Anyway, the solution to this is just to go to your project settings (right click on project and select properties) and under the Application tab (top tab) like so:

Lambda variable scope & execution confusion

Noah — Wed, 14 Jul 2010 18:31:10 GMT

I’ve been conducting interviews over the last couple weeks and have noticed a general lack of understanding regarding Lambda expressions (even in those candidates with senior role titles). So here’s some information about Lamdas that if you don’t already know, you should!

First of all, Lambda expressions are just inline methods. Unlike delegates, lambdas have access to variables that are scoped the same as the lambda. So if you have a method with a variable v and a lambda l, the statements inside l will have access to v as in the following example:

        1: void SomeMethod(){

       2:     int v = 1;

       3:     Func<int> l = () => { return v + 1 };

       4: ...

       5: }

simple enough, right? Except, and here’s what I believe separates the good from the great, do you know how that works? Think about it; in that example, l is a method and it doesn’t take v as a parameter, so v should be out of its scope. Like it would be with a delegate. But we already said it’s not. And the reason has to do with a cute compiler trick which pulls out the necessary variables and tacks them onto their own class. This is called variable lifting. The lambda expression is also tacked onto this class, so it has access to these variables (which otherwise would be out of scope).

The reason the how-it-works matters is all in the side effects. Now that you know how it works you should be able to figure out what the output would be from the following snippet:

       1: int i = 1;

       2: Func<int> f = () => { return i; };

       3: i = 10;

       4: Console.WriteLine(f());

Did you get it? The output is 10. Because by the time the Func is called, i was set to 10. Since i is lifted into its own class, the same i is changed a line after the initialization of f. So by the time we execute f, i is 10.

Consider the difference this makes for the following code:

       1: string[] strings = {"this is a sentence", "here's another one", "this should be plenty of sentences", "we're probably done here" };

       2:  

       3: var query = strings.AsQueryable();

       4: foreach(var s in new string[] {"s", "p"})

       5: {

       6:     query = query.Where(x => x.Contains(s));

       7: }

       8: foreach (var item in query)

       9:     Console.WriteLine(item);

Most developers I’ve spoken to assume that the query will have 2 predicates (where contains “s” and contains “p”) so the output will be:

this should be plenty of sentences

but in fact, the output is:

this should be plenty of sentences
we’re probably done here

because the query that’s built will basically look like:

       1: query.Where(x => x.Contains(“p”)).Where(x => x.Contains(“p”))

since the s variable will be lifted and will change through each iteration. By the time you get around executing the query, s will be set to the last string in the array. The way around this is to create a placeholder variable inside the foreach loop like this:

       1: ...

       2: foreach(var s in new string[] {"s", "p"})

       3: {

       4:     var localString = s;

       5:     query = query.Where(x => x.Contains(localString));

       6: }

       7: ...

Huge EFing Bug!

Noah — Tue, 08 Jun 2010 18:49:52 GMT

Thanks to Keith (comment #1 at http://weblogs.asp.net/johnkatsiotis/archive/2010/04/28/huge-ef4-inheritance-bug.aspx#7464062) for that title, btw.

After coming from NHibernate, EF4 is an enormous pain to work with. Unlike NHibernate, which basically lets you map any relationship you can dream of, EF4 has a lot of issues with inheritance. Here’s a pretty big one IMO:

Take a look at this object hierarchy:

Before I continue, if you think we’re working on an edge case I have to disagree. One of the tenets of OOP is the idea of encapsulation – there’s no reason a base class shouldn’t be able to encapsulate the logic of a subclass. For instance, in this case, the LeaveWork method on Employee would be abstract, but would be implemented on SomeoneWhoActuallyWorks to make sure they don’t leave too early and on SomeoneWhoDoesNotWork to allow them to leave at any time. I know I’m stretching it here with the example, but I wanted something simple that I could test & show you. I actually am working on a project with this type of hierarchy and EF4 is making my life painful… Take a look:

Mapping with Entity Framework (NOT)

Here’s an Employee table. This should really be all we need. Employees just have a FullName and a type. We just want to map it to the right classes based on the EmployeeType, but that won’t actually get mapped as a property on the class.

So you might be naive and try mapping this relationship like so (in the edmx file):

Of course you’d make sure that Sucker had a Condition “When EmployeeType = ‘Sucker’” and Manager when EmployeeType = ‘Manager’.

But that causes a runtime exception on insert:

Mapping and metadata information could not be found for EntityType 'EFTest.Manager'.

You see, the fact that Manager and Sucker both inherit from SomeoneWhoActuallyWorks is actually very important to Entity Framework 4. I’m not sure why. I mean if I didn’t specify a middle-man class, what’s the difference? If it has properties, I’m obviously not trying to map them, so who cares? Well EF4 cares… But wait, it gets better (or worse, depending on how you want to look at it)

So, your next try might be to map middleman -- SomeoneWhoActuallyWorks

You can map SomoneWhoActuallyWorks to the Employee table and set its Abstract property to True, this will compile without a problem. And it will run too. And inserting a Manager or a Sucker will work.

But this method explodes when you add another branch off of Employee:

This mapping (SomeoneWhoDoesNotWork is abstract, btw, and Owner is mapped to the Employee table with a Condition “Where EmployeeType = ‘Owner’”) won’t even compile:

Error 1 Error 3032: Problem in mapping fragments starting at lines 47, 66:EntityTypes EFTest.Manager, EFTest.Sucker, EFTest.Owner are being mapped to the same rows in table employees. Mapping conditions can be used to distinguish the rows that these types are mapped to.

Yea…

Now you can “fix” this by taking out those middle men. But deriving Owner, Manager and Sucker from Employee directly might have you writing a lot more code, and potentially duplicate code at that. And it’s a hack.

EntityFramework caching woes

Noah — Tue, 01 Jun 2010 16:25:25 GMT

When you retrieve an entity from the database, Entity Framework handles lazy loading for you by creating a proxy for your model class. There is, however, a little caveat. Take the following code:

        1: using (MSEntities ent = new MSEntities("name=MSEntities"))

       2: {

       3:     // I already added Microsoft to the database, with ID 1

       4:     Employee billGates = new Employee()

       5:     {

       6:         // We're not setting Company here, but there is a Company property

       7:         CompanyID = 1,

       8:         EmployeeName = "Bill Gates"

       9:     };

      10:     ent.Employees.AddObject(billGates);

      11:     ent.SaveChanges();

      12:  

      13:     int employeeId = billGates.EmployeeID;

      14:     var retrievedBill = ent.Employees.Single(x => x.EmployeeID == employeeId);

      15:  

      16: }

On line 14 we set a variable called retrievedBill to an employee that we retrieve from the DB* with the same ID as the one we just added – so in theory we should have a full Employee object setup and ready to go. Except that’s not what happens. In fact, retrievedBill’s Company property is null! However, if we try the same line -- ent.Employees.Single(x => x.EmployeeID == 1) – inside it’s own Entites context, we’ll get the full Employee and its Company property will not be null. What’s going on here?

Well, the reason is simple but far from expected (IMO). As mentioned, EF handles lazy loading for you by creating a proxy for your model class. This proxy inherits from your class, so a proxy for Employee might be EmployeeProxy123456789 (run through your code and you’ll notice the actual type for a retrieved employee looks something like this) and it overrides the getters for navigation properties and enables lazy loading (I went a little more depth on how this works at http://statichippo.com/archive/2010/05/28/entityframework-should-throw-more-exceptions.aspx).

However, EF also has a cache so that you don’t have to retrieve your entities from the DB on every call. This is smart, right? I mean if we have a 4 places that retrieve Employee 1, why should we have to do 4 SQL statements? On line 10 above, when we add billGates to the EF context, billGates is cached. Now if you step through this code, you’ll notice that on line 14 retrievedBill is not a proxy – it’s the actual Employee that we instantiated! That’s why Company is null -- there’s no proxy to override the getter and do the lazy loading for us!

Let’s just say that this is not how I would have build EF. In my opinion, EF should wrap even cached objects in proxies (if they’re not already) so that lazy loading works as expected…

EntityFramework should throw more exceptions!

Noah — Fri, 28 May 2010 16:05:50 GMT

I’m working on a new project using .NET4, MVC2 and EntityFramework. For the last couple years I’ve been using NHibernate as my ORM of choice but there are a lot of similarities between NHibernate and EntityFramework actually so the learning curve is quite small. There are a number of things that make me long for NHibernate, but one stands out in my mind: lazy loading. One of my first issues with EF was trying to track down a bug: ClassA has a property reference to ClassB, but the property is always null even when ClassB exists in the database. After debugging for what seems like hours, I tracked down the issue – the ClassB property on ClassA was not virtual. This was never a problem with NHibernate because NHibernate threw an exception if the property was not virtual.

Why does the property need to be virtual?

Suppose you have two tables in the database, TableA and TableB. And suppose you map your classes, ClassA and ClassB to those tables. For lazy loading to work there has to be some proxy – some class of type ClassA that has a property associated with ClassB that when it’s first accessed will do the actual query to get ClassB data. The way to accomplish this is to create a class, let’s call it ClassAProxy, that inherits from ClassA and overrides the properties, something like this:

   1:  public class ClassA

   2:  {

   3:     public ClassB ReferenceToClassB { get; }

   4:  }

5:

   6:  public class ClassAProxy : ClassA

   7:  {

   8:     private bool _IsRetrievedClassB = false;

   9:     private ClassB _ReferenceToClassB;

  10:     public override ClassB ReferenceToClassB

  11:     {

  12:       get

  13:       {

  14:         if (!_IsRetrievedClassB)

  15:        {

  16:          // RETRIEVE FROM DB & SET _ReferenceToClassB

  17:         _IsRetrievedClassB = true

  18:         }

  19:       return _ReferenceToClassB;

  20:       }

  21:     }

  22:  }

Notice that ClassAProxy has an override on ClassB. That’s to see if you’re paying attention – ClassA does not define ReferenceToClassB as virtual, so this wouldn’t compile. Change that one line (add ‘virtual’) and you get a working lazy loading proxy.

 
The TakeawayNHibernate throws an exception if you enable lazy loading but don’t set your properties as virtual.  Entity Framework just ignores this.  That means that you won’t know that a property isn’t lazy loaded (and will always be null) until you access it.  I can’t imagine why someone would want this “feature”!

OrderToList: Fix for LINQ-to-SQL Order dilemma

Noah — Wed, 07 Apr 2010 21:04:32 GMT

UPDATE: I’ve cleaned up the code a bit and put it up on CodePlex at http://ordertolist.codeplex.com/

A team member recently wanted LINQ-to-SQL to perform a query with a Contains() clause and return the results in the same order. For example, he was passed a list of People:

        1: List<int> idsOfPeopleToDisplay = new List<int> { 578, 291, 788, 230, 45 };

and he wanted to select all People from the database with those IDs, in that order. The following will result in a non-correctly ordered result set:

       1: var results = dataContext.People.Where(x => idsOfPeopleToDisplay.Contains(x.ID))

this is equivalent to:

       1: SELECT * FROM People WHERE ID in ( 578, 291, 788, 230, 45 );

The reason the results will not be ordered correctly is because any result with an ID that’s contained in the list will be returned in the order it was found. This presented a dilemma. Doing a JOIN onto the ID list was out of the question because of the size of the Article table.

He ended up with a temporary fix. I believe I went the rest of the way.

The temporary fix

I wrote a little tester to test my solution against his temporary fix. This isn’t exactly his code, but it should pretty much perform the same logic:

       1: public static List<People> GetSortedPeopleFromList(IEnumerable<int> ids)

       2: {

       3:     using (DatabaseDataContext dataContext = new DatabaseDataContext())

       4:     {

       5:         var people = dataContext.Peoples.Where(x => ids.Contains(x.ID));

       6:         List<People> sortedPeople = new List<People>();

       7:         foreach (var i in ids)

       8:         {

       9:             sortedPeople.Add(people.Single(x => x.ID == i));

      10:         }

      11:         return sortedPeople;

      12:     }

      13: }

As you can see, I’m retrieving the People from the database and then plucking them out of their list one by one in the order of my IDs. Not very efficient, I think we can all agree.

A better solution:

The final solution I came up with was to wrap objects in order to implement IComparable and then do an Array.Sort on them. I have an extension method to support this called GetSortedPeopleFromList. It’s used like so:

       1: public static List<People> GetSortedPeopleFromList(IEnumerable<int> ids)

       2: {

       3:     using (DatabaseDataContext dataContext = new DatabaseDataContext())

       4:     {

       5:  

       6:         var people = dataContext.Peoples.Where(x => ids.Contains(x.ID));

       7:  

       8:         return people.OrderToList(ids, x => x.ID).ToList();

       9:  

      10:     }

      11: }

that’s also code from my tester app. On line 8 you can see the extension method in play here. The first argument is the list of IDs (which is already sorted to our liking) and the second is a Func that expresses how to retrieve the ID from each element.

The difference is pretty substantial. I loaded 1k random records (ID, FirstName, LastName, City, State, Zip) into a SQL Express DB and ran some random tests (each pass had a random list of IDs, the IDs are outputted and so are the results of each method in order to confirm that it did, indeed, order the results correctly):

Notice the bottom lines: 443ms vs 39ms (averaged over 10 iterations) – that’s over 11x speed increase! So how does it work?

OrderToList extension method

       1: public static IEnumerable<TObj> OrderToList<TObj, TKey>(this IEnumerable<TObj> list, 

       2:     IEnumerable<TKey> ids, Func<TObj, TKey> getKey)

       3: {

       4:     var idsList = ids.ToList();

       5:     var wrappedList = list.Select(x => 

       6:         new IComparableWrapper<TObj>(x, idsList.IndexOf(getKey(x)))

       7:     );

       8:     var arr = wrappedList.ToArray();

       9:     Array.Sort(arr);

      10:     return arr.Select(x => x.Item);

      11: }

This basically does 3 things:

Wraps each item inside a class called IComparableWrapper (no, it’s not an interface and it should probably be renamed)
Performs an Array.Sort on the result (which will use a Binary sort algorithm)
Returns an IEnumerable of the original results which are now sorted (that’s the Item property of the IComparableWrapper)

Supporting parties:

The IComparableWrapper is a simple class that just wraps an object and implements IComparable (in order to achieve our Binary Sort):

       1: public class IComparableWrapper<T> : IComparable

       2: {

       3:     private T _Item;

       4:     public T Item

       5:     {

       6:         get

       7:         {

       8:             return _Item;

       9:         }

      10:     }

      11:     private int _Index;

      12:     public IComparableWrapper(T item, int index)

      13:     {

      14:         _Index = index;

      15:         _Item = item;

      16:     }

      17:     #region IComparable Members

      18:  

      19:     public int CompareTo(object obj)

      20:     {

      21:         if (!(obj is IComparableWrapper<T>))

      22:             throw new InvalidOperationException();

      23:         IComparableWrapper<T> typedObj = (IComparableWrapper<T>)obj;

      24:         return _Index.CompareTo(typedObj._Index);

      25:     }

      26:  

      27:     #endregion

      28: }

The CompareTo method compares the _Index field of each object, which is just an integer which represents where this item belongs in the list. Notice in the OrderToList method (above), on line 6 we instantiate a new IComparableWrapper – the second constructor argument is just the Index of the Person’s ID in the list of IDs. The reason I used a Func instead of just passing in an int was to allow for different situations where perhaps the IDs are accessed differently.

QA Based Testing

Noah — Sun, 28 Mar 2010 04:00:00 GMT

Over the last couple weeks I’ve spoken to a few people about something I’m calling QA Based Testing, and I figured it would be a good time to write about it.

There’s been a lot of attention around Test Driven Development in the last couple years. TDD does take some practice to get used to and it requires team participation. If your boss tells you to “do it right now” while he looks over your shoulder waiting to alert the business that it’s ready, you might have a hard time practicing TDD ;).

But there’s another way to get your code coverage up, and it doesn’t require practice or team participation. In fact, it’s very helpful for working with legacy systems. The idea behind QA Based Testing is that the moment QA fails a certain part of the app, the first thing to do is write a test. The test should fail because it should produce the issue that QA had a problem with. Then fix your code. Red-Green-Resubmit to QA.

At my current company I’ve used this with great success. First of all, I don’t always do TDD. And second of all, we’re building on top of a legacy system that constantly surprises us with new issues. Even modules I built with TDD have failed to account for certain situations because I just could not think of all the possible (invalid) data permutations we would have to account for. As issues arise, the code coverage increases. And it requires no getting used to and no team participation.

There’s really no excuse for not testing your code!

Object tracking != Lazy Loading!

Noah — Fri, 26 Mar 2010 17:36:42 GMT

LINQ-to-SQL DataContexts have a boolean property called ObjectTracking which is used to determine whether the framework should exert the overhead necessary to remember previous states. This defaults to true and is used to determine the SQL UPDATE command to run when persisting the object.

We have a project with data that’s read-only and so we decided to turn ObjectTracking off (set to false). We figured this was a good idea since, the thinking went, it would save overhead and since we’re not persisting the data anyway (read-only) we’ll have nothing to worry about. Boy were we wrong.

All of a sudden things started breaking! Some thing. Not others. Wierd. We figured out that it had to do with relationships not being loaded – for instance, if we have Person objects that have a collection of Car objects, the collection would be empty.

Well, apparently this is to be expected. The MSDN article on DataContext.ObjectTrackingEnabled states:

If ObjectTrackingEnabled is false, DeferredLoadingEnabled is ignored and treated as false. In this case, the DataContext is read-only.

(Interestingly enough, the next line states: “If ObjectTrackingEnabled is true, DeferredLoadingEnabled is false. In this case, DataContext allows you to load an object graph by using LoadWith directives, but does not enable deferred loading.” which clearly is not the case because it would mean there’s no difference in regards to DeferredLoadingEnabled whether ObjectTrackingEnabled is true or false! What they meant to say was “If ObjectTrackingEnabled is true AND DeferredLoadingEnabled is false…”)

So, I looked at the DeferredLoadingEnabled article and sure enough:

When the code accesses one of these relationships, null is returned if the relationship is one-to-one, and an empty collection is returned if it is one-to-many. The relationships can still be filled by setting theLoadOptions property.

(In fact, this article clears up the ObjectTrackingEnabled issue above by stating “[ObjectTrackingEnabled and DeferredLoadingEnalbed are][b]oth are set to true. This is the default.”

I guess we didn’t expect that because ObjectTracking and Lazy Loading are two very different things. Just because I don’t want to track my object does not mean I don’t want to enable lazy loading! This is just one of many quirks in Linq-to-SQL.

BitlyDotNet just got more awesome

Noah — Thu, 11 Mar 2010 19:17:07 GMT

Mike Gleason Jr has a bit.ly client library written in C# called BitlyDotNet (it’s hosted on here on Google Code). If it wasn’t cool enough before, it just got more awesome.

I submitted a new version with the following improvements:

Added support for error code 208 ("you have exceeded your hourly rate limit for this method")
Overload to shorten multiple URLs in one shot
Fixed bug that occurred when hourly limit was exceeded (SingleOrDefault throws InvalidOperationException because of multiple errorCode nodes)

This library saved me time on a recent project and I’m glad I could contribute. Go check it out!