Alec Lazarescu's Blog : Unit Testing

Syntactic Sugar for Thrown Exception Checking with Generics

alecl — Sat, 25 Oct 2008 15:52:00 GMT

Using attributes like [ExpectedException] can obscure intent because they apply to the whole method instead of a specific call you're checking. To make intent totally clear and prevent unexpected bugs, you should wrap just the statement you're expecting an exception from in a try/catch:

int y = 0;

try
{
    int x = 1 / y;
    Assert.Fail("Exception not thrown DivideByZeroException");
}
catch (DivideByZeroException)
{
}

In addition, at times the same exception is thrown, but with different descriptive messages and in the interest of checking correctness we need to check the message as well. To help achieve DRY, clarity, and expressiveness, I've created some helper methods for my unit tests to deal with testing proper exception throwing. Note that to be totally by the book, you likely should be creating special custom exceptions for each type rather than checking the message strings, but this can be superfluous at times so the option is here.

public class SpecHelpers
{
    public static void ExpectException<T>(Action methodCall) where T : Exception
    {
        bool caught = false;

        try
        {
            methodCall();
        }
        catch (T)
        {
            caught = true;
        }

        if (!caught)
            Assert.Fail(String.Format("Expected exception {0} not thrown"));
    }

    public static void ExpectExceptionMessageStartsWith<T>(Action methodCall, string startsWith) where T : Exception
    {
        bool caught = false;

        try
        {
            methodCall();
        }
        catch (T ex)
        {
            if (!ex.Message.StartsWith(startsWith, true, null))
                Assert.Fail(String.Format("Expected exception {0} thrown, but message didn't start with \"{1}\".  Message: \"{2}\"", 
                            ex.GetType().FullName, startsWith, ex.Message));

            caught = true;
        }

        if (!caught)
            Assert.Fail(String.Format("Expected exception {0} not thrown"), typeof(T).FullName);
    }

    public static void ExpectExceptionMessageContains<T>(Action methodCall, string contains) where T : Exception
    {
        bool caught = false;

        try
        {
            methodCall();
        }
        catch (T ex)
        {
            if (ex.Message.IndexOf(contains, StringComparison.InvariantCultureIgnoreCase) < 0)
                Assert.Fail(String.Format("Expected exception {0} thrown, but message didn't contain \"{1}\".  Message: \"{2}\"", 
                            ex.GetType().FullName, contains, ex.Message));

            caught = true;
        }

        if (!caught)
            Assert.Fail(String.Format("Expected exception {0} not thrown"), typeof(T).FullName);
    }
}

some example calls:

private bool ThrowEx()
{
    throw new Exception();
}

private bool ThrowExContains()
{
    throw new NullReferenceException("a special message");
}

[Test]
public void TestExceptionWrapper()
{
    // .NET 3.5 style with lamba notation
    SpecHelpers.ExpectException<Exception>(() => ThrowEx());
    SpecHelpers.ExpectExceptionMessageContains<NullReferenceException>(() => ThrowExContains(), "special");

    // .NET 2.0 style with anonymous delegate
    SpecHelpers.ExpectExceptionMessageContains<NullReferenceException>(delegate() { ThrowExContains(); }, "special");
}

Share this post: Email it! | bookmark it! | digg it! | reddit!

Functional Programming and Unit Testing

alecl — Sat, 13 Sep 2008 17:21:00 GMT

These code samples are from:

http://www.ayende.com/Blog/archive/2008/06/06/Scratching-an-itch-NMemcached.aspx

   public class SystemTime
   {
       public static Func<DateTime> Now = () => DateTime.Now;
   }

At first I was wondering what the point of that was. Does he really hate the name DateTime? Was he planning on changing the class one day to not use DateTime.Now and wanted a layer of indirection? Turns out that was on the right track, but not exactly it.

https://rhino-tools.svn.sourceforge.net/svnroot/rhino-tools/experiments/NMemcached/NMemcached.Tests/Memcache_FlushAll_Tests.cs

After seeing his unit tests I figured it out. He changes the value of the static function SystemTime.Now from being a function call that is evaluated each time to returning a burned in value of January 1, 2000. Now his code that checks for the current time to see if something is expired will see that date. He tests this in "When_getting_item_that_has_been_expired_will_return_empty_result()". You could do something messier like adding a cache item that is set to expire in 2 seconds, sleeping 2 seconds, and then checking that it's gone, modifying your classes to allow passing in a DateTime or setting it via reflection, but this is so much neater and doesn't introduce delays into your unit tests which should run as fast as possible.

Share this post: Email it! | bookmark it! | digg it! | reddit!

Good Overview of Types of Testing and TDD

alecl — Sat, 23 Feb 2008 00:58:00 GMT

Test-driven development, Unit Test, VSTS, NUnit, TestDriven.NET, whats all this?

Good Overview of Types of Testing and TDD

Share this post: Email it! | bookmark it! | digg it! | reddit!

Testing Theories Part 4 - TypeMock Isolator - The Dark Horse of Unit Testing?

alecl — Tue, 19 Feb 2008 03:17:00 GMT

Another in a series of posts that in no small part will serve as summaries/reminders to myself on various readings on the topic of automated code testing. Perhaps others will find them useful though. I can't take credit for any terribly creative conjectures in my conclusions/summaries. They almost exclusively come from the writers in the referenced material. I would strongly encourage reading the original source materials for far more details and a better understanding of the topics.

Testing Theories Part 1 - Classical vs Mock Testing
Testing Theories Part 2 - Unit Testing with Inversion of Control Pro's/Con's
Testing Theories Part 3 - IoC Containers (Castle Windsor, Spring.NET), AutoMocking Containers
Testing Theories Part 4 - TypeMock Isolator - The Dark Horse of Unit Testing?

A lot has been discussed on the pro's/con's of IoC for the purposes of unit testing and in general it has been framed as for or against TDD. This is actually not entirely correct. TypeMock Isolator has the notion of Reflective Mocks that allow setting up expectations on actual classes and not just interfaces. I've been fascinated with the idea of TypeMock and it introduces the concept that you don't HAVE to modify your design for testability using IoC, interfaces, and factories strictly for testing purposes. At the Philly .NET Code Camp in January the ALT.NET track was very heavily into testability via IoC. Also as part of that track, I attended Don's presentation on design patterns and I had asked him afterwards how he achieved testability in his sample code when he did not implement IoC and the design pattern implementation would have made it difficult to do so. He mentioned using TypeMock, a product I had been reading a lot about, and it definitely made me think of this product more seriously.

This article, Stop Designing for Testability by TypeMock Isolator evangelist Eli Lopian also promotes the bold assertion that IoC is not the only path to testability. The article itself is very thought provoking as well as the discussion in the comments. One particular comment by narsyseth (sadavoya) draws a great a parallel to white box and black box testing.

"Hi Guys,
This has been an interesting thread to watch. I see this moving toward a general discussion on how to test. It seems to me that there are two types of unit testing, white box and black box. Both are equally useful at different times. Black box unit tests would test an object using it's interface. The onus is on the production code to provide the flexibility to inject mocks, but those mocks require limited knowledge of what they are "mocking". White box tests would involve mocks that have more intimate knowledge of exactly how the unit they are "mocking" works. It seems to me that Typemock leans toward the white box approach, while the use of factories is more black box. I realize it's not so clear cut, but that's a pattern I'm seeing here. Regardless of whether or not that's true, where I see something like Typemock really shining is when writing tests for legacy code, when you have no tests to begin with. Assuming the legacy code is not written in a test-friendly way, using the black box approach is extremely difficult: you can't refactor the legacy code to support mock injection because without tests you can't be sure you won't break it. On the other hand you can't inject mocks because the legacy code doesn't support it. Typemock breaks this vicious circle, it seems, by allowing the injection of mocks without touching the production code. This way you can write tests, then safely refactor the production code. Voila, the legacy code can now be brought into a testing cycle and the 21st century. For me, this is the most exciting aspect of Typemock."

Some detractors mention that since TypeMock will let you test nearly anything, you'll have no incentive to adhere to any of the solid principles that come along with testability via IoC such as designing to interfaces, isolation, lower coupling, and single responsibility. That simply doesn't seem to be a clear argument. Having a more flexible tool doesn't automatically make you a sloppier programmer. Ideally your own discipline should motivate you, not the limitations of a tool.

It is still possible to use TypeMock just as a regular interface based mock tool using natural mocks. The reflective features aren't forced on you.

Roy Osherove, formerly on the fence about the power of TypeMock and now affiliated with it, also provides some insights on why TypeMock shouldn't be avoided just because it's flexible.

Personally, while I have done research and information gathering from various online sources and in-person conferences, there is no substitute for sufficient hands on experience with Unit Testing and TDD. I'm still new enough to it to have a revelation that come about only when I was trying to retrofit testing into legacy code. Had I been using TypeMock, I very well may have just used Reflective Mocking and not realized that my design wasn't very well layered nor were my classes following single responsibility very well.

I do have some reservations about instituting a non-free component into our codebase. The cost is pretty reasonable. The main concern is that the community usage is less than free alternatives like Rhino Mocks so you're more likely to run into a blogger or another developer at a user group and discuss Rhino Mocks than TypeMock. The basic principles of mocking seem fairly similar though, especially with the Natural Mock support so I'm willing to give it a shot. There is a downside to the latest open source trends as well. They can be quickly replaced by the next big free thing.

For now I will be doing IoC and using non-reflective Mocks through both TypeMock and Rhino Mocks for a few features while I'm still getting the hand of it to get a good frame of reference on both. For legacy code that I am simply adding tests to because I'm not making significant feature changes (can't justify the full refactor and QA cycle of a rewrite), I'll use reflective mocks from Type Mock.

After I become more experienced in making testability design decisions, I'll consider which framework to use for new code as well.

Share this post: Email it! | bookmark it! | digg it! | reddit!

Testing Theories Part 3 - IoC Containers (Castle Windsor, Spring.NET), AutoMocking Containers

alecl — Tue, 19 Feb 2008 00:41:00 GMT

IoC can certainly be done manually without the help of a container, but there are advantages to using one.

Can configure external dependencies in config for class so each time it's used the dependencies are automatically wired up. No setting properties or passing lots of interface reference parameters each time.
Can you separate config for unit test, pre-production environment staging, and production so no code changes for where your dependencies/resources come from. Integration tests on a test environment can use a local copy of the DB or completely mock the data layer altogether. Code based config right in the unit tests is also a possibility.
The container saves you the trouble of writing specialized factory classes to configure your objects for various roles.
Dependency lists serve as a form of implicit documentation for your classes and how they are connected.

AutoMockingContainers automatically create dynamic mocks for all dependency references needed for the class in test and the specific portion of functionality being tested can then be setup with expectations. It cleans out the setup part of your tests a bit and cuts down on repetition.

JP Boodhoo and David provide similar BDD base class implementations using Windsor AutoMocking container to further clean up redundant test code.

---

Resources

Castle Windsor

http://dotnetslackers.com/articles/designpatterns/InversionOfControlAndDependencyInjectionWithCastleWindsorContainerPart1.aspx
http://dotnetslackers.com/articles/designpatterns/InversionOfControlAndDependencyInjectionWithCastleWindsorContainerPart2.aspx
http://dotnetslackers.com/articles/designpatterns/InversionOfControlAndDependencyInjectionWithCastleWindsorContainerPart3.aspx
http://dotnetslackers.com/articles/designpatterns/InversionOfControlAndDependencyInjectionWithCastleWindsorContainerPart4.aspx
http://msdn2.microsoft.com/en-us/library/aa973811.aspx

Windsor AutoMockingContainer

http://blog.eleutian.com/2007/02/23/TestsAutoMockingIoCContainer.aspx
http://blog.eleutian.com/2007/08/05/UsingTheAutoMockingContainer.aspx
http://www.ayende.com/Blog/archive/2007/06/08/The-Auto-Mocking-Container.aspx

Spring.NET

http://www.developer.com/net/csharp/article.php/10918_3722931_1
http://forum.springframework.net/showthread.php?t=4201 - AutoMock Spring
http://www.ibm.com/developerworks/library/j-mocktest.html - Java example but still useful for reference

Helper Base Classes for TDD/BDD
http://codebetter.com/blogs/david_laribee/archive/2007/12/17/approaching-bdd.aspx
http://codebetter.com/blogs/jean-paul_boodhoo/archive/2007/12/19/bdd-specification-base-class.aspx

Share this post: Email it! | bookmark it! | digg it! | reddit!

Testing Theories Part 2 - Unit Testing with Inversion of Control Pro's/Con's

alecl — Mon, 18 Feb 2008 21:04:00 GMT

Inversion of Control (IoC) is often achieved by Dependency Injection and programming to interfaces and can be facilitated using IoC containers such as Castle Windsor or Spring.

Pro

Improves modularity and extensibility through interface based programming - it's easier to add/swap out implementations in the future and mock frameworks are well suited to mocking interfaces
Reduces coupling and dependency through isolation and decoupling into layers - layers can be replaced in the future or if that is not needed, they still gain the benefit of individual development and testing brought about by the reduction in dependencies
Promotes Single Responsibility Principle - easier to prevent unexpected breakage from unrelated functionality in the same class
All of the above facilitate creating fine grained unit tests that test only one thing and run quickly, especially using Mocks.
Explictly defining your external dependencies in constructors makes them clear to users of the class serving to improve implicit documentation

Note: #1-#3 above result from a solid implementation of IoC principles. Nevertheless, aside from the facets of isolation that result strictly from dependency injection, the rest of the principles mentioned in #1 - #3 above are not specific to inversion of control. This is an important distinction as often arguments FOR IoC are just arguments for some of those good, base programming principles and IoC is not required for their implementation

Con

Leads to a lot of classes generated for interfaces and factories on objects that are not required for production code and many objects are perhaps known to be simple and very unlikely to change. Violates YAGNI.
Tends to deviate from some facets of encapsulation given that dependencies which can arguably be considered to be implementation details are made external and must be known to be passed in/set.
Testability via IoC doesn't deal well with static methods or singletons. While there may be certain good reasons for avoiding these at times, at others they are useful and do serve a purpose, but you have no choice
Similarly, while programming to interfaces is often useful and recommended, there are times when inheritance lends itself more easily to the design and IoC becomes kludgy to implement when dealing with a dependency that can consist of any of a inheritance chain of classes. [Need example for this - not as clear in my mind as when I had thought of this a while back]
Difficult to retrofit legacy code to use IoC

---

Resources
More Debate: http://kohari.org/2007/08/15/defending-dependency-injection/

Pro

Con
http://www.codeproject.com/KB/dotnet/StopDesign4Tests.aspx
http://scruffylookingcatherder.com/archive/2007/08/07/dependency-injection.aspx
http://scruffylookingcatherder.com/archive/2007/06/20/mocking-developments.aspx
http://blog.eleutian.com/2007/08/18/DependencyInjectionTypeMockWindmillsAndOtherStuff.aspx
http://scruffylookingcatherder.com/archive/2007/08/16/tilting-at-windmills.aspx
http://www.elilopian.com/2007/08/19/dependency-injection-keep-your-privates-to-yourself/
http://colinjack.blogspot.com/2006/08/typemock-helps-improve-your-designs.html
http://qgyen.net/archive/typemock-is-wonderful/

Share this post: Email it! | bookmark it! | digg it! | reddit!

Testing Theories Part 1 - Classical vs Mock Testing

alecl — Mon, 18 Feb 2008 18:01:00 GMT

I'll begin a series of posts that in no small part will serve as summaries/reminders to myself on various readings on the topic of automated code testing. Perhaps others will find them useful though. I can't take credit for any terribly creative conjectures in my conclusions/summaries. They almost exclusively come from the writers in the referenced material. I would strongly encourage reading the original source materials for far more details and a better understanding of the topics.

Classical Testing favors using real objects when possible and stubs/doubles when using the real code is not practical due to complex dependencies, slowness, resource availability, etc. In this way, it borders on integration testing more so than unit testing. You are testing more "real code" in each test. Care must be taken to not make tests too coarse grained, or at least also provide fine grained tests to reduce confusion and the time taken to determine the root cause of test failures. There is more of a tendency for coarse grained tests with classical testing than mock testing, but it is up to the diligence of the programmer to do classical testing well.

In some ways your unit tests are less fragile when refactoring than tests strewn with many mocks which can pepper your tests with implementation details when ideally the tests should know/rely on as little implementation knowledge as possible. This is somewhat reduced by using Dynamic Mocks which only need to set expectations for the code being tested. Conversely, there maybe be some degree of implementation detail needed to make a stub/double so classical testing is not immune from this potential pitfall either.

Mock Testing, especially using a toolkit such as Rhino Mocks or TypeMock Isolator, really limits the amount of real code dependencies needed to run a test trending towards finer grained tests. Using an interface (or in the case of TypeMock just the class being tested itself), expectations for method calls, property setting, and return values can be defined and verified. While faking return values can be done quite easily with a stub as well, it would take a fancy stub to be able to verify that a method was simply called and this is trivial and compactly done with mocks. This amount of fine grained control and rigorous testing comes at the price of needing to know and specify the method names and their behavior in the implementation in your test code. Older frameworks relied on mocking using string parameters for class names but type safety has been improved in the latest frameworks and generics support in .NET has helped in this area as well.

---

There is absolutely room for both methodologies even within the same project. For instance, while it may not be desired as part of your run on every compile test suite, a layer of integration tests in the classical style using as much real code as possible would make an excellent verification that your objects are truly interacting properly in context.

----

References

http://martinfowler.com/articles/mocksArentStubs.html

http://codebetter.com/blogs/ian_cooper/archive/2008/02/04/classicist-vs-mockist-test-driven-development.aspx

http://codebetter.com/blogs/jeremy.miller/archive/2005/12/19/135757.aspx

http://weblogs.asp.net/rosherove/archive/2005/06/30/416872.aspx

Share this post: Email it! | bookmark it! | digg it! | reddit!

Assigning Responsibilities

alecl — Mon, 14 Jan 2008 15:04:00 GMT

It’s All about Assigning Responsibilities

This example really hit home for me. I am faced with a very similar legacy code design where some processing takes place and an e-mail is sent out. My current tests consist of integration tests that outright send me an e-mail that I have to confirm manually. Having these tests was far better than nothing at all, but their design never sat well with me. They were from an earlier attempt at writing tests before I truly understood or had time to delve into IoC, mocks or refactoring for testability.

Now half a year after having written the tests, I understood that I could use a mock to determine if an e-mail was sent without sending an actual e-mail. But how would that help me determine if the contents of the e-mail were correct. After reading Jeremy's article it became clear that my code design itself was flawed and I need to simply separate the generation of the e-mail text from sending the e-mail. It's such a simple idea and makes so much sense.

Share this post: Email it! | bookmark it! | digg it! | reddit!

Promises and Paradigms

alecl — Sun, 25 Feb 2007 22:37:00 GMT

I've always tried to make it a point to read a lot about the industry and constantly shifting and improving methods and best practices. Sometimes the buzz dies down after a bit -- I was particularly amused to find people tired of the lofty promises of EJB and going back to basics with POJO. Other times it grows to a roar.

The promises of unit testing and refactoring were being extolled for a while, but I must admit it took some time and much reading for it to sink in. If you're smart enough to make tests to exercise your code cases properly and didn't miss anything, you probably would have handled all those cases properly when you were coding anyway, right? Perhaps, but what about changing that code 6 months from now? What about someone else changing that code? I must admit I find a certain comfortable simplicity in writing unit tests to exercise the boundaries and coverage of a piece of code. After doing a few, it starts to become automatic and not take as much time as you would think. It keeps you honest. Even the best programmers can be burned when making just a quick last minute change that wasn't expected to change another area of code, but invariably does and most people are terrible at methodically manually testing their code countless times. Too many habits of doing things the same way every time leads very often to surprised cries of "But it worked for me.". Unit tests also add a margin of comfort to refactoring. Too often we let bad code persist and create maintenance headaches for years because it's not well suited for automatic or manual testing.

So I'm starting to believe now and writing my unit tests where I can going forward with new code and adding them to existing code when making changes. As time goes on, existing code will be retrofitted for testing and testing code coverage will increase. I'm definitely still learning and experimenting on testing methods and will occasionally post snippets or ideas I'm working on.

Share this post: Email it! | bookmark it! | digg it! | reddit!