If you’re familiar with JUnit and Test::Unit then HUnit is the Haskell equivalent for unit testing. As I started to work with it I found it syntactically different than the others, which was something I expected since Haskell is such a different language than Java or Ruby. But what I hadn’t expected, was that it might be lacking in some way compared to the others.

Haskell, being a functional programming language, dispenses with the need for declaring a top level class that inherits from the framework and writing test methods that start with “test” in their name. Instead, HUnit allows you to write tests in almost a data driven fashion, in fact, your tests are essentially bundled up as list(s) of lambda functions as we’ll soon see.

Here’s a very basic test to see that we can turn strings to uppercase properly.

> import Char
> import Control.Monad
> import IO
> import System.Directory
> import Test.HUnit             -- only import needed, others are optional

> test1 = TestCase $ assertEqual "test upCase" "FOO" (map toUpper "foo")

To test this, use runTestTT in ghci

% ghci mytests.hs
   ___         ___ _
  / _ \ /\  /\/ __(_)
 / /_\// /_/ / /  | |      GHC Interactive, version 6.6.1, for Haskell 98.
/ /_\\/ __  / /___| |      http://www.haskell.org/ghc/
\____/\/ /_/\____/|_|      Type :? for help.

Loading package base ... linking ... done.
[1 of 1] Compiling Main             ( mytests.hs, interpreted )
Ok, modules loaded: Main.
*Main> runTestTT test1
Loading package haskell98 ... linking ... done.
Loading package HUnit-1.1.1 ... linking ... done.
Cases: 1  Tried: 1  Errors: 0  Failures: 0
Counts {cases = 1, tried = 1, errors = 0, failures = 0}

It’s a lot shorter than the equivalent code would be in Java or even Ruby, but HUnit also defines a number of shortcut operators that make your tests even shorter. In this next example we define a list of two tests, with a label, an expected value and then the function we’re testing.

> test2 = TestList [ "test upCase"   ~: "FOO" ~=? (map toUpper "foo")
>                  , "test downCase" ~: "baR" ~=? (map toLower "BAR")
>                  ]

If we run this in ghci, it gives us an error and uses the label to help identify which testcase failed.

*Main> runTestTT test2
\### Error in:   1:test downCase
user error (HUnit:expected: "baR"
 but got: "bar")
Cases: 2  Tried: 2  Errors: 1  Failures: 0
Counts {cases = 2, tried = 2, errors = 1, failures = 0}

The labels become important for discovering which test failed, because as you can see, this is really a list of lambda functions. Without the labels it would be impossible to see which was the problem test.

Some of you may feel we’re cheating a bit, all we’ve tested so far are pure functions. How about testing some monadic IO?

Let’s add a test that will create a file.

> test3 = TestList [ "test upCase"   ~: "FOO" ~=? (map toUpper "foo")
>                  , "test downCase" ~: "bar" ~=? (map toLower "BAR")
>                  , "create file"   ~: do let fn = "testfile.txt"
>                                          writeFile fn "HUnit rocks!\n"
>                                          b <- doesFileExist fn
>                                          assertBool "no file" b
>                  ]

So this code writes a file and then checks to see that it exists. The standard assertion assertBool is provided by HUnit. I’ve written this routine using the do notation only because it’s easier for many to understand.

But here’s the same test written in more idiomatic Haskell without do notation. We’ll also leave out the other tests to save some space, now that we’ve shown it’s no problem to combine tests of pure functions with impure functions.

> test4 = TestList [ "create file" ~: return "testfile.txt" >>= \fn ->
>                                     writeFile fn "HUnit rocks!\n" >>
>                                     doesFileExist fn >>=
>                                     assertBool "no file"
>                  ]

The syntax and format of a HUnit test file is different from JUnit and Test::Unit, but from a feature perspective we haven’t seen a difference. But there is one as we’re about to see.

When you get out of ghci and look on your filesystem, you’ll notice that our test file is left hanging around. In Java or Ruby, we would have written a setup or teardown method to handle pre and post conditions. In this case a simple teardown method would have ensured that the file was deleted after the test was run.

So how do we go about writing setup and teardown methods in HUnit? Well, it’s unclear. The HUnit docs make no mention of setup or teardown – not a good sign since the other frameworks have the responsibility of invoking setup and teardown at the proper times. If the framework doesn’t invoke them, how does one hook new methods into the call chain? Searching Google, I found some people asking questions, but no answers.

Well Haskell is pretty powerful, there must be a way to do this without drastically changing HUnit, let’s do some digging and see what we find out. Everything in Haskell begins and ends with types. So the first place to look is at the types provided by HUnit.

type Assertion = IO ()

assertFailure :: String -> Assertion

assertBool :: String -> Bool -> Assertion

assertString :: String -> Assertion

assertEqual :: (Eq a, Show a) => String -> a -> a -> Assertion

What we can see from this is that assertions are essentially actions (aka IO computations). This means that they can be combined, something we’ve already seen since we combined several of them as part of our file writing test.

What’s cool about Haskell is the ability to compose functions. Given two functions f and g, we can create another function h that is their composite.

With the help of bracket and some helper functions,

> setupFile :: FilePath -> IO String
> setupFile fn =
>   doesFileExist fn >>=
>   flip when (assertFailure ("setupFile: (" ++ fn ++ ") already exists!")) >>
>   return fn

> teardownFile :: String -> IO ()
> teardownFile fn =
>   doesFileExist fn >>=
>   flip when (removeFile fn) >>
>   return ()

> withFile fn = bracket (setupFile fn) teardownFile

we can compose our test action with a setup and teardown.

> test5 = TestList [ "create file" ~: withFile "testfile.txt" $ \fn ->
>                                       writeFile fn "HUnit rocks!\n" >>
>                                       doesFileExist fn >>=
>                                       assertBool "no file"
>                  ]

So what have we done here? Well we’ve hooked in setupFile as our setup method which will check to see if our test file already exists on the filesystem. Secondly, we’ve hooked teardownFile as our teardown method which will ensure that the file gets deleted after the test is run, regardless if whether any code threw exceptions. The HUnit framework still knows nothing about setup and teardown methods, what we did was compose our setup and teardown around our test method.

While it seems that we had to do this work just to get back to where JUnit and Test::Unit are, we are actually in a much better place. Those other frameworks are limited to having just one setup and teardown per set of tests in a testfile … we’re able to arbitrarily mix and match setup and teardown methods for each test in a testfile! We have almost unlimited choice in how we want to structure our tests.

So in this psuedo example we can have two tests that use a common teardown method, but their own unique setup methods, while in the same test file. How cool is that?

> setupCache :: FilePath -> String -> IO String
> setupCache fn url =
>   -- download web page and store in file
>   return fn
>
> withWebCache fn url = bracket (setupCache fn url) teardownFile
>
> test6 = TestList [ "create file" ~: withFile "testfile.txt" $ \fn ->
>                                       writeFile fn "HUnit rocks!\n" >>
>                                       doesFileExist fn >>=
>                                       assertBool "no file"
>
>                  , "process web page" ~:
>                    withWebCache "testfile.txt" "http://reddit.com/" $ \fn ->
>                    -- parse cached web page
>                    assertBool "parsing web page" True
>                  ]

Implementing this capability in JUnit or Test::Unit is left as an exercise for the reader.

Initially working with HUnit, people who are used to working with JUnit or Test::Unit will find it comparable, but the lack of setup and teardown methods to be a major block to usability. But with this new way of providing arbitrary setup and teardown methods on your tests, HUnit not only catches up to JUnit and Test::Unit, but it surpasses them with added flexibility.

Update (2007-12-09): I edited out the assertion that the code without do notation was idiomatic Haskell. I haven’t read or written enough to say one way or another. But the coding style I’m trying to use at the moment is staying away from do notation as much as possible. Without getting into the arguments about whether it is harmful to newbies, I’ll just say that getting away from it has helped me quite a bit.