FsCheck: An introduction

Simon Reynolds -

We've all heard of unit testing and some of us have even been known to practice it. It has a simple premise

  • Have a method or some other "unit" of code you want to test
  • Pass in some contrived test data
  • Compare the result to what you would expect it to be

This is essentially example based testing. All we can guarantee with it is that the output is as expected for the given input. That may be a single case or a handful of test cases.

If we were to imagine we had a requirement for a method that would reverse any given string of text then we could have a particularly obtuse developer who only wants to implement exactly what is in the requirement, no more and no less.

If the requirement states

For example, a given string "abcdef" would be reversed to "fedcba"...

then it's possible that our developer may write something like this.

test "String reverses correctly" {
    let expected = "fedcba"
    let input = "abcdef"
    
    Expect.equal (reverse input) expected "Should be equal"
}

This all looks straightforward enough until we look at the implementation of reverse

let reverse (input:string) =
    "fdecba"

This is clearly a contrived example but we can see how one test is not exhaustive. But what's the alternative? We clearly can't create a list of every possible string of text. Especially when we consider the amount of whitespace and non-printable characters.

We need random input, this is where we need property based testing. We can consider the properties of a method that would reverse a string.

  • The characters that make up the original string will all be in the reversed string, but in the opposite order
  • So if we count each character in both, we should never have a character that only appears once
  • Both strings should be the same length
  • If we join the two strings into a new string, the same character should be repeated in the middle
  • Reversing a string twice should give us the original again

Reversing the string again is a useful test but by itself does not prove that the reverse method is complete. After all if it just returned the string unchanged then the string would be the same after two calls to reverse.

A more useful test would be to compare the reversed string and iterate backwards through each character. Each character should match if we were to iterate forwards through the original string.

Let's use a property based testing framework called FsCheck. It allows us to test the assumptions in our methods and find edge cases we never even considered. 

let reverse (input:string) =
    if isNull input then
        null
    else
        let charArray = input.ToCharArray() |> Seq.rev |> Seq.toArray
        String(charArray)

let properties =
    testList "ReversePropertyTests" [

        testProperty "Char at index[length-n] of reversed string equals index[n] of original" <|
            fun (input:string) ->
                let reversed = reverse input

                if isNull input then
                    reversed = input
                else
                    // Account for zero-based index
                    let reversedLength = reversed.Length - 1
                    let reversedAsArray = reversed.ToCharArray()
                    input
                        |> Seq.mapi(fun i c -> c = reversedAsArray.[reversedLength - i])
                        |> Seq.forall id
                
       // This will likely fail, can you see why?
       testProperty "Reversed string should have same length as original" <|
            fun (input:string) ->
                let reversed = reverse input
                reversed.Length = input.Length
                
       // Think about what other testProperty could be defined here

    ]

This way, we are not testing any individual input. Instead FsCheck will, by default, run the test 100 times using progressively more complicated strings that include whitespace, line breaks, unprintable characters etc.

If a test fails, the framework will then restrict the inputs it generates based on the failing tests to try identify ranges of failure. When a failure is encountered it will attempt to reduce the failure to the minimum changes from passing tests required for it to fail.

In practice this means that if we were testing a method with DateTime parameters, it might be able to show us that there is an error in the method when dealing with dates in specific ranges or that only occur before or after particular dates.

It is also very adaptable, we can create custom data generators, allowing us to create random data within ranges or of specific types. For instance, you may want to create random email addresses or some other constrained type.

These are some of the properties we can test using a library like FsCheck. Obviously in this example the first property above is enough to define the method entirely but in more complex scenarios, it can be better to have a number of tests that each check an individual property of the scenario. An example of this is Mark Seemann's excellent Introduction to Property Based Testing on Pluralsight which utilises the Diamond Kata as a demonstration.

FsCheck is a .NET library so although it is written in F# and primarily with the F# ecosystem in mind it can be used just as easily in C# or VB.NET. It uses a PropertyAttribute or any custom attribute which inherits from it. Although the examples here are written using Expecto and its FsCheck plugin, there are also plugins for xUnit and NUnit.

Property based testing is a different method of thinking about your tests but something well worth considering, especially for those areas of your codebase that seem to always have more edge cases every time you look at them. By making you stop and think about the more generalised properties of what the method should be doing it flushes those edge cases out into the light where they can be found.