24 days of Hackage, 2015: day 15: IOSpec: testing IO; and some QuickCheck tricks

Dec 15, 2015 · 7 minute read · Comments
Haskell Hackage IOSpec IO testing QuickCheck quickcheck-unicode coercion

Table of contents for the whole series

A table of contents is at the top of the article for day 1.

Day 15

On day 11, I tangentially mentioned feeling embarrassed that unlike all of my other code examples, because I was using IO, I didn’t approach it by writing tests up front. I didn’t want to distract from the topic at hand, which had nothing to do with IO but to do with general monadic combinators.

But now I’m back to fill in the gap by showing one way of “mocking out” IO, the IOSpec library. This isn’t a perfect solution, and I wish there were some kind of standard way of testing IO in the Haskell ecosystem, rather than the various different methods (often reinvented) floating around out there, but at least gives the flavor of one approach.

Ironically, I found and fixed a bug in my code for day 11 in the process of writing a test for it!! Sometimes I feel I’m super unlucky because when I do last-minute edits of code without tests (in this case to turn one print statement into two, in the course of revising the day 11 article), I inevitably introduce bugs. This is precisely why I like to write tests. I do not trust myself without tests.

The day 11 code to test

Below is a copy of the introductory description of the task in day 11, with the first version of the Haskell code. We will test this first version:

Let’s write a function to simulate a user login process, in which

the user is prompted to enter a password
a loop is entered in which
- a guess is read in
- if it is not correct, we prompt again and loop
- if it is correct, we exit the loop
- we print congratulations

logIn :: IO ()
logIn = do
  putStrLn "% Enter password:"
  go
  putStrLn "$ Congratulations!"

  where
    -- Use recursion for loop
    go = do
      guess <- getLine
      if guess /= "secret"
        then do
          putStrLn "% Wrong password!"
          putStrLn "% Try again:"
          go
        else
          return ()

Code changes in order to use `IOSpec`

I copied the entire module for day 11 to a new module IOSpecExample.hs, with only a few modifications:

Changing the module name

-- | Copied with modification from MonadLoopsExample
module IOSpecExample where

import Prelude hiding (getLine, putStrLn)
import Test.IOSpec (IOSpec, Teletype, getLine, putStrLn)

Changing type signatures

We change IO a to IOSpec Teletype a everywhere, in order to set up to use the “teletype” simulator for reading and printing characters, since it happens that our logIn only uses those operations. If we used other operations, there are other simulators available, and IOSpec uses a “data types a la carte” type mechanism for mixing and matching.

How `IOSpec` works

IOSpec works by making everything build up a data structure that is then run with a Scheduler to produce an Effect. We use evalIOSpec:

evalIOSpec :: Executable f => IOSpec f a -> Scheduler -> Effect a

We (the tester) can then interpret the Effect however we want. We’ll just use a basic single-threaded scheduler for this example.

The test

For our test, we’ll use QuickCheck to generate random user input, subject to some constraints. Since we’re simulating a teletype, we assume that the user enters a stream (we’ll represent this as a list) of strings that do not contain newlines. We want to verify that if one of them is the word "secret", then the collective output will simply be the introductory prompt, the correct total number of re-prompts upon a wrong guess, and the final congratulations.

Here’s the spec, with auxiliary code discussed below:

{-# LANGUAGE ScopedTypeVariables #-}

module IOSpecExampleSpec where

import IOSpecExample (logIn)
import qualified Test.IOSpec as IO

import Test.Hspec (Spec, hspec, describe)
import Test.Hspec.QuickCheck (prop)
import Test.QuickCheck
import Data.Coerce (coerce)

-- | Required for auto-discovery.
spec :: Spec
spec =
  describe "IOSpec" $ do
    prop "logIn outputs prompts until secret is guessed" $
      \(notSecretLines :: [NotSecretString]) (anyLines :: [NotNewlineString]) ->
      let allLines = coerce notSecretLines
                     ++ ["secret"]
                     ++ coerce anyLines
          outputLines = ["% Enter password:"]
                        ++ concatMap
                           (const [ "% Wrong password!"
                                  , "% Try again:"])
                           notSecretLines
                        ++ ["$ Congratulations!"]
      in takeOutput (withInput (unlines allLines)
                               (IO.evalIOSpec logIn IO.singleThreaded))
         == unlines outputLines

`newtype`s to drive QuickCheck

We’re using a standard QuickCheck trick to create domain-specific random generators for already-existing data types.

One of our types is NotSecretString, representing a line typed by the user that is not equal to "secret"; it also should not have a newline. The QuickCheck Arbitrary instance is boilerplate to maintain the desired invariants.

-- | User input without a newline, and not equal to "secret".
newtype NotSecretString =
  NotSecretString { getNotSecretString :: NotNewlineString }
  deriving (Show)

instance Arbitrary NotSecretString where
  arbitrary = NotSecretString <$>
              arbitrary `suchThat` ((/= "secret") . coerce)
  shrink = map NotSecretString
              . filter ((/= "secret") . coerce)
              . shrink
              . getNotSecretString

We refine further down to the Char level, to generate NotNewlineChar:

type NotNewlineString = [NotNewlineChar]

newtype NotNewlineChar =
  NotNewlineChar { getNotNewlineChar :: Char }
  deriving (Show)

-- | Quick hack. Ideally should write specific generator rather than
-- filtering off the default 'Char' generator.
instance Arbitrary NotNewlineChar where
  arbitrary = NotNewlineChar <$>
              arbitrary `suchThat` (/= '\n')
  shrink = map NotNewlineChar
              . filter (/= '\n')
              . shrink
              . getNotNewlineChar

A note on random strings

We used a quick hack for getting an “arbitrary” string. It’s not so arbitrary, as you can see from the QuickCheck source code. For quality Unicode string generation, use quickcheck-unicode.

A note on `coerce`: it’s just an optimization hack

If you haven’t seen coerce from Data.Coerce before, you may wonder what it is. It’s just an efficiency hack to take advantage of Haskell’s guarantee that a newtype’s data representation is identical to the type it wraps. coerce solves the problem of ensuring that wrapping of wrapping (of wrapping…) is still recognized as having the same representation, such that there is no runtime cost in treating a value as being of a different type. It’s an ugly hack but convenient. If something can’t be coerced, that’s a type error, so this is safe.

If you don’t like using coerce, you have to use a lot of map and unwrapping and rewrapping to convert between all the different types, such as

notSecretStringsAsStrings :: [NotSecretString] -> [String]
notSecretStringsAsStrings = map notSecretStringAsString

notSecretStringAsString :: NotSecretString -> String
notSecretStringAsString = map getNotNewlineChar . getNotSecretString

It was easier if less principled for me to just think “they’re all just Char and [Char] underneath”. If anyone has guidelines on when to use coerce and when not, I’d be happy to link to them from here (and change my code as appropriate).

Interpreting an `Effect`

We interpret the Effect in a way taken from an example coming with the library. We convert Print and ReadChar how you would expect if generating a stream of characters or reading from one.

Taking output is straightforward, interpreting Done and Print:

takeOutput :: IO.Effect () -> String
takeOutput (IO.Done _) = ""
takeOutput (IO.Print c xs) = c : takeOutput xs
takeOutput _ = error "takeOutput: expects only Done, Print"

Reading input is trickier, because what is really happening is that an input stream of characters is used to convert one Effect into another. The ReadChar constructor has type Char -> Effect a and therefore when its value is called on a Char, goes to the “next thing that happens”.

withInput :: [Char] -> IO.Effect a -> IO.Effect a
withInput _ (IO.Done x) = IO.Done x
withInput stdin (IO.Print c e) = IO.Print c (withInput stdin e)
withInput (char:stdin) (IO.ReadChar f) = withInput stdin (f char)
withInput _ _ = error "withInput: expects only Done, Print, ReadChar"

And with this test, I found the bug in my code, in which the output was not what I expected because of a percent sign that went missing during a hasty edit.

A final note: there is a popular pattern called “free monad”

If you already know about free monads, you probably wanted to tell me “what about the free monad” as this article began, but I didn’t want to go there, yet. Maybe later.

Conclusion

Today I presented one way of adding a layer of abstraction to IO operations in order to be able to interpret them in an alternate way than actually doing them, in order to be able to write and run interesting tests.

All the code

All my code for my article series are at this GitHub repo.

The Conscientious Programmer