Top 3 Unit Tests Causes of Death

One challenge in writing unit tests is making them survive the test of time.

Sooner or later your code will change and if your tests aren't resilient they'll break. That usually happens right before a deadline so evrybody's too busy to fix the tests, and as life goes those tests could continue to fail for quite some time.

Following a list of 3 major unit tests causes of death and some tips to avoiding them.

Reason 1: Special Privileges

As both code and its unit test is written by the same developer, that developer's deep knowledge of how the code works could lead to writing tests that rely on private interfaces. A variant on this happens when the developer leaves special "back doors" for the tests.

Take for example this python guessing game class:

import random

class NumberGuessingGame:
    def __init__(self):
        self._value = self.get_new_number()

    def guess(self, guess):
        if guess < self._value:
            return 'too low'
        elif guess > self._value:
            return 'too high'
        else:
            return 'Bravo!'

    def get_new_number(self):
        return random.randint(1,1000)

Writing the tests we know its _value field has the secret number and so can use it in our test:

class TestGame(unittest.TestCase):
    def setUp(self):
        game = NumberGuessingGame()
        game._value = 20
        self.game = game

    def test_low(self):
        self.assertEquals('too low', self.game.guess(10))

    def test_high(self):
        self.assertEquals('too high', self.game.guess(50))

    def test_bingo(self):
        self.assertEquals('Bravo!', self.game.guess(20))

Which works but not sure for how long. Because it's easy to imagine future code renaming _value or rewriting something about how this class works internally.

A better approach would be to rely on public interface only, and change the public interface to allow tests. In the above game example, the constructor could take an optional initial value as an argument. If passed it'll be used as the value, otherwise a value is randomized:

import random
import unittest

class NumberGuessingGame:
    def __init__(self, initial_value=None):
        self._value = self.get_new_number() if initial_value is None else initial_value

    def guess(self, guess):
        if guess < self._value:
            return 'too low'
        elif guess > self._value:
            return 'too high'
        else:
            return 'Bravo!'

    def get_new_number(self):
        return random.randint(1,1000)

class TestGame(unittest.TestCase):
    def setUp(self):
        self.game = NumberGuessingGame(20)

    def test_low(self):
        self.assertEquals('too low', self.game.guess(10))

    def test_high(self):
        self.assertEquals('too high', self.game.guess(50))

    def test_bingo(self):
        self.assertEquals('Bravo!', self.game.guess(20))

unittest.main()

Now the test still passes and it has a better change to survive the test of time.

Reason 2: Relying on Moving Parts

Another problem in the above test code is its reliance on hard coded texts. UI elements are considered moving parts, since they'll probably change often.

You should also careful relying on dates, network status, online services or anything that may change as part of the normal development of your application.

Moving the texts to a global variable in the module would let us use them without relying on their specific content:

TEXTS = {
        "TOO_LOW": "Too Low",
        "TOO_HIGH": "Too High",
        "BRAVO": "Bravo!",
        }

class NumberGuessingGame:
    def __init__(self, initial_value=None):
        self._value = self.get_new_number() if initial_value is None else initial_value

    def guess(self, guess):
        if guess < self._value:
            return TEXTS["TOO_LOW"]

        elif guess > self._value:
            return TEXTS["TOO_HIGH"]
        else:
            return TEXTS["BRAVO"]

    def get_new_number(self):
        return random.randint(1,1000)

Now the test code can use the same texts and verify just the logic used:

self.assertEquals(TEXTS["TOO_LOW"], self.game.guess(10))

A second example is internet connection. Consider a download function which uses requests to download a URL:

import requests

def download(url, filename):
    r = requests.get(url, stream=True)
    if r.status_code == 200:
        with open(filename, 'wb') as f:
            for chunk in r:
                f.write(chunk)
    else:
        raise Exception(r.reason)

How would you go about testing it? One option could be to download a known url and check the file was created with the correct content. But such a test relies on moving parts: It assumes we have internet connection and that the file is still there.

Alternatively we can mock requests to always return known data:


class TestDownload(unittest.TestCase):
    @patch('requests.get')    
    def test_dl(self, get_spy):
        mock = MagicMock()
        mock.status_code = 200
        mock.__iter__ = lambda self: ['hello world', '\n'].__iter__()
        get_spy.return_value = mock
        download('https://foo/bar/demo.txt', 'demo.txt')
        self.assertEquals('hello world\n', open('demo.txt').read())
        os.unlink('demo.txt') 

Reason 3: Mocking

And as much as the above example may feel "cool", it's actually an anti pattern I'd be really careful using in real life. The reason is its reliance on the behaviour and API of requests itself.

My main problem with over mocked tests is that they tend to break often (whenever any of the code they mock breaks), and they're hard to maintain.

A better approach is to test just our own logic and not the interaction with 3rd party libraries. By breaking the above function we can have simplere code and more focused tests:

def download(url, filename):
    r = requests.get(url, stream=True)
    if r.status_code == 200:
        write_result_to_file(r, filename)
    else:
        raise Exception(r.reason)

def write_result_to_file(r, filename):
    with open(filename, 'wb') as f:
        for chunk in r:
            f.write(chunk)

When working on medium or large projects, automatic unit tests are definitely a necessary part of our work. But for the tests to be effective we need to write code that's easy to test.

Comments