Category: Unit testing

To test a unit thoroughly¹, one has to throw different sort of things at the unit to make sure it behaves well in different situations. Throw it with some – or all – normal things, with a bunch of abnormal things, and finally with a few why-on-earth-would-you-do-that things to see how it reacts.

With all of the possible permutations, it’s tempting to write one or a few unit test that tests everything in the unit. They may be slightly complicated, but loops are cheap and copy-pasting is boring. The “master unit test” is born. One test result per unit is enough.

Before doing so, don’t. Separating the permutations to different tests will be helpful in the longer term.

1. or at least try to ↩︎

A mock object is by definition not a real object. It implies that it’s possible that the expectations set in the mock object for a unit test do not correlate with the real object, and that means that the test may be making sure that the tested unit is malfunctioning.

This may happen with the natural evolution of code, at least with some of the mocking frameworks that don’t care to validate the mock objects with the real ones – that may be sometimes even impossible. Even if the expectations in a mock were correct in the beginning but changes are done to the real object, the existing mocks may be testing that your tested unit works with an old version of the implementation but not necessarily with the updated one.

This may mean more engineering time to figure out why things don’t work even if the unit tests passes, and more time to update all the mocks and tests that are not directly related to the area that was changed – but that had mock objects of the changed area.

One of the major reasons to use mocks is to isolate the units. And that they do well. But is absolute isolation something to thrive towards to?

Mocks in unit testing are quite popular in at least some circles. Mocks are surely a powerful tool for unit testing, and they really enable having a great unit testing code coverage, but there’s a catch or a few with using them.

Mocks may introduce a burden of maintenance if the mocks have bad expectations to focus on the the code (coverage) rather than the behavior of the tested unit. Making mocks that behave or expect non-realistically is easy.

So what is the problem with suboptimal mocks? More maintenance than expected. It may mean that one has to re-write (all) the unit tests after fixing a bug or refactoring the code that is tested – even if the behavior wouldn’t change. Why? Because the expectations, the return value, or the side-effect in the mocks were expecting the exact, original, implementation of the unit (and not the behavior) and you fixed or optimized it.

There are times when to use mocks, but it’s easy to use them too much.

Test doubles are fake implementations which are used to replace real implementation during testing, for example to isolate the tested entity from the rest of the system. This enables testing without the use of real databases, services, and other complex and fragile dependencies – and let’s one focus the testing to the entity one is interested in.

There are various different types of doubles, for example fakes that have some logic build in, stubs that have fixed logic build in, and mocks that have expectations build in. There are times when to use one specific type – and there may be times when to not to use any of them at all.

Using test doubles is, at least in most of the cases, completely optional. They may make testing more stable and faster, though. But they can also become a challenge, if they’re used too naively.

Let’s face it, humans are bad in observing our own mistakes; we miswrite, misread, misunderstand, and miscommunicate often, without realizing it.

As unit tests are often written by the same person who writes the code that is being tested, there is a chance that any mistakes in the code will be duplicated in the unit tests as well. Or vice versa in case of test driven development.

It’s not that uncommon, that some of the unit test makes sure that the code misbehaves. But at least such tests make sure the misbehavior happens in a systematic manner.

First of all, there’s no doubt that unit testing can be a great form of testing. But it is not the silver bullet it is often promised. The lack of unit testing is not likely the reason for all of the issues in your project-gone-bad.

As automated unit testing is, or should be, fast, efficient, assertive, and repeatable it’s easy to run into conclusions and focus all efforts in unit testing – and reduce higher level testing. Shift-left testing is a trend. Metrics like unit test code coverage have become the most important metrics of the project.

However, the real users of unit testing are the developers. Unit tests focuses on units that are likely too small to tell anything about if the system works. Instead unit testing is tool for the developers to

1. Assist with development of new code; it enables the developers to design, debug, and test new code fast and easy without the rest of the system setup to test their code
2. Enable checking for any side-effects, regressions, they would introduce when they are touching any old code for a reason or another – as long as that area of code is covered by existing unit tests

For any other purposes, unit testing may be the wrong tool even if it would be for the right reasons. But have you already heard of snake oil?

Surely, everyone talks about the same unit in the context of unit testing, right?

It may be that the world hasn’t completely agreed on it yet.

There are likely two main religions about the unit. The first group, let’s name them as the ideologists, believe that the unit is always the absolutely smallest unit that can be tested. This means that unit means a function or a method¹.

The second group, let’s name them the pragmatists, consider that the unit means the smallest usable unit in the system. This means that the unit is more of a component or a module – a larger entity than a function or a method.

But do the religions matter in the end?

They may – while the ideologists are testing the implementation of a function with unit testing, pragmatists are testing behavior of a component with unit testing.

1. Or a term of your preference ↩︎

Unit testing? You mean those automated fast tests, right?

That is, hopefully, often the case. Unit testing is wired in our brain to mean automated tests and maybe you’re already thinking about your favorite unit test framework…

But let’s zoom out a bit. Unit testing is testing pieces, the units, of software more or less separated from the rest of the software. The tests are focusing on testing the operation of the unit with different scenarios. Often not all the scenarios, such as how does the unit behave with all the different inputs, are interesting but the focus is to select some well-selected scenarios that relate to normal operation, corner cases, or error situations of the unit.

Now. Unit testing could be manual work as well – compile an executable with (just) the unit and run the tests for it manually! However, it’s likely easier and faster to just use an existing unit testing framework and write automated test cases for your units.