High Code Coverage Means You Have Quality Tests, Right? Wrong!

In Agile Testing, you want to adhere to the testing pyramid when designing your automated tests. For this discussion, I will be focusing on the bottom tier of the pyramid, Unit Tests, where you want to have the largest percentage of your tests.   

I have been a software developer for a long time, and I've seen a lot of unit tests; unfortunately, that doesn't always mean I have seen tests that are of quality.

All too often, when I point this out to the developer(s) who wrote the tests, I am confronted with a common response, "my tests are fine, I have 100% code coverage!"  That's the point when I sit down with the team and walk through an example similar to the one below, showing how high code coverage does not automatically mean you have quality tests.

Consider the following simple application.  In this application, we have a UserService that exposes an addUser method, taking a first name and last name as parameters.  It then passes the two parameters to an implementation of a UserRepository, which in this case is a simple implementation that prints out the first and last name to standard output.  However, this code is written to include a defect: the addUser method passes the first and last name parameters to the user repository in the wrong order, resulting in the output having the first and last name transposed. 

public interface UserRepository {

    /**
     * Add a user.
     * @param firstName users first name.
     * @param lastName users last name.
     * @return ID for the user that was added.
     */
    String addUser(String firstName, String lastName);
}

public class SimpleUserRepository implements UserRepository {

    public String addUser(String firstName, String lastName) {

        System.out.println(String.format("Storing user --> first name: %s, last name: %s", firstName, lastName));

        // do something with the user
        return UUID.randomUUID().toString();
    }
}

public class UserService {

    // the user repository
    private UserRepository userRepository;

    /**
     * Construct the user service
     * @param userRepository the user repository used by this service.
     */
    public UserService(UserRepository userRepository) {
        this.userRepository = userRepository;
    }

    public String addUser(String firstName, String lastName) {

        return userRepository.addUser(lastName, firstName);
    }

    public static void main(String[] args) {

        UserService userService = new UserService(new SimpleUserRepository());

        userService.addUser("John",  "Smith");
    }
}

Ideally, our unit tests should reveal this defect and fail the tests.  However, all too often, small defects like this work their way into production despite claims of High Code Coverage metrics.  This is illustrated by the following example unit test.

@RunWith(JUnit4.class)
public class UserServiceTest {

    @Test
    public void testAddUser() {

        String firstName = "John";
        String lastName = "Smith";

        UserService serviceUnderTest = new UserService(new SimpleUserRepository());

        String userID = serviceUnderTest.addUser(firstName, lastName);

        assertNotNull("Expected a user ID to be created", userID);
    }
}

As you can see by the results below, the unit tests resulted in 100% code coverage and all tests passed, but the results are wrong. Why?  The answer is easy; it is not a quality test!

A quality test will not only provide high code coverage, but it will also identify defects and fail the tests when detected.  There are lots of ways to write quality tests, but almost all revolve around isolating the code that you are trying to test.  One way to do that is through the use of mock objects. The updated test code below illustrates the use of an open source mock framework, Mockito, that is used to "mock" the user repository.  By mocking the user repository, we can verify that the correct parameters are being passed into the user repository.

@RunWith(MockitoJUnitRunner.class)
public class UserServiceTest {

    @Mock
    private UserRepository mockUserRepository;

    @Test
    public void testAddUser() {

        String firstName = "John";
        String lastName = "Smith";
        String expectedUserID = "test-id-12345";

        when(mockUserRepository.addUser(isA(String.class), isA(String.class))).thenReturn(expectedUserID);

        UserService serviceUnderTest = new UserService(mockUserRepository);

        String userID = serviceUnderTest.addUser(firstName, lastName);

        assertNotNull("Expected a user ID to be created", userID);
        assertEquals("Wrong user ID", expectedUserID, userID);

        // verify that the mock user repository was passed the correct values
        verify(mockUserRepository).addUser(firstName, lastName);
    }
}

You can see in the results below, these unit tests failed, correctly identifying the defective code, this is a quality test!

The example above is simplistic, for illustrative purposes only, but shows how easy it is to miss defects if you are concentrating solely on code coverage. Imagine how many defects could be missed in a large code base. 

How can you make sure your tests are of quality, not just quantity?  Here are a few suggestions:

• Require Peer Reviews of your unit tests, they are code and should be treated as such
• Practice Test Driven Development (TDD), writing your tests first makes you put more thought into them and helps drive design
• Practice Pair Programming, another set of eyes always helps
• Use Mock Frameworks to better isolate the code you are testing

Remember, it's not the number of tests you write or how much code you cover, it's the quality of those tests.  I hope this article helps you on your Agile Testing journey. Find out more about our sevice offerings related to Agile Testing, TDD, and Test Automation.

To learn more about automated testing and to become an ICAgile Certified Professional in Agile Testing, check out our courses.