Using Design for Test Principles to Write High Quality Tests

Have you ever made changes to your code, ran your unit tests, and deploy only to find it breaks unexpectedly? “But my unit tests pass!” you cry in despair, yet the reasons why elude you. Situations like these can be avoided by following Design for Test (DFT) principles that help us not only make sure our use cases work, but also avoid situations that give us the false sense that our code is working as intended.

Let’s start with a simple example. I have been tasked with creating an app that allows users to search for products in a catalog. Products have a name and a description field. The requirements are as follows:

1. Users can input a keyword in a search field.

2. The user can search for the keyword in the name field, the description field, or both.

3. The application returns a list of products that match the search criteria.

Here are the contents of our product catalog:

Ring Name: Ring Description: A beautiful round ring, 24K gold plated band.

Necklace Name: Round Necklace Description: Round, 18K gold alloy.

DESIGN FOR TEST

Design for Test (DFT) is a principle that is applied before we implement the application. Tests are designed in a way that represent the functionality of the application. This principle will give clarity on which specific tasks our implementation should handle, keep the developers in scope, and provide a better overview on the structure of the code.

The first requirement indicates that the user provides a keyword to search, and the third requirement indicates that a list of products is returned. Before we start developing the app, let’s use the requirements explained above to stub out a test.

[TestMethod]public void SearchByKeyword() { var result = Catalog.Search(Keyword); Assert.isTrue(result.length>0);}

Looking at the second requirement, we learn that the user can also select which fields to search. That means we expect a List<string> parameter. We also learn that the search can be configured to match one or all fields, which can be represented with a bool MatchAllFields. Let’s recreate our tests using this newly acquired information:

[TestMethod]public void SearchByKeyword_AnyField(){ //Will match if keyword is found in at least 1 field string Keyword = “gold”; ListFields = {“Name”, “Description”}; bool MatchAllFlds = false;   var result = Catalog.Search(Keyword, Fields, MatchAllFlds); Assert.isTrue(result.length>0);}   [TestMethod]public void SearchByKeyword_AllField(){ //Will match if keyword is found in all fields string Keyword = “round”; ListFields = {“Name”, “Description”}; bool MatchAllFlds = True;   var result = Catalog.Search(Keyword, Fields, MatchAllFlds); Assert.isTrue(result.length>0);}

Now we have an idea what our code should be able to handle when it comes to parameters and returns. Using the DFT principles of scaffolding tests before writing our app provides a clear starting point for the developer to begin coding.

TESTING FOR FAILS: A SUCCESSFUL PLAN

In the snippet above we wrote tests in which we assume we are always going to find an expected result. But what happens if the user provides a keyword that returns no results? We should always include tests that we know will fail to cover these type of scenarios. In our case, a common fail scenario can be when the keyword provided does not exist in any of our products.

For such a situation, we need another test that is as inclusive as possible to maximize the chances of catching unforeseen situations.

[TestMethod]public void SearchByKeyword_AllField_NoResult(){ //Will not match any record because zzzzzz does not exist string Keyword = “zzzzzzz”; ListFields = {“Name”, “Description”}; bool MatchAllFlds = false;   var result = Catalog.Search(Keyword, Fields, MatchAllFlds); Assert.isTrue(result.length<1);} 

AVOIDING CONFIRMATION BIAS

When deciding what values or conditions to test our code with, it is important to choose carefully. Values or tests that will always pass or are a superset of what we are testing for are poor choices because they will undermine the purpose of our tests. For example, let’s take a second look at our public void SearchByKeyword_AllField(){} test. When searching with the term “round” as a search criteria, it is easy to just test to make sure our return collection is not empty ( > 0 ), however this is a trap, as the result can include other elements that do not fit the criteria. Our results collection should only contain Necklace because it contains the word “round” in both its description and name fields, but Ring only contains the word “round” in its description field. Rewriting the test to avoid this issue we end up with:

[TestMethod]public void SearchByKeyword_AllField(){ //Will match if keyword is found in all fields string Keyword = “round”; ListFields = {“Name”, “Description”}; bool MatchAllFlds = True;   var result = Catalog.Search(Keyword, Fields, MatchAllFlds); Assert.isTrue(result.length==1 && result.First().Name==”Round Necklace”); } 

Confirmation bias is tricky to avoid, and requires careful foresight. Picking situations where our tests always pass makes us feel good and gives us the false sense of security that our code is working as intended. To test out your own confirmation bias, take a look at this cool online puzzle from the New York Times.

FINAL REMARKS

DFT principles help teams keep code lean, prevent scope creep, and provide a clear visualization as to what the ultimate goals for the application should be. Thinking ahead about what possible cases may arise (even simple success/fail cases) and how we can accurately test them (knowing that confirmation bias exists) increases the chances of catching unforeseen situations before they break our code or confuse us with pesky false positives.

If you have questions or comments about this blog post or other development topics, please leave a comment below, send us a tweet at @CrederaMSFT, or contact us online.

• Microsoft Solutions
• Technology Solutions
• Design For Test