nUnit is not Unit Testing

Let me explain the title of today’s blog entry. A coworker stated today “I don’t know how to get this through people’s heads, but nUnit testing is not unit testing.” I understood what he meant, but figured I had to enter my two cents, as the statement is a bit too general to be of any value.

What is nUnit?

nUnit is a framework for adding code that tests methods in your code. The same functionality is given in Team System with its test harnesses. As such, I would consider this blog entry to include Team System tests, as well.

Now, to a few statements that came up in our conversation:

nUnit is not unit testing

This largely depends on how you write your tests. It also depends on what you are calling a unit and what you define “testing a unit” to mean. nUnit (and Team System tests) provide a means of testing both success and failure on a method. The level of the test depends on where the test points to.

If you place nUnit tests in your deliverable, you can test methods/functions at all levels, including protected, internal and private. I generally do not advocate this level of granularity, but I do know those that do. If you adopt this level of testing, you are best to branch code so a non-debug build does not reference nUnit and/or contain tests (it is easy enough to make the tests disappear, but not as easy to get rid of the reference).

If you place nUnit tests in a separate project, you are pretty much stuck to public methods, unless you add a wrapper to your test harness and inherit. Even in this case, private and internal methods are not touched. Is there value in this? Certainly, as you are testing code at the interface level, which is where others will poke in when they use your libraries (services, etc.).

I agree with the idea of labeling nUnit tests as developer tests, as I concur that this is not the only type of “unit” testing you should use to test your code. Proper nUnit tests will, however, cover a great deal of your code and provide you a decent idea of the current quality of your code. If you also follow the discipline of writing a test for each bug, the level of certainty that your code is quality increases tremendously.

nUnit testing is not functional testing

In general, when you build tests in nUnit or Team System, you are focus on success/fail and exceptional cases for a particular method on an interface. If you are running off of the public interface only, however, there is a certain bit of functionality that is tested. It is not GUI testing, but it does test what goes on when the GUI is exercised. As such, I both agree and disagree with the statement.

You can test GUI with tools like nUnitAsp, which tests web GUIs. This is a form of functionality testing. I will agree, however, that nUnit is not, in general, a functional testing tool, and that it cannot replace functional testing, nor should it. Used properly, however, nUnit will aid in testing functionality and improve the quality of your software.

It is difficult to cover all eventualities in nUnit, so I state that I would provide other means of functional testing than automated developer tests in a tool like nUnit.

nUnit testing is not regression testing

I would disagree with this overall. The very nature of having an automated suite of tests provides some form of regression testing. Every time you run a group of tests against new code that was working, and they succeed or fail. Comparing to previous states, you know whether the quality of your software has gone up or down. If you have 100% of your tests passing today and run again with a code change and have 50% working, you have a decrease in code quality.

nUnit is incremental. In order to have it effectively regression test software, you have to add a test for every (yes, EVERY) bug you encounter. This test, initially, should fail. You now have a state where 100% of old tests pass and 0% of bug tests pass. This is the current state of your software.

If you now fix your bugs and push a build where 80% of your old tests pass and 100% of the bugs pass, you have a regression test that indicates your software is worse than it was before.

Does this mean nUnit replaces other forms of regression testing? Certainly not, but it does give you a sanity check for future changes, which is what regression testing does.

Summary

A big portion of the problem is we (computer geeks) constantly redefine words to get more granularity. Thus, a test suite that is aimed at GUI is functional testing, while nUnit is unit testing. By the same token, we see regression testing as an external test suite that shows the software works the same way. I disagree with these redefinitions as much as I do the verbal magic surrounding SOA (which is, simply put, adding service interfaces and message transports to software — if you already do this, you are building SOA).

To me

• Unit testing is testing a unit of code. What "unit" means brings out arguments, but generally you find unit to either mean a single function or a single call to an interface. I would generally focus on a single function, as it is more precise. nUnit can do this, but you then get to argument #2: should you place tests in your assembly or not. I would say, in general, NO!
• Functional testing is testing the functionality of software. In most instances, the most thorough tests work with UI. There are nUnit add ons for web applications (nUnitAsp), but I am not familiar with any for Windows Forms.
• Acceptance testing is ensuring the functionality of an application is acceptable. What this actually breaks down to depends on whether you are Agile or not, in many ways. In Agile terminology, it is a set of functional tests with different input and output expectations. An example of an Agile framework for acceptance testing is the FIT Framework (FITnesse is the latest wiki-fied implementation).
• Regression testing is ensuring software still performs all of the functions it used to in the same manner. In regression testing, you are focused on repeatability.

If you disagree with the definitions, feel free to post a comment and we can hash it out. 🙂