nUnit is not Unit Testing


Let me explain the title of today’s blog entry. A coworker stated today “I don’t know how to get this through people’s heads, but nUnit testing is not unit testing.” I understood what he meant, but figured I had to enter my two cents, as the statement is a bit too general to be of any value.

What is nUnit?

nUnit is a framework for adding code that tests methods in your code. The same functionality is given in Team System with its test harnesses. As such, I would consider this blog entry to include Team System tests, as well.

Now, to a few statements that came up in our conversation:

nUnit is not unit testing

This largely depends on how you write your tests. It also depends on what you are calling a unit and what you define “testing a unit” to mean. nUnit (and Team System tests) provide a means of testing both success and failure on a method. The level of the test depends on where the test points to.

If you place nUnit tests in your deliverable, you can test methods/functions at all levels, including protected, internal and private. I generally do not advocate this level of granularity, but I do know those that do. If you adopt this level of testing, you are best to branch code so a non-debug build does not reference nUnit and/or contain tests (it is easy enough to make the tests disappear, but not as easy to get rid of the reference).

If you place nUnit tests in a separate project, you are pretty much stuck to public methods, unless you add a wrapper to your test harness and inherit. Even in this case, private and internal methods are not touched. Is there value in this? Certainly, as you are testing code at the interface level, which is where others will poke in when they use your libraries (services, etc.).

I agree with the idea of labeling nUnit tests as developer tests, as I concur that this is not the only type of “unit” testing you should use to test your code. Proper nUnit tests will, however, cover a great deal of your code and provide you a decent idea of the current quality of your code. If you also follow the discipline of writing a test for each bug, the level of certainty that your code is quality increases tremendously.

nUnit testing is not functional testing

In general, when you build tests in nUnit or Team System, you are focus on success/fail and exceptional cases for a particular method on an interface. If you are running off of the public interface only, however, there is a certain bit of functionality that is tested. It is not GUI testing, but it does test what goes on when the GUI is exercised. As such, I both agree and disagree with the statement.

You can test GUI with tools like nUnitAsp, which tests web GUIs. This is a form of functionality testing. I will agree, however, that nUnit is not, in general, a functional testing tool, and that it cannot replace functional testing, nor should it. Used properly, however, nUnit will aid in testing functionality and improve the quality of your software.

It is difficult to cover all eventualities in nUnit, so I state that I would provide other means of functional testing than automated developer tests in a tool like nUnit.

nUnit testing is not regression testing

I would disagree with this overall. The very nature of having an automated suite of tests provides some form of regression testing. Every time you run a group of tests against new code that was working, and they succeed or fail. Comparing to previous states, you know whether the quality of your software has gone up or down. If you have 100% of your tests passing today and run again with a code change and have 50% working, you have a decrease in code quality.

nUnit is incremental. In order to have it effectively regression test software, you have to add a test for every (yes, EVERY) bug you encounter. This test, initially, should fail. You now have a state where 100% of old tests pass and 0% of bug tests pass. This is the current state of your software.

If you now fix your bugs and push a build where 80% of your old tests pass and 100% of the bugs pass, you have a regression test that indicates your software is worse than it was before.

Does this mean nUnit replaces other forms of regression testing? Certainly not, but it does give you a sanity check for future changes, which is what regression testing does.

Summary

A big portion of the problem is we (computer geeks) constantly redefine words to get more granularity. Thus, a test suite that is aimed at GUI is functional testing, while nUnit is unit testing. By the same token, we see regression testing as an external test suite that shows the software works the same way. I disagree with these redefinitions as much as I do the verbal magic surrounding SOA (which is, simply put, adding service interfaces and message transports to software — if you already do this, you are building SOA).

To me

  • Unit testing is testing a unit of code. What "unit" means brings out arguments, but generally you find unit to either mean a single function or a single call to an interface. I would generally focus on a single function, as it is more precise. nUnit can do this, but you then get to argument #2: should you place tests in your assembly or not. I would say, in general, NO!
  • Functional testing is testing the functionality of software. In most instances, the most thorough tests work with UI. There are nUnit add ons for web applications (nUnitAsp), but I am not familiar with any for Windows Forms.
  • Acceptance testing is ensuring the functionality of an application is acceptable. What this actually breaks down to depends on whether you are Agile or not, in many ways. In Agile terminology, it is a set of functional tests with different input and output expectations. An example of an Agile framework for acceptance testing is the FIT Framework (FITnesse is the latest wiki-fied implementation).
  • Regression testing is ensuring software still performs all of the functions it used to in the same manner. In regression testing, you are focused on repeatability.

If you disagree with the definitions, feel free to post a comment and we can hash it out. 🙂

Advertisements

2 Responses to nUnit is not Unit Testing

  1. Kevin says:

    Bravo! Here again, I see a trend in which new terminology is a goal, and to impress people as a developer (or architect) you have to adopt the new lingo and be able to sound convincing with it. What a crock!
     
    I’ve been "unit testing" for almost as long as I’ve been a programmer. Big things are made up of lots of little things. Complex things are made up of lots of simple things. "Loose coupling" (which seems to be a rapidly-fading buzz-word in the community these days) is a term which as been around in principle ever since we started fooling with these confounded adding machines. I prefer the term "modular programming" or "component programming." The whole idea is to create usable components so that one doesn’t have to keep re-inventing the wheel. It all started with functions. All you have to do is develop software for a half-dozen years before you figure it out all by yourself. And "unit testing" is one of those same sorts of ideas, which seem to be leaking like a torrent these days from the (less successful) Sun/Java community into the (more successful) Microsoft community (heaven knows why). Heck, it’s just a matter of testing each piece completely before you add it to the mix, and then testing the mix completely afterwards!
     
    Preach it, Cowboy. Neither a follower nor a lender be!

  2. Casey says:

    I was actually the guy Greg has this converstation with initially.  I tend to label all testing done by _developers_ into the term "unit" test.  This is a term used by many shops, perhaps incorrrectly.  

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: