Last week after the Microsoft Launch Event in Chicago (for the 2008 'wave' being in this case Windows Server 2008, SQL Server 2008, and Visual Studio 2008), there was another get together of the Chicagoland Alt.NET group (there isn't a web site yet but that is forthcoming) where we discussed all sorts of things, as well as planning out the initial stages of a Code Camp in the fall (more details forthcoming).
Somewhere in there, I was able to formulate in pretty brief terms a description of why I am skeptical of mocks and testing. It was not a formulation that was meant to convince anyone of anything, but I am going to relate it here so that it is written down and I can refine it (if I feel like it).
Who watches the watcher?
There are multitudes of places where one can get detailed explanations of mock objects, unit testing, integration testing, TDD, BDD, DDD, blah blah blah. Wikipedia is a good start. I will not pretend to give adequate presentations of any of them, because it is somewhat besides the point (and anyone who objects on that basis is just admitting to missing the point).
A point that has been made by Yegge (among many others) is (paraphrasing) that if you have really good developers, then you can implement most any methodology, most any development practices, and you will end up with good code. What should be obvious, but maybe isn't quite so, is that the opposite is also true.
A potential use of testing is to provide some verification of the intent of the code under test. A potential use of mocking is to allow your testing to avoid any dependencies on concrete implementations that would cause friction and interfere with the intent of the tests.
There are many other potential uses that I will not mention here.
But, if the code is such that it needs testing to verify its intent, and mocking is required to help bring this about, what happens if the same person who developed the original code also develops the tests and the mocks used in those tests?
To put it in its most coarse form, and in personal terms: "Suppose I'm not a very good coder. I've heard that by using testing and mocks within that testing, I can improve the quality of my code. But, I'm the person who will be writing the tests, and the mocks used in those tests. And we've already submitted that I'm not a good coder. So why should I think that by adding tests and mocks that things will get better? Isn't it just as likely that they will get worse?"
Less bad code is better than more bad code
Testing code is, ultimately, just more code. To use mocks, you need to know how to code with them. But writing tests that actually test what you think you are testing is a skill. Writing mocks that actually properly setup your expectations is hard.
I was recently in a situation where I pointed out that a particularly important piece of codebase was not under tests. So, the other people involved went off and created a bunch of tests. The only problem is that the tests they created were not able to prevent either false positives or false negatives. In my view, having poorly designed tests such as these is worse than having no tests at all.
There's no compile time check that your tests actually are correct, or that your mocks have their expectations setup properly. It requires some level of sophistication to do this.
To use another example, simply 'releasing' GOF patterns without guidance is just as likely to lead to disaster as to having never learned them.
How do we get to there from here, without having to take the journey?
I love integration tests. They test what will happen in my production environment (especially if I run the tests against that environment) with my production code. There is no leakage.
They are also incredibly slow, incredibly dangerous ("hey boss, we just placed 400 orders on your credit card, is that okay?"), and difficult to implement. For these and many other reasons, people try to find out how to produce tests that avoid these problems and still provide value.
But there is the potential for leakage. NUnit and Rhino Mocks are leaky abstractions (just to pick a couple).
When faced with an organizational decision of whether to embrace testing (and if you are really going to go beyond integration testing, you do really have to have some sort of mocking framework to use), I think this is a very difficult decision to make. Once you introduce unit tests that can be green while production bugs scream red, I think the battle is lost. Ensuring that this doesn't happen is as much a battle as whatever worth there is in testing itself.