Why I'm Skeptical about Mocks and Testing

Last week after the Microsoft Launch Event in Chicago (for the 2008 'wave' being in this case Windows Server 2008, SQL Server 2008, and Visual Studio 2008), there was another get together of the Chicagoland Alt.NET group (there isn't a web site yet but that is forthcoming) where we discussed all sorts of things, as well as planning out the initial stages of a Code Camp in the fall (more details forthcoming).

Somewhere in there, I was able to formulate in pretty brief terms a description of why I am skeptical of mocks and testing.  It was not a formulation that was meant to convince anyone of anything, but I am going to relate it here so that it is written down and I can refine it (if I feel like it).


Who watches the watcher?

There are multitudes of places where one can get detailed explanations of mock objects, unit testing, integration testing, TDD, BDD, DDD, blah blah blah.  Wikipedia is a good start.  I will not pretend to give adequate presentations of any of them, because it is somewhat besides the point (and anyone who objects on that basis is just admitting to missing the point).

A point that has been made by Yegge (among many others) is (paraphrasing) that if you have really good developers, then you can implement most any methodology, most any development practices, and you will end up with good code.  What should be obvious, but maybe isn't quite so, is that the opposite is also true.

A potential use of testing is to provide some verification of the intent of the code under test.  A potential use of mocking is to allow your testing to avoid any dependencies on concrete implementations that would cause friction and interfere with the intent of the tests.

There are many other potential uses that I will not mention here.

But, if the code is such that it needs testing to verify its intent, and mocking is required to help bring this about, what happens if the same person who developed the original code also develops the tests and the mocks used in those tests?

To put it in its most coarse form, and in personal terms:  "Suppose I'm not a very good coder.  I've heard that by using testing and mocks within that testing, I can improve the quality of my code.  But, I'm the person who will be writing the tests, and the mocks used in those tests.  And we've already submitted that I'm not a good coder.  So why should I think that by adding tests and mocks that things will get better?  Isn't it just as likely that they will get worse?"


Less bad code is better than more bad code

Testing code is, ultimately, just more code.  To use mocks, you need to know how to code with them.  But writing tests that actually test what you think you are testing is a skill.  Writing mocks that actually properly setup your expectations is hard

I was recently in a situation where I pointed out that a particularly important piece of codebase was not under tests.  So, the other people involved went off and created a bunch of tests.  The only problem is that the tests they created were not able to prevent either false positives or false negatives.  In my view, having poorly designed tests such as these is worse than having no tests at all.

There's no compile time check that your tests actually are correct, or that your mocks have their expectations setup properly.  It requires some level of sophistication to do this.

To use another example, simply 'releasing' GOF patterns without guidance is just as likely to lead to disaster as to having never learned them.


How do we get to there from here, without having to take the journey?

I love integration tests.  They test what will happen in my production environment (especially if I run the tests against that environment) with my production code.  There is no leakage.

They are also incredibly slow, incredibly dangerous ("hey boss, we just placed 400 orders on your credit card, is that okay?"), and difficult to implement.  For these and many other reasons, people try to find out how to produce tests that avoid these problems and still provide value.

But there is the potential for leakage.  NUnit and Rhino Mocks are leaky abstractions (just to pick a couple).

When faced with an organizational decision of whether to embrace testing (and if you are really going to go beyond integration testing, you do really have to have some sort of mocking framework to use), I think this is a very difficult decision to make.  Once you introduce unit tests that can be green while production bugs scream red, I think the battle is lost.  Ensuring that this doesn't happen is as much a battle as whatever worth there is in testing itself.

posted on Sunday, March 16, 2008 9:31 PM Print
# re: Why I'm Skeptical about Mocks and Testing
Matt Hinze
3/17/2008 6:29 AM
There's something to be said for the vast improvements in design that can be had by thinking about how you want to consume your code before you produce it. That's TDD. Verifying algorithms are another matter, and I think your conclusion is mostly well founded within those bounds.
# re: Why I'm Skeptical about Mocks and Testing
3/17/2008 8:08 AM

Perhaps, but TDD doesn't come sui generis either. You have to learn it. And what exactly do you test?

Using pseudo code, no one does:

int i = 0;
i = i + 1;
Assert.AreEqual(1, i, ".NET framework int incrementing broken)

So where do you start? Constructors? Property getters/setters?

Practicing TDD may be helpful, but will practicing it poorly get you better code than if you just hacked away without it?
# re: Why I'm Skeptical about Mocks and Testing
3/17/2008 8:36 AM
If you suck and you know enough to know you suck, trying to do TDD is a good way to begin to not suck. You'll write some awful tests with 200 lines. If you have the potential to not suck, your methods will soon get shorter and your tests will become more concise. Either that or you'll get fed up with TDD, stop doing it, and continue to suck. That's why a bad coder should practice TDD.
# re: Why I'm Skeptical about Mocks and Testing
3/17/2008 9:12 AM
There are two facets to using integration testing, both of which provide value to on going development. They aid in design and they provide a safety net for refactoring.

Using mocks has a learning curve during which you are exposed frequently to the problems you've outlined. Simply using a mock is not a promise that code works correctly. It is only an assurance that the code works according to the expectations you have defined. Because of this, integration tests are critical in ensuring that the expectations you have of the code lead to the correct outcome.

Once integration testing has calibrated the expectations in the test code, the tests then become an important feedback mechanism for refactoring. With good test coverage you can at least verify that refactoring has not broken the existing behavior you intended to preserve.

Over time, the repeated cycle of finding bugs and refining test coverage starts to become a reliable proving ground for the code. This process takes time though so, as you mention, anyone unit testing without verification and calling it done will be in for a wake-up call when they found out they didn't complete understand the problem domain.
# re: Why I'm Skeptical about Mocks and Testing
3/17/2008 6:16 PM

It's implicit in what you said that doing TDD (if you have the potential to not suck) will help you not suck by its practice. I'm skeptical of that (if you had said BDD, I'd reject it completely). Not that you'd be surprised.

But, let's accept what is implicit as a given.

Let's also accept that the developer in question has the potential to not suck (might be more of a presumption, but you know that).

There is still the question of how the lessening of suckitude is to be achieved in reality.

As it often/usually happens, the developer in question has a couple of months of time when they are rolling off of a project and have access to a different developer who seems to know what they are doing and so the original developer in question can pick latter developer's brain, ask annoying questions, read their code, etc.

And if developer in question then has a couple of months in rolling into another project to greenfield stuff, retrofit stuff at leisure, etc. then maybe that's a good thing.

But if trying to introduce testing and mocking into an existing group of 'potential to not suck' developers when there is no time or organizational support to doing so, letting them try to hack it out on their own isn't necessarily a good thing.

Which is part of my point.
# re: Why I'm Skeptical about Mocks and Testing
3/18/2008 12:27 PM

If you give apprentice level developers free reign in a project you are going to wind up with apprentice level code. That applies just as well to anyone picking up a new technique or technology.

There is really a larger question there of how are you supposed to pick-up tool or technique improvements if you can't afford to cross the improvement ravine on the job? Is the danger there a reality or is any incremental improvement ultimately a good thing? I suspect asking the developer and the guy paying him would result in two different answers.
# re: Why I'm Skeptical about Mocks and Testing
7/15/2008 6:31 PM
I couldn't of said it better myself...and you know it goes back to what the elitists who worship Fowler and Beck and that whole group in general conveniently forget about what those they worship say - don't add complexity just for testing.

In fact I once won an argument with my boss on this; he wanted to override Equals in 260 DTOs and time of 3 weeks of a developer resource for testing purposes...I wrote Kent Beck and email and Kent replied back with 'I'd never override Equals just for testing purposes'. Awesome.

If there was less hero worship and more focus on improving the quality of one's code through plain old fashioned 'thinking for yourself' instead of trying to use the internet and the some odd million patterns/blogs/wiki's/etc out there as a recipe book, perhaps our industry wouldn't be so full of crappy programs that every business in every city goes through cycles of 'off-shore' 'out-source' 'in-source' trying to find proficient developers who can get the job done right the first time without introducing so many open source frameworks that they're bound to hire an elitist group of programmers in the future just to maintain the damn thing. Couple that with self-imposed SOX nightmares and people wonder why Shadow IT is so attractive!

This too is why I'm skeptical of mocks - it will a significant amount of time on my part breaking hard-wired rules about the amount of code complexity relating to TCO in my head before I add a bunch of moving parts (i.e. interfaces, making things public or internal that should be private) just for a testing framework.

Post Comment

Title *
Name *
Comment *  
Please add 4 and 4 and type the answer here: