Over at CodeBetter, Patrick Smacchia (the NDepend dude) recently has blogged a couple of posts about “Tests Driven Development” (not sure if the extra ‘s’ is supposed to signify something important or if that’s just what he calls it).
I’ve written at other times about why I’m not a big fan of TDD so won’t go through all of that blah blah blah here, but some more events at clients have re-iterated to me why Integration Tests are often much more important than unit tests.
Patrick talks about using code contracts as integral, and I agree with this. He also talks about some variations of the ‘80/20’ rule:
So now we have another side of the 80/20 law. 80% of the effort is spent writing tests to cover the 20% of the code remained uncovered, but 80% of the issues and bugs are found during this effort.
I’ve always found the idea of trying to achieve 100% code coverage to be, well, kind of nutty, but in all honesty, this is, in my mind, as well-thought out a defense of the idea as I’ve seen in a while. And succinct too. I doubt this defense is provable in any way, shape or form, but that’s true about many things.
And yet, it still seems to miss a fundamental point.
At around the same time, it turns out that over at LosTechies, Ryan Svihla has posted about an ‘anti-pattern’ which is neatly summed up in the title, “Too much of your application is about interacting with external resources.” Well, neatly summed up except that I think it would be better described as “if you can’t test your application at all without connecting to live external resources, you’re kinda f%cked.” The items that he lists are things I agree with, such as:
The majority of ‘unit tests’ require a database, or web and application server to be up and running.
This is very true, and yet, it still seems to miss a fundamental point.
It’s all about the data and your external systems
I’m going to go ahead and make a claim that is just as provable as Patrick’s, which is that I think most bugs in software do not come from bugs within the software itself, but bugs that arise because of the data that is entered into the software from external sources.
Which is to say that you can unit test your brains out to your heart’s content (mixed metaphor anyone?) but it won’t actually prove that your software works. Why? Because oftentimes the ‘bugs’ that appear in actual production operations are due to the nature of the data that you get from your external systems and/or how your software interacts with the nature of the data you send to your external systems.
What everyone who emphasizes ‘unit testing’ or TDD gets right is that, *of course*, given assumptions about the data you receive, your software should behave in predictable and testable ways, and you should be able to, well, test this in predictable ways. That’s why interfaces are so neato keen. You build your system to use interfaces so that you can have a healthy set of tests where you can dictate the input data without having to have to connect to external systems to do so, and can do so in an automated, predictable, reproducible fashion. Since *so* many systems can’t do this as they are built, emphasizing unit testing, TDD, or whatever gets you on the right track to fixing the sort of issues that arise. I’m totally on-board with that.
But, building your systems to get you to this level solves only a subset of the issues that software development faces. What really matters, in the end, is finding out what will happen when you actually connect to external systems and run your software, and integration tests are the only way to really figure this out.
I’ll give some more specific examples in a second, but imagine an incredibly vague, abstract scenario like this:
1) Some data comes in from an external system.
2) Your ‘receiving application’ does stuff with that data.
3) Your ‘sending application’ sends the data-with-stuff-done-to-it to an external system (often different from the system in step 1)
4) The external system does stuff with the data-with-stuff-done-to-it and then sends it back to you.
5) Another ‘receiving application’ takes that data and does some other stuff.
Rinse and repeat various steps.
You can unit test step 2 by using mocks/stubs/whatever to dictate the shape of the data that you assume is going to come in via step 1. You can unit test step 5 in a similar fashion, and you can unit test step 3 (more or less). You *cannot* reliably and predictably unit test steps 1 or 4, because you cannot reliably and predictably test what external systems will *actually* send you, by definition.
And yet, steps 1 and 4 (which are multiplied when you are dealing with multiple external systems) are what actually determine if your software works in Production, which is really all that anyone cares about.
Let’s talk about some examples.
External systems are finicky mean things
Over the last several years, the types of clients I’ve worked with have involved either e-Commerce systems or financial systems. ‘Financial systems’ is a bit vague, so for the sake of the discussion, let’s say that they involve supporting trading operations (but *not* real-time “gotta get this automated trade out to the market in microsecond” operations, which is another ball of wax).
When dealing with e-Commerce systems, you have to deal with product feeds, tax feeds, inventory feeds, changing APIs etc. etc. etc. When dealing with financial systems, you have to deal with company information, holding information, asset information, changing APIs etc. etc. etc.
Let’s talk at a high level about some of these things:
1) In an ideal world, when an external system changes the data that they send you, either in terms of format or schema, you will know this ahead of time. This doesn’t actually happen as often as you would hope. In an e-Commerce system that I’ve worked with recently, the external source of tax information added a column which required a change in the internal logic of the ‘receiving application’ to act upon it. We discovered this, of course, when the external source actually started sending the new data.
2) In an ideal world, when an external system claims to support a new API, you should be able to use the written documentation of the new API to change how your ‘internal’ applications work and send new data based on the new API. Then you discover that they support the new API but only in their UAT environment, but not PROD. Or only in PROD, but not UAT.
3) You’ve been sourcing data for domestic asset information from certain columns sent to you by the external system, and things work swimmingly. And then some international asset information starts coming in, and it turns out that you really should have been sourcing the data from totally different (but related) columns, but you didn’t know it at the time.
These are just a few of the scenarios I’ve faced recently, there are many others. The (I hope) obvious point is that you have no way of knowing from unit tests that your ‘internal’ software will actually work in PROD or UAT, until you actually interact with the external systems. And yet these are the common causes of ‘bugs’ that cause the production issues that you have to deal with, oftentimes at 3 in the morning.
How to setup your Specs
What I typically do when designing software is to start with Specs. ‘Specs’ is just another word for ‘well crafted tests’, where they are well-crafted from agreed upon design requirements, hopefully designed by the business users. I create separate code to deal with ‘Unit’ and ‘Integration’ specs.
‘Unit’ specs test your internal business logic (whatever there is of it) and also test that you have wired up things properly. Since I’m a cqrs fanboi, I tend to create code that follows very predictable patterns, such as, query hits service hits repository hits dataAccess, blah blah blah. My ‘Unit’ specs define the path of the code and that the right methods are called in the right places. Where I have intensive domain logic, I create specific specs around that logic, and this is where you do all of that mocking/stubbing/whatever stuff that we all know and love to know that given scenario A, we get output B, or whatever.
‘Integration’ specs actually hit your external resources. Given that I send external resource data A, data B should be returned and returned in the format and data I expect, or whatever. These are the tests that actually matter. The actual interaction between your ‘internal’ software and the external resources is what determines if your software actually works.
Unit tests are good things. Software systems that are built around predictable inputs that pass unit tests are good systems.
But at the end of the day, integration tests that actually hit your external systems are more valuable. You can never predict for certain how your external systems will behave until you actually hit them, and to a certain extent, there’s no integration test that will protect you completely. Vendors do all sorts of wild and crazy things. But integration tests are what tell you is the current state of your software.