Posts
1150
Comments
891
Trackbacks
1
The Importance of Integration Tests

The ‘red-green-refactor’ crew often seems to denigrate the importance of so-called ‘integration’ tests.  This isn’t always on purpose, but the idea seems to be that integration tests are ‘slow.’  As a matter of fact this is often true.  Integration tests do take longer, because you have to hit those databases and browsers and whatnot, and that takes time.

But this is the actual application.  The application is not the series of unit tests, and can never be covered by them.  Your end users care only about how your application works in real time.  This is why you *have* to have tests to cover the actual user experience if you expect your tests to be relevant.

Here’s an actual example from Apollo 11:

The PGNC System malfunctioned during the first live lunar descent, with the AGC showing a 1201 alarm ("Executive overflow - no vacant areas") and a 1202 alarm ("Executive overflow - no core sets").[6] In both cases these errors were caused by spurious data from the rendezvous radar, which had been left on during the descent. When the separate landing radar acquired the lunar surface and the AGC began processing this data too, these overflow errors automatically aborted the computer's current task, but the frequency of radar data still meant the abort signals were being sent at too great a rate for the CPU to cope.[7]

Happily for Apollo 11, the AGC software executed a fail-safe routine and shed its low-priority tasks. The critical inertial guidance tasks continued to operate reliably. The degree of overload was minimal because the software had been limited so as to leave very nearly 15% available spare time which, wholly by luck, nearly matched the 6400 bit/s pulse trains from the needless, rendezvous-radar induced Pincs, wasting exactly 15% of the AGC's time. On the instructions of Steve Bales and Jack Garman these errors were ignored and the mission was a success.

The problem was caused by neither a programming error in the AGC nor by pilot error. It was a procedural (protocol) and simulation error. In the simulator, the astronauts had been trained to set the rendezvous radar switch to its auto position. However, there was no connection to a live radar in the simulator and the problem was never seen until the procedure was carried out on Apollo 11's lunar descent when the switch was connected to a real AGC, the landing radar began sending data and the onboard computer was suddenly and very unexpectedly tasked with processing data from two real radars.[8]

Thankfully, the real world experience turned out to be okay, but the important point from my perspective is that the real-world experience didn’t match the expected one.

Obviously, nothing I’ve ever worked on matched the importance of Apollo 11, but it is a common experience in projects that I’ve worked on that production bugs occur precisely because you have no way of testing what will happen in production due to lack of a testing that matches production.

I don’t want to suggest that it isn’t important to test developer code.  It is.  But, in the end, production code matters more, and that requires important and relevant integration tests.

posted on Tuesday, March 17, 2009 10:45 PM Print
Comments
Gravatar
# re: The Importance of Integration Tests
Troy Tuttle
3/18/2009 11:15 AM
I'm a red-green-refactor sort of guy, and I think integration tests are extremely important. I just think it is beneficial to segregate your fast-running logical tests from the slower-running integration tests. So, if you are working on getting some business logic correct in your domain layer, it is nice to have the domain tests return their results in 3 seconds instead of 4 minutes (with integration tests mixed in).

Gravatar
# re: The Importance of Integration Tests
jdn
3/18/2009 11:26 AM
@Troy

I wouldn't disagree with you about that. But some people seem to express the attitude that integration tests aren't really that important, or just something you do in the end, etc.

In fact, they seem to view integration tests the way that non-TDD people sometimes view testing itself as something you do at the end, after the build phase, etc. I think integration testing should be done all the time.

I also think you should test production itself, but that isn't always possible, and a much longer story.
Gravatar
# re: The Importance of Integration Tests
Troy Tuttle
3/18/2009 2:01 PM
@jdn
I agree. Integration tests should be written when the integration code is written.

BTW, did you just make a post on the Kanban yahoo board under "jdn3times"? Just curious.
Gravatar
# re: The Importance of Integration Tests
jdn
3/18/2009 2:25 PM
Yes.

Post Comment

Title *
Name *
Email
Url
Comment *  
Please add 2 and 1 and type the answer here: