Blog posts in English

Acceptance tests are not enough!

Acceptance testing is a key method in Agile. One way of defining acceptance tests are Gojko Adzic‘s ”Specification by example” paradigm which has gained quite a bit of momentum lately. I personally found it to be both refreshing and nice when I heard him present it at Agile Testing Days 2009, and I also found his book Bridging the Communication Gap a nice read.

Photo of Gojko Adzic demonstrating communication gaps in his keynote presentation at Agile Testing Days 2009
Gojko Adzic demonstrating communication gaps at Agile Testing Days 2009

I’m sceptical of the concept of acceptance testing. Not because verification of agreed functionality is not a good thing, but because it tends to shift attention to verification instead of exploration.
This will shift attention from testing to problem prevention. Is that bad, you ask? Isn’t it better to prevent problems than to discover them?
Well, most people think ”why didn’t I prevent this from happening?” when problems do happen. Feelings of regret are natural in that situation and that feeling can lead you into thinking you should improve your problem prevention. And maybe you should, but more examples aren’t going to do it!
Real testing is still necessary.
To explain why, I’ll consult one of the great early 20’th century mathematical philosophers: Kurt Gödel. In particular his first incompleteness theorem. It says that no consistent system of axioms whose theorems can be listed by an “effective procedure” is capable of proving all facts about the natural numbers.
What does this mean to us?
It means that we will never be able to list all things that can be done with this particular set of data.
A specification is a kind of listing of ”valid things to do” with data, thus Gödel’s theorem teaches us that there are infinitely more things to a system than any long list of requirements. This also applies when the requirements are listed as examples.
If you’re in the business to deliver products of only ”agreed quality” to a customer, you can be all right only verifying things which are explicity agreed. If something goes wrong you can always claim: ”It wasn’t in the specification!”
But if you’re striving for quality in a broader sense, verifying that the system works according to specifications is never going to be enough.
Gojko has made a good contribution to agile. Examples can be useful and efficient communication tools, and if they are used correctly they can help making users and other interested parties better aware of what’s going on on the other side. His contribution can help bridge a communication gap. It can also produce excellent input for automated unit tests.
Just don’t let it consume your precious testing time. The real testing goes far beyond verification of documented requirements!
If you want to learn more about this, I recommend you sign up for one of the Rapid Software Testing courses offered by James Bach and Michael Bolton.
Photo from Rapid Software Testing course in London November 2010 as offered by Electromind
Michael Bolton with one of the many interesting and challenging test exercises at the RST course

0 replies on “Acceptance tests are not enough!”

Hej Sigge
Thanks for linking to your recent post. It’s an interesting read (- comments too!).

Acceptance testing is a horrible concept. What happens of the users “accept” the software and some time later a major problem is found? Are they stuck with it?
What if the users “reject” the software due to a defect or missing feature? Can they walk away from the software and find another solution?
It’s not about acceptance and rejection. It’s about making the software do what the users need in a simple, reliable way.

Hi Vin,
Thanks for your comment!
I actually don’t agree about acceptance testing being horrible in general. It can certainly be misused, but I strongly believe in Quality Control of software, and acceptance testing is one way of controlling, e.g. as a QC on the customer side. Many (most?) software contracts specify some kind of “acceptance test” to be conducted before the customer accepts the delivery, usually as part of the legal handover of ownership of the system.
But you have a very valid point in saying that it’s not about acceptance/rejection. There’s a lot more to it than that. The concept of “quality” exists only in the grey area between those two extremes.

Leave a Reply

Your email address will not be published. Required fields are marked *