Blog posts in English

The Communicative Power of Counting

Michael posted the following two comments on Twitter shortly after I published this post:

There’s nothing wrong with using numbers to add colour or warrant to a story. Problems start when numbers *become* the story.
Just as the map is not the territory, the numbers are not the story. I don’t think we are in opposition there.

I agree, we’re not in opposition. Consider this post an elaboration of a different perspective – inspired by Michaels tweets.

Michael Bolton posted some thought provoking Tweets the last few days:

Trying to measure quality into a product is like measuring height into a basket ball player.
Counting yesterday’s passing test cases is as relevant to the project as counting yesterday’s good weather is to the picnic
Counting test cases is like counting stories in today’s newspaper: the number tells you *nothing* you need to know.

Michael is a Tester with capital T and he is correct. But in this blog post, I’ll be in opposition to Michael. Not to prove that he’s wrong, not out of disrespect, but to make a point that while counting will not make us happy (or good testers), it can be a useful activity.
Numbers illustrate things about reality. They can also illustrate something about the state of a project.
A number can be a very bold statement with a lot of impact. The following (made up) statement illustrates this: The test team executed 50 test cases and reported 40 defects. Defect reporting trend did not lower over time. We estimate there’s 80% probablity that there are still unfound critical defects in the system.
80%? Where did that come from. And what are critical bugs?
Actually, the exact number is not that important. Probabilities are often not correct at all, but people have learnt to relate the word “probability” to a certain meaning telling us something about a possible future (200 years ago it had a static meaning, by the way, but that’s another story).
But that’s okay: If this statement represents my gut feeling as a tester, then it’s my obligation to communicate it to my manager so he can use it to make an informed decision about whether it’s safe to release the product to production now.
After all, my manger depends on me as a tester to take these decisions. If he disagrees with me and says ”oh, but only few of the defects you found are really critical”, then fine with me – he may have a much better view of what’s important with this product than I have as a test consultant – and in any case: he’s taking the resonsbility. And if he’s rejecting the statement, we can go through the testing and issues we found together. I’ll be happy to do so. But often managers are too busy to do that.
Communicating test results in detail is usually easy, but assisting a project manager making a quality assessment is really difficult. The fundamental problem is that as testers, by the time we’ve finished our testing, we have only turned known unknowns into known knowns. The yet unknown unknowns are still left for future discovery.
Test leadership is to a large extent about leading testers into the unknown, mapping it as we go along, discovering as much of it as possible. Testers find previously unknown knowledge. A talented ”information digger” can also contribute by turning ”forgotten unknowns” into known unknowns. (I’ll get along to defining ”forgotten unknowns” in a forthcoming blog entry, for now you’ll have to beleive that it’s something real.)
Counting won’t help much there. In fact it could lead us off the discovery path and into a state of false comfort, which will lead to missing discoveries.
But when I have an important message which I need to communicate rapidly, I count!

0 replies on “The Communicative Power of Counting”

Hi Anders,
Excellent post, I also relate with your sentiments that sometimes it does come down to a numbers game due to time constraints or the inability of management to read more than a paragraph or listen to you for more than a few minutes.
The use of visuals in reporting (something which I’ll need to write about myself at a later date) can greatly improve such statistical reporting into something more meaningful to the documents or visual communication debriefs target audience.
So for defects I can relate this easily. For test cases I think I’m still on the fence here, as it’s dependant on the type and style of test cases your team executes.
For example one feature area of our product could be covered by one to three of my lean test cases when compared to say fifty of another tester at another companies test cases. Most likely mine will give better coverage due to my approach in designing them, but how does a number convey that?
If we look at numbers split into severity of defects it does become more meaningful. For test cases I’d still can’t see their value, unless you can easily convey coverage provided by each.
Anyway rather than ramble on, I’m glad you shared your thoughts, as always they’re very interesting.

Thanks a lot for your comment, Darren!
Oh yes, I think you should certainly remain on your fence regarding test cases. My point is that it’s not important how or what you count – the important thing is the message you want to convey: Count branches in your mind map, rows in your test data Excel sheet, individual findings in your Rapid Reporter log – or minutes of your test session. It’s not important.
Visuals are certainly very powerful, but usually don’t survive well in rapid reporting to management through e-mailing. But if you’re into it, I’m rereading Edward Tufte’s Envisioning Information at the moment – very recommended!
I’ll look forward to reading your expeirences about visual communication.
PS: Thanks for pointing out that defect on my blog… I’ll have to write about that brainual memory failure real soon.

Leave a Reply

Your email address will not be published. Required fields are marked *