Blog posts in English

Why the dichotomy of testing versus checking is the core of our craft

Please note: This post is being updated.
In his Why the testing/checking debate is so messy – a fruit salad analogy, my good friend Joep Schuurkes posts an absurd dialogue in which two persons become confused because they cannot distinguish between apples and fruit. He claims the dialogue could still happen if apples is replaced with checking and fruit with testing.
He is trying to show that in the same way that apples are a sort of fruit, checking is a sort of testing. And that discussing testing *versus* checking is bullshit.
I think Joep is wrong, and I shall discuss why and how here.
A little “versus”
The core of the discussion is the little “versus” between testing and checking, which Bolton and Bach insists on. And I insist on it too: It introduces a dichotomy, which is not only important, it is even necessary.
And it is necessary because it shapes our thoughts about testing.
To be precise, it leads us to think on a conceptual level instead of just an activity level. Once we accept the little ”versus” between the two, accept the dichotomy, we can start thinking about our craft. We are no longer forced to only think about the activites we do.
And just as important: We can distinguish our craft from something that it is not.
It’s like the way more and more people discriminate between leadership and management. Once you accept that the two are conceptually different, something interesting happens: A whole new understanding of the act of playing ”the boss” reveals itself.
In the same way, when we start discriminating between testing and checking, the way we talk about what we do as testers, change. And we change.
A humanistic and value producing view on testing has revealed itself to us through this dichotomy:
Testing was, but is no longer…
Testing is no longer a necessary evil, only done because programmers are sloppy, don’t read requirements and make mistakes. Instead, testing has become a craft, carried out by humans. A craft that adds value to the product, the organisation and society as a whole.
We are no longer little machines working under detailed instruction. We are testers, and therefore everything we do, our job satisfaction and even the value we produce, depends on this very dichotomy.
I will not let the confusion confuse me
So why the confusion? Well, I think the confusion arises because we confuse concepts with activities when we talk our daily, ambigous language.
As a tester, I carry out checks when I test, but when I do, the checks I am doing are elements in the testing and the whole activity is testing, not checking.
But if, on the other hand, I program a computer to run through a number of input combinations to a software program, have my program verify the results by comparing them to something ”expected”, and produce a report of boolean results on the basis of this, the whole activity of running that and distributing the report from the computer program is checking, not testing.
However, letting this confusion lead us to discard the difference between testing and checking would be a pity. The dichotomy is core to Boltons and Bachs testing philosophy. If I reject it, I have to reject more or less everything they say about testing.
And worse: I will have to give up my profession.

PS: Joep tweeted to me that the only dichotomy he can think of with regards to testing is between good testing and bad testing. To me, the difference between good testing and bad testing is about performance. Good testing produces more knowledge and better knowledge than bad testing. Bad testers can improve and hopefully become good through training.
I could imagine a dichotomy for good versus evil testing, though. Evil testing would be testing that knowingly produce incorrect results, doesn’t test, but make up lies, and communicates untrue facts.

PPS: Another implication of the idea that there is a dichotomy is that if we for a moment assume that there could be something called ”scripted testing” or ”automated testing”, then it would not really be scripted testing or automated testing, as in someone really following a script in deail or something is completely automated by a machine. Then it can only be checking.
This is not because of any definitions of the words checking or testing, but because of the idea of the dichotomy. In other words: It is only when we seperate the two, that they materialise as real concepts.
On the other hand: If someone, who has hired me as a tester, but is not aware of the dichotomy, or who will not accept it, comes to me and says: Please set up an automated test suite, I will accept the order and know that what he really asks me to do is to set up a tool assisted test: A set up, where a tool does checking and I, the tester and user of the tool, evaluates the output of the tool and commuincates the testresults. And adjusts the tool in a forthgoing process.
I think Bolton and Bach will – in their latest terminology – call this ”tool assisted exploratory testing”, but the word ”exploratory” is really not necessary.

0 replies on “Why the dichotomy of testing versus checking is the core of our craft”

Hi Anders,
I’m afraid you’re interpreting my blog post in way I didn’t intend. So thank you for engaging with me in this discussion and making me think better. 🙂
You wrote:
“He [Joep] is trying to show that in the same way that apples are a sort of fruit, checking is a sort of testing. And that discussing testing *versus* checking is bullshit.”
I am not trying to show that checking is a sort of testing. What I am trying to show is that checking and testing are on different conceptual levels, because checking is a part of testing. Checking is one of the many activities one may do while testing.
Hence the fruit analogy, of which you did convince me that it’s less apt than I initially thought. I can say: ‘Apple’ is a member of the set of fruits, just like ‘checking’ is a member of the set of testing activities. And this illustrates the difference in conceptual levels. However, to make that work I had to say ‘testing activites’ instead of ‘testing’. So perhaps I should have chosen a different analogy, because I see now that my analogy only works in a very limited and particular way.
With the remainder of your post you seem to suggest I believe that the distinction between testing and checking is bullshit. This is not true. I appreciate the distinction, because it allows me (as you say) to think and talk more precisely about what I do as a tester. I do believe that the checking versus testing discussion is bullshit, as far as people discuss it as if checking and testing are the same kinds of things. They are not (see paragraph above), which is also the reason I wouldn’t call it a dichotomy. (On the other hand, in the context of your post I see why you would call it that. Anyway, this is not the most interesting part of our discussion.)
Finally in your PS you wrote: “Joep tweeted to me that the only dichotomy he can think of with regards to testing is between good testing and bad testing.” That was a bad tweet of me, because it requires more context than fit in a few tweets. So here’s a brief clarification. I hope you don’t mind, because it’s tangential to your blog post.
It’s virtually impossible for a human being to do only checking without doing any testing at all. Now that testing may consist of 99% checking and the remaining 1% may be very poorly executed, but that still means this person is doing testing, not checking – albeit bad testing. So when characterising a person, I don’t see any value in describing someone as ‘a checker’ versus ‘a tester’. When characterising an acitivity, I do see value in checking versus testing – although I do wonder if in many such cases we couldn’t use a more specific word than ‘testing’.

Thank you,Joep
Thank you for your reply. Thank you for replying in the way you do.
I have an apology to make. I apologise for not approving your comment for display here until now. It was only a press of a button, but I forgot. Shame on me! I guess, in this context, it helps me forgiving myself that I was simply too busy testing and leading and managing testing, but still.
It seems we agree completely, but fly into the discussion with slightly different terminology and understanding of the words we use.
And perhaps, this is the whole cause of the “messy discussion”, “the bullshit” and everything, but it might also the source of reflection and a deeper understanding?
I love the discussion, but need time to think.
Thank you, Joep. I like this 🙂

Leave a Reply

Your email address will not be published. Required fields are marked *