News in English Nyheder

Join our Panel Discussion: Engineering Quality in Your Stakeholders’ Point of View

I’m having a virtual panel discussion on April 9th with Venkat Ramakrishnan of Ministry of Testing Bangalore. We’re going to talk about quality engineering, specifically about quality as perceived by the stakeholders of the product or service. Follow the link to register – or read on if you want to know a little more!

Here’s an exercise for you in two simple steps:

1) Think about the project or system you’re working on. 2) Then try to put yourself in your users’ shoes and say this out loud:

The test of the system is the satisfaction it gives me. There isn’t any other test!”

Do you feel worried, happy, hopeful, or confident? All three?

I recently worked on a project where we drew a definition of quality as a three layer pyramid. In the middle we put what we knew important about the system: The calculations, decisions, and processes it was implementing had to be correct and follow the rules defined for the domain in which the system was to be implemented.

Below that level we elaborated the many requirements for the system from its specifications.

That’s quality, you might say, and that was in fact our first intuitive reaction: If the system is correct, adhere to requirements and rules, then that would mean that the system is of good quality.

Of course there would be other important things too: Usability for example. But usability would be consequential of correctness since users are important for correctness as they are inputting data into the system, and otherwise using the system.

However, we thought a little more about that, and then we decided to put something else on top of all the correctness and requirements: We put “Happy Users” on top of the pyramid. Everyone in the project wanted happy users, and it was clearly more than a mission. “Happy users” is a quality of the system.

This took some courage as making users happy is really difficult. Robert Pirsig defined quality as the event in which the subject realizes the object. Put in daily langauge terms what he meant was that quality is cognitive, relative, even emotional, and that quality is something you experience and probably even feel, it in a moment working with a product.

Quality is thus not in the product in itself, but something that happens.

Will correctness make a user happy? I sure hope so! But I know from myself that sometimes I’m not happy with correctness. I can become frustrated when people follow rules. Yes, you read me right.

For example I remember the times when I have received a parking fine because I did not notice the parking rules in the area where I was parking. My first reaction has always been anger against the meter man, then anger against the person who hid the sign, and finally anger directed against those who made the rules, put the signs up where I did not notice them, and hired the meter man to issue the fines.

Pirsig also said:

The test of the machine is the satisfaction it gives you. There isn’t any other test. If the machine produces tranquility it’s right. If it disturbs you it’s wrong until either the machine or your mind is changed.

He was talking about a motorcycle, but it applies more or less universally, I think. The final test is in the satisfaction.

Satisfaction and happiness depends on a lot more than corretness. It depends on things like implementation, clarity, communication, reflection, and education. Just to name a few things that comes to my mind when I think about what helps me when I’m trying to make myself happy about something. For example being accepting that I got myself a parking fine. What are my feelings? Why might I feel them? What can I do? (Probably nothing except paying and moving on with other, more important things in life.)

So think of the motorcycle as a metaphor for the system you’re working on and then try to relate to the following two statements: The test of the system is the satisfaction it gives the users and stakeholders. There isn’t any other test.

Please join Venkat and I to discuss if there are ways to engineer happy stakeholders in software systems, and what the role of testing might be in this.

Registration for the event is necessary. Please go here:

Quality Engineering In Action: A Panel Discussion

Friday, Apr 9, 2021, 5:00 PM

Online event

15 Members Attending

Please note: RSVPing here is not sufficient. You need to register at the following link to attend this free event: About the talk: “Achieving superior Quality has been focus of organizations for decades. A lot of methodologies and processes have been designed in the past, and have been followed to a…

Check out this Meetup →

Blog posts in English

Quality and the Art of Motorcycle Maintenance

Robert F. Pirsigs “Zen and the Art of Motorcyle Maintenance” is a classic about the authors’ motorcycle trip across North America with his son and two friends. It is also, however a philosophical dissertation about quality.

The trip through the changing landscapes during which he experiences and rediscovers himself, and heals his relation to his son is a strong and fascinating story. I’m an avid cyclist and I can deeply relate to the value of riding through landscapes, and letting the changing landscapes shape both thinking and reflection. This part of the book has a function too, however, as it serves as underpinnings for his dissertation. The book is fascinating and beautiful.

The motorcycle plays an important role in the book being a piece of technology that requires constant care and maintenance. But the motorcylce is also a medium.

Thanks to technology, we can experience ourselves in new ways. It’s fair to say that technology extends us. Pirsig rides and maintains his motorcycle, but the motorcycle also physically drives his thoughts and reflections, and eventually helps him heal himself. The motorcycle is thus both an object which Pirsig is working with, and a subject working back on him.

Pirsig identify two different views on things like motorcycles: First is the classic view, rooted in Socratic thinking, where things are understood by their components and individual elements. Second is a romantic view where the emotions and experiences are allowed to stand alone.

Through the first view on things the motorcycle can be maintained and tuned, the trip can be planned and executed, and the relation to his son and himself can be explored. But it is only through the second view that he can really understand what the motorcycle, landscape, and relations are in themselves. The two views are complementary.

Technology has evolved since Pirsig published his book 46 years ago. Both in the classic and romantic views modern information technology extends and affects us in ways that technology could’nt in the 60’s. Also, modern information technology is highly dynamic. While a motorcycle is a machine requiring materials and tools, and it takes time to perform even the simplest maintainenance job, a software system is pure design and can be changed in a moment.

People (like myself) working with software and IT systems often talk about quality as something that is built in: Cast in metal. Welded or painted on. Written in the code. Configured.

It’s not totally wrong, of course, but it reflects a purely classical view where technology is understood by breaking it down and analyzing it, and we assume that’s it: Once we’ve assembled the components, the job is over. This view is wrong. Systems theory, for example, teaches us that: Quality, in systems theory, is an emergent property.

Pirsig starts out personally favoring the classic perspective, but as he drives through the landscape he understands there’s also the alternative romantic view, and that it is just as valid. This, I think, is what leads him to start searching for a joining factor between the two perspectives.

Quality is an event in which the subject realizes the object, he says. Pirsig then concludes there is Zen at the core.

I read Pirsigs book during the first lockdown. I’m not sure I allowed myself to be completely absorbed by the book as I found the whole lockdown- and Covid-situation quite stressful. I’m looking forward to rereading one day.

However the book kept me thinking and has opened my mind to a deeper understanding of the nature of Quality.

Pirsig finds Zen at the core of quality. I think that might reflect his classical preference: There has to be an inner component, an object from which Quality emerges.

I think he should have been more careful identifying and naming it. At the core of Quality there could just as well be no object, only a subject.

After all, our very existence and personal abilities to do things like judging, deciding, acting, speaking, and even to think and reflect, seem to prove that there is something about us as humans that doesn’t necessarily need a name to be recognized, but still allow us to experience concepts like Quality as real and true to us. Putting Zen at the core of quality is a somewhat speculative theory.

But that’s just a minor objection about the book. I really enjoyed reading it and having my thoughts stimulated. I even wrote a poem in my diary after finishing it. This is an edited version:


It is sensibly comprehensible but objectively slippery

Did I find it?
Did I recognize it?
Did I shape it?
Or did I prove it?

Quality is the event in which
the subject realizes
the object.
I realize that.

Returning and revisiting
over prairies and mountains
through rain and snow always
anticipating, planning, solving,
predicting, projecting, caring.

Expectation turns into experience
A priori becomes a posteori
Time passes and in hindsight it is
dissapointingly often
encouraging sometimes
more complicated.

Indivisable, tacit, not now, here, there
A Never
A Scene
Where stories will never play and events never unfold.

Did I prove it?
Or did it prove me?
Uncontrollable and undivisable
Acceptable and passionable.

Quality is that
Which sets me responsibly free.

Blog posts in English

Quality is…

In a recent post, Anne Marie Charrett discusses quality as an emergent property.
I see quality as a term for value, and it is in itself complex. It can be read as being synonymous to aesthetics, beauty, and probably even to complexity, but it can also – as a term – refer to existential properties such as safety, existence, life etc: doing no harm is considered a fundamental quality in health care, for example. In the real, especially in the reality of projects, quality is indeed emergent: The product begins its life as an idea, we assemble the parts, and in the end the whole becomes something of greater value than the sum of the individual pieces.
But no matter how it emerges, quality, to me, is fundamentally tied to the human. We can define metrics for quality if we work hard, but even the best metrics will never be more true that the humans who relate to quality. This perspective can be debated, but as a premise for my work, I don’t want to engage in such a debate. Rather, I’m interested in debating how to make the human perspective operational in my work with quality and testing software systems.
And for that I have 12 models that can be applied, on three levels in four dimensions each.
I draw it like this (explanations follow, and yes, it’s a pyramid):

First order quality

Quality is something to someone, somewhere at some time. What is that something then composed of. Based on Ole Fogh Kirkeby’s “Greek Square” as described in his books on proptreptic value based coaching, Cynefin’s domains and Kant’s categories of terms, I think of quality as something on a compass disc with four corners:

  1. Quality as factual, obvious and true. Correctness is a quality in arithmetics, and subjecting a pupil to a number test with a number of addition assignnents allows us to evaluate how good she or he is adding numbers simply by comparing based on the fact that given a+b=c, given a, b being numbers of a known number system, c is a defined value. Sometimes, facts require more than algorithms, as in evaluating “North is that way”, but the principle is still the same, that a presupposition can be proven right or wrong given a concrete context of testing using someone, somewhere at some given time.
  2. Quality as relational, as when we for evaluate fairness and agreement with something or someone. Still in the context of someone in a given time and place, this kind of quality is relatively simple: We can ask the person, or we can put ourselves in her shoes, understand her, and evaluate for her. Agreement can probably always be evaluated mechanistically, but we still have to make sure that we know what to evaluate against: is it under our control, or a moving target? Fairness is more difficult, as even legislation needs to be interpreted in terms of both its written word, and its intention and ‘spirit’.
  3. Quality as an emergent, aesthetic, complex property is what we see in software Projects, especially if we look at the human values of the software. In that, people experience wealth of experiences existing on a continuum ranging from bad to good, ugly to beautiful.
  4. Finally, we sholdn’t rule out the safety aspect of quality. Even when we consider quality in the most personal and individualistic view, one that is rooted in a single persons’ experience with the product, we should not rule out the anxiety present: Will it destroy my idea (WordPress’s App deleted most of this blog post while I was editing it)? Will it destroy my day? Will it destroy me? As I mentioned above, a quality in healthcare is not harming patients. It’s fundamental, but it’s important, and a basis for ensuring that cure can emerge.

That was the first four models of quality. There are four more one level up:

Second order Quality

When I was a child, I played for my own sake. The quality of my play emerged, but it was personal. As I grew up, I began playing with others, and for them: At work, and as a parent, what I do is more serious: Part of growing up is learning to do things for others. Part of learning product engineering (which I did) is learning to engineer things for users.
The second level in the model is tied to the group of users, the team, an organization, the stakeholders etc. Quality is social here, but still bounded. But it can refer to the different domains in which the product has value: In the project, to sales, and to users. There are several ways to move to this level of abstraction, but we need to know what kind of abstract level we’re moving to for this to work.
My experience is that we can only move our thinking to this level if we know what we’re addressing. It doesn’t require absolute certainty, we can create props to represent users by describing them, or we can describe users in abstract terms. But there needs to be some kind of concrete knowledge available about the target of our idea of quality before we can talk rationally about quality at this levels.
But once we have that, we can start working with quality within the same Compass system of fundamental quality values I described above: Factual, relational, and emegergent quality, and safety. (Or as they are named in the Greek Square: The true, the just, the beautiful, and the good.)

Universal quality?

At the top of any pyramid is a smaller pyramid. In this case, this is where we move to the universal perspective, the quality that applies to everyone, everywhere, at all times. This would be God’s view on quality.
Factual qualities are often easy to work with, even at this level. Take math for example, which doesn’t depend on neither time, space or humans. Math is a thing in itself, it seems. Therefore it makes perfect sense to judge an attempt to solve an equation as universally good or bad depending on simple correctness:
2 + b = 4 <=> b = 3
The above is obviously incorrect and false.
It get’s much harder when we move further around in the value compass: Can we talk about good relations in a universal context? Is “the just” a universal thing? It’s not as easy as the above example, and I think this is where we really need to call in an expert who has done research in the quality of relations, or someone who knows about law and justice.
Even worse about emergence, complexity, aesthetics: Is there such a thing as universal beauty? I think there is research supporting this, but as a tester, I’ll skeptical about it: I’d rather test with a group of potential users, i.e. stay on the lower level, than employ rules for verifying aesthetic or emergent properties of a thing. Also, I would always suspect such rules are heuristic in nature, not universal.
Safety seems more obvious: Avoid chaos at all cost, and chaos is a reasonably well defined thing, I think, perhaps even mathematically.
But still: In the real it’s complicated. The Picture below is of the UN Building in New York. This is where universal rules for everyone on the earth are negotiated, but even those that have been decided on are often up for debate: We agree we want peace, but how? We agree we don’t want climate change, but at what cost, and who should do it? It’s hard, and as a tester, I want to avoid talking about this level, except as an abstract level I should avoid expressing myself too concretely about.
But that’s all right: I can say a lot of other things. Most of the quality pyramid is readily accessible in my work testing software and the relations real humans have with it.

Blog posts in English

Immanuel Kant and the Hallucinating Tester

Quality is an illusion. That may seem like a bold statement, but there is a deeper truth to it that I will discuss in this blog. I’ll also discuss how we can approach the real.
We can think of testers as doctors, scientists, or researchers whose job is to research, explore, or examine some software, gather, analyze, and communicate factual knowledge based on observations.
But science teaches us that when we research and observe things, including software, what we “see” is not reality. At TED 2017, University of Sussex neuroscience professor Anil Seth called what we see “hallucinations”.
This gives the hopefully scientific tester some severe epistemological challenges: As she is a person, and is hallucinating, how can she (or we) trust her observations?
The problem for her is that the images that she experiences as real are a synthesis, an intuitive product of her observances based on a minimal amount of sensory data. The critical mindset is important in testing but doesn’t help by itself.
Fortunately philosophy has a solution for her (and us). Before I explain it, let me share a daily life story about intuitive illusions and assumptions.

Walking on Black Ice

I was out walking my poodle Terry a few days ago. A car came against us, but as we were on the sidewalk and the car on the road, the situation was intuitively safe.
Unfortunately, my intuition turned out to be wrong as only a moment later my foot slipped on the sidewalk and I realized that the wet road was not wet; both the road and the sidewalk were covered in black ice.
When another car approached I was aware of the danger, and made sure to keep myself and my dog safe.
There could be a moral in this story about always being cautious about cars and roads, but it might end up in over-cautiousness of the type that grandmothers sometimes impose on their grandchildren.
Instead I consider it a reminder that we don’t see things as they are: The road was wet until my foot slipped and I realized it was icy.
Already the Stoic philosophers in Rome 2000 years ago had figured this out.

Immanuel Kant’s Model of Mind

In 1781 the German philosopher Immanuel Kant published his mammoth work Critique of Pure Reason in which a key concept is the transcendental, which can be thought of as a bridge between the real and the hallucination.
Let me explain: Something that is only realized by intuition, dreams, myths etc, and which doesn’t link to experience, is transcendent. Something realized by pairing sensing and experience is transcendental.
Kant’s model is simple and straightforward, as Kant was pedantic, but it still needs some explanation:
Outside us is of course the objects which we sense. Kant calls them “the things in themselves”. It could be the road I was walking with my dog.
Kant thinks of us as rational beings who act on the basis of the thing in itself, and that has caused much debate. Skepticism will claim that the thing in itself is not available, and that there is only the subjective experience. Logical positivism will claim that the thing in itself doesn’t exist at all. Realism will doubt the subjective. We can probably all appreciate that the always rational human doesn’t exist.
But Kant’s bridge is interesting. What he says is that even though “the thing in itself” is not available to us, we can still say rational things about what we see.
So the mind is connected to the real in a way so we can gain and share experience. Does it sound weird? In a way it is, but Kant’s arguments has generally stood the test of time and critical philosophers – and even neuroscience.
So let me tie this to testing.

Belief as a Premise

There are different ways to test: In exploratory testing, we do it interactively by operating and observing the software. In automated testing we “outsource” the actions to another piece of software, and our task is then reduced to making sense of data from the tests and suggest and possible implement changes to the test-software. Scripted and chartered testing sits somewhere in-between the two “extremes”.
However,no matter how we practice testing, we need to make sense of what is observed. And since observing is subjected to sensing, the only thing we have available is our intuitive image about the thing we are testing.
James Bach is quoted as saying “Belief is a sin for testers.” I like the quote as it is an important reminder to be careful what we think: It’s not reality. The road might not only be wet. The software probably doesn’t always do what it did this time. I probably missed something. My mind is hallucinating.
So with a bit of wordplay in Kant’s home language, German, I’ll say that “die Sinne ist die Sünde.”
Our senses are the sinner, but as they are also our only hope to see some tings about reality belief is not an option. It’s a premise.
But since we know, we can establish the transcendental: Think the real rationally by testing our beliefs.
In other words: The realist approach to testing is to test the product. The transcendental approach is to test beliefs.

On Common Terms

There is something missing in the above as so far I’ve only talked about sensing, imagining, and experiencing. The brilliant part of Kant’s philosophy is that he explains how we can collect experiences.
Kant develops four categories of terms that we think by, and argues how they are given to us a priori, i.e. before we experience anything. He argues how they come from the existence of time and space. Back in his time Newton had just published his theories. Today, we’ve progressed, and it probably makes better sense to think of the terms as a result of the experience of space and time.
But what’s important is that although our experiences vary, we’re on common terms, so to speak.
This is important as it means we can think and express our knowledge about experiences generally.
Let me give some examples: I told you about the black ice on the road above, and while cannot be certain what I said is true, you can understand my experience. I can also share a testing problem, and we can imagine solutions together. I can try them out afterwards, and share experiences with you. We can even talk about testing in general, and imagine solutions to certain testing problems in general.
In other words: The terms allow us to relate, connect, discuss, collaborate, learn, reflect, prospect etc.
This makes the transcendental model of experience complete: We can sense, imagine, think, and express our thoughts into words and actions that we can share with others, who can do the same.

The Two Things I Want to Say

So what do I want to say with all this? I want to say two things:
The first is that yes, we are trapped in hallucinating minds. We might theoretically be able to escape them if we subject our testing to strict scripted procedures, make sure what we do is repetitively accurate, and only communicate what we can verifiably record and therefore objectively observe. But we’ll essentially be turning ourselves into machines and miss intuitive and tacit knowledge. And one way or another, we’re still stuck in a mess where at every and any judgement and decision made will be based on hallucinations.
But we’re not lost as we can explore the product and our intuitive ideas about it transcendentally, i.e. by realizing that both are in play when we test. Although we can’t get access to the “thing as it is”, we can experience it. Our expeirences do not have to be transcendent, i.e. disconnected from real, but can be transcendental.
And this is the second thing I’ll say: Since we are not alone in the trancendental, our roles as testers become clearer.
People are different, but I think a fundamental, perhaps even genetically coded, qualification for testers is to be sensitive people with intuitions which are easily disturbed by reality. On top of that, great testers need critical thinking skills, i.e. courage to doubt intuitive illusions, and creativity to come up with test ideas useful in the context. The rest is about interaction and communication with teams and stakeholders so that the good hallucinations about the software that we develop through our testing are shared.

Testing Transcendentally

In the spirit of Anil Seth, the neurology professor, let’s be honest: Software quality is a hallucination.
We can’t escape our minds and the apparent mess created by the hallucinations we think of as real. But we can experience quality transcendentally by testing.
To me testing is not so much an exploration of a product.
I see testing first and foremost as the transcendental practice of exploring of our own, and our team colleagues’ and stakeholders’ hallucinations about the product.