Categories
News in English Nyheder

I'll be at Let's Test

Let’s Test Conference is the first conference in Europe on context driven testing – it looks like it will be a really great event. I’ll be there, speaking on day 2 about one of my favorite subjects: Black swans – things that happen, even though everyone thinks they can’t:

Nassim Taleb introduced the concept of Black Swans in his influental 2007 book: Rare events, in principle extremely unlikely, but very dangerous – and they occur much more often than we expect.IT incidents are true Black Swans. Incidents are tricky to test for, and this is probably why they are rarely treated exploratorily in testing.
I would like to see that change, since with increasing complexity and the ubiqtous nature of IT systems, failures in todays’ IT systems are much more consequential than they used to be, and incidents are frequently the cause of great losses; in some systems they even put human lives at risk.
I will present my view on the ”art” of testing for system incidents. My view diverts fundamentally from the non-functional tests, robustness tests, and performance tests which are often carried out. Instead, and based on a systems perspective, I will introduce heuristics and models which will allow explorative testing and learning – even in the Black Swan Domain. There are no bullet proof ways to do it, but by understanding the nature of incidents and by being able to identify vulnerabilities in the context in which testing is taking place, I beleive that you as a tester will be better able to focus your exploration and observation on what’s important: The system vulnerabilities.

Let’s meet there!

Categories
Blog posts in English

About Working Memory and Testing

Cognition is the word used to describe all the mental processes happening in our brains. Cognition relies on working memory, which is the memory we are using when we are solving problems or performing tasks.
Testers, for example, often rely on working memory to keep track of what we are doing and the observations we are making while we test.
Working memory is fallible. You will probably have experienced going to the fridge to check something, yet when you get there, you find that you have forgotten what you were lookoing for. It usually helps returning to the place where you first thought about it, but sometimes it’s gone completely. It can be quite frustrating!
Adults can usually hold up to five things in working memory at the same time over a time span of several minutes. Small children can only hold one. A friend of mine, who used to work in a nursery, told me about a small boy, who had just learnt to walk. He was good at it, but when someone called his name, he fell on his behind. Every time! It’s was quite cute, she said.
Some people have very poor working memory. My 13 year old son Aksel has Asperger Syndrome and a severe attention deficit disorder (ADD). His working memory is very poor. In school, he usually looses focus on what he is doing after less than a minute, unless something or someone is helping him. By using a lot of mental energy, he can keep focusing over about 10 minutes, but it exhausts him and he has to take a break afterwards.
Yet, he can build the most fantastic Lego creations, particularly on the computer, and be concentrated about it for several hours. He is also excellent in a go cart, where he can stay 100% focused for more than 30 minutes – and is still completely relaxed afterwards. I’m usually exhausted after just 10 minutes!
So what’s the difference?

The difference, I beleive, is that the Lego creation and the go cart isn’t loading his working memory: He does not have to remember what he is doing as it’s in the context. The Lego building application is even helping him keeping all the bricks in order: He doesn’t have to focus on looking for that missing brick, but can quickly browse for any brick without loosing focus on the thing he is building.
At school, Aksel and his teachers are working hard to find strategies for off loading his working memory.
Aksel and I share a lot of genes, and though my working memory is far better than his, it isn’t quite as good as I would like it to be. I’ve learnt myself a few strategies to overcome it, and it’s usually not any big problem, though.
But since testing, and in particular explorative testing, is very demanding on working memory, I sometimes feel working memory impaired and I have developed a few strategies for myself to overcome it.
Test scripting is one of those strategies: Scripts are working memory aids for me.
Scripting by mind maps, however, doesn’t work well for me. The linear script is probably easier to navigate for me. (I love mind maps for taking notes and for manuscripts when I’m speaking, though.)
Scrips are also excellent providers of a ”safe home” when I’m diverting off to explore something: I don’t have to worry about forgetting what I was doing. I usually don’t even have take notes, which is very good, since note taking is something I’m not very good at (though, with discipline, I have improved over the years).
Developing strategies to assist your working memory can a good thing for anyone, even if you have excellent working memory, but there’s no one solution that works for everyone: Some like scripts, some hate them.
Experiment and stick with things you find work for you: Handwritten notes, mind maps, scripts, drawings. And if you are in the mood, try to use your body: Most of our brain cells are allocated to interacting with our muscles and senses, and they can work for you too: Count with your fingers, move yourself (and your laptop) around, talk to yourself about what you’re doing, etc.
In case you want to learn more about the concept of working memory, I can recommend the following book. It’s about working memory in children (for education), but the concept is the same for everyone:
Working Memory and Learning: A Practical Guide for Teachers

Categories
Blog posts in English

More on testing terminology: Types of testing

Carsten Feilberg posted the following on Twitter yesterday:

Since I started translating ‘testing’ to ‘demonstrating’ when talking to non-testers like mgrs, it all makes much more sense.

The post sent my mind off thinking about types of testing: I realised that demonstrating functionality is a common task in testing, and that demontrating is an important task in the project I’m working on at the moment.
And after dreaming about this all night, I have come up with four types of (knowledge producing) work commonly associated with ”testing”:

  1. Testing
  2. Checking
  3. Demonstrating
  4. Assessing Qualty

Let me briefly describe each:
Testing has the objective of finding issues and knowledge about a product. You don’t do testing if you know everything you need to know about a product, so part of the ‘charter’ involved in testing (whether it’s written down or not) is that you are expected to find out things about the product that wasn’t known before: E.g. bugs. It might be better termed ‘learning’, as suggested by Michael Bolton.
Checking is something you do to verify that things are in order. A pilot goes through a check list before takeoff. Checking implies that you already know exactly what you are checking, so a check will only produce knowledge about the items on the checklist. In software, checks are often automated. As described frequently by Bolton, Bach, Weinberg and others, much of the testing work carried out in software projects is actually checking.
Demonstrating functionality is about showing that a product can do something – and how it does it. Most ”acceptance testing” and ”user testing” are actually demonstrations, and demonstrations are very useful when you need to document that something is ”ready for delivery” from supplier to customer.
Assessing quality is something I think deserves more attention: Testing is usually associated with the production of ‘bugs’ or ‘issues’, which do not by themselves say anytnhing about ‘quality’ of something: If you’re an optimist (e.g. project manager), you’ll tend to say: ”ah, that’s just one bug, they’re fixing it”, whereas pessimists (e.g. testers) will say: ”oh, this is all rotten, just give me more time to find all the other bugs”. Assessing quality is about qualifying either statement – or something in between.
These are four types of activities – are there more?

Categories
Blog posts in English

The many are smarter than the few: How crowds can forecast quality

This is a blog post which I’ve had underway since early May. It is about a new way of assessing quality. Let’s start with how we normally work:
Testers usually work alone or in small teams checking and exploring functionality, finding bugs, issues, and other artifacts. These artifacts do not by themselves say anything about the quality of the whole product, instead they document things which are relevant to quality.
In this blog, I’ll propose a different kind of testing, one which is organised in a way which is radically different from traditional testing – and which can produce a quite different type of result.
My inspiration is the 2004 book by James Surowiecki: ‘The Wisdom of Crowds‘ with the subtitle ‘Why the many are smarter than the few’. In the book, Surowiecki presents a thought provoking fact: That while some individuals are very good problem solvers or excellent forecasters, a diverse crowd of ordinary people can always do better than any single individual.
Surowiecki explains this in a very convincing manner and the book is an enlighting read. I find Surowiecki’s thoughts a welcome diversion from what most seems to be concerned about these days: The performance of the individual. Too often, we forget that most good solutions are not invented or implemented by any single person, but by groups of people. And that the performance of teams often depend more on the composition of the team than on the individuals in it.
As a tester, I enjoy working alone as well as in teams, but reading Surowiecki’s book made me think of ways to apply his thoughts to make quality assessments of a different kind than those traditional testing can make.
James Surowiecki: The Wisdom of Crowds
Let me start the description with an example of a question which traditional testing cannot easily answer, but which I think a new kind of assessment can:
A client approaches us with a product which is still under development and therefore not yet on the market. The client tells us that he needs a holistic quality assessment of the product and he asks us to provide the answer to a very simple question: Will it be a good product?
Though I can produce a wealth of information about a product I’m testing, answering this question is not possible by ordinary testing alone. I may be able to make up an opinion about the product based on my testing, and I can communicate this to my client, but it will always be a fundamentally subjective view.
And there is no practival way of assessing whether my view of the product matches that of the collective intelligence of the population of users of the future product. An expert in the field of forecasting product successes may do better than me, but in principle he may be just as wrong as I am – and the worst thing is that we will not know whether he’s right or wrong.
Humans are actually very good at coming up with answers to open ended questions: Quality is something that everyone tends to have an opinion about! But while a single human can (and according to Surowiecki will) make interpretation errors, Surowiecki points out that in a crowd, the errors will be evened out. Aggregated opinions can be a very reliable prediction of the quality of the finished product.
The crowd does not have to be a team of experts. Surowiecki points out that rather than focusing on maximizing the individual members’ domain knowledge and level of experience, the crowd should be put together to be as diverse as possible.
Obviously we have to supply some information about the product to the group – they can’t make up their minds about quality without knowing something about the product. Collecting information has to be done by someone and provided to group members. This is an important task which a ‘moderator’ has to do.
In the ideal situation, we will provide all available information the group: Prototypes, design documents, concept descriptions, ideas, diagrams – even code! The idea is to allow each individual member of the crowd use his own heuristic making up his mind about the question.
But that won’t work in practice. Asking all group members to read everything is just not effective. Besides, the documentation could lead them in wrong directions: They will focus on the most easily accessible parts and will avoid information for which they have to work a little to get to it.
So the moderator will have to make a ‘flat’ (as opposed to hiearachical) binder of different information from the product. What should it contain?
When I was learning sketching and drawing, I was introduced to the problem of drawing leaves on a tree or hair on an animal. I was taught a trick, which is to draw every 100th leave or every 10,000th hair accurately. It will then look correct to the viewer.
I suggest making the ‘information collection’ in the same way: Pick some documents, some diagrams, some code, some tests. Or even, pick some pages from some documents.
The idea is that the crowd members actually doesn’t need to see everytning – they only need enough to formulate an opinion. And then they should see different things, so we’re most certain that they will form different opinions about the system.
How about questions – what questions should we ask? We will have to ask them in a way so answers can be aggregated into a combined result. We may want ask them to give a score, which can then be averaged or in other ways analysed.
Surowiecki points out some important pitfalls that should be avoided. I’ll focus on what is often referred to as collective thinking. This is what happens when a group of people turns out to be ‘dumber’ than the individual members. A bullet proof way to get people to think in collectives is to let some members of the crowd influence other members: E.g. by putting a very charismatic person in the role of chairman or manager for the group. Surowiecki refers to several examples of how group thinking has lead to wrong decisions, and it is obvious that if we want to make an assessment which can be trusted, we have to avoid it. By all means.
So ‘voting’ should be secret, and we should generally prevent members from communicating with each other. If we do allow them to communicate, we should moderate the communication to ensure that individual members are not allowed to influence the opinions of other members.
Is crowd involvement in testing a new thing? I think so. I don’t think the concept has been described before.
On the other hand, many beta test programs have traits of it.
But where the crowd based quality assessments (or forecasts) can take place at any point in the development process, beta testing by definition takes place on an almost finished version of the product. And beta test programs produce the same types of results as ordinary testing: Bugs and other artifacts.
Holistic crowd testing is not an efficient bug factory. Its power is its ability to answer holistic questions about a product under development.
I’d like to set up a workshop in a forthcoming software testing conference where the idea can undergo further development. Let me know if you’re interested in participating, and I’ll let you know when and where.

Categories
Blog posts in English

Are we doing all we can to support young talents?

I’m reading Lee Smolin The Trouble with Physics. It’s a book about the crisis of physics, which has not produced new experimentally verified theories the past 20 years. All despite more people working with physics than ever before.

To me, this sounds like something I know from the testing profession: we are more testers than ever, yet we are not generally producing new knowledge about our contribution to software development and value generation.

You disagreee? Well I hoped you would, because there are people in our business who try – but they are not generally listened to. The software industry is in trouble, but so is testing.

Smolin clearly identifies what has ruined his profession: The fact that one area, which cannot be proven by experiment, is dominating physics: string theory. We have our own ‘string theories’ – tools and techniques which don’t deliver what they promise, yet are what everybody is talking about.

Smolin has one particular worry which I’d like to quote because I think if we can fix it, we will be much better off – in physics and testing.

“So the question […] I ask myself every morning: Are we doing all we can to support and encourage young scientists – and, by virtue of this, ourselves – to transcend what we have done these last thirty years and find the true theory that solves the five grand questions of physics.”

Categories
Blog posts in English

The problem with test documentation

The agile manifesto explicitly values ”working software over comprehensive documentation.” In test, this means that actual testing is valued over test documentation. I would have put it this way: Focus is on the quality of the product, not on the quality of the documentation of the product.
We can probably all agree that it’s more fun to test and explore software than writing documentation. But it would be too radical, if we skipped writing documentation at all!
I think, however, that we testers should be more critical about the documentation we actually do produce, and that we should look for ways to improve the documentation we’re making.
The problem with documentation is not that too much time is spent writing it instead of actually testing the product. The problem is that the quality of the documentation is often just not good enough.
Most organizations have standards for test documentation and require their testers to write specific types of documents based on mandatory templates.
Templates are generally a good thing, but they can be problematic if you limit your writing process to ”filling in the gaps.” A good document contains useful information to the reader, so the most important step in the writing process is finding out what information is actually needed.
Not all sections of a template can be equally useful in all contexts (or useful at all), and very often you need to document things which there has not been left space for in the template.
But, finding out what to document is not trivial. Basically, it requires that you know the context of the project you are working on. Determining context can be difficult if you are new on a project, but asking questions can give you answers which will help you define the context.
Here are some questions, which I find useful:

  • Who will be reading the document? How will they be reading it? The challenge is to include content which the actual readers will find useful and to structure the content in a way so that they can find that information.
  • What information are they looking for? Try to get people to answer in concrete terms rather than using abstract vocabulary. The problem with written documentation is that stake holders will often not read it – most testers seem to prefer to explore systems on their own rather than reading documents, and managers will often not have time to read everything, but will only check the headline and certain details. But if readers get what they look for, chances are that they will read the document.
  • What kind of analysis do I need to carry out on the system to better understand it and test it? Writing a document about the system can often help me understanding it. I will discover missing knowledge and knowledge which I didn’t see initially. The writing process is part of the analysis.
  • Are there regulatory requirements which specify that testing should be documented to a certain detail and in a particular way? In this case, the test documentation is actually a product of the project.
  • Should the document assist knowledge transfer once I’m no longer on the project? In that case, the big question is what kind of knowledge should be ”transferred.”

I checked the section about documentation in Kaner, Bach and Pettichord’s Lessons Learned in Software Testing a few days ago. They have a better and longer list of context-free questions to ask about documentation which will help you find out what kind of documentation you need. They also list a few references to other lists of useful questions, so I suggest you take a look at that too.
Are there any specific types of documentation which is particularly useful? Indeed, there is:

  • Mind maps are becoming increasingly popular with testers, but ‘old style’ software diagrams like swimlane diagrams, state diagrams, and flow charts etc are still working well. The problem with mind maps is that they are often personal to the author of the map and not very easy to read.
  • Test scripts can be very useful in initial stages of knowledge transfer: A script can help another tester finding out how a certain procedure is performed in the system. However, a script will by itself tell the tester anything about the context of the script, and this is something which is often missed: Knowledge about a system under test is much more than knowing how to do things.
  • Check lists are actually much more useful than the name implies: A check list will list things to verify, but unlike a script will not specify in detail how to verify them. That information has to be available elsewhere e.g. in user manuals.
  • I always look for a document describing the system context in a top-down manner: What is the system doing, for who, and how? If it isn’t there, I don’t mind writing that document myself.
  • A catalogue of tools used in testing is often also very useful. Test tools are often not well documented (or not documented at all), and that can be a hurdle for new testers when they come aboard a project. A well written ”tips and tricks for testing system X” will get them up to speed faster and can act as a platform for sharing knowledge about testing specific parts of the system. I like Word documents for their self-contained’ness, but a Wiki could be better in many situations – the important thing is that such a document is actively maintained.

What are your preferred test documents?

Categories
Blog posts in English

It's fortune Friday

It’s fortune(6) Friday… what my server said when I logged in on the console today:

Welcome to NetBSD!
WARNING!!!
This machine is subject to breakdowns during periods of critical need.
A special circuit in the machine called “critical detector” senses the operator’s emotional state in terms of how desperate he/she is to use the machine.  The “critical detector” then creates a malfunction proportional to the desperation of the operator.  Threatening the machine with violence only aggravates the situation.  Likewise, attempts to use another machine may cause it to malfunction.  They belong to the same union.  Keep cool and say nice things to the machine.  Nothing else seems to work.
See also: flog(1), tm(1)

Thanks for warning me!

Categories
Blog posts in English

Computer history

As a teenager, I spent hours reading datasheets for CPU components like the bit slice processors AM2901. I also designed hypothetical CPUs in block diagrams. A 16 bit CPU was my objective, and it should have had a clock of 20 MHz. The instruction set was to be small and beautiful. My first computer was based on a National Semiconductor SC/MP processor, which had a very nice and simple instruction set, and I’ve always liked things which are the simple, beautiful and logical.
I never did build my own CPU, but at university, I spent hours playing with the old RC4000 mainframe we had in the electronics club. It was (is!) a 24 bit machine from the late 60’s, a very nice and advanced design. Its concurrent programming system was among the first in the world (appearing at about the same time as Unix was developed in the US), and my father, who used to work at Regnecentralen, gave me one of the original books about the system, which I read with great interest. It was a message passing based system, and I think the design had a few drawbacks, but the beauty of it was fascinating.
Yesterday I saw my old friend again. She is now residing in the rooms with computer historic enthusiasts. I visited the club with 14 year old son Frederik.

Photo of the RC4000 computer from 1971 in the basement of Dansk Datahistorisk Forening
The RC4000 computer was designed in the 1960's and was built from the late 60's to mid 70's when it was replaced by RC8000. It was a 24 bit architecture with up to 384 Kb core memory.

The RC4000 still works, but the enthusiasts are more focused now on its predecessor: A GIER computer, which has recently been brought back to life. The first GIER was delivered on July 31st 1961, so the design is turning 50 years old this year. About 50 of these machines were built and the club owns two: One early model and a late model. It is the late model which they are bringing back to life.
Photo of the now running late model GIER computer
GIER, a 50 years old computer design, now being brought back to life.

There are very few test programs for GIER, and this makes the repair process a bit more complicated. Poul-Henning Kamp, the author of the Varnish Cache, who is one of the active enthusiasts in the club, mentioned that it was probably because the designers found test program development to be too boring. These guys were innovators! Poul-Henning has unit tests for Varnish with great code coverage, but admits that writing tests is not the fun part of that project.
I used to program for a living and I can agree with this. Unit testing isn’t fun!
The smell of the old computers, their physical size, the noise they make, and their sheer lack of computing power (by todays standards) suggest that these machines are of a different era. Technology has evolved a lot, but people are still people. And the people matter – the technology is just a means to an end.
I still love working hands-on with technology, but my view on IT has grown a lot wider since I was young.

Categories
Blog posts in English

Acceptance tests are not enough!

Acceptance testing is a key method in Agile. One way of defining acceptance tests are Gojko Adzic‘s ”Specification by example” paradigm which has gained quite a bit of momentum lately. I personally found it to be both refreshing and nice when I heard him present it at Agile Testing Days 2009, and I also found his book Bridging the Communication Gap a nice read.

Photo of Gojko Adzic demonstrating communication gaps in his keynote presentation at Agile Testing Days 2009
Gojko Adzic demonstrating communication gaps at Agile Testing Days 2009

I’m sceptical of the concept of acceptance testing. Not because verification of agreed functionality is not a good thing, but because it tends to shift attention to verification instead of exploration.
This will shift attention from testing to problem prevention. Is that bad, you ask? Isn’t it better to prevent problems than to discover them?
Well, most people think ”why didn’t I prevent this from happening?” when problems do happen. Feelings of regret are natural in that situation and that feeling can lead you into thinking you should improve your problem prevention. And maybe you should, but more examples aren’t going to do it!
Real testing is still necessary.
To explain why, I’ll consult one of the great early 20’th century mathematical philosophers: Kurt Gödel. In particular his first incompleteness theorem. It says that no consistent system of axioms whose theorems can be listed by an “effective procedure” is capable of proving all facts about the natural numbers.
What does this mean to us?
It means that we will never be able to list all things that can be done with this particular set of data.
A specification is a kind of listing of ”valid things to do” with data, thus Gödel’s theorem teaches us that there are infinitely more things to a system than any long list of requirements. This also applies when the requirements are listed as examples.
If you’re in the business to deliver products of only ”agreed quality” to a customer, you can be all right only verifying things which are explicity agreed. If something goes wrong you can always claim: ”It wasn’t in the specification!”
But if you’re striving for quality in a broader sense, verifying that the system works according to specifications is never going to be enough.
Gojko has made a good contribution to agile. Examples can be useful and efficient communication tools, and if they are used correctly they can help making users and other interested parties better aware of what’s going on on the other side. His contribution can help bridge a communication gap. It can also produce excellent input for automated unit tests.
Just don’t let it consume your precious testing time. The real testing goes far beyond verification of documented requirements!
If you want to learn more about this, I recommend you sign up for one of the Rapid Software Testing courses offered by James Bach and Michael Bolton.
Photo from Rapid Software Testing course in London November 2010 as offered by Electromind
Michael Bolton with one of the many interesting and challenging test exercises at the RST course

Categories
Blog posts in English

Covering test coverage

Rolf Østergaard @rolfostergaard suggested on twitter when I posted my previous blog that instead of counting defects and tests we take a look on test coverage. Certainly!
Mathematically, coverage relates the size of an area fully contained in another area, relative to the size of that other area. We could calculate the water coverage of the Earth or even how much of a floor a carpet could cover. Coverage can be expressed as a percentage.
But coverage is also a qualitative term. For example a book can cover a subject, or a piece of clothing can give proper (or improper!) body coverage.
So what is test coverage? Well, the term is often used to somehow describe how much of a system’s functionality is covered by testing.
Numbers are powerful and popular with some people, so a quantified coverage number would be nice to have. One such number is code coverage, which is calculated by dividing the number of code lines which have been executed at least once by to the total number of code lines in a program.
Another measurement relies on business requirements for the system being registered and numbered, and tests mapped to the requirements which they test. A suite of tests can then be said to cover a certain amount of requirements.
Numbers can hint something interesting. E.g. if your unit tests exercise only 10% of the code and it tends to be the same 10% on all of them, the chances are that something important will be missing from the unit tests. Or you could even have a lot of dead legacy code. This would be similar if you found that you actually only tested functionality in a few of the documented business requirements: Could the not-covered requirements be just noise?
No matter what, a coverage number can only give hints. It cannot give certanity.
Let’s imagine we can make a drawing of the functionality of a system; like a map. Everything on the map would be intended functionality, everything outside would be unaccepted. Let’s make another simplification and imagine for the moment that the map is the system, not just an image of it. Here is an example of such a simple system:
Drawing of a system being tested. Some tests verify valid functionality of the system, other tests verify that there are not functions in the system which should not be there. But tests are points.
The blue area is the system. The red spots are checks carried out as part of testing. Some of the checks are within the system, others are outside it. The ones within are expected to pass, the ones outside are expected to fail.
Note that there is no way to calculate the test coverage of this imaginative system. Firstly, because the area outside the system is infinite and we can’t calculate the coverage of an infinite area. Secondly, because the checks don’t have an area – they are merely points – so any coverage calculation will be infinitesimal.
Ah, you may argue, my tests aren’t composed of points but are scripts: They are linear!
Actually, a script is not a linear entity, it’s just a connected sequence of verification points, but even if it was linear, it wouldnt’ have an area: Lines are one-dimensional.
But my system is not a continous entity, it is quantified and consists only of the features listed in the requirement document.
Well that’s an interesting point.
The problem is that considering only documented requirements will never consider all functionality. Think about the 2.2250738585072012e-308 problem in Java string to float conversion. I’m certain there are no requirement documents on systems implemented in Java, which actually listed this number as being a specifically valid (or invalid) entry in input fields or on external integrations. The documents probably just said the system should accept floats for certain fields. However a program which stops responding because it enters an infinite loop is obviously not acceptible.
A requirement document is always incomplete. It describes how you hope the system will work, yet there’s more to a system than can be explicitly described by a requirements document.
Thus any testing relying explicitly on documented requirements cannot be complete – or have a correctly calculated coverage.
My message to Rolf Østergaard is this: If a tester makes a coverage analysis of what he has done, remember that no matter how the coverage is measured, any quantified value will only give hints about the testing. And if he reports 100% coverage and looks satisfied, I strongly suggest you start looking into what kind of testing he has actually done. It will probably be flawed.
Intelligent testing assists those who are responsible for quality in finding out how a system is actually working, it doesn’t assure quality.
Thanks to Darren McMillan for helpful review of this post.