Blog posts in English

The problem with test documentation

The agile manifesto explicitly values ”working software over comprehensive documentation.” In test, this means that actual testing is valued over test documentation. I would have put it this way: Focus is on the quality of the product, not on the quality of the documentation of the product.
We can probably all agree that it’s more fun to test and explore software than writing documentation. But it would be too radical, if we skipped writing documentation at all!
I think, however, that we testers should be more critical about the documentation we actually do produce, and that we should look for ways to improve the documentation we’re making.
The problem with documentation is not that too much time is spent writing it instead of actually testing the product. The problem is that the quality of the documentation is often just not good enough.
Most organizations have standards for test documentation and require their testers to write specific types of documents based on mandatory templates.
Templates are generally a good thing, but they can be problematic if you limit your writing process to ”filling in the gaps.” A good document contains useful information to the reader, so the most important step in the writing process is finding out what information is actually needed.
Not all sections of a template can be equally useful in all contexts (or useful at all), and very often you need to document things which there has not been left space for in the template.
But, finding out what to document is not trivial. Basically, it requires that you know the context of the project you are working on. Determining context can be difficult if you are new on a project, but asking questions can give you answers which will help you define the context.
Here are some questions, which I find useful:

  • Who will be reading the document? How will they be reading it? The challenge is to include content which the actual readers will find useful and to structure the content in a way so that they can find that information.
  • What information are they looking for? Try to get people to answer in concrete terms rather than using abstract vocabulary. The problem with written documentation is that stake holders will often not read it – most testers seem to prefer to explore systems on their own rather than reading documents, and managers will often not have time to read everything, but will only check the headline and certain details. But if readers get what they look for, chances are that they will read the document.
  • What kind of analysis do I need to carry out on the system to better understand it and test it? Writing a document about the system can often help me understanding it. I will discover missing knowledge and knowledge which I didn’t see initially. The writing process is part of the analysis.
  • Are there regulatory requirements which specify that testing should be documented to a certain detail and in a particular way? In this case, the test documentation is actually a product of the project.
  • Should the document assist knowledge transfer once I’m no longer on the project? In that case, the big question is what kind of knowledge should be ”transferred.”

I checked the section about documentation in Kaner, Bach and Pettichord’s Lessons Learned in Software Testing a few days ago. They have a better and longer list of context-free questions to ask about documentation which will help you find out what kind of documentation you need. They also list a few references to other lists of useful questions, so I suggest you take a look at that too.
Are there any specific types of documentation which is particularly useful? Indeed, there is:

  • Mind maps are becoming increasingly popular with testers, but ‘old style’ software diagrams like swimlane diagrams, state diagrams, and flow charts etc are still working well. The problem with mind maps is that they are often personal to the author of the map and not very easy to read.
  • Test scripts can be very useful in initial stages of knowledge transfer: A script can help another tester finding out how a certain procedure is performed in the system. However, a script will by itself tell the tester anything about the context of the script, and this is something which is often missed: Knowledge about a system under test is much more than knowing how to do things.
  • Check lists are actually much more useful than the name implies: A check list will list things to verify, but unlike a script will not specify in detail how to verify them. That information has to be available elsewhere e.g. in user manuals.
  • I always look for a document describing the system context in a top-down manner: What is the system doing, for who, and how? If it isn’t there, I don’t mind writing that document myself.
  • A catalogue of tools used in testing is often also very useful. Test tools are often not well documented (or not documented at all), and that can be a hurdle for new testers when they come aboard a project. A well written ”tips and tricks for testing system X” will get them up to speed faster and can act as a platform for sharing knowledge about testing specific parts of the system. I like Word documents for their self-contained’ness, but a Wiki could be better in many situations – the important thing is that such a document is actively maintained.

What are your preferred test documents?

9 replies on “The problem with test documentation”

Hi Anders,
Good tips, and thoughts.
If find looking at test documentation as a whole difficult at times. You have however did a good job here in keeping the points generic enough to cover most instances.
A good question to ask is “What types of test documentation are there?”
I’d find by looking at these types you can apply different techniques and approaches to writing most of them. No universal technique or approach will apply to all. Likewise what applied in one workplace might not transverse to the next.
Lean, documentation which provides a quick sharp interesting points only summary before delving into more detail (the detail should also be kept short and to the point) I find works best. With a good use of visuals to highlight key points and allow easier digestion of more complex or lengthy information.
Whilst thinking in terms of test documentation, perhaps a mind map of types of test documentation and techniques / approaches that you could apply for each could be handy? I’ll have to try and find some time to put one together.
Thanks Anders,

Thanks for your comment, Darren.
Lean documentation – that’s a concept I haven’t seen defined… But you have a very good point about summaries – IMO time spent writing a good summary before discussing the details is usually spent very well.

Nice post, Anders.
As we all seem to understand here, different jobs will have different audiences with different expectations, placing different documentation challenges on you.
The poorer end of the scale are the organisations which have templates carved in stone that you’re expected to adhere to, and even worse, they have no real understanding of what they’re asking for and it’ll sit under change control gathering virtual dust.
The better position to be in is undoubtedly where you can have an open dialogue with the audience so the (lean) documentation can be crafted and tailored to serve a purpose rather than simply checking a box.

Thanks, Del. Good point about differences.
Documentation that nobody ever reads is waste! Per definition.
Checking boxes can help show progress, but doesn’t define any kind of quality. Yes, open dialogue is always good!

Hi Anders
Well done, I really enjoyed reading your blog, I think you brought up some very good points and gave some interesting thoughts and tips on how to go about documenting you’re testing.
I certainly like Darren’s comment with regards to “quick sharp interesting points only summary before delving into more detail”.
As for templates, just to be little indifferent to Del’s comments. I can see no reason why certain types of templates should not be set in stone by companies i.e. “exit and entry criteria’s” and “Test reports”. These sorts of templates probably need to be set in stone for some companies, so that the tester enters only the information and comments that are required and nothing more or less.

Hi Mark
Sorry I haven’t answered you before. Your comment got caught in my holiday 🙂
Thanks for your comment, Mark. Yes, Darren has a good point about summarizing information before providing detail. That’s always a good idea when you are trying to communicate something to someone: Provide a conclusion – and put it first.
I think Del is referring to the ‘hardcore’ lean thinking that waste should be eliminated. If a company has a template ‘set in stone’, but nobody finds the template useful, then what argument would you have not to throw away the stone?
That said, I can see your point and if you look in the ‘Lessons Learned…’ book, you’ll find that Back, Kaner and Pettichord lists plenty of good reasons to use templates. And I’m sure you can easily come up with at least the double.

Hi Mark,
It’s all down to context ultimately and you’re correct in saying that some companies may well prefer to have stone carved templates for any number of reasons.
My personal experience of stone carved templates has been that no-one really paid attention to the document when I rigidly adhered to the template. It was only when I started to deviate from the template (after discussing with my audience what they really wanted to see) and started to present (what was perceived to be) more useful information that people started to really take an active interest in such documentation.
Horses for courses, essentially.

Interesting point, Del.
But not surprising. (Normal functioning) brains eliminate information. If you want people to pay attention, make sure you say something they didn’t expect – or say it in a way they didn’t expect!

Thanks for this article it’s great.
I often talk with the teams I coach about the need to “story tell” more. It is for this reason I like combining tests with specifications and then automating them with Fitnesse or a similiar tool. And story telling is even more important after tests are automated since testers no longer get to learn what the tests do by actually performing them.
Story telling is not new to software engineering. It is something programmers have started doing several years ago when they identified code smells such as code comments that can be removed by grouping parts of code into a method with a descriptive name.
To me it’s about creating maintainable and retestable software. So keep in mind someone relatively new to the project picking up the documentation.
Clare McLennan

Leave a Reply

Your email address will not be published. Required fields are marked *