The Next Problem in Black Swan Testing

The pervasive, but intangible nature of Black Swans means that that practical testing with the aim of demonstrating actual problems is probably either not going to give any useful evidence, or so much resources will be spent proving a point, that one might be missing the point itself completely.
This was the fundamental problem I was facing when I finished reading Talebs book and wanted to apply his philosophy into actual testing. I realised that I needed a model, and the Skype incident of december 2010 led me in the right direction.
The model that I’ve come up with is that black swans are system errors. This may not be true in all circumstances, but it’s a good model and it’s helping me come up with solutions to the testing problems.
Unfortunately, treating black swans as system errors also mean that instead seeing Black Swan Testing as a practical testing activity, I’m moving it to a meta level, where the ‘root causes’ are of a more abstract nature and often not directly observable.
In my speach here at Let’s Test yesterday, I introduced three classes of system attributes and suggested that practical testing, with the aim of learning about potential black swan incidents in a system, should focus on these attributes:

• Complex versus Linear Interactions
• Tight versus Loose Couplings
• Barriers

The two first come from the work of sociologist Charles Perrow, in particular his book Normal Accidents, the third on I owe to psychologist James Reason, author of Human Error. I’ll come back to these attributes in later blog posts, but for now you just have to accept them as system attributes that play parts in system errors and Black Swans.
But we’re at a conference with all sorts of things going on: My presentation was well received, the discussions afterwards were great, but Let’s not just talk… let’s do it, Let’s Test.
I think James Lyndsay and I got the idea at about the same time yesterday: Let’s take Black Swan Testing into The Test Lab.
So I did, and it was great. I had a great team of very brave testers, and the mission was clear: Find indications of Black Swans, look for tight couplings, complex interactions, and barriers.
Did we succeed? Not really. But it was loads of fun and we learned a lot!
In particular, I learned that while I think I have a very good idea of what Black Swan Testing is, I need to work on the practical aspects: Making useful charters, coaching and teaching testers efficiently on the subject, reporting… Black Swan testing must be communicated and operationalized.
That’s the next problem, I’m going to address.