{"id":2303,"date":"2017-01-23T21:14:38","date_gmt":"2017-01-23T19:14:38","guid":{"rendered":"http:\/\/asym.dk\/?p=2303"},"modified":"2017-01-23T21:14:38","modified_gmt":"2017-01-23T19:14:38","slug":"where-might-the-black-swans-be-lurking","status":"publish","type":"post","link":"http:\/\/www.asym.dk\/index.php\/2017\/01\/23\/where-might-the-black-swans-be-lurking\/","title":{"rendered":"Where might the Black Swans be lurking?"},"content":{"rendered":"

Summary:<\/strong><\/em> Last year in September, I spoke<\/a> at Anna Royzman\u2019s New Testing Conference \u201cReinventing Testers Week\u201d in New York about testing in Black Swan domains. The title of the talk refers to Nassim Taleb\u2019s book \u201cBlack Swan\u201d and concerned testing in contexts where improbable risks can have disproportionate effects. This blog contains an invitation to a peer conference<\/a> in New York on the subject Sunday April 30th.<\/i><\/p>\n

\n

Sometimes things happen which appear to be beyond “the possible\u201d.
\nThis awareness haunts us in testing: We aim to get those important bugs to materialize. We want to qualify real and serious risks. Yet,\u00a0our stakeholders\u00a0have to accept that no matter how much testing is done, we cannot cover everything.
\nLogically, testing has to be focused on what is important to the business, and what might go wrong in reality. To me, that is at the core of risk based testing.
\nBut saying that is one thing; doing it in reality is quite another. Certain risks seem almost impossible to qualify through testing. How should our stakeholders interpret the absence of clear testing results, for example, when we are trying our best to dig out quality information about quality? Could there be a serious problem lurking? The thought may seem paranoid, but experience shows it is\u00a0not.
\nYears ago, I read Nassim Taleb’s \u201cThe Black Swan – The Impact of the Highly Improbable\u201d<\/a>. The book blew my mind and set me on a path to find out what we can do in testing about what he writes about.
\nThe book is about \u201cthe random events that underlie our lives, from bestsellers to world disasters. Their impact is huge; they\u2019re nearly impossible to predict; yet after they happen we always try to rationalize them.\u201d (from the backcover of the 2010 paperback edition)
\nAs an engineer and a human, I think testers and test managers should not give up and leave to product owners, project managers or developers to interpret testing results take care of potential Black Swans. As a tester, I wish to embrace the possibility of Black Swans and<\/strong> do quality testing with the aim to qualify them.
\nI think, however, that we\u00a0need new models in testing. The problem is that most of our techniques and heuristics tend to support us best on the functional testing level.<\/p>\n

Accident Focused Testing?<\/h2>\n

The first part in solving a problem is accepting it. It sounds basic, but acceptance implies understanding what we are dealing with.\u00a0Reading Talebs book asserted to me that we have to accept the fact that really bad things can happen in the world. Knowing what I do about information technology, I appreciate that his philosophy can be applied on technology. I also believe that the functional testing will not help us muc.
\nMentally examining what I do as a tester, I understood that the idea of Black Swans is fundamental to the very nature of what we do and the systems we work with.
\nThat much about acceptance.
\nThe problem is that in some contexts – banking, healthcare, industrial plant management, safety systems, public sector, transportation etc – accidents and Black Swans could be of a nature where they cause irrecoverable losses, set lives at stake or otherwise be fundamentally unacceptable.
\nLet me give an example:
\nI recently came across a story of an interesting IT breakdown in a hospital in Sweden. It concerned something most people do on a regular basis: Apply the newest updates to our pc\u2019s.
\nAs updates were rolled out in the hospital, performance of pc\u2019s started degrading. During the rollout the problems became worse, and before the rollout could be stopped all computers in the hospital became useless.
\nNow that the computers stopped working, undoing the rollout became extremely difficult and had to be carried out manually, one pc at a time.
\nIn the end it took IT-operations several days to get everything back to normal. Meanwhile the hospital had to be run “on paper”.
\nThe hospital used an uncommon Windows network configuration, which is not recommended by Microsoft which in combination with the Microsoft update triggered a problem in the network. What is interesting here is not the root cause, however: The outcome of a seemingly trivial update in a complex system turned out very bad.
\nIt is easy to imagine how the stress experienced by doctors and nurses due to this situation could have affected patients. Someone could have been hurt.
\nWe can shrug and blame Microsoft or the hospital IT operations. However, as skilled testers, I think we need to be able to provide some kind of answer as to how we can constructively contribute\u00a0to hospital safety by qualifying even Black Swan-types of risks.<\/p>\n

Systemic risks<\/h2>\n

Before moving on, let me dive into the subject of risk. Risk is something we all talk about, but do we really\u00a0know what it means? I’m not sure. Risk is a common thing to talk about, but the concept of risk<\/strong> is in no way simple.
\nThere seems to be at least three “risk domains” in software projects:<\/p>\n