{"id":160,"date":"2010-12-31T08:43:06","date_gmt":"2010-12-31T06:43:06","guid":{"rendered":"http:\/\/asymaps.wordpress.com\/?p=160"},"modified":"2010-12-31T08:43:06","modified_gmt":"2010-12-31T06:43:06","slug":"skypes-first-black-swan","status":"publish","type":"post","link":"https:\/\/www.asym.dk\/index.php\/2010\/12\/31\/skypes-first-black-swan\/","title":{"rendered":"Skype's first Black Swan"},"content":{"rendered":"

Skype went out for about 24 hours just before Christmas. Skype management is embarrassed and promise this will not happen again, which of course is true. The particuar situation is now prevented. However, the question is: Will Skype never be out again?
\nSkype’s CIO explains what went wrong in this post mortem<\/a>\u00a0of what I’d\u00a0call Skype’s first Black Swan.
\nTo summarise, it was a high load on Skypes infrastructure which triggered a bug in a certain version of the Windows client for Skype which again increased the load on the infrastructure, thereby rapidly taking the entire network down and making it almost impossible to get it up again.
\nThe bug was always there of course, and it was probably already known internally at Skype. It is also possible that the risk of server overloading\u00a0and\u00a0service degradation\u00a0had been identified, but obviously not in the context of making a complete system crash a likely possibility (if so, they would have prevented it). Further, I’m quite certain that the risk of the client bug affecting the server load had not been identified. Humans are positive thinkers, as Taleb<\/a> documents in his book: The Black Swan: The Impact of the Highly Improbable<\/a>.
\nSo Skype’s challenge now is to prevent outages in general by identifying and preventing Black Swans in general. This will involve a cross organisational backwards thinking process, which the innovation driven company has probably not been focusing on at all until now. (I may actually be wrong here, Janus Friis, one of the\u00a0founders of Skype,\u00a0used to work in a support function of an ISP so he may have been involved in preventing problems, but generally, startups think very positively, and even if Skype has millions of users, it’s still a very young company – a startup.)
\nOne may think that this is going to be extremely expensive for Skype since they will have to predict every possible way their system can go wrong. It does not have to be that expensive, although it will cost money.
\nWhen securing a nuclear facility, engineers don’t have to analyze every possible way a disaster can happen, instead\u00a0they think: How can we prevent failure at every level? This is what I mean with “backwards thinking” – start assuming something is failing, then work backwards identifying ways to prevent it becoming worse.
\nThis is done on multiple levels: On component level, asking what can go wrong here and how can we prevent a bug or incident\u00a0from affecting the rest of the system? And on system level, assuming that disaster is happning, how can we prevent it from developing.
\nI assume that’s what Skype is doing now.
\nWishing everyone a happy 2011!<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"

Skype went out for about 24 hours just before Christmas. Skype management is embarrassed and promise this will not happen again, which of course is true. The particuar situation is now prevented. However, the question is: Will Skype never be out again? Skype’s CIO explains what went wrong in this post mortem\u00a0of what I’d\u00a0call Skype’s […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[3],"tags":[15,68,89],"_links":{"self":[{"href":"https:\/\/www.asym.dk\/index.php\/wp-json\/wp\/v2\/posts\/160"}],"collection":[{"href":"https:\/\/www.asym.dk\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.asym.dk\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.asym.dk\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.asym.dk\/index.php\/wp-json\/wp\/v2\/comments?post=160"}],"version-history":[{"count":0,"href":"https:\/\/www.asym.dk\/index.php\/wp-json\/wp\/v2\/posts\/160\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.asym.dk\/index.php\/wp-json\/wp\/v2\/media?parent=160"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.asym.dk\/index.php\/wp-json\/wp\/v2\/categories?post=160"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.asym.dk\/index.php\/wp-json\/wp\/v2\/tags?post=160"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}