#WeekendWisdom 098 Lessons from the Facebook Outage

Welcome to #WeekendWisdom number 98. This week we’re going to talk about Lessons from the Facebook Outage.

No. There is no Facebook bashing here. Just the fax mam. 😎

Where were you on the late afternoon, early evening when Facebook and Instagram and WhatsApp were all down for nearly 6 hours?

What had happened to them was that they were making a routine change to their network configuration. But there was a bug somewhere in it, that caused them to basically disconnect completely from the internet. So nobody on the internet could see Facebook, Instagram, WhatsApp or any of their services. But not only that, this bug also cut off their internal access to their systems. So they could not even see their own email or messaging systems internally and this also prevented the engineers who had made the change from being able to get access to the equipment that they put the change onto and to try and reverse it.

Lessons from the Facebook Outage

So what can we learn about this? Well, I suppose the thing is, the way Facebook are set-up, they seem to have everything is in one big giant bubble that they all use internally. So all their eggs are in one basket. But if you were in an environment where a change like that could take out your entire environment, you might want to consider having some other alternatives that are not based on your internal network.

What could you use

So instead of using WhatsApp you might consider using something like Signal for messaging or even come back to good old fashioned text messages. Have some other kind of, what they call, out-of-band way of being able to communicate internally and be able to get around any kind of outage like this that might occur.

So that’s it for this week. Lets be careful out there and we’ll talk to you again next week.