Well I guess that - having decided to take the plunge and write a blog - it's got to be a no-holds-barred one. And guess what - last night, for the first time in 2007, we suffered an unscheduled outage on Netroster. Here's what happened:
Just about the only single point of failure in the system is one that we can do very little about - the domain name, Netroster.biz, only has one point of registration on the internet. It's the same for all domains. Our registrar suffered an outage at around 9pm UK time on Friday night, the 28th.
Our servers were happily running - responding to heartbeat checks and therefore not raising any alarms with our monitoring. However, because the domain had gone off the air, clients weren't able to get to online.netroster.biz to use the service. Various clients reported this to our helpdesk and - joy of joys - they decided to escalate this by email rather than by phone. It was only when a client called me directly on my mobile that the ...