Two Business Disaster Stories, & What You Can Learn From Them

[img src=”/wp-content/uploads/sites/2/2014/07/when-disaster-strikes-acoupleofgurus.jpg” class=”aligncenter”]

Today we have a tale of two disasters.  Each story has a different cause and a different outcome.  Both stories we are dealing with catastrophic data failures that happened within the past year.  The reason I want to tell these stories is not to capitalize on someone else’s pain, but so that you can learn and plan accordingly.  The first story involves an associate of mine in another state.  The second story involves one of our clients and our ability to help them.  Think your business is impervious to a catastrophic disaster?  Think again.

Story 1

A service company (not one of our clients) decided to reuse a decommissioned (outdated) piece of hardware temporarily, even though this piece of equipment was far beyond its warranty and production value.  The problem they ran into was they needed more capacity within their SAN (a Storage Area Network, which is essentially a group of hard drives set up as a large external drive for a group of servers).  Subsequently, they were using this SAN to run some of their core business systems, including their time entry and billing software.

This company had a situation where their internet connection went down.  They had to “fail-over” to their secondary internet connection.  The problem was that their secondary connection was not big enough for them to send their backups offsite.  So, one of their engineers decided to put the backup on hold, at least until their primary internet connection was back up online. Their internet problem did eventually get resolved, and they moved on from that… except that engineer never remembered to re-enable their backup to be delivered off site.

After a couple weeks of undelivered backups, this SAN eventually experienced a multi-drive failure due to old hardware.  They lost all of their servers.  This was a cataclysmic, catastrophic failure.

Since they lost all of their servers, and they didn’t have a current backup, they had to do two things. First, they had to send their SAN off to a data recovery specialist company, which ended up being a two-week process costing them $15,000 in data recovery services.  Second, they had to conduct a system recovery from their last good backup, which at this point was a few weeks old.  Eventually, they did get their SAN & data back from the recovery company, but now they were looking at two sets of data that were very out of sync.  Once they got their data back, they had to go through the very tedious process of re-merging their data back together.  That took another week.

Ultimately, this company not only had the cost of the data recovery service, but they also had the loss of productivity by working with old data and then having to spend nearly a week re-merging their two sets of data back together and finally reconnecting their offsite backup.

Story 2

The second scenario actually also involves a SAN, but from a different manufacturer and with a more positive outcome.  This company had a continuous replication backup system in place that was being sent to an offsite location.

One stormy night, in classic Minnesota fashion, a major thunderstorm went through the Twin Cities, knocking the power out in many industrial areas.  This company experienced an outage which knocked their systems down.  Power interruptions and storms can be very hard on electronic equipment because of the surges of electricity that can go through the systems.  Due to the power outages caused by this particular storm, the company had a multi-drive failure on their SAN.

As this company’s Managed Services Provider, we immediately received an alert that their systems were down.  Once the power came back on, we took a look at their systems and discovered that the SAN failure had occurred.  We immediately activated their Business Continuity Plan, which called for running their production systems.  We brought up the offsite replication and turned on their production systems at their offsite location.  Once they were up and running, a process that took just a couple of hours, we were then able to remove the damaged SAN and work on its recovery.  Once we completed the recovery, we did a “fail-back” to their production facility on-site to continue usage of their systems.

There are some lessons we can learn from these two stories:

Lesson 1: Disaster Will Strike… and it will strike for different reasons at any moment.

The outcome of a disaster is going to be determined by the kind of backup system and business continuity solution that was put into place.

A business continuity solution is a plan to have your production servers and systems turned on within a matter of hours at an offsite location, whether that is in your own offsite facility or in the cloud.  Business continuity solutions are affordable for nearly any type of business.  Just because you run a small company doesn’t mean you can’t afford this.  I would say that you can’t afford NOT to have this.

Lesson 2: Old Hardware Will Fail

The first scenario was avoidable.  It’s important to evaluate and assess your equipment and critical production systems and to know what your continuity plan is for those assets.

Lesson 3: Investing in a Business Continuity Solution pays off

It also pays to have those systems monitored as well, to ensure a quick recovery.

So the question is… are you prepared?  Is your system ready to handle a catastrophic disaster?  If you’re not sure or wish to reevaluate your strategy, then we would be happy to sit down with you and have a conversation.