DR – A tale of three organisations

**Organisation One – The Stubborn Ostrich**

The first organisation we’ll look at is a NSW not-for-profit. They did not have a DR environment and placed an order for DR equipment to be housed at one of their locations on a Friday afternoon. That very Friday night in 2008, their entire head office including the server room was burnt to the ground.

Dedicated engineers over many days were able to cobble together borrowed legacy equipment and locate some backups and restore some of their environment at a friendly competitors site. Data loss was inevitable.

In due course, the organisation found rental accommodation and the ICT equipment was relocated to the new location. The server room environment was less than standard but they were able to keep the business functioning on a limited basis for a few months.

The borrowed equipment needed to be returned to its rightful owner and new equipment was purchased. To allow data migration between the two storage devices, the borrowed equipment needed to have a code upgrade. Recommended procedures were not followed by the organisation resulting in data corruption of the entire storage array.

Backups were located and failed to restore data as licenses for backup agents had expired some months before. Again due to the dedication of engineers, some data was restored but the amount of loss was substantial for the organisation.

**Organisation Two – The Lucky Laggard**

Our second organisation in the spotlight is a Victorian health organisation. They were recently upgrading their IT infrastructure and adding a DR environment. Alongside the current production workload was newly commissioned production equipment. The uncommissioned DR equipment was sitting at their DR site. During the course of commissioning the new production equipment, the unthinkable occurred. Two floors above the server room, a mains water pipe burst.

It flooded the floor above the server room then made its way into the server room, saturating the current production equipment alongside the new production equipment. Water was trapped within the server room to a height about half way up the racks.

With no DR environment and no DRP (disaster recovery plan) the situation looked grim. It was only with some very good luck and dedication of engineers that the DR site was urgently commissioned and data replicated from damaged equipment to the DR equipment, allowing the organisation to remain online. More good luck than good management.

**Organisation Three – The Boy Scout**

Our third organisation, a national health benefits provider migrated their production environment into the Cloud earlier this year. The service provider maintains high availability services, meaning the likelihood of an ICT outage is minimal. Organisation three felt the risk to their business on a minimal likelihood of an outage occurring, was too great, and mitigated the risk by creating another Cloud based environment, at their service provider’s disaster recovery Cloud, replicating data near real time, from production to DR.

They built up a DRP (disaster recovery plan) which provided all the businesses context for declaring a disaster and the technical steps for recovery of environment at the DR site. Further to this, they exercised the DRP over a weekend, while maintaining critical online business services. They are in a position that when a disaster occurs, they have proven confidence that their DR solution will allow the business to maintain service to their clients. The truth of the matter, each of the above stories is true.

We often think of disasters as the major types of incidents associated with the environment e.g. floods, bushfires and earthquakes. Closer to home, the usual culprits are the unplanned for or unseen events as highlighted in the first two stories.

Should we be prepared or wait until the event to respond? Unfortunately, most business still believe “it won’t happen to me” and therefore do not prepare for a disaster, taking on the risk and the subsequent recovery activities once an incident occurs; potentially placing the survival of the business at risk. Smart businesses plan for and exercise for the unexpected. A well written and exercised plan will ensure business survival no matter what scenario causes a major incident.

Would you not insure your house?
It is unthinkable today that we would not insure our house against a disaster. Even our cars for that matter are, on the whole, insured.

Why?
We want to be protected. We want to know if something goes wrong, we have an “out” and have financial backing should the improbable happen.

Business insurance is no different and proper disaster recovery and business continuity plans are a form of insurance. Maybe it’s time to take DR out of the too hard basket.

**If you would like to speak to TD about DR, or any other technology, please [contact us here](http://www.thomasduryea.com.au/contact/).**