Disaster Recovery Plan value proven in a real-life situation

A company’s Disaster Recovery Plan value is often questioned until it called into use. This was demonstrated during the February 2022 floods in Lismore, which were among the largest the nation has ever seen. As a result, the flood affected 85% of Adaptive Computer Services clients in some way. We spoke with Peter Power of M-Power Computing about one case.

Peter is the front-line IT specialist consultant for a number of Northern Rivers businesses. He uses Adaptive Computer Services to supply server and network support and to source new IT components for his clients. Peter was kind enough to answer our questions about the Disaster Recovery plan value of his client Southside Truck Centre, and how it proved itself in the floods. Of particular interest is the role that Datto Cloud Backup played in quickly restoring operations using cloud-based servers.

What was your first indication that this one was a biggie?

I had just started a 10-day holiday at Coolangatta Sunday 27/2/2022 and received a phone call from Steve Robinson around 2:45am Monday morning 28/2/2022 – flood water was rising fast, what do we do with the equipment in the computer cabinet? We agreed to power off the server and associated equipment in the cabinet, disconnect equipment from power, label cables with masking tape and permanent marker then disconnect, and use their fork-lift to relocate the cabinet to the 1st floor storage area. This was the first flood to inundate the premises. [ACS: Luckily the flood didn’t make it to the first floor, so the server and associated equipment were saved, although some parts did need replacing.]

How did things look when you arrived?

I did not attend site until return from Coolangatta, 9/3/2022,. However, I did work remotely from 28/2/2022 until then, to ensure the operation of Southside’s five remote sites (Port Macquarie, Tamworth and three at Coffs Harbour) and remote access for the Lismore users.

The site was still being cleaned, and there were no power or internet services other than the 4G/5G mobile phone network.

What damage was done?

1.8 metres of water in the premise affecting the whole site.
14 PC’s, 17 monitors, 4 notebooks, 3 Ubiquiti Wi-Fi devices, several vehicle diagnostic toughbooks, data and power outlets were either lost or flood damaged.
Spare parts and oils inventory, workshop equipment, vehicles, and paper-based records were also lost.
No power, NBN or wireless internet services (mobile phones remained working).

What had to be done?

The priority was to activate the Datto cloud backup servers and get the five remote sites operational. [ACS: The Datto business continuity and disaster recovery, or BCDR solutions allow you to create a cloud-based replica of a server within minutes].
Clean-up flood mud and recover whatever could be salvaged at Lismore.
Verify data and phone cabling, power switchboard and outlets.
Reinstatement of power and internet services.
Replace lost PC’s, monitors, notebooks, vehicle diagnostic toughbooks and printers.

When did you contact ACS, or did they contact you?

I contacted ACS on Monday morning 28/2/2022 to initiate the Datto SIRIS cloud disaster recovery.

What were the first steps?

Activate DATTO cloud server backups, configure access via OpenVPN, obtain VPN configurations and install OpenVPN on all remote site computers.
Activate Maillguard Live for email continuity.
Setup access to DealerPro from local PC’s at remote sites (DealerPro is Southside’s 3rd party cloud hosted dealer management system).
Configure remote computers for local printer access.
Configure remote computers for OpenVPN and RDP to cloud backup server when it became available.

How long before operations could get underway again?

I arranged for Mailguard Live (a mail management solution) to be activated 1/3/2022 to provide email continuity.

The five remote site branches and general remote access were operational 2/3/2022. The Lismore HQ was still affected by flooding.

How long did the full process take?

The business operated successfully from the Datto cloud server from 3/2/2022 until 1/5/2022 when the Lismore physical server was reinstated. The server required a replacement smart array RAID controller and fan damaged by the moist conditions.

How were ACS involved? How did they do?

ACS activated the Datto cloud backup servers, provided open VPN configuration files, sourced replacement PCs, monitors, laptops and replacement parts for the physical server. They also reinstated the physical server.

They did an excellent job, considering this was the first time they had to recover using Datto cloud backup, from a disaster of this magnitude.

Can you comment on ACS partners in the project (hardware suppliers, security, backup and monitoring systems, email, etc.)

Datto BCDR was crucial to the recovery process.

Dicker Data supplied the replacement computer equipment.

If ACS weren’t involved, how would the process have been?

Business continuity would not have been achieved.

What things were in place before the event that made the recovery easier?

Physical host server running two virtual machines (A domain controller running Exchange Server and company file shares and a Terminal Server).
All sites were operating by RDP connection to the Lismore server with access to all company software applications, printers, and digital information.
Datto BCDR for servers, files and documents was crucial.
MailGuard for mail management.
DealerPro – Southside’s 3rd party hosted dealer management system.

What did you learn from the process?

Power, internet and phone services are critical. The Lismore offices relied on standalone generators for power and on the 4G/5G mobile phone networks to operate during the recovery process.
Reliable cloud server backup is critical.
The computer server room should be permanently located to the 1st floor (and this has been done).
Consider cloud hosted servers going forward. We are looking at this now in conjunction with ACS.