TRENDS & TECHNOLOGY

What It Takes to ‘Thread the Needle’ on Industrial Disaster Preparedness

POSTED BY TSI TURTLE SERVICES BLOG IN PERSPECTIVES, SERVICES


 
There is plenty of reporting on the heavy toll of natural disasters. From 2006 to 2015, the United States had 645 natural catastrophes, with Hurricane Sandy standing out. That storm destroyed 650,000 homes and left 8 million people without power. What’s troubling today, though, is that many industrial facilities and infrastructure remain woefully unprepared to recover from disasters like Hurricane Sandy.
 
“Many people have forgotten the tremendous efforts to recover from Sandy and are falling back into the same old bad practices,” warns Chris Price, Division Director at Turtle & Hughes.
 

Are You Truly Ready?

When KPMG surveyed a global sample of executives in 2016, they found that over half of surveyed companies had losses from severe weather, and more than a quarter had business interruptions from flooding. More telling is that 56 percent said they needed to activate their business continuity plans in the past year, but only 30 percent said they completely satisfied their recovery times.
 
So, what’s going on? Why are utility and industrial plants not rising to the disaster recovery task?
 
Complexity is one of the culprits, especially when it comes to critical industrial infrastructure. People who manage modern industrial control systems often don’t know the true depth of applications and configuration settings that are present on these systems.
 
“We perform system-wide assessments and analyses of industrial technology and automation at facilities so that they understand what exists at their facility and why it’s important,” Price explains. “When people familiar with the systems review these assessments, they’re often surprised to see applications, devices with configuration settings and infrastructure they knew nothing about. In many instances, legacy or obsolete systems exist that were installed a long time ago, but because they still function, receive little oversight or interest. It’s the old adage, ‘If it’s not broke, don’t fix it.’ The time to focus on this infrastructure is before disaster strikes, plan ahead, stay ahead. How many people budget for a disaster?”
 

Lack of Reliable Backup

The problem of complexity is eclipsed by the problem of subpar backup plans. It’s not that companies didn’t have backup plans for their critical infrastructure when Sandy came ashore, but that their backup plans were flawed. During Hurricane Sandy, one Turtle & Hughes customer lost their onsite computers and industrial control systems because they were under water, only to find that the backups that were stored in an adjacent building were also under water! To make matters worse, the vendor that installed the systems had their backups distributed among onsite computer systems – all of which were under water. Fortunately, Turtle & Hughes Consulting Division, TSI Turtle Services, was able to recover the main computer and control system configurations through the use of forensic hard drive data recovery.
 
In the industrial plant sector, the horror stories of recovery often included processes that couldn’t be restarted because the settings on industrial equipment and devices were wiped and not easily replicated. Simply replacing components wouldn’t work to get back up and running quickly because those components required someone to program them from scratch. And, that’s assuming the components were even available.
 

Using Planned Replacements is Not a Recovery Strategy

Companies often make the mistake of depending on a pipeline of “available” replacement components for disaster recovery. This includes automation equipment, computers and server infrastructure. The problem is this infrastructure may have lead times of weeks. TSI Turtle Services assists customers in taking a proactive approach to disaster recovery through its understanding of customer infrastructure and relationships with technology partners.
 
TSI Turtle Services recommends backup servers or Network-Attached Storage units that are strategically staged to support zones or areas of systems, so, a particular server can satisfy a failure in a number of different systems. By leveraging technology, such as Virtualization and Private Cloud Technology, it is possible to shorten recovery time to as little as 15 minutes. Additional steps may be taken to incorporate full redundancy. Normally the right mix of resiliency and redundancy is the way to go.
 

The Benefits of Standardization

Facilities with disparate industrial infrastructure can delay the recovery process. Relying on different manufacturers supporting different systems complicates disaster recovery because the various vendors have different lead times, different availability or different vintages — and some don’t even have components available. TSI Turtle Services assists customers with standardizing industrial infrastructure to ensure systems can be maintained, critical components are available, and consistency exists across the operation.
 

Closing the Disaster Recovery Loop

Industrial customers face more difficulty with disaster recovery when documentation, testing and training are subpar. Not only does this open the door to confusion during stressful times, but it also raises the risk of grave mistakes during recovery. It’s critical for people to understand their operations and be able to identify their critical infrastructure. But it doesn’t stop there. Verifying backup recovery strategies and teaching personnel how to recover from disasters is just as critical.
 
TSI Turtle Services brings expertise and specialized tools to standardize disaster preparedness and recovery through the use of assessments, documentation, testing and training. Learn more.

Subscribe to
Trends & Technology:

Connect with Us:

Archive

  • 2017
  • 2016
  • 2015