How Effective is Your Data Center’s Disaster Recovery Plan?

Disaster Recovery is an essential component of every business plan that provides a framework that plans for the worst in the event of a natural or man-made disaster. Within Information Technology:

Disaster recovery (DR) is an area of infrastructure and security planning to minimize the imact of significant negative events within your organization. A disaster recovery plan is a structured document that instructs your staff on what to do in the event of significant, unplanned incidents. Disaster recovery plans enable businesses to maintain, replace, or resume mission-critical functions following a disaster.

Is Your Data Center Disaster-Proof?

With hundreds of Delta Air Lines flights recently grounded in the wake of a power outage, companies of all sizes started taking a hard look at their own IT operations. At the top of the list is an assessment of disaster recovery plans.

More than likely, the overall damage to Delta has yet to be assessed. According to the Wall Street Journal, the carrier not only is facing millions of dollars in lost revenue; it is suffering a major blow to its brand reputation as one of the nation’s leading international airlines.

Companies the size of Delta will survive. Organizations any smaller are not as likely to quickly get back on course after a significant setback.

For anyone operating a business in the Midwest, the threat of snowstorms, tornadoes, hail, and floods is constantly at the forefront of any emergency management plan. Indiana, for example, has had more than 18,200 weather extremes during a 60-year period leading up to 2010, with a good portion of them involving those weather events.

Earthquakes? Unlike California, which had 1,987 earthquakes with magnitudes of 3.5 or above during that period, Indiana only experienced one significant earthquake since 1950. That earthquake, which measured at a magnitude of 4, struck in 1984.

However, as you reassess your company’s emergency management plan, you’d be wise to consider the worst possible scenario involving an earthquake. According to the Federal Emergency Management Agency, the Midwest could face the “highest economic losses due to a natural disaster in the United States” because of its location on the New Madrid Seismic Zone, one of the most active faults in the nation.

More than seven years ago, FEMA had issued the warning for the states of Alabama, Arkansas, Illinois, Indiana, Kentucky, Missouri, Mississippi, and Tennessee, pointing to a study that showed an earthquake with a 7.7 magnitude is possible. Some scientists had reported the event could happen within 50 years.

Hurricanes, floods, hackers or any other natural or human-made disruptions can pose a threat to your data centers with little or no warning at all. Therefore, having a solid Disaster Recovery plan is no longer a choice – unless you want to start over from scratch.

Most organizations today have a secondary data center or external wholesale colocation backup facilities, but having a planned Disaster Recovery colocation blueprint is critical to assess the vulnerabilities of various risks affecting the data center reliability and functionality. The DR blueprint should have careful risk assessment keys, which illuminate the impact of a disaster on direct users, stakeholders, etc. facilitating efficient prioritization and scoring of risk assessment.

Business Continuity Plans vs. Disaster Recovery Plans

Neither big or small organizations can afford downtime in their data center operations. The losses due to downtime have jumped to almost $138,000 on an hourly basis in 2012, which is three times more than the loss calculated in the initial days of data center usages way back in 2004 when the figures were just around $42,000 an hour.

Taking the downtime factor out of your operational formula is impossible because, even if you have the necessary resources, you still must deal with Mother Nature. The recent examples of Hurricane Sandy and the devastating Japanese Earthquake are testimonials to this fact.

Therefore, ensuring business continuity is vital, and this calls for radical steps. But you should not confuse a business continuity plan with a disaster recovery plan since these are two different concepts that ensure data protection.

Disaster recovery is just a piece of the bigger business continuity plan. The aim of disaster recovery is to restore your data after an unexpected disaster in your data center facility.

On the other hand, business continuity combines all efforts and managerial principles that are involved in making sure that key business functions like IT, resource management, and finance are not impacted in the event of a disaster. Without a business continuity plan, it would be very difficult for organizations to get things back to normal. As IBM pointed out in 2011, nearly 43 percent of companies experiencing a big data loss never reopened their business when disaster recovery alone was not enough to undo the damage.

So, foresight and planning for inevitable threats and determining the impact of disasters on your facilities are quite important to ensure continuity. In other words, your data center operations need to have a business continuity plan if they are to survive significant downtime threats.

How Effective is Your Data Center’s Disaster Recovery Plan?

More than likely you’ve invested the resources, time, and cash to insure your business properly in the event of a fire, PR disaster, or other emergency/catastrophe.

When it comes to data centers, it’s important to implement a sound disaster recovery plan as part of that equation. According to recent statistics, data center outages cost companies in the United States a total of $700 billion every year. And the typical data center can expect to lose about $9,000 a minute because of an unplanned data center outage.

Fire, storm, or flood can immobilize a business, especially when servers and other IT equipment are vulnerable to damage. The best way to protect your data and essential equipment is to perform a disaster recovery risk assessment, which can guide you in determining the types of disasters that could hit your business, the solutions that may be required, and the overall expenses you’ll need to recover. As part of your disaster recovery (DR) plan, it’s also important to consider backup contingencies, commonly called redundancies, for your essential IT infrastructure.

Consider the following points sound reminders for a secure, quick, and efficient DR plan:

1. Assess the impact on business – Risks can vary with industry, geography, and various other factors, but there are four general categories: Financial loss, operational commotion, reputation damage, or regulatory penalties. The assessment should answer two major questions in the event of a disaster: how much data would suffer and what would be the resultant financial loss, and how soon are you required to resume the operations?
   • Prioritization plans. As with any company, some areas of your operations will require more immediate data recovery attention than others. Determine which areas are a priority for getting back online based on your company’s objectives in ensuring that clients receive services.
   • Take inventory. Review your equipment, both hardware, and applications, taking note of what will need to be replaced in the event of damage during an event. This ensures that components can be quickly addressed by contacting the vendor for replacements or solutions to help your company recover.
   • Assess your downtime tolerance. What is your level of dependence on your servers? Are you operating primarily online? Or are you operating a service agency with limited use of technology for specific functions? Depending on your answer, you may be better able to determine your recovery point objective and recovery time objective.
   • Categorize your functions. Determine which applications are a priority under your disaster recovery plan. One category should include applications that are critical and must be addressed urgently in the event of a disaster. Other categories can range from several hours to several days in priority, based on how critical they are to the success of your operations.
2. Team communications and having a trained team on standby – While the goal is to have 99.995% uptime, running a data center means planning for disaster and requires a recovery plan in the event something catastrophic, whether digital or natural. Two of the most common occurrences a data center may experience is a facility becoming too hot or too cold, which will affect operations. Both of these situations can cause damage to supporting hardware, and they each require different techniques to remedy the situation. What most people don’t think about is that different types of disasters call for different staffing requirements. Different disasters call for different staffing requirementsRecovery in a hot site means all your supporting hardware infrastructure and data are available in the event of a disaster, making it easier to perform the recovery process. In this situation, almost everything is already set up to perform a recovery procedure, so you may only need a few staff members to be available during this type of disaster. Having a support staff during recovery in a hot site isn’t always needed, but having a few experts will suffice for the operation.Recovery in a cold site means that there might not be any infrastructure or data, which requires more “all hands on deck.” A large supporting staff is needed to meet the requirements for a recovery in a cold site. This type of disaster can also be costlier due to the time and hardware it takes to resolve the issue.

   Although each of these scenarios requires different staffing requirements, there needs to be a disaster recovery plan in place for a variety of scenarios, so your staff knows what to do when disaster hits. Knowing how the staff will get to the site promptly and what type of accommodations they will need to make are important factors when coming up with a plan. When choosing the right IT staff for different scenarios, make sure you know which professionals you will be able to call on immediately after a disaster.

   • Establish an emergency plan. Your employees are the most valuable assets of your company. Regularly keep them trained on what to do in the event of an emergency, regularly updating them on safety drills to perform during tornadoes and other extreme weather. If you haven’t done so already, incorporate earthquake drills. Numerous businesses throughout the Midwest have already performed a massive earthquake drill in connection with the U.S. Department of Homeland Security. For tips on performing these drills, see the checklist for the Great Central U.S. ShakeOut.
   • Assign roles of responsibility. Identify the members of your team who will be contacted for different roles and at different times. Critical team members should be contacted immediately to handle specific responsibilities under your disaster recovery plan. Remember that employees come and go. You could be in a disaster situation, and a previously identified employee for your plan may no longer be with the company. Frequent testing will uncover these issues. Also, identify any third-party consultants who are critical to your plan
   • Establish a communications plan. How do you plan to get the word out to employees on next steps in the wake of a natural or human-made disaster? Outline those guidelines in your plan. Under challenging circumstances, how will employees receive information on where to go next? If systems are down, including phone and the internet, you may need to establish alternative methods of communication. You can also distribute protocol in writing beforehand. Train employees on how to access those details.
   • Train your employees to ensure adoption. Engage your employees in the disaster recovery plan testing. It’s not enough to test your disaster recovery plan every couple of years. Going through the exercise may be disruptive, but it could be the key to keeping your operations viable when disaster does strike.
   • Do we have a plan for safety? Safety measures include more than the hardware and appliances at your location – any staff who might be unaware of a sudden disaster or the safety plan can be detrimental during a disaster. Make sure that your plan for disasters and recovery is circulated among all the personnel at your data center so that everyone knows what to do during a crisis.
   • Do we have a trained team on standby? You can’t expect all personnel to know how to react to a disaster. It is important to have an experienced and trained team who are ready to take control. A lead team member who can get to the data center even in harsh natural conditions is beneficial to have in case of an emergency.
3. Technology Risks – To have proper data restorations, the original backup data should be relevant, validated, and free of errors. Issues like pulling data from old versions and legacy based systems, or version control issues, etc. should be avoided.
   • Have we planned for a power back up? It is unlikely to have a data center that doesn’t have a backup plan, but it is important to check the details of how long the backup would last in case of a prolonged outage. You may also need to identify priority servers, as well as the inactive ones so that you could distribute the backup power efficiently.
   • Have we planned for Data redundancy or back up? Geographical redundancy of your data is vital, and with the advent of the cloud, you no longer need to depend on multiple locations for your centers to achieve this. If your data is backed up regularly and accessible from the cloud, you are home safe
4. Always be prepared – Is your structure disaster-ready? In most cases, your most valuable resources are stored in a well-insulated and safe structure but think about the implications a natural disaster such as a flood or storm might have in your surrounding areas. Any small repair that is left unattended, such as a blocked roof drain, could have major impacts during an unexpected natural disaster. Remember, you may need to physically relocate, ramp up operations in an emergency, account for transport, power outages, etc. at any time So, be ready with sufficient, efficient, and quick team members.
   • Assess your physical space. Next to your employees, protecting your physical investment is a top priority. Get an assessment of how secure your building will be in the event of flooding, tornadoes, and other natural disasters. If you’re planning to move into a new office or warehouse space, make sure you understand how well it was built to withstand strong winds, floods, and other extreme weather. After Superstorm Sandy in 2012, several businesses in New York realized that basements aren’t ideal locations for servers. Floodwater poured into the basements of several office buildings, disabling servers along with diesel fuel pumps needed for backup power for generators. Because of the storm, data center representatives started to re-examine the scope of their disaster plans, considering more severe storms as potential risks. Don’t wait until the next disaster to assess whether your servers are stored in a secure environment. If stored in a basement, for example, research the feasibility of moving them to ground level or higher. If that’s not possible, consider moving your server to a safer offsite location. Also, consider hiring a professional consultant to help you make the right choice.
   • Basic operations. Where will your employees work if a disaster destroys your office building, or you lose power for a prolonged time? After a weather-related disaster, road conditions and other hazards may mean working remotely is the only option. But if it’s safe for employees to travel, having a disaster recovery hot site – with servers, laptops, and office space – can allow you to continue running your business. A cold site, one that offers only office space, can be a viable alternative if you’re able to effectively install the equipment you need to resume operations. It may be a less expensive option overall, especially in the event of an extended outage.
   • Develop a backup plan. Whether a natural disaster or other events cripple your business, it’s wise to protect your mission-critical data by setting up a secondary support infrastructure that allows you to continue your operations seamlessly. Colocation centers can not only provide automated backup and recovery services, but they can also provide a location for your employees to work if your business location is inaccessible. 

Steps to Choosing Colocation for Disaster Recovery

When creating a disaster recovery (DR) plan for your company, you have numerous options. But one of your main objectives is moving data to a location where it can be secure while, ensuring that your team can monitor the data, or at least its security. While you could build an additional data center offsite, another viable alternative, as many companies have discovered, is using colocation as a cost-saving and effective solution. However, using a third-party service provider requires plenty of research on your part to ensure that you have the right fit.

Here are some of the things you should consider:

• Analyze the location. Is it far away enough from your facility to ensure that you’re not facing the same challenges if a natural disaster strikes? Also, check to see if the location is an area that has a low risk for natural hazards like flooding.
• Ask about experience. You want to know if the provider has experience working with clients like yours, in size and challenges. Also, check its track record for compliance and industry standards as well as its ratings for uptime. Don’t forget to ask for referrals.
• Understand all fees. Make sure the company provides an extensive explanation of costs and fees as well as policies. Have your legal team or an attorney review the contract beforehand.
• Ask about other services. Do you need experts outside of your company to manage your IT assets? Ask the provider if this is provided. Some colocation providers offer a wide range of services, while some are more focused on specific areas and needs. Analyze your needs and choose a provider that best meets them.[/section]

Disaster Recovery Tests and Updates

Conduct semi-annual tests for your DR plans depending on the changes taking place within your environment to refine it constantly.

With major floods hitting states across America, including Oregon, Illinois, Missouri, and Tennessee, scientists are now predicting there’s plenty more to come over the next five years putting residents, businesses and other organizations at risk. That’s why experts are increasingly concerned about the need for data centers to put disaster recovery plans in place to minimize disruptions.

“Sea-level rise has already doubled the chances of extreme flooding in locations around the U.S., and that will only accelerate in the coming decades,” says Benjamin Strauss, vice president for climate impacts and director of the Program on Sea Level Rise at Climate Central. Strauss said, by 2020, coastal areas in states like Florida and Louisiana could be significantly affected by the sea-level rise. More than likely, you already have a disaster recovery strategy in place. However, that process does not end with making sure you have an effective plan to protect your company against significant loss. It also includes developing steps to test, review, and adjust if necessary, to ensure that you’re not at risk for overlooking flaws in the DR plan.

Here are five crucial steps to make sure that your operations can successfully recover in the event of flooding or another critical event.

1. Appoint employees to a planning committee. Designate employees and key managers to be part of a DR planning committee that review and update your DR strategy. Make sure you have a team that represents different areas of your organization to address unique needs.
2. Perform a risk assessment. One of the top priorities of the planning committee is to conduct a risk assessment that covers a variety of scenarios, including the inability to assess data or to communicate with one another. Be exhaustive in determining the risks, as well as determining the costs with each scenario.
3. Prioritize functions. Obviously, not all functions are created equal. Determine which departments and department functions are a priority for recovery after a disruption. Rank them in order of importance and include that in your recovery plan.
4. Assess your backup plans. One of the key components of a disaster recovery plan is developing backup resources. Many companies choose to build a backup plan through a third party to avoid the expense of building out the facility or purchasing the equipment themselves.
5. Test and test again. With your DR plan finalized, it’s important to regularly test and review it for areas that may need updating. Also, make sure that you keep a printed copy in a safe area outside of your facility that you can reach in the case of a disruption.

Update Your Disaster Recovery Plan by Answering These 2 Questions

Every business, whether large, medium, or small, should have a comprehensive disaster recovery plan that guides employees on how to recover in the event of a breach caused by a natural or human-made event. Most of us realize that. But when was the last time you reviewed your plan and identified any gaps that may require tweaks or significant changes?

Whether you’re just now developing a disaster recovery plan or at a stage where you need to review it, start the process by asking these two questions:

1. How fast do you need to recover after an event?
2. What do you need to recover quickly?

By outlining the answers to those questions, you can start implementing a strategic plan based on your specific needs and requirements, according to Alex Carroll, who discusses the important elements of a disaster recovery plan in this video. Carroll said company officials would need to bring numerous team members to take a granular look to determine which areas are more critical than others.

The solutions could vary extensively depending upon which functions of your company you need to be recovered immediately and, as a result, will be the most expensive recovery projects in your plan. You also may determine that other data can wait a week before being required to get your company back to normal or return to operations, as described in the industry.

Companies also may want to determine if they need work group recovery — another location that allows employees to physically gather to keep working in the event of a disaster or other event that prevents them from working at the company.

• Determine the cost. If your business has experienced growth since you last reviewed your disaster recovery plan, it’s time for an update. A solid plan should consider what is required to get back to service as quickly as possible, as well as the expenses involved with different disaster recovery scenarios.
• Look beyond nature. With cyber attacks increasing, your threats have expanded beyond acts of nature like floods and earthquakes. Factor in how malicious attacks can impact your systems and outline a plan for recovery.
• Identify your key people. It happens in every business. People come and go. Some move up in the company. Make sure your plan reflects the key employees who will be critical to addressing issues with the data center in case of an emergency. Make sure all contact information is current, even if you are dealing with the same list of employees.
• Retrain employees. Review your disaster recovery procedures to make sure instructions on what to do in the wake of an event still make sense. Once the plan is solid, remind employees of their roles. Consider it a tornado drill. Everyone needs a reminder.

Floods, tornadoes, severe storms, earthquakes, and hail were among the natural disasters hitting the United States in 2016. Some regions of the country, of course, are better than others when it comes to the damaging effects of Mother Nature.

But companies globally must deal with the increasing threats caused by cyber-attacks. No one is immune. These criminal acts have impacted small companies to big names such as Wendy’s, the FBI, Verizon, LinkedIn, Citibank, and the Democratic National Convention, according to The Heritage Foundation.

With threats coming from all sides, companies need to focus on developing effective disaster recovery plans — ensuring that they’re updated and regularly tested to minimize the fallout caused by disastrous events. Doing so could be the difference between your company being down for days, even weeks, or being able to fully recover and operate within an hour or two of an event crippling your system.

Entrusting your facility operations to a reliable data center service provider like Lifeline Data Centers can certainly help you considerably reduce downtime concerns in your operations. Take a quick look at our services and our client testimonials.

Alex Carroll

Managing Member at Lifeline Data Centers

Alex, co-owner, is responsible for all real estate, construction and mission critical facilities: hardened buildings, power systems, cooling systems, fire suppression, and environmentals. Alex also manages relationships with the telecommunications providers and has an extensive background in IT infrastructure support, database administration and software design and development. Alex architected Lifeline’s proprietary GRCA system and is hands-on every day in the data center.