The power of technology: OMES, Dell and others secure Oklahoma's most critical data

Like many states, Oklahoma contains its fair share of history-defining tragedies. What sets the state apart is its response and recovery from disasters like the Moore tornadoes in 2013, extreme floods in May 2015 and the 1995 Oklahoma City Bombing, which served as the foundation of the Oklahoma Standard. This standard is defined by Oklahomans uniting on-site to help each other during great times of need. However, with a catastrophe like COVID-19, the Oklahoma Standard looks starkly different.

Safety during a pandemic means separation: Staying home unless absolutely necessary, avoiding physical contact, distance from loved ones, mask-covered faces to slow the spread. In-person relationships, including family, friends and coworkers, become digital. Resources must be delivered virtually, whenever possible. Therefore, a secure, reliable network is crucial to continue essential services for Oklahomans during this crisis.

Without a reliable data network, Oklahoma would lose all its critical data during its citizens’ biggest time of need. There would be no access to public safety, health or tax information. There would be no human services, unemployment information or other government resources Oklahomans use every day. Life in Oklahoma would stand still. It would take months, if not years, to rebuild the state’s data network.

That recovery effort would have fallen on the shoulders of the Office of Management and Enterprise Services, whose Information Services division is responsible for protecting Oklahoma’s networks and digital assets to support public safety and cybersecurity. OMES leadership realized if a natural or cyber disaster were to hit Oklahoma during the darkest part of 2020, the state would shut down indefinitely.

With over 30,000 employees working from home, OMES Executive Director Steven Harpe and Chief Information Officer Jerry Moore knew the state needed a huge change to continue protecting Oklahoma’s most valuable assets.

For decades, state data was stored on tapes in Stillwater, Oklahoma, and the data backup process was flawed. If a disaster were to occur, especially in addition to the pandemic, it would have taken weeks, possibly even months, to get state services back online.

OMES set out to implement a modern Disaster Recovery project for Oklahoma to protect digital assets and deliver seamless services in the face of crippling natural or cyber disasters. The project required three primary components: redesign and build the state network from scratch, transition massive amounts of data to a central location, and establish a sustainable backup process.

The sheer amount of state data meant OMES couldn’t accomplish this feat alone. While partnering with local businesses is always the state’s first choice, only a handful of companies worldwide have the capacity to handle the scale of Oklahoma’s DR project. Director Harpe said that was the primary reason the agency chose to work with tech giants Dell Technologies, VMware and NTT DATA Services.

“Oklahoma is a $22.9 billion entity,” Harpe said. “It was imperative that we partner with companies capable of making our success possible in a matter of months.”

While not Oklahoma-based businesses, they contribute to the state’s economy and workforce. Dell, alone, contributed an estimated $233 billion to the gross state product during fiscal year 2017, and employed over 1,450 people in Oklahoma last year.

Digital dependence during the pandemic highlighted the importance of a resilient network. Once Oklahoma implemented telework to keep state employees safe from COVID-19, new cybersecurity measures were essential to protect the network from bad actors looking to take advantage of the pandemic. OMES and its partners made numerous improvements to physical and cybersecurity processes – such as Windows Virtual Desktops, cloud-based drives and workflows, and digitized mainframe systems.

Moore said the increase in security threats required technology the state previously lacked when its workforce remained on-site.

“There would be no way to recover quickly from a cyberattack with the current network in place,” Moore said. “It was fundamentally designed for state employees to be working in a state office building, using a state computer on the state network.”

In addition to replacing Oklahoma’s entire core network equipment, OMES implemented Windows Virtual Desktops to empower more state employees to return to productive work faster without having to go into the office. These virtual desktops allowed a connection through any computer, eliminating the need for state-owned laptops with VPN clients on them, which were almost impossible to obtain early in the pandemic due to global supply chain demand. The WVDs were also a big help for pandemic call centers and access to other agencies.

OMES moved shared drives from old mapped drives that were only accessible on state property to Microsoft Office 365 and OneDrive, which allowed employees to access their data from any location.

Moore said they virtualized several old physical mainframe systems – like the Oklahoma Employment Security Commission’s – that were vulnerable to crashing under heavy traffic during the pandemic.

“Since there was no way to increase the mainframes’ sizes,” Moore said, “we put them in the cloud to make them resilient and recoverable.”

One big objective of the Disaster Recovery project was to establish a central location for the state’s information. The DR team upgraded the Lincoln Data Center in Oklahoma City to fulfill this role. OMES and its partners deployed the latest data center technology and established high-availability infrastructure in the LDC, which can also withstand 200 mph winds.

The LDC would store critical information from many different state agencies. However, each agency’s data requirements are as unique as the services they provide. For example, the Oklahoma Employment Security Commission’s needs vary significantly from the Department of Public Safety’s. OMES, Dell and NTT designed, built and implemented 6,000 servers to meet the diverse needs of Oklahoma state agencies.

The DR team migrated 2,661 terabytes – equivalent to over 665,000 average-length movies – of data, including health records, tax information and other confidential materials onto those servers in just four months.

“What we did in a few months usually takes two years,” Dell Account Executive Christina Bivins said. “We could not have done that without OMES’ workforce.”

A new backup process was crucial once OMES and its partners moved everything to the primary data center. This investment ensured the state could access its digital assets in case the LDC lost connection, which Moore said could have happened multiple times in the past three months, alone.

The DR team replicated everything stored at the LDC to back up the information to cloud services and TX1, a secondary data center in Garland, Texas. This best-in-class data center promotes environment sustainability but can also withstand an EF-3 tornado (winds up to 136 mph). Like the LDC, TX1’s backup generators run on diesel fuel to continue services in the event of power outages.

The state chose Texas because it is out of region but within driving distance should OMES need to be on-site. Natural and cyber disasters could impact local networks, telecommunication and utilities, so outside systems allow continued delivery of essential services.

The new process benefits all Oklahomans because it’s not limited to scheduled copies of the state’s data. Instead, the LDC and TX1 sync information continuously, ensuring not even a crippling disaster could interrupt state services.

“2020 proved that the state’s technology infrastructure must be ready to support any disaster at a moment’s notice,” CIO Moore said. “By investing in TX1, we are able to provide our customers with the guarantee they can continue delivering core services to Oklahomans, no matter the circumstance.”

Moore said, previously, any information that wasn’t backed up was at risk from natural or cyberattacks. Unlike the new automatic system, the former backup method was manual, labor-intensive and unreliable.

“The systems that were backed up were either on tape or at a remote backup site,” he said, “but they were simply data, not a functional system. We now have a redundant server for each system in TX1 and they are in a constant state of synchronization. If we were to lose connectivity to the Lincoln Data Center, we could route users to connect to the same systems in TX1.”

This truly redundant, high-availability system goes beyond a simple copy of the state’s data. OMES will be able to access its information within a matter of seconds in a network resilient to potential disruptions.

To underscore this Herculean effort, OMES and its partners didn’t merely replace the state’s entire core network. They also transitioned Oklahoma’s comprehensive digital assets – $22.9 billion worth – to 6,000 servers, each one with its own unique data. They established a backup process with continuous live replication between two physical locations and cloud services. And they finished the entire project in 5 1/2 months.

Bivins said completing the full migration in less than six months required all hands on deck and compiling tens of thousands of work hours to support Oklahoma’s needs.

She said Dell and NTT engaged to the highest executive levels of all companies to procure equipment, ship hardware to the two data centers, set up the necessary software and migrate agency workloads with minimal downtime.

NTT Delivery Senior Director Ingrid Guthrie said NTT’s division president personally oversaw the DR project’s successful implementation and migration in Oklahoma City.

The DR project is one of the first of its kind in the nation, and Bivins said other states are taking notice.

“Oklahoma has become the gold standard for other states looking to improve their disaster recovery process due to COVID-19. Many states have reached out to Dell wanting to replicate the Oklahoma plan.”

Harpe said, while he has completed data migrations on a massive scale during his career as a global technologist, Oklahoma’s Disaster Recovery project was a unique level of collaboration and overall success.

“I have managed data transitions on a global scale for companies like Sabre and ABN AMRO Bank, one of the largest banks in the world at the time,” he said. “However, in 31 years, I’ve never seen such an efficient move of such a large amount of data. Oklahoma has set a new standard of excellence in this space that other states are looking to mimic.”

Dell Vice President Tami Booth, an Oklahoma resident, said the implementation and migration of the state’s DR project was effective and swift, especially for a project of its size.

“It was really cool to be one of the first states in the U.S. to do this large of a transformation,” Booth said. “The process was fast and furious to put together a plan for Oklahoma to provide premiere services to its citizens. I’ve never seen a state do it that quickly.”

The Disaster Recovery project was a crucial solution to protect Oklahoma’s information and allow state agencies to continue mission-critical services without interruption. This project ensured enhanced security, reliability and reduced recovery times from weeks to hours when faced with cyber or natural disasters like a pandemic.

With a new core state network, an upgraded central data center, 6,000 unique servers, a backup online and a backup in Texas, OMES has proved its commitment to the safety of Oklahoma’s most important data.

OMES’ partnership with global tech giants and other state agencies on this project shows the Oklahoma Standard lives on and has adapted to the era of physical separation. Our communities have united, even while quarantined and socially distanced. The State of Oklahoma meets its citizens’ greatest time of need with continued access to government resources and has raised the bar of service on a global level.

“We’ve forever changed the landscape of working in Oklahoma,” Harpe said. “We truly could not have done this without Dell and NTT. OMES and our partners created this system out of necessity, and it will benefit states for years to come.”