Severe weather in South Carolina comes with the territory, and for a local communications service provider like HTC, that translates to service outages for the internet, streaming video, phone, and wireless networks it provides to predominantly rural communities.
When these events strike, the Emergency Operations Center (EOC) at HTC faces enormous pressure not only to maintain services, but to prioritize customer support efforts and restore services as quickly as possible. Successfully achieving these goals requires an intelligent automation platform to provide actionable monitoring and reporting on the many different devices that HTC maintains.
During major storms and severe weather events, HTC’s EOC was looking at too many interfaces and struggling to match alarms to locations where services were being disrupted. Power outages often occur over entire counties, and the NOC needed a resource that would help instantaneously prioritize its response. Critical information about which facilities lost power, which ones had fixed generators versus those running on commercial power, and the status of battery capacity or generator fuel levels were not readily available.
With a limited supply of generators, which provide a lifeline during these extreme weather events, HTC had to prioritize deployment based on a quick assessment of areas most in need. Once generators are deployed, they require crews to remain onsite to protect against theft, but those crews had no way of knowing when power returned and when they could move on to the next site.
Before implementing Resolve, the alarms that came through to the NOC provided just basic information that lacked indicators of severity or available backup power. Additionally, the task of manually checking generator fuel levels was time consuming for personnel while urgent customer issues also required attention.
Beyond storm-related activities, HTC has a unique service agreement with a national wireless carrier to provide the last mile of fiber to cell sites and data circuits. Under their contract, HTC must provide monthly reports on a number of key performance indicators (KPIs), including frame loss, jitter, latency, and many others. These reports were very time consuming to produce manually.
HTC’s EOC needed to easily and quickly aggregate and analyze many disparate data points during storms and major weather events in order to get complete visibility across their network operations infrastructure. In doing so, they could reach their ultimate goal of providing uninterrupted services regardless of weather-induced outages or restoring services as quickly as possible in the event of any disruption.
Achieving the latter requires quickly dispatching technicians to sites that require the most urgent attention. HTC needed to reduce manual efforts and automate the gathering and analysis of critical information to accelerate the process and to enable smarter, data-driven decision-making. HTC also needed to generate monthly KPI reports more efficiently to meet their SLAs with the national wireless carrier.
The biggest barrier to achieving HTC’s goal was the time-consuming process of manually collecting, aggregating, and analyzing data from many different sources. Staff had to login to individual pieces of equipment and disparate systems, pull input from customer calls, and then attempt to connect the dots between those data points by hand.
The process was extensive and time consuming. Ultimately, the EOC was forced to make decisions informed by estimations, and they knew they needed real-time data.
The Resolve Actions intelligent automation platform was implemented to perform proactive network testing, capturing detailed health and diagnostic information that is then used to build dynamic reporting maps for a wide range of equipment, including power, cable modems, generators, DLC batteries, and more. Resolve automatically pulls extensive data from these devices, including fuel levels, environmental conditions, latency, frame loss, jitter, IP pool usage, and battery voltage.
All of these reports and maps provide HTC’s network operations a central view of precisely where and when power goes down and where a storm is having the most impact. The maps include dynamic links to spreadsheets showing the expected life of a battery and diesel generator fuel levels by volume and percent, so they can determine exactly what’s running on generator power and respond accordingly.
HTC’s EOC is now able to make data-driven decisions about where to focus emergency response efforts, from truck rolls to internal diagnostic and remediation processes. The right resources are now deployed more efficiently to the places most in need. For example, when Resolve alerts the team that a DLC battery has just an hour of life remaining, HTC can dispatch technicians ASAP while batteries with longer life can be safely serviced at a later time.
With Resolve Actions, generator crews also receive alerts immediately when power is restored, enabling them to switch back to commercial power sources and complete their jobs much quicker and move on to the next customer in need of support. Additionally, Resolve enables HTC to intelligently select the best mix of equipment based on specific customer needs.
To meet their SLAs with the national wireless carrier, HTC now creates detailed reports automatically in just a few minutes. Resolve’s ability to automate reporting on performance, health, and diagnostic data also helps HTC determine when staff must be deployed physically to locations during dangerous weather conditions. “One of the most significant benefits of using Resolve is being able to efficiently manage storms in a manner that keeps people out of harm's way,” said Joey Billings, NOC supervisor, noting how the cable maintenance manager is now able to perform his job remotely during emergency weather.
Network operators have an abundance of information available in a single pane of glass at their fingertips, complete with links detailing battery life and other health and diagnostic indicators.
“We’ve got a wealth of device data all centralized now, so people don’t have to manually log in to individual pieces of equipment. They can just look at Resolve and see everything in one place,” Billings said.
Ultimately, Resolve helps HTC improve customer service, especially during emergencies. While Resolve Actions creates the most value for HTC during extreme weather events, the team relies on intelligent automation to manage mission-critical resources every day, saving countless hours by automating a wide range of processes. According to Billings, “What used to take us hours to do, Resolve does every five minutes.”
AppDynamics + Resolve help solve challenges with full-stack observability and automated remediation.
Learn five benefits an automated DDM solution can bring to your CMDB management process.