<\/p>\n\n\n\n
<\/p>\n\n\n\n
A leading provider of advanced network communications and technology solutions for consumers, small businesses, enterprise organizations, and carrier partners across the U.S. wanted to become more powerful, using automation, as to better understand the customer impact of bad weather and proactively improve their customer experience. <\/p>\n\n\n\n
The Fortune 500 company would need to overcome a handful of pain points: <\/p>\n\n\n\n
And they had their sights set on two big business goals, including advancements in scripting and reducing alert volume to 10,000 per week. <\/p>\n\n\n\n
<\/p>\n\n\n\n
<\/p>\n\n\n\n
The company started in its Network Operations Center (NOC) as its ideal location for automation opportunities, and what would be come transformative change. Company leaders mapped its automation needs to its business goals, and identified four use cases that made the most sense for the organization. <\/p>\n\n\n\n
Automation, built onto those already existing, led to a successful transformation of the NOC. The company improved its alarm triage processes and made it much more efficient, as the need for a NOC surveillance organization was completely eliminated. With the help of automation, each IT professional who was bogged down triaging alarms could then focus on remediation issues and more important tasks that supported business goals. <\/p>\n\n\n\n
1. Circuit Enrichment:<\/strong> Looking up Circuit ID numbers had been a manual process that took up an astounding amount of time. It limited the response team\u2019s efficiency and productivity, especially considering the overwhelming quantity of alarms coming in. <\/p>\n\n\n\n Automation took place of the IT team, automatically looking up Circuit IDs and adding them into the Netcool alarm. <\/p>\n\n\n\n 2. Maintenance Correlation:<\/strong> During regularly scheduled maintenance windows, and as expected, hundreds of alarms were generated. Each alarm had to be factored into the IT team\u2019s time and effort. <\/p>\n\n\n\n Again, to remove tedious busy work from IT\u2019s workloads, automation was implemented to tag each alarm appropriately and once the window closed, clear the alarms out. <\/p>\n\n\n\n 3. Power Alarm Processing:<\/strong> The company relied on its IT team to recognize alarms from different locations whenever a power outage occurred. The IT staff risked making mistakes, as they had to pay attention to each and every alarm. <\/p>\n\n\n\n With automation, the staff no longer carried out this process. Verifying the alarms and appropriately escalating them to a technician was done for them, which allowed technicians to be dispatched immediately. <\/p>\n\n\n\n 4. TDM Switch Diagnostics:<\/strong> The company\u2019s IT team was also responsible for running diagnostics on the switches to identify fault packs. <\/p>\n\n\n\n Once automation was implemented for this case, the fault packs were automatically identified and important details were escalated to technicians, enabling dispatch of techs to the right locations right away, for faster remediation of the issue. <\/p>\n\n\n\n RELATED BLOG: <\/strong>The NOC of the Future: What Businesses Must Know Now<\/strong><\/a><\/strong> <\/p>\n\n\n\n <\/p>\n\n\n\n <\/p>\n\n\n\n Weather was the unstoppable force that struck the organization’s operations and created an unwavering storm of alerts. With what was called \u201cStorm Mode Automation,\u201d the company was able to learn more about the impact on customers and get a handle on the quantity of alarms per week. <\/p>\n\n\n\n The company also correlated network events and found they were all related to a single site. Automation made the events relatively easy to verify, provided a straightforward process to follow with minimal remote work, and gained control for the team by securing the process for handling the events. <\/p>\n\n\n\n Phase 1:<\/strong> The company started using automation for alarm acknowledgement, verification, ticket creation, and routing to the right person for follow-up <\/p>\n\n\n\n Phase 2:<\/strong> They Increased scale of automation with additional devices; and therefore, automation touched 85 percent of the total alarm volume. <\/p>\n\n\n\n Phase 3:<\/strong> The IT team increased the scope of automation with additional functionality for technician dispatch. Right after dispatch, automation follows up, and cancels or closes the ticket. <\/p>\n\n\n\n Phase 4:<\/strong> The \u201cStorm Mode Automation\u201d correlated with Digital Subscriber Line Access Multiplexer (DSLAM) monitoring to examine and comprehend the alarms\u2019 impact on customers. Automation also verified customer services upon receipt. <\/p>\n\n\n\n Phase 5:<\/strong> The company added in lower volume devices until 100 percent of the alarm volume was touched by automation. <\/p>\n\n\n\n <\/p>\n\n\n\n <\/p>\n\n\n\n It was Aug. 25, 2017 when Hurricane Harvey made landfall along the Texas coast near Port Aransas, according to the National Weather Service (NWS). The Category 4 storm brought devastating impact, and continued its damaging path inland, to Victoria, Texas. The hurricane slowed its forward motion, and dropped tremendous rainfall as it paved forward for five more days. <\/p>\n\n\n\n The company, fearing Hurricane Harvey\u2019s catastrophic damage and how it would affect customers, was skeptical of automation during such a storm that would produce a surge in alarm volume. <\/p>\n\n\n\n Hurricane Irma followed closely behind Harvey, and wreaked havoc in the Caribbean after forming in the Atlantic Ocean on Aug. 30. Named a Category 5 storm on Sept. 5, Irma\u2019s wind speeds reached a rare 185 mph, making it only the fifth hurricane to ever reach that speed. By the time Irma reached the Florida Keys on Sept. 10, its winds had slowed to 130 mph, and it\u2019d fell to a Category 3 intensity when it made landfall near Marco Island. As Irma hit Florida, tropical storm force winds extended outward up to 400 miles from the center, and hurricane force winds extended out to 80 miles. <\/p>\n\n\n\n Seeing the opportunity for automation during Hurricane Irma, which would come from an unsettling spike in alarms, the company changed its operations philosophy. For four days after Irma hit the Florida Keys, the company leaned on automation \u2013 testing its ability to hold up, and still provide benefits, when the alarm volume was unimaginable. <\/p>\n\n\n\n It was a test worth taking. The company saw a drastic change for the better, including 2,244 clear events, 1,207 correlations, and 710 tickets created. <\/p>\n\n\n\n <\/p>\n\n\n\n <\/p>\n\n\n\n Fiber cut marked another use case for the organization. Resolve worked with the organization\u2019s L-3 engineers to identify key indicators, as well as the types of alarms that came in when a fiber was cut. The indicators found were codified into a Resolve Workflow, and it then successfully started correlating and identifying the issue. The four key indicators include: <\/p>\n\n\n\n During the automated remediation, in this use case, an alarm came in for a customer port, from which an alarm was triggered. The Resolve engine identified alarms as they came in from customers. Next, the alarms were automatically triaged and diagnosed, and finally, automation ran diagnostics to check for a loop, and it attempted to drop a loop and rebuild the circuit, as to see if it came back up. <\/p>\n\n\n\n <\/p>\n\n\n\n <\/p>\n\n\n\n Using automation at the height of severe weather proved to be of great value to the company, starting with a jump in return on investment (ROI), and direct and indirect cost savings. As with many automation cases, the company started with the tasks that took too much time to complete and were too many for humans to manage. Not only can the company accomplish more with fewer resources, but it benefits from having human IT engineers available for more powerful work that supports the business and moves the needle. <\/p>\n\n\n\n After all, the company set out to better understand its customers\u2019 experiences during weather changes, and the company\u2019s staff was then free to contribute to goals like these that require a human\u2019s deep thinking and analysis. <\/p>\n\n\n\n The company also saw a bright change in meeting service level agreements (SLAs) and returning service to a normal state fast enough. <\/p>\n\n\n\n READ MORE: <\/strong>Coming in Hot: XLAs Fire Up Business Results as User Expectations Rise<\/strong><\/a><\/strong> <\/p>\n\n\n\n No volume of alerts is completely preventable, and customers will inevitably have to deal with power outages when thunderstorms, increased precipitation, and high wind speeds roll through. However, with too many alarms to manage, the outages couldn\u2019t be addressed quick enough to meet the SLA\u2019s terms. By failing to stay within the acceptable outage range, as indicated in the contract, the company risks a lot. A breach of SLAs will cause financial penalties and can lead to damaged relationships and legal troubles. <\/p>\n\n\n\n Resolve\u2019s automation enhanced the entire alarm management process, by adding consistency throughout, from step to step. As a result, the company improved and maintained data integrity, refined ITSM records and resolution, and built detailed audit trails to help determine root cause analysis. <\/p>\n\n\n\n Resolve\u2019s solutions were remarkably easy for the company to use, and developers were trained quickly \u2013 it only took four weeks to get the company\u2019s automation up and running fully, and functioning without flaw. <\/p>\n\n\n\n RELATED CUSTOMER STORY: <\/strong>One Year of Automation, 100K Staff Hours Saved: A Telco Giant\u2019s Big Gain<\/strong><\/a> <\/strong> <\/p>\n\n\n\nWeathering Heavy Alarm Volumes: 5 Phases to Reach More Automations and More ROI <\/strong> <\/h3>\n\n\n\n
Hurricane Harvey and Hurricane Irma: True Tests of Automation Strength<\/strong> <\/h3>\n\n\n\n
Resolve Automation in Action: A Fiber Cut\u2019s 4 Key Indicators <\/h3>\n\n\n\n
\n
The Calm After the Storm: The Benefits of Resolve Automation<\/strong> <\/h3>\n\n\n\n