Security is not a one-size-fits-all discipline. Far from it. One thing that is always consistent, enterprise to enterprise, is the fact that security is complex. Especially for large organizations. In a large enterprise, in particular, addressing security incidents is muddy terrain because they tend to spill across teams and involve systems and technologies beyond the security operations center’s (SOC) purview. We see that security incidents in really big companies often require the input of subject matter experts in other parts of the organization.
And, as sophisticated threats grow more quickly than ever, security teams are faced with resource constraints. Ask any enterprise Incident Response leader about their workload and you’ll quickly understand the vast amounts work they face. On top of the fact that these security professionals have more work than they can handle, they are further constrained because the tools intended to help them – like improved threat detection solutions – often inadvertently add to their workload by creating unmanageable volumes of alerts and false positives. What happens to any of us when we are bombarded with false alarms on a regular basis? Eventually we become desensitized. And in the security world, that means staff doesn’t respond as quickly as we’d like to alerts – false or otherwise.
The fact is too many organizations keep their incident response approach in a silo. They position their SOC as an island, focused only on the task at hand. Automation is helpful, but it’s not a fix-all. The real solution is: incident response is a team sport. To mount a smart and effective defense, the right approach goes beyond individual technologies or the SOC to collaborate across the enterprise.
Are you ready for a Security Incident Response Platform? Assess your Readiness!
When the SOC works alone and apart from other operations teams, the speed of responding to an incident is almost guaranteed to go up (by hours or even days) while the security team goes through the motions of opening a ticket, sharing information via email and waiting for other teams to prioritize the information against their daily objectives. To reduce risk and accelerate investigation and incident remediation times, the SOC should leverage the most potent blend of information, assets, staff and technology in five ways.
Action: Give your security team the control they need to orchestrate all activities. Incident response may be a team sport, but, especially for security-related incidents, you need a quarterback that can handle pressure. And that’s your SOC.
Why? Intelligent orchestration is the key to enterprise-wide incident response. You’ve probably faced this scenario: a malicious process or operation has been found and now multiple teams are crowding into a war room, on a call or email chain, and it’s quickly turning into a stressful situation. Everyone is jumping in with their experience and opinions to fix the problem, but it’s only creating chaos.
Action: Develop integrated working relationships before an incident. If the teams understand each other’s workflows and use the same applications, they’re going to have tighter alignment and better visibility when it comes to addressing their incidents in a timelier fashion.
Why? Security doesn’t work in a vacuum and is a multidisciplinary function. Addressing security incidents requires input and action from multiple non-security teams. Yes, the security team usually takes the lead by initiating an investigation as the result of detecting the incident or threat, but they’ll have to work with the IT, network, and/or applications infrastructure teams in collecting information and taking action on the correct tools for investigation, containment and remediation. Flip this idea on its head and it’s easy to see that network operations and other teams often need help from their security practitioners to review and respond to their own incidents.
Action: To make sure your team learns from incident-to-incident, you have to remember how situations have been handled previously. Start by prioritizing documentation. You want to capture critical incident information, document the right subject matter experts (SMEs) who understand the processes, detail the most effective actions, and wrap it all up in a prescriptive playbook or automation that strengthens your future capabilities. You’ll reduce your vulnerabilities, spend less time researching and remediate more quickly and effectively knowing that you’re doing what the experts would do.
Why? In order to involve other teams to create a robust incident response program, you have to tie it all together. I call this “memorializing” the process and capturing the “tribal knowledge” that exists across the organization. It’s very easy for disconnected or fragmented processes to settle in, which only hurts the SOC when the next incident occurs. But when you capture and utilize all the SME expertise and cross-team knowledge and actions in a given incident, you can more quickly address incidents and not have to rely on an SME who might have left the company, be on vacation, or preoccupied at the time.
Action: Automate in bite-size chunks that make sense. Look for places where automation can save your team time, which, in the beginning, are often repetitive and time-consuming tasks. You don’t have to automate everything at once; automating the tasks that make sense for you right now is a smart, logical approach to deploying automation.
Why? Automation can be the secret sauce to swift response, but is often misunderstood. Some teams make it more complicated than it needs to be, while others assume they need to automate every activity in their incident response workflow, resulting in an overwhelmed team and an automation process that’s not as useful as it could be.
Also, as I pointed out earlier, most teams are exhausted by trying to respond to the ever-rising list of false alarms. Quantity of alarms are still only increasing given the rise of new systems and IoT devices, so implement automation that can separate the real and actionable alarms from the false threats that distract your team.
Action: Build the skill sets of your SOC members by assigning important work that matters to the organization. Automation and orchestration tools can help to eliminate some false alarms, can fully remediate other incidents, and, more importantly, empower your front-line and Level 1/2 analysts solve some many of the incidents that typically would have required more experienced security engineers. This step frees your more experienced (and well paid) security staff to focus on tougher, more complex incidents or on activities such as threat hunting.
Why? There’s a hierarchy in most SOCs that keeps Level 1 analysts doing less-skilled, repetitive work while, if they have time, senior analysts get to do the glory work of hunting and finding threats. On top of the benefit mentioned above about using experienced staff on the most challenging problems, letting junior analysts solve real challenges improves their job satisfaction and, therefore, tenure with your company. You will have also positioned them for deeper training and involvement in the future. This capability frees up the mundane and repetitive tasks from your more experienced security experts so they can do what they enjoy doing, as well.
We know that enterprise security faces complex challenges. What we need to learn and embrace is that security teams don’t have to respond to them entirely on their own. By taking a holistic approach to incident response, SOC leaders can leverage the right personnel and speed major incident resolution. Collaboration across teams will result in optimized, enterprise-wide incident response and increased efficiencies that benefit the company – and its staff – today and tomorrow.
AppDynamics + Resolve help solve challenges with full-stack observability and automated remediation.
Learn five benefits an automated DDM solution can bring to your CMDB management process.