Situational awareness for Effective IT Incident response

Incident response (IR) is a process used by ITOps, DevOps, and dev teams in reacting to IT disruptions such as cyberattack, security breach, and server downtime etc.

An incident response process is something you hope to never need, but when you do, it’s critical that it provides you with all the information you need for you to take important decisions or necessary steps for the response to go smoothly and seamlessly.

Information Overload 

In today’s always-on digital world, IT infrastructure is monitored by multiple monitoring solutions that generate events and alerts, which indicate changes to the IT environment or a monitor being in a failed state. Many IT and development teams get hundreds of emails a day from their monitoring systems due to alert storms that flood their inboxes. This type of notification traffic creates  ‘alert fatigue,’ which makes it very difficult for incident responders to properly diagnose the incident and respond effectively.  

To improve the Incident response time, it's imperative to have situational awareness of all the IT Services such as upstream / downstream dependencies of each IT service, current status of Incidents associated with each IT service and mapping of IT Service to Business service. This will enable the Incident responder to take his response decision easily and swiftly. 

Situational awareness provides Incident responders, business responders and leaders a live, shared view of system health and incident impact to improve real-time decision making when operational issues affect customers.

What are the important elements of IT Operations Situational awareness

Following are the three categories of important elements of situational awareness are : 

• Information gathering – know the typical sources of Incident by collecting information available such as ITSM, Monitoring tools

• Understanding and correlation of information – be able to interpret the information gathered to IT services and present it using the Dashboards , Status Pages, Dependencies service graphs 

• Root cause Anticipation – be able to anticipate the root cause of  an incident

Lets elaborate more on this and see how it works 

Information gathering 

Though the monitoring tools have sophisticated features such as reducing duplicate alerts or flapping, it’s still hard for the monitoring tools themselves to determine which information is vital for an incident responder to make a decision.  Adjusting rules or thresholds or setting up maintenance mode at the source can be an arduous and manual process, and forces teams to look to multiple sources of truth to see all their data.

At the same time operations teams perform best when they can see all their data in a single pane of glass. To get to the balance of providing a single pane of glass and also at same time not to overload the incident responder with overwhelming monitoring data, information from various monitoring tools and ITSM needs to be  fused properly to show actionable events only, needed for that specific incident. 

Understanding and correlation of information

Obtained information needs to fused and presented in a manner to make which operating picture via following status and dashboards 

Service Activity :

Service Activity allows users to see incident activity across many of their services at once, so that they can quickly understand their digital operations health, and identify if there’s a widespread major incident. 

Status Dashboard :

Status dashboard provides technical responders, business responders, and leaders a live, shared view of system health to improve awareness of operational issues. It displays the current status of key business services and sends notifications to alert users when business services are impacted. This feature improves communication between response teams and stakeholders during incidents. 

Response Status :

Oncall Response status provides a quick and easy way to search which users are on call across all escalation policies. 

Timelines :

Incidents and Changes Timeline provides a timeline of incidents and change events across all services 

Anticipation

Anticipation of the root cause of an Incident can be achieved using ML alogothium, which leverages historical Incident data to create vector representations (or embeddings) of those incidents in a multi-dimensional space.  Historical incidents that are similar (consistently occurring within the same time window, triggered from dependent resources or from the same source) are clustered together in the muti-dimensional space as calculated by the distance between the vectors. When new alerts are triggered, the ML algorithm converts them into their vector representations. Alerts related to the same incident share similar context (e.g., occurring within a time window) and hence, will be converted to vectors that fall within the same cluster and grouped together. 

Zapoj IT Event Management is known as the digital Operations platform for driving real-time incident response. Our customers leverage the core Zapoj CEM platform for coordinating incident response processes when operational issues occur. Many customers also use our situational awareness  features to act as a central nervous system through which digital signals from their various distributed systems can be routed, correlated, and surfaced automatically.