Incident Response

Incident response is how a team responds to active threats and attacks on their system. Security incidents will happen—it is a matter of "when", not "if". When they happen, it is important to have an incident response plan which can be activated and used to contain and recover from the attack.

The first step is to establish an Incident Response Team (IRT). The team should be comprised of individuals with a variety of skill sets which will allow each to have a specific role and allow the team to responds to many different types of incidents. IT and technical staff are an obvious choice, but it should also include team members with legal, human resources, and public relations expertise. These members can provide vital advice and assistance during an incident and mitigate the impact of the incident on the organization.

Incident Response Phases

An incident response plan can be subdivided into six phases.

  1. Preparation
  2. Identification
  3. Containment
  4. Eradication
  5. Recovery
  6. Lessons learned

1: Preparation

The first phase is preparation. This is the most important phase because it determines how all other phases will operate and ensures the teams preparedness for implementing each phase efficiently.

The organization should publish security and data-handling policies and distribute them to all staff. Written policies keep users from making bad choices which can cause incidents but, more importantly, it gives legal and human resources clear authority to act if policies are violated.

The Incident Response Team (IRT) should develop a written response plan and a series of checklists to ensure that no detail is skipped while moving quickly to respond to an incident. The team should gather any documentation which might help their response. This might include technical manuals, maps of the physical facility, and maps of the network and infrastructure. The IRT should be given whatever online and physical access they might need to respond, or a member of the IRT should be an admin who would be able to quickly grant the privileges they need. It is important to train the IRT and drill them regularly on various scenarios.

The IRT should plan all communications in advance. They should know who to contact, when it is appropriate to contact them, and how to reach them (via multiple means). The "contact list" should include all members of the technical team, senior management, external vendors and third-party resources, and law enforcement.

The IRT should put together a "jump bag" with various technical tools which can be useful during an incident. These tools include:

  • Hard drives, USB drives, cables
  • Bootable USB drives or CDs
  • OS, hardware, and software utilities
  • Anti-virus software
  • Forensic software
  • Physical tools (screwdrivers, wrenches, etc.)

2. Identification

Identification begins when an incident first begins. The IRT should be notified and begin their communication protocols.

Their first job should be to identify the threat and gather details on the incident. These are the classic questions: who, what, when, where, why. When and where did the incident start? When was it first noticed? Who noticed it and how was it discovered? What do the error messages and log files report about the incident?

The goal of these questions to identify the nature and scope of the incident, and to begin to understand the current and future impact it may have. How urgent is the incident? How likely is it to spread or get worse? Which files have been accessed or edited?

At this phase, the IRT should also start an incident timeline to keep track of what has happened so far and to log of all response steps performed going forward. This timeline is important in understanding a current attack and for reviewing the incident and response afterwards.

3. Containment

The containment phase may begin before the identification phase is complete. The goal is to intervene and limit damages from the incident. This means isolating any affected systems to prevent problems from spreading. Network traffic can be re-routed and servers should be removed from the network.

Once systems are isolated, they should be backed up using forensic software. Forensic software is specially designed for this purpose. It is important to have a perfect backup which will preserve all details of the current state for technical review. A forensic copy also preserves evidence which can be used by law enforcement. The forensic copy should be immediately stored in a secure location to keep it safe, to prevent tampering, and to guarantee its integrity.

4. Eradication

With the incident contained, the IRT can take a deep breath and begin the work of eradication. This phase may take days or weeks to complete and should not be rushed. The technical team will remove any accounts or backdoors created by an attacker. Affected hard drives should be completely wiped and re-imaged—do not attempt to track down malicious code and risk missing something. The system and files should be scanned with anti-malware software. And whenever possible, install defenses or configuration changes to prevent the same vulnerabilities from being exploited again.

5. Incident Response: Recovery

Once the threat has been eradicated, the systems should be brought back online. This should be done after planning and consultation with the IRT to coordinate a careful re-introduction. The team should determine which tools will be used to validate that eradication was successful, how the system will be tested, and for how long it will be monitored to ensure it remains clean.

6. Lessons learned

The last step is to review the incident and consider the lessons learned. This can prevent future incidents and improve the team's responses. After a major incident, it is not uncommon to want to take break rather than re-hash recent events. However, this is a vital step which will pay dividends in the future. The IRT should schedule a meeting for soon after the incident, within two weeks. That is long enough to give everyone a break but short enough to talk while memories are fresh.

In the meeting, the IRT should expand on the documentation taken during the event, completing the timeline with step-by-step details. The IRT's performance and decisions should be evaluated to determine what worked and what needs improvement. Action items should be developed and followed up on to improve readiness for the next incident.

Fork me on GitHub