Skip to content

CivicActions Security Incident Response Procedures

Table of Contents

Introduction

This document describes the process that the CivicActions Incident Response Team follows when responding to security incidents and other disruptions that may affect the Confidentiality, Integrity, Availability (CIA) or Privacy of system resources and data. It explains:

  • roles and responsibilities during and after incidents
  • overview of the steps to follow for resolution

During an incident, the IRP checklist may be more useful as it contains bulleted, actionable items for the IR Team to follow. For most non-project-related incidents, see the Security Incidents page.

Roles and Responsibilities

Individual and team roles are described below.

Responder

A Responder is a member of the CivicActions IR Team who investigates and remediates an event or incident.

First Responder

The First Responder is the first IR Team member who becomes aware of the incident.

  • Frequently the First Responder is also the Incident Reporter.
  • The First Responder assumes the role as the Incident Commander (IC) until handing off IC duties.
  • For the first 15-30 minutes, the First Responder may work alone. If needed, the First Responder begins forming the IR Team. See Initiate.

IR Team Responders

During incident response, Responders do the following:

  • Assume primary responsibility for the Assess and Remediate steps.
  • Document in real time the measurements, theories, and steps taken using the Slack channel #general or other channels provided by the Incident Commander (IC). Use @security to trigger a Slack notification for the Security team.
  • Designate an Incident Commander (IC), if the incident might require more than 15-30 minutes to resolve, and do an explicit handoff.

Incident Commander

The Incident Commander (IC) remains uninvolved in remediation efforts, and performs three major duties:

  1. IR Team creation and management, ensuring that the IR Team:

    • Includes team members who are capable of containing, investigating, and remediating the incident.
    • Remains focused on resolving the incident.
    • Uses the most appropriate media/communication channels for recording actions. During business hours, Incident Commander (IC) may create a dedicated Slack channel (for example, #fire-team) for IR Team communications.
    • Utilizes work shifts if the incident lasts longer than 3 hours.

  2. Documentation, including all actions taken during investigation and remediation, using the following methods:

    • Slack channel #general (Use @security to trigger a Slack notification for the Security team.)
    • Project JIRA ticket or Gitlab issue (if applicable)

  3. Communication, ensuring that internal and external entities stay informed. For communication duties, the Incident Commander (IC) may designate a Communications Officer (CO) and do an explicit handoff.

Communications Officer

Communication is critical as the IR Team works to contain, investigate, and remediate an incident.

The Incident Commander (IC) manages communications regarding the incident until handing off IC duties to another IR Team member or designating a Communications Officer (CO).

The Communications Officer (CO) manages external communications with:

  • Management, developers, users, and anyone affected by the incident
  • Client stakeholders (if applicable)
  • Additional Project team members and/or the Product Owner (if applicable)
  • CivicActions Legal team, and US-CERT if escalation is required

Communication channels

The Incident Commander (IC) determines the most appropriate communication channels during incident response. Any of the following may be used:

  • Slack channel #general. Use @security to trigger a Slack notification for the Security team.
  • During business hours, Incident Commander (IC) may create a dedicated Slack channel (for example, #fire-team) for IR Team communications.
  • A JIRA ticket or Github/Gitlab issue for the incident (if applicable) will be the final location for all incident reporting, with links to other documents as needed.
  • Video conference: Zoom, Google Meet, Microsoft Teams, Skype, etc. (Be sure to record the call for documentation purposes.)
  • Email to security@civicactions.com.
  • Email/telephone to the CivicActions IR Team for an incident that has potential Project impact.

Incident response process

There are six major processes of incident response, detailed below:

During an incident, the IRP checklist may be more useful as it contains bulleted, actionable items for the IR Team to follow.

1. Breathe

No one's life is in danger.

2. Start documenting

Begin documenting all steps and findings. Documentation makes handoffs and Responder onboarding easier. The Slack channel #general is recommended because it is most widely accessible, but other communication channels may be used. When posting messages to Slack, use @security to trigger a Slack notification for the Security team.

3. Initiate the response

At this stage, the First Responder is usually working alone, and is also the Incident Commander (IC).

A. Allocate 5 minutes and determine whether this event is a potential incident or false alarm. Consider any potential Project impact.

An incident begins when someone becomes aware of a disruption in expected normal system operations. An incident is defined broadly, following NIST SP 800-61: Computer Security Incident Handling Guide, as "a violation or imminent threat of violation of computer security policies, acceptable use policies, or standard security practices". This definition encompasses any scenario that might threaten the security of system resources and data. For more information, see the CivicActions guidebook: What is an incident?

B. Respond accordingly:

  • Potential incident

    1. Issue a broadcast notification via one or more of the following:

    An example message follows. The format is not important, but the information fields are useful.

      **Description**: [Short description of the event and its impact]
  **Status**: investigating
  **Severity**: unknown
  **Incident Reporter**: [name of the person who reported the issue]
  **Incident Commander**: [your name]
  **Responders**: [names of other _Responders_]
  **Details**: [Extra details about the event]

    Observe the following guidelines for communications:

    - During this stage of incident response, the event status is "investigating".
- An unconfirmed issue is called an _event_. A confirmed issue is called an _incident_.

    1. For an incident requiring more than 30 minutes to resolve:

    More information on incident response roles and responsibilities:

    - [Responder](#responder)
- [Incident Commander (IC)](#incident-commander)
- [Communications Officer (CO)](#communications-officer)

Use the [Explicit Handoff Ceremony](#explicit-handoff-ceremony) when transferring/changing roles.

  • False alarm

Conclude the incident. Proceed to 6. Conclude the incident.

4. Assess the incident

IR Team responsibilities during assessment

A. Confirm the incident.

  1. Gather information, and document your findings.

    • Was the event triggered by an external dependency?
    • Is a system failure causing the disruption?

  2. Proceed to the next step for a confirmed incident. (For a false alarm, conclude the incident. Proceed to 6. Conclude the incident.)

B. Assess the severity.

  • Use the rubric for determining severity. Project incidents are generally "Low severity".
  • Does it affect system or data Confidentiality, Integrity, Availability and/or Privacy?
  • Note that severity can change over the lifespan of an incident, and it is acceptable for the IR Team to assess the initial severity quickly.

C. Determine whether the IR Team needs to activate the Contingency Plan. Consider whether Disaster Recovery is required.

The IR Team should record all actions and observations in an appropriate communication channel.

Reminder: Use the Explicit Handoff Ceremony when transferring/changing roles.

Incident Commander responsibilities during assessment

  • Post an initial situation report (sitrep), in the following locations:

    • Slack channel #general (Use @security to trigger a Slack notification for the Security team. Include link to the ticket/issue if applicable.)
    • JIRA ticket or Gitlab issue (if applicable)
    • Any other communication channels as specified by the Incident Commander (IC) (or Communications Officer (CO)).

    Here is an example sitrep:

    **Subject**: \[sitrep\] Chickens are escaping
**Severity**: low
**Incident Commander**: Farmer Jane
**Responders**: Spot the Dog, Farmer Dave
**Description**: We've confirmed reports of escaped chickens. Looks like a fox may have tunneled into the run. Dave is working to fix the fence. Spot is tracking the fox.

  • For an issue with potential Project impact, ensure that a ticket/issue has been created. This should be done, even if the First Responder/IC manages the incident fully, for example, by simply re-starting a service.

5. Remediate

Remediation is about resolving the issues caused by an incident. Remediation will be situation-specific, and timelines vary based on the assessed severity.

Remediation and service disruption

Remediation may require service disruption. If it does, the IR Team should proceed in a different way depending on the severity:

  • High severity: Take action immediately, even if this causes disruption. Send a notification about the disruption as soon as possible. The CivicActions IR Team, or Project IR Team if applicable, does not need permission to take action at this level.
  • Medium severity: Consult the other members of the CivicActions IR Team and agree on the best course of action. For an issue with Project impact, notify the Project leads about the planned action, and help them assess the relative risk of disruption versus security. If the leads are unavailable on Slack, contact them using the phone numbers in their Slack profiles. The Project team should reach a collaborative decision on action, with a bias towards disruption. If they cannot be reached within an hour, the Project IR Team may take action without them.
  • Low severity: Consult the other members of the CivicActions IR Team and agree on the best course of action. For an issue with Project impact, notify the Project leads as described above. Do not take action until a mutually-agreed course of action has been determined.

Remediation requiring more than 3 hours

Remediation takes time. If the issue progresses for more than 3 hours without being resolved, the Incident Commander (IC) should plan for a long remediation. This means:

  • The Incident Commander (IC) determines whether remediation efforts will occur during business hours only or be continuous. This depends on the severity of the issue, and whether breaches are ongoing.
  • For a continuous response, the Incident Commander (IC) should plan shifts. This allows Responders to take breaks and insures continuous coverage. Shifts should be no longer than 3 hours. Also, the Incident Commander (IC) duties should rotate in shifts no longer than 3 hours.

IR Team responsibilities during remediation

  • Determine the cause, implement a resolution, and return the system to normal operations. Make every attempt to identify the cause; this can prevent incident recurrence.
  • Maintain a list of informational leads from the incident — actionable information about any security breaches, stolen data, etc.
  • Develop a list of remediation steps. These can be tracked as checklists in Slack, shared Google Docs files, a JIRA ticket, Gitlab issue or another communication channel as specified by the Incident Commander (IC).

If suspicious activity is suspected or other unanswered questions exist, do the following before making any changes:

  • Make snapshots of relevant volumes and data.
  • Preserve logs.
  • Take screen captures of anomalous activity that can be used in post-remediation forensic analysis.
  • Consider implementing a containment strategy. For example, reconfigure firewall rules for the affected instance to drop all ingress and egress traffic, except from specific IPs like your own, until forensics can be performed.

Incident Commander responsibilities during remediation

At a high level, the Incident Commander (IC) tracks remediation actions, ensures they are assigned and followed, and verifies them when they are completed.

The Incident Commander (IC) must distinguish between immediate concerns, which need to be completed before the incident is considered resolved, and long-term improvements/hardening, which can be deferred to the Retrospective.

The Incident Commander (IC) does do the following:

  • Maintains current information in Slack, shared Google Docs files, a JIRA ticket, or another communication channel. Be sure to include:

    • IR Team members and their roles, and/or Project team leads and members (if applicable)
    • Remediation items and their assignees

  • Establishes and documents work shifts for an incident longer than 3 hours.

  • Maintains communications with stakeholders, or designates a Communications Officer (CO) via explicit handoff.

  • Shares sitreps on a regular basis:

    • High severity: hourly
    • Medium severity: 2x daily
    • Low severity: 1x daily

  • Focuses on coordination, communication, and information collection -- not remediation.

Communications during remediation

The Incident Commander (IC) or Communications Officer (CO) does this following:

  • Coordinates with the CivicActions managers to apprise them of the situation.
  • Coordinates with the Project Product Owner (PO), if applicable, to notify affected customers.
  • Ensures that the IR Team is recording all actions in the appropriate designated communication channels.
  • Shares sitreps on a regular basis in Slack, in the ticket/issue (if applicable), and with stakeholders. See the section on incident severities for suggested time intervals based on severity level.

6. Conclude the incident

Closing the ticket

When the incident is no longer active, for example, the breach has been contained, the issue has been fixed, etc., the Incident Commander (IC) should conclude the incident. There might be longer term remediation required, and possibly more investigation, but when the incident is no longer active, these activities can proceed at the regular pace of business.

To conclude an incident, the Incident Commander (IC) should:

  • Set the status of the ticket/issue to Ready for QA.
  • Send a final sitrep to stakeholders, including CivicActions managers and the Security team.
  • Thank everyone involved for their service.

Conducting a retrospective

An Incident Commander (IC), or another designated party such as the Communications Officer (CO), should lead a retrospective and develop an incident report.

Developing the incident report

The incident report should contain:

  • a timeline of the incident
  • details about how the incident progressed
  • information about the vulnerabilities that led to the incident, also called a cause analysis (The cause analysis is an important part of the incident report. Tools such as Infinite Hows and Five Whys can help the IR Team explore potential causes, prevention, and improved incident response.)

Additionally, the incident report should include basic response metrics:

  • Discovery method: How did the IR Team become aware of the issue?
  • Time to discovery: How much time passed from the time the incident became active until someone became aware of it?
  • Time to containment: How much time passed from the time someone became aware of the incident until the incident was contained?
  • Threat actions: What actions were taken by the actor? For example, phishing, password attacks, etc.

The incident report should be posted in Slack, or in the ticket/issue as a final comment before the ticket is closed.

Incident severities

The incident severity level determines the actions of the IR Team. Severity usually changes during the lifecycle of the incident.

High severity

A high severity incident does one or more of the following:

  • compromises the confidentiality/integrity of Sensitive Personally Identifiable Information (SPII),
  • impacts the availability of services for a large number of customers, or
  • has significant financial impact.

Examples include:

  • Confirmed breach of SPII
  • Successful root-level compromise of production systems
  • Denial of Service attacks resulting in severe outages

Guidelines for incident response:

  • Remediation efforts will likely be continuous until the issue is contained.
  • Responders may take any action required to contain the issue, including complete service degradation.
  • Sitreps should be shared every hour, or more frequently.

Medium severity

A medium severity incident can be an unsuccessful attempt to breach Personally Identifiable Information (PII), an event with limited impact on the availability of services for a large number of users, or an event with limited financial impact. Examples include:

  • Suspected PII breach
  • Targeted but unsuccessful attempts to compromise production systems
  • Spam/phishing attacks targeting CivicActions or Project staff
  • Denial of Service attacks resulting in limited service degradation

Guidelines for incident response:

  • Response should occur during business hours.
  • Responders should attempt to consult stakeholders before causing downtime, but may proceed without consent if stakeholders do not respond in a reasonable time frame.
  • Sitreps should be shared approximately twice per day.

Low severity

A low severity incident does not affect PII, and has no availability or financial impact. Examples include:

  • Attempted compromise of non-important systems, for example, staging or testing instances
  • Denial of Service attacks with no noticeable customer impact

Guidelines for incident response:

  • Response should occur during business hours.
  • Responders should avoid service degradation unless stakeholders agree.
  • Sitreps should be shared daily.

Explicit Handoff Ceremony

To transfer Incident Commander (IC), Communications Officer (CO), or Responder duties:

  1. Outgoing ROLE initiates the handoff and briefs the incoming ROLE on the situation.
  2. Incoming ROLE confirms the handoff and assumes responsibility.
  3. Incoming ROLE documents the handoff in Slack or the JIRA ticket/Gitlab issue.
  4. Incoming ROLE shares a sitrep, which notes the handoff.
  5. Outgoing ROLE remains available for 15-20 minutes to ensure a smooth handoff and then logs off.
This page was last updated on November 3, 2023.