CivicActions Incident Response Procedure Checklist
This is a short, actionable checklist for the Incident Commander (IC) and Responders to follow during incident response. It's a companion to the Incident Response Plan where you can find the full details of each step.
Step 1: Breathe
- No one's life is in danger.
- Document your steps and findings! (A client specific or
#generalslack channel is recommended -
#generalwill be used in the rest of this document.) This eases communications and handoffs.
- Consult with the Incident Commander (or other team members) if you have questions.
There is often overlap between these two roles, especially at the beginning of an incident response.
Incident Commander (IC)
- is the first CivicActions team member to notice or respond to the incident
- forms the team (for the first 15-30 minutes the IC may also be the only Responder)
- escalates communication to additional team members and/or the Product Owner when needed.
- ensures steps are documented (usually in
- ensures a ticket is created and a record of the Incident process is maintained
- primarily responsible for the Assess and Remediate steps
- document in real time measurements, theories, steps taken in
- seek out a separate Incident Commander if the incident will take longer than 15-30 minutes to resolve
At this point, the Incident Commander (aka the first Responder) is usually working alone:
- Determine if you can whether or not this may be a false alarm.
- If you believe it may be a real incident (best to err on the side of "it is an incident") and you can't resolve yourself in under 15-30 minutes, find either a Responder or replacement Incident Commander to help share the load:
The Responders (more than one is OK) work to:
- Confirm the incident — is it a real incident?
- Assess the severity, using the rubric in the IR guide (CivicActions system incidents are generally Low severity.)
The Incident Commander:
- Forms a team (responders) to determine if the event is actually a confirmed incident, and if so assesses the severity.
- Determines whether to also activate the contingency plan - is a system failure causing the disruption?
- Posts an initial situation report ("sitrep") (example sitrep) to
#generalslack including the a descriptive name, Commander and Responders
The Responders work to determine cause, find resolution and return the system(s) to normal operations.
- If suspicious activity is suspected or other unanswered questions exist, create database dumps, take disk snapshots of relevant volumes, get screen captures of anomalous activity before making changes such that post-remediation forensic analysis is supported.
- Determining the cause can drive follow-up measures to prevent incident reoccurrence.
- A containment strategy may be implemented by the Information Security Group for the instance to drop all ingress and egress traffic except from specific IPs (like yours) until forensics can be performed.
The Incident Commander coordinates activity:
- Keep the ticket/docs updated as people work, tracking:
- Leads, and who's following them
- Remediation items, and who's working on them, including customer notification (if appropriate to the situation)
- Send out sitreps on a regular cadence (high: hourly; medium: 2x daily; low: daily).
- Go into work shifts if the incident lasts longer then 3 hours.
- Hand off IC if the incident lasts longer than 3 hours.
Once the incident is resolved:
- Update the ticket, set status → "Ready for QA".
- Schedule a retrospective.
- Send a final sitrep vie email to stakeholders.
- Thank everyone involved for their service!
Extra checklists for special situations that don't always occur during incidents:
Follow this checklist if an event turns out not to be a security incident:
#generalof the false alarm.
- Update the GitHub/GitLab/JIRA issue (if one exists) setting status to
- If any sitreps have been sent out, send a final sitrep to all previous recipients, noting the false alarm.
Handing off IC
Follow this checklist if you need to hand over IC duties:
- Old IC: brief New IC on the situation.
- New IC: confirm that you're taking over.
- New IC: update the GitHub/GitLab/JIRA issue, noting that you're now the IC.
- New IC: send out a sitrep, noting that you're taking over IC.
- Old IC: stick around for 15-20 minutes to ensure a smooth handoff, then log off!