What to Put in an Incident Plan

Most plans fail before they start because nobody wants to be the one who declares. The engineer who found something odd is not sure it is serious. The manager does not want to wake people at midnight over what might be nothing. By the time everyone agrees, four useful hours have gone.

Fix that with a short ladder written in things a tired person can check against reality. Not major and minor, but sentences. Customer data may have been read by somebody outside. Nobody can place an order. One person has had a strange sign in alert.

Then say who is allowed to declare, and make that list long and junior. An incident declared and stood down in twenty minutes costs you a little wasted time. One declared four hours late is the entire reason the plan exists.

A job given to a team is a job given to nobody. Every role needs a name and a deputy, and the plan has to say how to reach both of them outside working hours.

Four jobs matter in the first hour. Somebody runs it and makes the decisions, and that person should not also be typing, because the two compete and the decisions lose. Somebody handles communication, inside and outside. Somebody keeps a log with times against it. Somebody leads the technical work.

In a small company one person may hold two of those, which is fine as long as it is written down. The failure is never too few people. It is four people each assuming that somebody else is keeping the log.

Under pressure nobody reads paragraphs. They follow lists. The first hour should fit on one page and be specific enough to act on without interpreting anything.

Set out the containment options and what each one costs before you need them. Cutting a network off, disabling an account, blocking an address, taking a system down: every one of those stops somebody working. The time to weigh that is not while the clock is running.

Say plainly what not to do. Do not rebuild the machine. Do not delete the suspicious mail rule before somebody has captured it. Do not pull the power out if what is in memory might matter. Well meant tidying destroys the evidence that answers the question everyone asks afterwards, which is how far it went.

Communication is not something the person running the incident does in the gaps. It has its own audiences. Staff who need to know what to stop using. Customers who need to know whether this touches them. Regulators who may have a clock running. Your insurer, whose policy may require an early call. And the board, who will hear about it from somewhere.

Draft the first message to each of them now, while nothing is happening. A holding line written calmly beats one written at speed with the facts still moving, and it stops you going silent, which is the behaviour that turns an incident into a story about your company.

Decide the channel in advance. If the plan runs on company email and email is the thing that is compromised, there is no plan. Agree something outside the affected systems, keep the contact list somewhere that survives them, and check that people actually look at it.

An hour around a table with a realistic scenario finds more holes than another month of writing. The usual finds are the same everywhere. Nobody knows who holds the insurance policy. The out of hours contact list is two leavers out of date. The person named to run it sits in the team that would be under investigation.

Run some of them without warning anybody. A plan that works when everyone read it that morning is not the plan you have.

Most of all, rehearse getting back, not only talking about it. Table exercises produce optimistic recovery times every single time. Actually restoring a system, timed, from the backups you really hold, produces a real number. It is usually a surprise.

The review afterwards looks for the conditions that allowed the mistake rather than for the person who made it. Where people get punished for errors, the next problem gets reported later, and lateness is the thing you were trying to design out.

Actions from the review go into the same list as everything else, with owners and dates, and get reviewed at the same meeting as everything else. Actions kept inside the incident document do not get done.

Then update the plan while the detail is fresh. The best source of improvements to a plan is the last time somebody had to use it.