Whether you can investigate a breach was decided months ago

During an incident, everyone in the room wants the answer to one question: is it over. It is the wrong question, and it is not the one that will be asked afterwards. Afterwards, a regulator, an insurer, a customer’s security team and possibly a court will want to know what was accessed, when it started, what left the building, and how you know. Those are questions about evidence, and evidence is not something you can decide to have once you need it.

This is the uncomfortable structure of the discipline. Almost everything that determines whether an investigation succeeds was decided months earlier by someone configuring log retention, choosing an endpoint agent, or deciding that DNS query logs were too voluminous to keep. Those decisions are made on cost grounds, quietly, by people who are not thinking about incidents. They are the whole ballgame.

Retention is the failure, not coverage¶

Most organizations we look at are logging more than they think. Authentication events, endpoint telemetry, proxy traffic, cloud audit trails: the data is being produced. It is then kept for thirty days, or seven, because that is what the licence tier covered and nobody revisited it.

That interacts badly with how intrusions actually run. An attacker who established access in February and was noticed in June leaves an investigation that can see only the last few weeks of a five-month engagement. You can describe what they did recently. You cannot establish initial access, you cannot bound the scope, and you therefore cannot tell a regulator what was affected. The honest report says a period cannot be reconstructed, which is a considerably worse conversation than the retention invoice would have been.

The first hour is where evidence is usually destroyed¶

The instinct of a competent operations team facing a compromised machine is to restore service: reimage it, bring it back, move on. It is the right instinct for availability and it is catastrophic for the investigation, because the machine was the record. Volatile memory in particular holds a great deal of what you would want to know, including things that were never written to disk, and it does not survive a power cycle.

Isolate rather than power off. Stop log rotation on anything relevant before it rolls. Acquire before you remediate, even when acquiring feels like a delay you cannot afford, because the alternative is remediating an incident whose extent you will never be able to establish. NIST’s guide to integrating forensic techniques into incident response is largely a long argument for exactly this sequencing, and it has been making it since 2006.

Write down what you do, as you do it. This sounds like bureaucracy and it is the single cheapest thing on this list. An action taken by a responder and not recorded is, three weeks later, indistinguishable from an action taken by the intruder. We have watched teams spend days investigating their own colleague. The current NIST incident response guidance, rewritten in 2025 around the Cybersecurity Framework, treats this preparation as continuous risk management rather than as a phase you enter when the alarm goes off, which is the right framing: readiness is a property of the system, not a stage of the response.

What to build, specifically¶

Forensic readiness is not a product. It is a short list of engineering decisions, most of which cost very little at build time and are expensive to retrofit.

• Process lineage on endpoints: what executed, from where, with which parent. Without it you can see that a machine was compromised and not how.
• Authentication logs covering every plane, including the identity provider, the VPN, the cloud console and the database. Most interesting attacker moves are logins that worked.
• DNS and egress records, because exfiltration is a network event and rarely anything else.
• Application audit trails that record who read something, not only who changed it. Almost every application logs writes and almost none log reads, and “what did they access” is the question a regulator asks.
• Immutable or append-only storage for the audit trail, held somewhere an administrator of the source system cannot alter it.
• Synchronized clocks everywhere, which sounds trivial until you try to build one timeline out of six sources that disagree by minutes.
• Retention set from the dwell-time question above, with the expensive sources kept longer in cheaper cold storage rather than dropped.

Test it the way you would test a backup¶

Nobody trusts a backup that has never been restored. Logging deserves the same skepticism, and gets it far less often. The exercise takes an afternoon: pick three incidents that are plausible for your business, a compromised administrator account, a departing employee taking data, ransomware reaching a file server, and for each one write down the questions you would have to answer. Then check, source by source, whether you could.

The result is reliably worse than the team expects, and it is far more persuasive than a maturity score, because every gap on it is attached to a question somebody will actually be asked. That list is the remediation backlog, and it is usually short.

The part that is not technical¶

Two decisions belong to people who are not engineers, and both are better made in advance. The first is who is authorized to disconnect a production system, and on what evidence: during an incident this question generates an hour of deferring upward that you do not have. The second is whether the investigation runs under counsel, which affects the privilege position of everything the investigation produces, and which is difficult to arrange retroactively.

Check the insurance policy too, before you need it. A number of cyber policies require the use of a panel response firm, or prior written approval, and engaging outside that can affect coverage on the one occasion it matters.

None of this is exciting work, and none of it will ever be the reason a project gets funded. It is simply the difference between an incident you can describe and an incident you can only apologize for.