We built software to take the busywork out of compliance, and we use it on every engagement, so we are not against automation. We are against the claim that comes with much of it: that automation makes the assessment, or the assessor, unnecessary. Here is where it genuinely helps, and where it does not.
Where it helps
Producing dated records on a schedule. The assessment runs on evidence that a control operated throughout the period. A configuration export taken on the first of every month, by a job, with a timestamp, is better evidence than the same export taken by hand the week before the assessment. Access lists, firewall rules, patch status, scan results, log review outputs: anything that can be exported on a schedule should be.
Noticing drift. A mechanism that compares the running state with the intended state and tells someone when they differ is what the change-detection requirements actually ask for. Note the difference from infrastructure-as-code: templates define the intended state, and they prove nothing about the running one. Drift detection with an alert and a review record is evidence. Templates alone are a description.
Reviewing logs at scale. Daily log review is a requirement nobody meets by hand on a large estate. Alerting that produces a ticket, and a record that the ticket was handled, is the realistic form of the control.
Linking evidence to requirements. One export serves four requirements. A system that stores the link, so that the item is asked for once and found four times, saves more assessment time than any other single thing.
Where it does not
Judging adequacy. A tool can tell you multi-factor authentication is configured on the VPN. It cannot tell you that the management API has no such control and is reachable from the office network. That is a question about scope and architecture, answered by someone who understands the environment.
Observation. Four people walking through a single-entry mantrap on four badges. A reception described as staffed around the clock, with nobody there. A backup documentation that describes an infrastructure rebuilt since. No tool sees these. An assessor in the building does.
Reading documents for what they do not say. A provider’s attestation that lists every site but yours. A scope document that puts the key vault out of scope. Reading those correctly is the job.
Interpretation. Where the standard leaves room, a tool picks one reading and hides it. An assessor shows the reasoning.
The claim to be wary of
Some compliance platforms say, or imply, that their dashboard is the assessment. It is not. For a Report on Compliance the Council requires a Qualified Security Assessor, and no automation vendor is one. For a self-assessment, the questionnaire still asks whether controls are in place in your environment, and a dashboard showing green for the systems it is connected to says nothing about the ones it is not. Automation is the evidence layer. The judgement is still a person’s, and the signature is still a QSA’s.
Automation is the evidence layer. The judgement is still a person’s, and the signature is still a QSA’s.
What people get wrong
They automate the collection and skip the review, so the records pile up with nobody reading them, which is its own finding. They mistake a template for a control. They buy a platform and assume the scope is whatever it monitors.