Nobody wants a Non-Compliant Report on Compliance. Not the client, who has customers and an acquirer waiting. Not the assessor, who would rather sign something everyone is happy with. This is the story of one, and of why it was the right finish.
The situation
An international e-commerce payments company had validated with a self-assessment questionnaire for years. Its transaction volume crossed the line, and the acquirer asked for a Report on Compliance. It was the company’s first.
The spring gap assessment raised 151 findings. That is not unusual for a first full assessment of a cloud environment that grew faster than its documentation. What matters is what happens to the list. By late summer, 54 findings were still open, and the company wrote to us about the pace of the engagement. We agreed a close-out plan: a three-day workshop at the company’s offices to work through the open items with the people who owned them, then a cut-off date after which no further evidence would be reviewed.
The last four days
The cut-off date came and submissions were still arriving. In the four days before it, we received six versions of the network security standard, five of the approved-services inventory, and five of the hardening standard.
The company believed that more documents would close the remaining gaps. Six of the eight open findings needed a firewall rule changed, a service disabled on a running system, or a standing privileged access removed. We had written that down in July: more documents will not close them. A document written in the final days can show that a control is defined. It cannot show that the control operated during the assessment period, because it did not exist for most of it.
Two examples of what “a different document” cannot fix. A listening service on a container worker node, which the company’s own inventory of approved services listed as “not approved, must be disabled”, was declared closed because network rules now blocked traffic to it. Blocking traffic does not disable the service; the requirement is about which services run, and a different requirement covers the network. Change detection on the container platform was evidenced with the infrastructure-as-code templates that define it. Templates describe the desired state. The requirement asks for a mechanism that notices when the real state drifts from it, and tells someone.
What we got wrong
Before issuing, we ran our own pre-issue quality review on the report. It turned seven of our earlier determinations into “not in place”. For three of them, the honest note to the client was that our earlier determination had been made too readily, and that correcting it was our responsibility, not theirs.
We put that in writing because it is true, and because an assessor who cannot say “we were wrong about this one” has no standing to say “you are wrong about that one”. The quality review exists so that the signed report is right, not so that the earlier emails were.
An assessor who cannot say “we were wrong about this one” has no standing to say “you are wrong about that one”.
The conclusion
The final position was fifteen requirements not in place, a Non-Compliant Report on Compliance with an action plan, and no further evidence rounds. The note that went with it said what we meant: this is the point at which an assessment concludes. It is not a judgement on the company’s effort.
A few lines from the exchange that we stand behind, and that any client should expect from us:
- If you do not have what is asked for, say so. A stated gap is quicker for both of us than a file we have to open only to rule it out.
- A concluded report with a clear plan is a better finish for both of us than another open round.
- For the next attempt: remediate first, let each control run long enough to leave evidence, submit once per requirement, and have one named owner per finding.
What happened next
The company accepted the conclusion and chose to keep going with us. It came back with a full remediation plan, with set milestones and clear closing objectives, which we agreed to. The work is ongoing, with a firm deadline. A hard conclusion led to a better plan, not a broken relationship.
What people get wrong
They believe the assessor’s job is to find a way to sign. Our job is to describe a scope the company stands behind and to say whether, during the period assessed, each requirement was in place. When it was not, the honest report says so, and the useful report says what would change it.
They mistake effort for evidence. The company worked hard, and that is in the record. The requirements ask what operated, not what was attempted.