• Why SOC 2 compliance is not enough for pentest data
  • Does this apply to your team?
  • What a SOC 2 Type II report certifies
    • Follow-up questions to ask your vendor
  • The audit window: you are reading last year's security posture
    • Follow-up questions to ask your vendor
  • Vendor staff access: "controlled" is not "prohibited"
    • Follow-up questions to ask your vendor
  • The subprocessor chain your SOC 2 does not cover
    • Follow-up questions to ask your vendor
  • Encryption at rest: what it protects, and what it does not
    • Follow-up questions to ask your vendor
  • SOC 2 is not a penetration test
    • Follow-up questions to ask your vendor
  • Breach discovery lag: the 72-hour notification clock starts at detection
    • Follow-up questions to ask your vendor
  • The exit: self-hosted deployment eliminates the question
  • Practical next steps
  • Related content
  • Frequently asked questions

Why SOC 2 compliance is not enough for pentest data

If you are mid-procurement on a cloud pentest platform and the vendor just handed you an 80-page SOC 2 Type II report, you have answered one security question. You have not answered many others. This article gives you several questions you should ask to dig deeper, explains what each one probes, and provides the exact follow-ups to ask your vendor before you sign.

Does this apply to your team?

This applies if you handle pentest findings that map attack paths, document authentication bypasses, contain proof-of-concept payloads, or include evidence of successful exploitation, and you are evaluating a cloud vendor whose primary security credential is a SOC 2 Type II report.

Skip this if your team's findings carry low regulatory exposure, your engagements produce vulnerability scan output rather than exploitation evidence, and you have already completed a vendor risk assessment that goes beyond SOC 2. For teams in that position, a well-vetted cloud vendor with SOC 2 may be the pragmatic choice.

Our data sovereignty brief covers the full architecture comparison for teams still deciding.

What a SOC 2 Type II report certifies

SOC 2 audits against the AICPA (American Institute of Certified Public Accountants) Trust Services Criteria. The Security criterion (CC1 through CC9) is mandatory. The other four criteria (Availability, Confidentiality, Processing Integrity, Privacy) are optional. A vendor can hold a valid SOC 2 certificate without the Confidentiality or Privacy criteria having been audited at all.

The audit itself is sample-based. The auditor tests a subset of access events, configuration changes, and controls. A control can fail in production and still pass an audit if the failure falls outside the sample. The report verifies that the vendor's control framework matches the description the vendor provided. It does not guarantee that every system handling your data was tested or that no incident occurred during the period.

For pentest finding data, this matters more than it does for most business data. A completed engagement produces a detailed map of the target's infrastructure: authentication bypasses, network segmentation gaps, attack paths, proof-of-concept payloads. If that data sits on a cloud vendor's platform, the question of who can access it is not an abstract compliance concern. It is a direct security question with a wrong answer that produces a roadmap for attacking your client.

Follow-up questions to ask your vendor

1. Which of the five Trust Services Criteria does your SOC 2 cover? Security only, or also Availability, Confidentiality, Processing Integrity, and Privacy?
2. Can we see the full auditor's report, including scope description, exceptions, and the auditor's qualifications, not just the summary certificate?
3. Which specific systems that process and store pentest findings are named in scope? Are there components that handle our data but are not in scope?
4. Has the auditor ever issued an exception or qualification in any previous SOC 2 period? If so, on what?

The audit window: you are reading last year's security posture

SOC 2 Type II covers a trailing period, typically six to twelve months. The audit closes one to three months after the period ends. The report you receive today may describe controls from a period that ended six months ago, assessed against a baseline established before that.

A vendor can migrate hosting providers, restructure access controls, change their engineering team, or overhaul their incident response process after the audit period closes. None of this appears in the current report. The report is a historical document.

IBM's Cost of a Data Breach Report 2023 found the average time to identify a breach is 204 days. A breach that started during the vendor's current audit period may not have been detected before the report was issued.

Follow-up questions to ask your vendor

1. When did your most recent audit period close, and when was the report issued?
2. What infrastructure, staffing, or configuration changes have you made since the audit period closed?
3. Do you conduct security reviews between annual audits? Who performs them and what do they cover?
4. If I sign a contract today, how many months of your infrastructure's current state will not have been reviewed under any audit?

Vendor staff access: "controlled" is not "prohibited"

SOC 2 Logical Access controls (CC6.2, CC6.3) require vendors to restrict and monitor staff access to customer data. They do not require that staff cannot access it. They require that access is controlled, logged, and reviewed. For most SaaS platforms, senior engineers, database administrators, and on-call infrastructure staff retain the technical ability to query production databases containing your findings. SOC 2 requires that this access is logged. It does not prevent it.

Verizon's 2024 Data Breach Investigations Report attributes a consistent share of breaches to insiders, including accidental exposure and negligent handling alongside intentional misuse. SOC 2 addresses this through controls. It does not eliminate the risk.

Follow-up questions to ask your vendor

1. How many staff members currently have the technical ability to query the production database containing customer pentest findings?
2. What would prevent a senior engineer responding to a production incident from reading a customer's findings in plaintext?
3. When was the last time a member of staff was found accessing customer data they were not authorized to access?
   • If the vendor confirms a date: How was this identified? What monitoring surfaced it? What disciplinary and technical controls followed?
   • If the vendor states this has never occurred: Who reviews access logs for unauthorized access? When was this review last performed? Is the process manual review, automated alerting, or neither?
4. Do you provide customers with an alert when any staff member accesses their data?

The subprocessor chain your SOC 2 does not cover

A SOC 2 audit covers the vendor's own controls, not the controls of every service they depend on. Their hosting provider (AWS, GCP, Azure), error tracking, logging, CDN, and support tooling are subservice organizations. Most SOC 2 reports use the carve-out method: they acknowledge these dependencies exist but exclude their controls from the audit scope. The vendor manages those relationships; the auditor doesn't test them. If you need assurance on those layers, you have to request each subservice organization's own SOC 2 report separately.

Most cloud pentest tools run on shared infrastructure from major cloud providers, use third-party error tracking that may capture stack traces containing data values when exceptions occur, and use support or chat tooling that may log ticket contents including customer-submitted screenshots. Each subprocessor is an additional link in the chain. GDPR Article 28 requires vendors to document subprocessors and ensure they meet equivalent standards, but this documentation is rarely surfaced to customers without being asked.

Follow-up questions to ask your vendor

1. Which subprocessors have access to pentest finding data? Include your hosting provider, logging services, error tracking, CDN, and support tooling.
2. Does any error tracking or monitoring service (Sentry, Datadog, New Relic) receive stack traces that could include finding content when exceptions occur? What controls prevent this?
3. Do all named subprocessors hold their own SOC 2? Can you provide those reports?
4. Have any subprocessors changed since your last SOC 2 audit period closed?
5. If a subprocessor experienced a breach, what is your contractual notification SLA to customers, and what would the actual notification process look like?

Encryption at rest: what it protects, and what it does not

SOC 2 requires data to be encrypted at rest and in transit. Buyers often read this as meaning their data is inaccessible to the vendor. Not necessarily.

Encryption at rest means the physical storage media is encrypted. It protects against hardware theft, data center intrusion, and storage-layer attacks. It does not prevent the application layer, which the vendor operates, from reading the data in plaintext.

Unless the customer holds the encryption keys (a feature called customer-managed encryption keys, or BYOK), the vendor's own stack can read data in plaintext at any time. The encryption protects against hardware-level attackers. It does not protect against the vendor itself, vendor staff, or an attacker who has compromised the vendor's application layer. Very few SaaS pentest tools offer BYOK.

NIST SP 800-53 Rev 5 distinguishes between data confidentiality (SC-28), which encryption at rest addresses, and access enforcement (AC-3), which governs who can read plaintext data through the application. SOC 2 requires both categories of control. But "both controls exist" does not mean "vendor staff cannot read your data."

Follow-up questions to ask your vendor

1. Who holds the encryption keys for customer data at rest: your company, your cloud provider's key management service, or the customer?
2. Is there per-customer encryption key isolation, or is all customer data encrypted under shared keys?
3. Under what circumstances would your staff decrypt or access a specific customer's data in plaintext? Has this occurred, and if so, for what reason?
4. Do you offer customer-managed encryption keys (BYOK)? If not, is it on the roadmap?

SOC 2 is not a penetration test

SOC 2 auditors verify that controls exist and that the vendor's description of those controls is accurate. They do not conduct penetration testing. They do not attempt to exploit vulnerabilities in the vendor's infrastructure. They do not test for cross-tenant data leakage, probe for injection vulnerabilities in the application layer, or red-team access controls.

A vendor can have a complete, well-documented SOC 2 control framework and simultaneously have an unpatched remote code execution vulnerability in their image processing pipeline, a misconfigured storage bucket, or a cross-tenant insecure direct object reference in their findings API. SOC 2 will not surface any of these.

NIST SP 800-115 (Technical Guide to Information Security Testing and Examination) and the Cloud Security Alliance both establish penetration testing as a distinct assurance activity from compliance auditing. Compliance auditing verifies that controls are documented.

Penetration testing provides evidence that controls work under adversarial conditions.

Follow-up questions to ask your vendor

1. When was your infrastructure last penetration tested, by whom, and what was the scope?
2. Can you share the executive summary from the most recent penetration test, redacted if needed?
3. How many critical or high-severity findings from your last penetration test are open or accepted as risk?
4. What is your patch management SLA for critical CVEs in infrastructure components that process customer pentest findings? Can you show the last three times it was met?

Breach discovery lag: the 72-hour notification clock starts at detection

Many vendor contracts include a breach notification clause, commonly 72 hours from discovery. Buyers read this as "if there is a breach, we will know within 72 hours." The clause says no such thing. It says: from the moment the vendor discovers the breach, they have 72 hours to notify you.

Mandiant's M-Trends 2024 report shows the global median attacker dwell time before detection has improved to approximately 10 days. IBM's 2023 data shows an average of 204 days across a broader population. Both numbers represent the detection gap: the period during which an attacker has access before the vendor knows. A targeted intrusion against a pentest reporting platform, a high-value target aggregating detailed vulnerability data across multiple organizations, would optimize for stealth over speed. The 72-hour clock does not start until detection. Detection is not guaranteed on any specific timeline.

For pentest findings, this matters more than it does for most data types. Findings are static records. They do not change. An attacker who exfiltrates a database of findings walks away with the complete set, and there is no post-breach revocation equivalent. The data cannot be rotated, reset, or expired after the fact.

Follow-up questions to ask your vendor

1. What is your current average dwell time (time from intrusion to detection) based on your internal monitoring?
2. In your incident response history, what is the longest period between an unauthorized access event and its detection?
3. Your contract commits to notifying us within 72 hours of discovery. What is your target SLA for detection itself, and what monitoring generates the initial alert?
4. If an attacker exfiltrated pentest findings silently over 30 days without triggering automated alerts, what would eventually surface that?

The exit: self-hosted deployment eliminates the question

The follow-up questions above are not rhetorical. They are real due diligence for any cloud vendor handling sensitive data. But they exist because the cloud deployment model creates the exposure they probe. Each question points at a residual risk that a SOC 2 certificate does not eliminate.

Self-hosted deployment eliminates the category. If the vendor never receives the data, the vendor's SOC 2 scope, staff access controls, subprocessor chain, encryption key management, penetration testing cadence, and breach detection timeline are all irrelevant. The security of your findings depends on infrastructure you control.

Open-source deployment adds a layer of verifiability that proprietary self-hosted tools cannot provide. Any customer, auditor, or regulator can inspect the source code to confirm that no telemetry, usage data, or findings are transmitted externally. The answer to "how do you know data does not leave your environment?" becomes "we read the code," not "the vendor told us, and their auditor confirmed the controls existed when last checked."

Dradis Framework has been self-hosted and open-source since 2007, with 19 years of continuous development across 1,177 security teams in 81 countries.

• The data sovereignty brief covers the full architecture comparison between self-hosted and cloud deployment models.
• For teams evaluating specific cloud vendors, the cloud vs. self-hosted comparison breaks down the operational trade-offs.
• For teams comparing pentest reporting tools more broadly, the deployment model question applies to every tool on that list.

For teams where the NIS2 directive applies, the SOC 2 gaps above compound with regulatory exposure.

Practical next steps

1. Request the full SOC 2 report (not the summary certificate) and check which of the five Trust Services Criteria are in scope.
2. Map the vendor's subprocessor chain and confirm which services touch your pentest finding data.
3. Ask the vendor directly: how many staff members can query the production database containing your findings?
4. Request the executive summary of the vendor's most recent penetration test and compare its scope against the systems that store your data.
5. Evaluate whether self-hosted deployment eliminates enough of these residual risks to justify the operational overhead for your team.

Related content

Data sovereignty

• Data sovereignty overview: architecture comparison between self-hosted and cloud deployment models
• NIS2 and pentest data: where SOC 2 gaps compound with EU regulatory exposure
• Shadow AI and pentest data leakage: vendor-access risks in platforms routing findings through external infrastructure

Comparisons and tools

• Cloud vs. self-hosted pentest reporting: operational trade-offs by deployment model
• Pentest report generator roundup: deployment model question applied across the tool landscape
• Self-hosted AI pentest reporting: local LLM processing that keeps findings on your hardware

Frequently asked questions

Ready to see how self-hosted pentest reporting works? Download Dradis Community Edition and run your next engagement on infrastructure you control.