HIPAA and SOC 2 govern different things. HIPAA is a law that follows protected health information; SOC 2 is a voluntary attestation about your controls. For an AI feature you are not certifying the model — you are scoping the system that touches sensitive data, signing the right agreements, and proving your controls actually operated.
This article is general engineering guidance, not legal advice. Have a qualified attorney or compliance professional review your specific scope, agreements, and obligations.
What these frameworks actually require of an AI feature
Neither framework asks you to audit a model’s weights. Both ask the same question in different vocabularies: what are the controls around the data, and can you prove they ran?
HIPAA is a statute enforced by HHS/OCR: the Privacy Rule governs what you may do with PHI and with whom, the Security Rule governs safeguards. There is no certificate at the end, only an obligation. SOC 2 is an attestation report from a licensed CPA firm against the AICPA Trust Services Criteria — you do not “get certified,” you define a system boundary and receive a report on the controls inside it.
The part teams miss: the unit of scope is the system, not the model. An LLM feature adds three things to your boundary that a CRUD app does not have:
- A third-party inference hop that receives raw payloads, including whatever users pasted in.
- A new persistent store of prompts and completions, usually holding the same sensitive data as the source system, in a tool nobody classified as a system of record.
- Non-deterministic output, which can leak across tenants if retrieval isolation is weak or a shared cache returns another tenant’s context.
Before writing code, draw a data-flow diagram and label every hop as PHI flows here, de-identified only, or no customer data. That diagram is the first artifact an auditor requests, and it determines which vendor needs which agreement. A hop labeled “de-identified only” is a claim you must be able to test.
HIPAA: PHI, the BAA, and where inference actually happens
PHI is individually identifiable health information created or received by a covered entity or business associate. The Privacy Rule’s Safe Harbor method at §164.514(b)(2) enumerates 18 identifier categories: names, geographic subdivisions smaller than a state, all date elements except year, phone numbers, email addresses, SSNs, medical record and account numbers, URLs, IP addresses, device and biometric identifiers, full-face photographs, and any other unique identifying number or characteristic.
Note what that means for AI infrastructure: a log line containing request IP address + full timestamp + a clinical question is plausibly PHI even without a name — and teams store exactly that shape of record in default-configured observability tools.
The BAA is a precondition, not paperwork
A Business Associate Agreement must be in place before PHI is disclosed — you cannot ship first and paper it later, because the disclosure already happened. If you build for a covered entity, you are a business associate, and a model provider receiving PHI is your subcontractor, which requires a second BAA in the chain. Two agreements, not one.
Three architectures for inference, and how to choose
- Send raw PHI to a model API. Requires an executed BAA plus a verified no-retention configuration. Many self-serve tiers will not sign a BAA at all; enterprise tiers generally will. Confirm which tier your production key belongs to before any PHI leaves your boundary.
- De-identify before inference. Data de-identified under §164.514 is no longer PHI, so the provider is not a business associate for that flow. If you must re-identify results, the mapping table stays inside your boundary — and becomes the highest-value object in your system.
- Self-host an open-weight model in a HIPAA-eligible environment. Maximum control and cost, and it shifts the burden from contracts to infrastructure hardening.
The decision criterion is whether identifiers are load-bearing. If the task needs them — “summarize this chart and name the prescribing physician” — de-identification breaks the feature and you must choose between a BAA-backed provider and self-hosting. If the task is triage or classification over de-identified text, the second architecture is far cheaper to comply with, because it removes a vendor relationship from scope entirely.
“We don’t train on your data” is not “we don’t retain your data”
These are different commitments; get all three in writing: no training on customer data, an explicit retention window for abuse monitoring (some providers offer zero-data-retention endpoints), and deletion on request with a stated SLA. Then verify it operationally — confirm the retention setting is enabled on the exact endpoint your production key points at, and keep that configuration in version control so a silent console change shows up as a diff.
Encryption, including the stores nobody classifies
TLS 1.2 or higher in transit, AES-256 at rest. Under the Security Rule, encryption at rest is an addressable specification: implement it, or document why it is not reasonable and implement an equivalent alternative — auditors ask for that rationale, so if you skipped encryption somewhere, the memo matters more than the decision. Extend it to every derived store: prompt and completion logs, vector databases, eval datasets, backups. Disk-level encryption is not enough if a query inside your own application can read another tenant’s plaintext. The Security Rule also expects unique user identification (§164.312(a)(2)(i)), automatic logoff (§164.312(a)(2)(iii)), audit controls (§164.312(b)), and — the one most teams fail — actual review of system activity (§164.308(a)(1)(ii)(D)). Collecting logs is not the control; reviewing them is.
SOC 2: mapping the trust services criteria to an AI feature
Security (the Common Criteria, CC1–CC9) is mandatory. Availability, Processing Integrity, Confidentiality, and Privacy apply only if you scope them in. Most AI SaaS starts with Security, Availability, and Confidentiality.
| Criterion | What it means for an AI feature |
|---|---|
| CC6.1 — Logical access | Per-tenant and per-environment keys, no shared production keys, MFA on the console, rotation on offboarding. A leaked key here does not just read data — it spends budget and can exfiltrate tenant context. |
| CC6.6 / CC6.7 — Boundary protection | Encryption in transit and at rest, egress allow-listing so only approved inference endpoints are reachable, network isolation between tenants. |
| CC7.1 / CC7.2 — Monitoring | Alerts on anomalous prompt patterns, token-spend spikes, and repeated near-miss refusals that suggest probing. |
| CC7.3 – CC7.5 — Incident response | A rehearsed runbook, severity definitions, and a documented evaluation after every incident. |
| CC8.1 — Change management | Prompts and model versions are changes. Version them, require review, record the resolved model version per request, keep a rollback path. |
| CC9.2 — Vendor risk | A maintained vendor register, current reports, and review of subprocessors. |
| A1.1 – A1.3 — Availability | Provider SLAs, a tested fallback chain, capacity evidence, and monitoring that supports your uptime claims. |
| PI1.1 – PI1.5 — Processing integrity | Output validation and eval gates in the deploy pipeline, not just human review. |
| C1.1 — Confidentiality | Retention limits and disposal controls across every store, including logs and vectors. |
CC8.1 is the criterion AI teams most often under-build. A one-line prompt edit can change what data a model emits, which tools it calls, and what it costs. Treat prompt templates as code: pull requests, review, version identifiers, and a rollback that does not require a deploy. For model upgrades, record the version actually served alongside each request — “we use the latest” is not an auditable statement.
Two structural facts matter more than any single control. Type I covers design at a point in time; Type II covers design and operating effectiveness over an observation period, commonly three to twelve months. And the scheduling trap: Type II evidence cannot be created retroactively. If access logging goes live in month two of a twelve-month window, months zero and one are a gap the auditor will report. Start collecting the moment a control is live, even if the audit is a year away. Our guide on LLM observability covers what is worth recording.
Vendor management: DPAs, BAAs, and what to demand in writing
A DPA and a BAA are not interchangeable. A DPA is the GDPR Article 28 instrument covering personal data generally; a BAA is HIPAA-specific and covers PHI. A DPA does not satisfy HIPAA, and a BAA does not satisfy GDPR. If you serve both EU customers and US healthcare customers, you need both documents with the same vendor. Our companion guide on AI data privacy and GDPR covers the EU side; this article stays on HIPAA and SOC 2.
Maintain a subprocessor list with advance change notification (30 days is a common contractual floor) and a right to object. Your model provider’s own subprocessors — inference hosts, cloud regions, moderation services — are part of the chain, and a change there can invalidate your residency assumption even though nothing in your code changed.
What to get in writing before onboarding any vendor that could touch sensitive data:
- An executed BAA, or written confirmation that the endpoint is not BAA-eligible so PHI must never be routed to it.
- Retention window, no-training commitment, and a deletion SLA with a measurable response time.
- Current subprocessor list, plus the change-notification mechanism and notice period.
- Breach notification timeline. HIPAA’s outer bound is 60 days without unreasonable delay; contract for 24–72 hours, because your notification clock starts when you learn, not when they finish investigating.
- Processing regions, and whether you can pin them per tenant.
- Encryption standards and key management — who holds the keys, and whether customer-managed keys are available.
- Annual right to receive the SOC 2 report and ask clarifying questions about it.
- A named escalation path for security incidents, not a generic support queue.
One architectural lever makes all of this tractable. If every model call goes through a single OpenAI-compatible endpoint you control, you have one BAA-eligible path, one place to enforce redaction, one audit-log format, and one place to pin a model version — instead of N integrations each with its own retention defaults. An AI API relay such as qoraapi.com sits in that position: one endpoint in front of many providers, so the compliance surface is a single boundary rather than a fan-out.
Read the vendor’s report properly. Section III (the system description) defines what was actually in scope, and Section IV usually lists complementary user entity controls — things the vendor asserts you must do for their controls to hold. Your auditor will test whether you implemented them, so copy the CUECs into your control matrix the day you receive the report.
Data-handling controls and the evidence that satisfies them
Controls without artifacts are opinions. If you cannot name the artifact, you do not yet have the control.
| Control | What it enforces | Evidence an auditor accepts |
|---|---|---|
| Ingress de-identification | Safe Harbor identifiers removed before any outbound call | Redaction middleware source, unit tests, a sampled redacted request |
| BAA register | Every counterparty touching PHI has an executed agreement | Signed BAAs with effective dates, owner, next review date |
| Retention enforcement | TTL on prompts, responses, vectors, and logs | Configuration export, deletion job run logs, one record shown past expiry |
| Encryption | TLS 1.2+ in transit, AES-256 at rest across all derived stores | Configuration export, key management policy, a posture or scan report |
| Access control | Per-tenant and per-environment keys, least privilege, MFA | Key inventory, IAM policy export, signed quarterly access reviews |
| Residency pinning | Inference and storage confined to approved regions | Per-environment region configuration, vendor region attestation |
| Change control for prompts and models | Every behavioral change is reviewed, versioned, reversible | Pull request history, prompt version registry, a rollback record |
| Audit logging | Immutable record of who accessed what, when | Log schema, retention setting, a sample query and its output |
| Incident response | Detect, contain, assess, notify — with timestamps | Incident tickets, breach risk assessment memos, notification records |
| Vendor review | Subprocessors and reports reviewed on a cadence | Vendor register, dated review notes, current SOC 2 report on file |
Evidence and logging: what auditors actually ask to see
Access logs should answer four questions per event: who (a unique user or key identifier, never a shared service account), what (resource and action), when, and outcome. HIPAA requires unique user identification and audit controls; SOC 2 CC7 expects you to monitor those logs, which means someone must be able to describe the alert thresholds.
Retention is set by the strictest applicable rule. HIPAA requires documentation retained six years from creation or the last effective date (§164.316(b)(2)(i)); SOC 2 evidence must span the entire observation period. Enforce the window by configuration, not convention.
The non-obvious trap: logging raw prompts creates a new PHI repository. The moment a prompt body lands in a general-purpose observability tool, that tool is in scope — encryption, access control, retention limits, and a role in your breach assessment. Two patterns are defensible: log metadata only (template ID, version hash, model, token counts, latency, outcome) plus a pointer into an encrypted, TTL-limited payload store; or log raw content and apply the full control set you apply to your primary PHI store. The middle ground — raw prompts with default retention in a shared tool — is what turns an incident into a reportable breach.
import hashlib, json, re, time
# First-pass identifier scrubbing. Safe Harbor also requires that all 18
# identifier categories are addressed AND that you have no actual knowledge
# that the residual data is re-identifiable -- a regex list is a starting
# point, not proof of de-identification.
IDENTIFIER_PATTERNS = [
r"\b\d{3}-\d{2}-\d{4}\b", # SSN
r"\b[\w.+-]+@[\w-]+\.[\w.]+\b", # email
r"\b(?:\d{1,3}\.){3}\d{1,3}\b", # IP address
r"\b\d{1,2}/\d{1,2}/\d{4}\b", # full date (keep year only)
r"\b(?:MRN|DOB)[:\s]*\S+", # labeled identifiers
]
def deidentify(text):
hits = 0
for pattern in IDENTIFIER_PATTERNS:
text, n = re.subn(pattern, "[REDACTED]", text, flags=re.I)
hits += n
return text, hits
def audit_event(**f):
"""Emit metadata only -- never the raw prompt or completion."""
return json.dumps({
"ts": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
"actor": f["actor"], # unique user id, not a shared key
"action": f["action"],
"tenant": f["tenant"],
"prompt_version": f["prompt_version"], # ties output to a reviewed change
"model_version": f["model_version"], # pin it; "latest" is not auditable
"payload_sha256": f["payload_sha256"], # pointer into the TTL store
"redactions": f["redactions"],
"retention_days": f.get("retention_days", 30),
}, separators=(",", ":"))
# Order matters: de-identify BEFORE the outbound call, log AFTER redaction.
safe_prompt, n = deidentify(user_text)
audit_event(actor=user.id, action="llm.invoke", tenant=user.tenant,
prompt_version="triage-v7", model_version="mid-2026-04",
payload_sha256=hashlib.sha256(safe_prompt.encode()).hexdigest(),
redactions=n)
Incident records follow the same principle. For a suspected breach, HIPAA expects a documented four-factor risk assessment: the nature and extent of the PHI involved, the unauthorized recipient, whether the PHI was actually acquired or viewed, and the extent of mitigation. Document it even when you conclude it is not a breach — that memo is the evidence your process worked. Timestamp each phase (detection, containment, assessment, notification) so the timeline is reconstructable. Monitoring design for the detection half is covered in our guide on AI API security.
A pre-audit checklist
- Data-flow diagram exists, with every hop labeled PHI / de-identified / no customer data.
- BAA register complete — including the model provider, not just the covered-entity customer.
- Retention configured and verified on every store: prompts, completions, vectors, logs, backups.
- Redaction tested against a labeled synthetic corpus; false-negative rate recorded.
- Prompt templates and model versions in version control, with review and a tested rollback.
- Per-tenant keys, MFA on admin access, and a signed access review from the last quarter.
- Audit logging enabled, retention set, and a dated monthly review note proving someone looked.
- Vendor register current, SOC 2 reports on file, CUECs copied into your control matrix.
- Incident runbook rehearsed at least once, with a written after-action record.
- Every control has a named owner — auditors ask “who owns this,” and “the team” is not an answer.
- Type II observation start date scheduled after all controls are live, not before.
Frequently asked questions
Does using a model provider with a SOC 2 report make my app SOC 2 compliant?
No. Their report covers their controls inside their boundary; yours covers what you built. A vendor report is one piece of evidence under CC9.2 (vendor risk), and it typically includes complementary user entity controls that you must implement for their assurances to hold.
Can I send PHI to an LLM without a BAA?
Not if the data remains identifiable. Either execute a BAA and confirm a no-retention configuration on the endpoint you actually use, or de-identify under §164.514 before the call so the payload is no longer PHI. Safe Harbor requires all 18 identifier categories removed and no actual knowledge that the residual data could be re-identified — a regex pass is a first line of defense, not automatic proof.
Is SOC 2 required by HIPAA?
No. HIPAA’s Security Rule is the legal requirement; SOC 2 is a voluntary AICPA attestation. Enterprise buyers request SOC 2 reports as third-party assurance, and many teams map HIPAA safeguards onto the same control set so one body of evidence serves both. Passing SOC 2 does not by itself establish HIPAA compliance, and vice versa.
How long must I keep audit logs and compliance evidence?
HIPAA requires documentation retained six years from creation or the last effective date. SOC 2 evidence must cover the full observation period, commonly three to twelve months. Contracts and state law can extend either. Keep the longest applicable period.
Conclusion
Compliance for an AI feature is not a property of the model. It is a property of the boundary you draw around it: which hops see identifiable data, which vendors hold agreements, which stores enforce retention, and which logs prove any of it happened. Start with the data-flow diagram, close the BAA gap for every hop that sees PHI, put prompts and model versions under change control, and begin collecting evidence before you need it — Type II evidence does not backfill.
If you are still designing the architecture, our AI API gateway guide covers how to centralize model access behind a single endpoint, which is also the cleanest way to keep your compliance surface to one governed hop.
Leave a Reply