Mark Aklian is Chiri’s Chief Information Security Officer and has vast senior security experience including Point72, Bank of America, and many more.
Having curated and battle-tested this checklist for over a year, it’s grown from a simple review aid into a board-ready framework I use with our clients, product teams, vendors, and red-team exercises. It’s been hardened against real incidents, mapped to emerging regs, and trimmed to what actually moves risk. Here’s the zero-BS version 👇
AI Security, Privacy & Risk — What I Actually Check:
🧠 Architecture transparency
• Where do models run (vendor/self-hosted)?
• Which FMs, fine-tunes, agents?
• Any third-party AI APIs, embeddings, vector DBs?
🔁 Data flow & retention
• Are prompts/outputs logged? For how long and by whom?
• Used for training/evals? Opt-out controls?
📚 RAG hygiene
• Sources, chunking/metadata, pre-index redaction
• Tenant-scoped indices, per-doc ACLs
• Vector deletion workflows
🔐 Privacy controls
• Legal basis (GDPR/CCPA), purpose limitation
• DSRs for prompts/vectors/model snapshots
• Handling of sensitive data (ATS/HR/health/finance)
🛡️ Guardrails against abuse
• Prompt-injection defenses, retrieval allow-lists
• Output filtering/citations, jailbreak/misuse protections
• Abstain on low confidence
🧰 Tool/agent safety
• Sandboxing & controlled egress
• Scoped credentials, auditable tokens
• Function-calling with least privilege
📋 Governance & change management
• Model cards & documentation
• Prompt/model version control, approvals, rollback
• Audit trails on safety rule changes
🧪 Testing & red-teaming
• Pre-prod & continuous evals (accuracy, leakage, injection resilience, bias)
• Independent red-teaming with concrete attack playbooks
🔎 Explainability & human-in-the-loop
• Source-grounded answers, rationale visibility
• Human review gates for high-impact decisions (hiring, finance, legal)
🔑 Keys, secrets & spend
• Central LLM gateway, vaulting & rotation
• No secrets in prompts
• Rate limits/quotas, cost anomaly detection
🧭 Regulatory alignment
• Map to NIST AI RMF, ISO/IEC 42001, and (where relevant) EU AI Act
• Sector overlays: EEOC / SEC / FINRA / HIPAA
🚨 AI incident response
• Playbooks: injection, RAG exfil, model/safety regressions
• Forensics: retain prompts/outputs, retrieval snapshots, versions
If your AI implementation (or third party that utilizes AI to enhance their services) can’t answer these with evidence, you’re likely carrying an inordinate amount of risk.
Reach out for guidance. We’re here to help.