Flight training runs on checklists because memory is a poor inspector, and operations teams deserve the same honesty. If your organisation runs any of its work through chat — a bot posting alerts, people firing commands from a channel — then that setup should survive an audit, not just a demo. ChatOps, meaning operational work conducted in a shared channel where bots post events and execute only allow-listed commands, is established enough to have known failure points, and this checklist walks through all twelve of them in the order an auditor would. The source material behind every item is this ChatOps field reference; keep it open while you score yourself.
What a ChatOps Audit Covers, and What It Skips
The audit tests structure, not enthusiasm. Structure means: can every automated action be traced to a verified request, a known human, an explicit permission and a durable record? Enthusiasm means command counts, bot personalities and channel volume, none of which belong in the inspection. Before item one, gather four artefacts: the inventory of actions the bot may perform, the permission policy, the audit destination, and the last quarter of refusal logs. If any of those four does not exist, that is already your finding — stop, write it down, and proceed anyway, because the remaining items will tell you how deep the gap runs. One working session with the bot’s owner and the channel’s heaviest users is enough to complete this checklist the first time; the discipline lies in doing it again, quarterly, with fresh eyes rather than fond memories.
The ChatOps Checklist: Twelve Items
- Every action is inventoried. The team can produce a current list of everything the bot may do, each entry owned by a named person. An uninspected command set grows quietly until it includes something nobody remembers approving.
- Proof of origin is verified on each platform. Slack requests must pass HMAC-SHA256 signature checks, within a five-minute replay tolerance and a three-second acknowledgement budget. Teams requests must validate as signed JWTs. Mattermost’s static token — which leaves the request body unprotected — must be flagged as a weak control and compensated, not waved through.
- Identity means immutable IDs. No permission decision anywhere keys off a display name, which anyone can change in a settings menu between two commands. Each request resolves to a permanent user identifier before authorisation happens, and that mapping is stored where auditors can read it rather than in the bot author’s head.
- Actions are graded into risk tiers. The lowest tier informs, the next reads, the third writes reversibly under role and channel limits, and the top tier covers consequential moves. The grading exists in writing, and recent changes to it are dated.
- Tier 3 enforces the two-person rule. Requester and approver are distinct, eligible accounts, with dual authorisation mapped to NIST AC-3(2). Reactions and emoji do not count as sign-off anywhere in the system.
- A break-glass path exists and is rare. Emergencies have a documented bypass — loudly logged, reviewed after every use. An audit finds break-glass events measured in single figures per quarter, each with a post-mortem note; a bypass that fires daily is not a bypass, it is the process, wearing a costume.
- No free text reaches a shell. Commands match an explicit allow-list of patterns; everything else is refused. Sample a month of logs and confirm the refusals were real refusals, recorded with reasons.
- The audit trail lives outside chat. Records land in an append-only store shaped along NIST AU-3 lines. If your evidence strategy is “search the channel”, the finding is critical: history expires, edits, and answers nothing under scrutiny.
- Secrets are managed, not pasted. Webhook URLs and tokens sit in a secret manager with rotation. Leak-by-screenshot is a threat, and a credential that lives in a channel description is a credential you have already lost — treat every visible token as compromised and rotate on schedule, not on suspicion.
- The logic survives a platform swap. Adapter, gateway and executors are separated; the catalogue, identity map and records belong to you, not the chat vendor. Platform retirements keep proving the point — HipChat and Stride are gone, Slack’s legacy custom bots ended in March 2025, Teams’ Office 365 Connectors go in May 2026, and Opsgenie follows in April 2027.
- AI assistance is leashed. Assistants propose; humans approve; prompts and tool calls are logged per OWASP’s LLM01 and LLM06. PromptArmor’s August 2024 prompt-injection disclosure against Slack AI is the reference incident, and default-permissive AI setups — Amazon Q Developer in chat famously leaning towards AdministratorAccess with guardrails unset — are the reference mistake.
- Review rhythms actually run. Weekly refusal reviews with attendees, quarterly permission-scope audits, policy kept as code, and a channel taxonomy that stops #general from becoming an operational dumping ground.
The Four Findings Audits Keep Finding
Run this checklist across enough teams and the same handful of failures recur, which makes them worth naming in advance:
- The orphan command. A command exists that nobody can explain — added during an incident, never reviewed, still armed. The inventory (item 1) and quarterly scope audit (item 12) exist precisely to catch these.
- The channel-as-archive. A team that answers “can you prove it?” with a channel search. Conversation history was never evidence, and the gap only becomes visible the first time a serious question arrives after retention has pruned the thread.
- The friendly shortcut. Approvals via emoji, decisions in direct messages, credentials pasted where colleagues can copy them. Each one saves seconds and quietly removes a control the team believes it has.
- The eager assistant. An AI integration granted broad permissions because the default was permissive, with no logging of what it was asked and what it touched. The August 2024 injection findings against chat AI turned this from a theoretical worry into a documented attack path.
Notice the pattern across all four: none of them involve exotic attacks. They are ordinary habits that outgrew their safety margin, which is why a checklist beats a security lecture — habits get inspected, lectures get nodded at.
Scoring the Audit Honestly
Mark each item pass, partial or fail — and treat partials as fails until the paperwork exists, because undocumented controls decay on contact with staff turnover. Score ten or better and the setup is trustworthy for its current scope. Score six to nine and you have structure but leaks; fix the security items (2, 3, 9) before any hygiene items. Below six, halt new commands entirely until items 2, 7 and 11 pass, since forged requests, vanishing evidence and unleashed AI are the three findings that end careers. Benchmarks exist for the recovered state: published operations report coverage of audited changes at 93.9% and a 98.1% command success figure, alongside an actionable-notification share of 23% against a 50% target — while unmanaged alert traffic of roughly four hundred weekly messages, spread over a dozen people at ten seconds each, quietly costs some thirteen hours of attention.
Where to Fix What the Audit Finds
Findings cluster into three repair tracks. Security findings (items 2, 3, 5, 9) get fixed first and fast — they are cheap to fix and expensive to ignore. Evidence findings (items 1, 7, 12) follow: inventories, append-only records and review calendars, all boring, all decisive when a regulator or a customer asks questions. Hygiene findings (items 10, 11 and the noise half of 12) come last, because they tune an engine rather than keep it attached. Sequence the work this way and each repair makes the next one easier, instead of fighting it. Budget them like any training decision — the sequencing logic resembles how students approach costs, weighing what pays back soonest, as laid out in ways to reduce flight training expenses and the funding options in a scholarship guide for flight training students; the aircraft-choice trade-offs in how aircraft type changes training cost teach the same lesson about matching capability to need. Re-run the audit quarterly. A ChatOps setup is never “done” any more than a pilot is ever done with checklists — the value is that the next flight starts from a known state, every time.
