Even well-behaved assistants need emergency brakes. A kill switch runbook keeps incidents controlled and auditable.
Kill switch tiers
- Per-embed: Disable a specific placement if it misbehaves (e.g., marketing page only).
- Per-tenant: Pause all embeds for a customer (billing, security, policy).
- Global: Turn off every embed if the LLM provider or data pipeline is compromised.
Runbook steps
- Identify trigger: security incident, hallucination spike, billing issue, or regulator request.
- Decide scope: embed, tenant, or global; document reason.
- Execute: flip flag in ops UI or API; log actor, timestamp, scope.
- Notify: send Google Chat alert, email tenant admins, and update status page if global.
- Mitigate: fix root cause (crawl, prompt, provider).
- Restore: re-enable embeds, notify stakeholders, and annotate analytics timeline.
UX considerations
- Show a friendly message in the widget: “Assistant temporarily disabled. Please contact support.”
- Provide admin dashboard banners explaining the status.
- For global incidents, add a public status page entry.
Testing
- Exercise kill switches monthly in staging and quarterly in production with low-risk tenants.
- Record results in the audit log (who tested, outcome, follow up).
- Ensure analytics continue logging “disabled” events for postmortems.
Threada controls
Threada exposes kill switches in ops tooling, logs every change, and blocks new chats while allowing existing conversations to wrap up gracefully. Adopt a similar runbook to keep AI incidents contained.***