Worksi — Runbook Library & Platform Test Programme
Thirty-three operational runbooks for the clients of an AI-gateway product, and an end-to-end test of every product area that fired all eleven incident detectors.
- Role
- Runbook programme lead and platform tester
- Team
- Worksi (XEOO) team
- Timeline
- Oct 2026
- Organisation
- Worksi (XEOO)
- Code
- Proprietary (employer)
- Reading time
- 4 min
For Worksi AI Control, the XEOO team’s AI gateway and monitoring product, I planned and wrote a 33-runbook library — incident response, security, administration and internal operations — with machine-importable versions, and ran a full platform test that exercised every product area.
Overview
Worksi AI Control sits between a company’s applications and its AI providers: a compatible gateway, incident detectors, policies, approvals and cost reporting. Clients going live needed to know exactly what to do when a detector fires — and the team needed evidence that every part of the product behaves as documented.
The problem
- Clients had no step-by-step procedures for incidents, security events or routine administration.
- Every incident type and error code needed a clear owner and a documented response.
- The product’s areas — gateway enforcement, detectors, policies, approvals, tracing and exports — had never been tested together end to end.
Context & constraints
The product had been live since late September 2026. I worked from the live console, its documentation and the public API specification (170 endpoints), staying read-only except for an approved test programme in the company’s own organisation.
Approach
- Plan — a programme of 33 runbooks in four tracks, piloted with three to fix the style, then built in four waves.
- Library — 12 incident runbooks, 3 for security and continuity, 12 administration procedures and 6 internal ones, on two fixed templates. The client edition is 27 runbooks over 149 pages, with an index mapping every incident type and error code to its runbook.
- Single source — Markdown masters with structured front matter produce the PDFs, the wave handbooks and JSON payloads that import straight into the product through its API (with a dry-run mode, and never self-approving).
- Test programme — 780 gateway requests through the product’s simulator, 26 incidents covering all 11 detectors, 12 policies covering all 10 policy types, approvals, tracing, the MCP server and exports — written up with findings and configuration lessons for onboarding.
- Enablement — a complete product handbook and a 164-question study guide for conversations with clients, investors and engineers.
Key capabilities
- 33 runbooks in four tracks, two consistent templates
- Index mapping every incident type and error code to a runbook
- Wave handbooks for client distribution
- API-importable versions with a dry-run import tool
- End-to-end platform test with a written report
- Product handbook and a 164-question study guide
Key decisions
ADR-01Pilot three runbooks before writing thirty
- Context
- Thirty-three documents written in different styles would be hard to follow in an incident.
- Decision
- Write one incident runbook, one security runbook and one procedure first, agree the templates, then build in waves.
- Trade-offs
- A slower start for a library that reads as one voice.
ADR-02One Markdown master per runbook
- Context
- Each runbook had to exist as a PDF, inside a handbook and as an importable record in the product.
- Decision
- Keep a single Markdown source with structured front matter and generate every format from it.
- Trade-offs
- A small build pipeline to maintain — but no copies that drift apart.
ADR-03Report product defects to the team, not to clients
- Context
- The test programme found product issues, including two high-priority ones in the approval workflow.
- Decision
- Write them up for the team with evidence, and keep client-facing runbooks to the documented behaviour.
- Trade-offs
- Some runbooks wait on fixes — but clients never receive procedures built around defects.
Challenges
Testing with real, billed traffic
Problem. Exercising every detector meant generating real incidents in a live product.
Solution. An approved programme inside the company’s own organisation, driven by the product’s simulator and limited to a single runner script.
Keeping 33 documents consistent
Problem. Shared steps, error codes and terminology repeat across the library.
Solution. Two templates, a shared index and structured front matter that the build validates.
Quality & delivery
Testing
The test programme covered 780 gateway requests (499 billed), 302 traced runs, 26 incidents with all 11 detectors firing, 12 policies across all 10 policy types, 14 runbooks approved and linked, gateway enforcement responses, OpenTelemetry, the MCP server and exports. It produced five product findings — two of them high priority — and four configuration lessons for client onboarding.
Deployment
Edition 1.0 was released on 15 October 2026, with a review due in April 2027. Import files are ready to load into a client’s organisation once each client is onboarded.
Outcome
- Runbooks
- 33
- Four tracks, four waves
- Pages in the client edition
- 149
- 27 runbooks
- Incident detectors exercised
- 11 / 11
- 26 incidents in the test programme
- Gateway requests in the test
- 780
- 499 billed
The library shipped as Edition 1.0 in October 2026 with client handbooks, an index and importable versions, and the test report gave the team a prioritised list of product fixes before wider rollout.
Lessons
- A runbook is only as good as the test that proved its steps work.
- Documentation scales when it has one source and many outputs.
Building something similar?
Tell me what you’re working on — I reply to every serious message.
Related work
HR Inbox Agent
A Python agent that reads an HR mailbox through Microsoft Graph, asks a language model for a structured classification and draft reply, and lets deterministic code decide whether to send, save a draft or escalate to Slack.
- Python
- SQLite
- Docker & Compose
- LLM APIs
- Agent guardrails & evals
- AI gateways
- +1