Playbook 4 — Rollout & Lifecycle

Why Gate the Rollout

An AI system that works in testing can still behave differently once real users and real data hit it. The fix isn’t more testing up front — it’s a controlled, staged rollout with a defined exit at each stage. This playbook runs alongside your adoption plan (readiness, use-case selection, adoption gates); this is the technical and risk gate, not the change-management one.

The Four Gates

Four points where the rollout has to earn the right to continue. The rest of this page fills in each one.

Before the Pilot: Requirements and Verification

The pilot checklist below assumes two things already happened. Skip them and the pilot tells you whether the system works — not whether it works for the reason you think.

• One paragraph, written down: why this system, what it has to do, and what data trains or feeds it.
• A test plan exists — what “pass” looks like, tested on data that actually resembles what production will see, not a clean demo set.
• You’ve agreed in advance on acceptable reasons the system might miss its target (an error rate under 5% for a first release, say) — not a pass/fail bar set after you see the results.

Controlled Pilot Checklist

• Limited to internal staff, or a small, defined customer segment — not everyone on day one.
• There’s a clear rollback trigger: a specific error rate, complaint volume, or outcome that pauses the rollout automatically.
• A human can review or reverse any individual decision the system makes during the pilot.
• Someone is actually looking at the outputs weekly, not just trusting the dashboard.

Monitoring Plan — What to Track After Go-Live

• Accuracy or quality of outputs — spot-check a sample monthly.
• Error rate or complaint volume, and whether it’s trending up.
• Whether the system still behaves consistently across the groups checked in Playbook 3.
• Whether the underlying data or business conditions have changed enough to warrant a re-check (new product line, new market, new regulation).

Keep a running log of system events, not just monthly spot-checks. For most SMB tools this is the vendor’s own activity log — check that it exists, and that someone can pull it if a decision is ever disputed. The spot-check tells you it’s still working; the log is what you hand someone who asks why it did something specific.

Signs It’s Time to Retrain or Retire

• Error rate or complaints have risen noticeably since launch.
• It no longer reflects current business reality (new pricing model, new customer base, new regulation).
• A better or cheaper alternative now exists.
• It’s being kept alive out of habit rather than because it still adds value.

If the System Keeps Learning

Most SMB AI tools are static between updates — the vendor retrains it, not you. If yours is the exception (a model that updates itself on your data, or a feature that adapts to user behavior in real time), the retrain/retire checklist above isn’t enough on its own. Add a standing check:

• Someone re-runs the original test data through the system on a fixed schedule — monthly, for most SMB use — not only when something looks wrong.
• A drift in output is treated as a trigger the same way an error-rate spike would be, not something you wait to notice.

If nothing about your system continuously learns, skip this. Most SMB deployments can.

Retirement Checklist

• The reason for retirement is documented.
• Anyone or anything relying on this system has a replacement or fallback in place before it’s switched off.
• Historical records and decision logs are preserved, not deleted, for as long as they might matter (audits, disputes).
• The AI system inventory (Playbook 6) is updated to reflect the retirement.

Next Steps

Update the AI system inventory and decision log (Playbook 6) at every gate — this is what turns “we were careful” into something you can actually show someone.