TRACE autonomy checklist
Five questions to answer before you let a system decide something on its own. You leave with what needs work, who owns it, and the next step.
About this tool
When to use it
Once per automated step, before you ship, with the person who owns the step. About 15 minutes.
What you get
A list of what needs work before launch, with owners and next steps, as Markdown for your team’s doc.
What it is not
A rule of thumb, not a standard, and not a replacement for testing. A written answer is not proof that the answer is good. Nothing leaves your browser, and your answers are saved in it.
Can you actually see it work?
- Where does this step write its log?
- Who can read that log without asking someone for access?
- If it goes wrong at 3am, how long until a person knows?
If the honest answer is “when a customer complains”, fix that before anything else. A log nobody can reach is not a log.
What does it claim, and how sure is it?
- Does the output carry a confidence, or just an answer?
- Can you write one line saying what would have to be true for this answer to be wrong?
If you cannot write the line, you do not have a ranking. You have a guess with a number stapled to it.
Who owns the result?
- Who is responsible when this is wrong? Name a person, not a team.
- Which severity level always goes to a human, even when the model is very confident?
- What does that person see: just the answer, or the answer plus the case against it?
Being sure is not the same as being responsible. Confidence sets the routing for the boring cases. Severity overrides it.
Can a person disagree cheaply?
- How many clicks to override this?
- Is the override saved with a reason, or does it vanish?
- Does the reviewer see the argument against the answer before they decide, or only after?
If disagreeing is slow, people stop. Your metrics then show high agreement, and what you built is a system that wore everybody down.
What closes the loop?
- Who reads the overrides, and how often?
- What is the rule for changing the threshold? Write it now, while nothing is on fire.
- If nothing has changed in three months, is the system working, or is nobody looking?
Both look identical on a dashboard.
What does a miss cost, and what does a false alarm cost?
- A miss is expensive, a false alarm costs a minute: favour recall, accept the noise, and tell the team you are doing it on purpose.
- Both cost about the same: look at the cost of the whole process before you decide whether to automate it.
- A false alarm is expensive: raise the bar, accept that you will catch less, and say so out loud.
What you have so far
Needs work before launch
Documented
The same, as Markdown for your team
Keep the filled-in version next to the code, not in someone’s head. The test is whether it still makes sense when the person who built the step is on holiday. The full text is on GitHub under CC0.