The most underestimated risk is a change that does not come from you. The vendor releases a new model version, retires the old one, and the agent’s behaviour changes without anyone deciding it should. Often the change affects only some cases, so it is hard to notice.
Someone would report an outage. A decline of a few percentage points shows up differently. The team starts checking the results more carefully, then checks all of them, and after a quarter stops using the agent.
The solution is a regression test – a check that a change has not broken what worked before. The whole reference set runs automatically and is compared with the previous result. If the result falls below the agreed threshold, the change does not go into day-to-day use.
The list below says when to run the test and which two habits are worth adding. They cost little but help you avoid surprises.