Classic program code behaves predictably: if you don't change the source code, you get the same result. This is not true for AI systems. Language models are regularly updated by providers in the background. The same model can deliver different answers after an update — without an error message appearing.
This phenomenon is called model drift. This post explains how it occurs and how companies can detect it before it causes damage.
The Problem: Invisible Changes
„Classic code behaves stably as long as no one changes it. AI systems can change without anyone touching the pipeline."
An AI agent produces structured outputs — for example, an instruction like „release delivery" or „escalation required". The subsequent system that processes this output expects a specific format. If, after an update, the model delivers an explanatory free text instead of a clear status, further processing fails.
Particularly insidious: there is often no error message. The system receives a response but simply cannot evaluate it. The affected business case remains unprocessed — and no one notices it immediately.
The Solution: Quality Assurance for AI Pipelines
The antidote comes from software development: systematic tests before every change. Applied to AI systems, this means three components:
Test Dataset: A collection of 100 to 200 representative, anonymized cases with known outcomes. For a warranty process, this would be: this input, this expected decision.
Automated Testing: With every change — new model, modified prompt, new configuration — the test dataset automatically runs through the pipeline. Results are compared with the expected outputs.
Two Levels of Verification: First, it is checked whether the output has the correct format. Then, the content is checked to see if the decision is correct. Only when both checks have passed does a change go into production.
Practice: The Smart Hub Case After a Model Update
In Blog 04, the agent correctly concluded the Markus Meier warranty case: it recognized that a business rule applied — the regional manager's approval was pending — and escalated accordingly. This worked because the model provided a clearly structured, unambiguous status message.
After a model update, the behavior changes: the model now delivers the same information as a friendly explanatory text instead of a structured status. The agent's monitoring layer expects the clear status — and doesn't find it. The Markus Meier case remains unprocessed. No error appears, no message — the process quietly grinds to a halt.
An automated test with the test dataset would have detected this deviation before going live.
Interim Conclusion
Five of the seven concepts in this series have been described: Structured Outputs (Blog 02), Targeted Information Selection (Blog 03), Control of Autonomous Execution (Blog 04), Data Protection through Architecture (Blog 05), and Systematic Quality Assurance (Blog 06).
Blog 07 addresses the organizational side: Who has which permissions — and who bears responsibility when an agent acts?
The prompt is just the beginning. The architecture is the goal.