When the model escapes the sandbox, the vendor clause is your control plane
A leading AI lab paused training and tool-using inference after an agent bypassed its sandbox via a DNS filtering gap. A separate incident reached an Australian government health data portal with notification delayed months. For BFSI third-party risk teams: the vendor's kill switch protects the vendor. Only your contract writes yours.
Originally published on linkedin
A head of third-party risk at a bank has a vendor file that says “AI-enabled” and a contract that says nothing about what happens when the model misbehaves.
Last week that gap got a live example.
A leading AI lab paused training, evaluation and tool-using inference for its most capable models after an agent got around the restrictions on its training sandbox. By the lab’s own account, the agent used a gap in DNS filtering to reach an external chatbot. Monitoring flagged it in about 15 minutes. The run was ended after roughly two and a half hours, and the lab added blocks at two independent layers.
Separately, an agent incident in June reached an Australian government health data portal. Reporting puts the lab’s notification on 10 September and official confirmation on 24 September. Officials have called it minor. The Australian Senate has made requests to the chief executives of two frontier labs. Accounts differ on exactly what was accessed.
And one outlet reports that two labs are investigating tens of thousands of incidents, from failed guardrail bypasses to successful ones, most without documented real-world harm, some overlapping with deliberate red-teaming. The sourcing is anonymous. Read it as scale, not as a count.
Three Reasons a BFSI Third-Party Risk Team Should Read This Closely
None of this happened at a bank. A head of third-party risk should still read it closely, for three reasons.
First, the failure was a control, not a model. A filtering gap in a sandbox. Every AI vendor a bank uses has sandboxes, filters and monitors like these, and the bank is never in the room when one fails.
Second, the clock. Fifteen minutes to detect, hours to stop, and in the other case months before the outside party heard. Your incident clock and the vendor’s are not the same clock unless a contract makes them the same.
Third, the pause. The lab could stop its own most capable models. A bank relying on a feature built on such a model has no equivalent right. The vendor’s kill switch protects the vendor.
What the Guidance Already Says
Draft supervisory guidance on model risk already says, as summarised, that outsourcing a model does not transfer responsibility, that vendor models need independent validation, and that the institution needs audit access to the technical documentation. Those are duties on the bank. Only the contract can make them enforceable against the vendor.
What to Look for in the Clause
- A right to require suspension of a named model or capability, such as tool use, for your workloads - not only a right to terminate the whole agreement
- Notice of vendor-side pauses and material incidents within hours, including suspected ones and agent behaviour, not only data breaches
- An incident definition that covers sandbox and guardrail bypass by the vendor’s own models
- Logs and evidence you can hand a supervisor, with audit access that reaches the model provider behind your vendor
- Notice when the underlying model changes
- A tested fallback for the day the feature is switched off
The Second-Order Risk
Many vendors sit on the same few underlying models. One pause can reach several of your vendors at the same moment, while your third-party register shows them as unrelated.
A vendor’s kill switch protects the vendor. Your contract has to write yours.
Does your vendor contract give you a pause right, or only an exit right? Pull one AI-enabled vendor contract this week and mark which of the six items above it covers.