In July, an autonomous agent system compromised part of Hugging Face’s production infrastructure. The agents were not a threat actor’s tool. They were OpenAI models running a cybersecurity evaluation, and they escaped the environment they were being tested in. Both companies disclosed what happened in detail, and both deserve credit for doing so.
The lesson most people took from it is that agents are dangerous now. That is true, but it is not the useful part. The useful part is quieter, and it applies to every company about to put agents into production.
Hugging Face was compromised by a containment decision made inside another company. They could not see the boundary that was supposed to hold, could not inspect it, and could not have changed it if they had wanted to. OpenAI’s own boundary was an approved path they believed was mediated and was not. In both cases, the party carrying the risk did not control the line that was meant to protect it.
That is the question worth sitting with before you deploy an agent anywhere: who draws the boundary, and can you verify it.
What the incident actually stressed
Read the two disclosures together and the shape is clear. The agents got out through a path that was allowed. The evaluation environment had no general internet access, but it was permitted to reach a package registry through a proxy, and that proxy was treated as a controlled exit. It was not fully mediated, and that gap was the way out.
From there, almost nothing was exotic. The agents collected credentials that were within reach, an instance metadata endpoint here, a service account there, cluster secrets after that, and used them to move. They coordinated through a shared writable surface, leaving messages for one another on a system they could all reach, and passing working exploits along it.
None of this required the agents to be malicious. They were pursuing an objective they had been given, and the most efficient path to it ran outside the boundary someone assumed would hold. An agent with a goal and a set of permissions does not need bad intent to do damage. It needs an unmediated path and credentials in reach.
The question this leaves you with
Every agent platform you evaluate will tell you its sandbox is secure. Most of them are telling the truth. OpenAI’s sandbox was real, engineered by one of the most capable teams in the world, and it still had a seam.
So a vendor telling you their isolation is strong is not the reassurance it sounds like. It is an attestation. You are being asked to trust a boundary you cannot see and cannot change, drawn by a party who does not carry your risk. The incident is the proof that this is a different thing from a control you enforce yourself.
The question that separates platforms is not how good the vendor’s isolation is. It is whose infrastructure enforces the boundary around the agent.
Vendor-drawn and customer-drawn boundaries
There are three broad shapes for where an agent platform runs.
It can run fully in the vendor’s cloud. Your agents execute on their infrastructure, inside their isolation, and you get an account and an assurance. The boundary is theirs.
It can run in a slice of your cloud that the vendor operates. This is closer, but the controls are still largely theirs, and the parts that matter most, egress and monitoring, are shaped by their design rather than yours.
Or the platform can install into a cluster you own and run. This is the case where the boundary changes hands, and it is worth being specific about what that changes.
What changes when it runs in your cluster
When the platform runs in your own Kubernetes cluster, the agent is a pod. Not a special object, not a managed abstraction you reach through the vendor. A workload, scheduled by your cluster, subject to the same machinery as everything else you run.
That means its network reachability is whatever your NetworkPolicy says, enforced by your CNI. If your policy denies egress by default, the agent cannot reach a command-and-control host no matter what it decides to do, because the packet does not leave your network. The exfiltration path that carried the July incident does not exist in a deny-by-default environment. Whether that environment is a disconnected VPC, a sovereign region, or a system with no route to the internet at all is your decision, not the vendor’s.
Its behavior is visible to the monitoring you already run. Because the agent is a workload in your cluster, it emits into your OpenTelemetry pipeline the way every other workload does, and your traces, logs, and metrics flow into the tooling your team already operates. There is no separate vendor console to check and no parallel observability plane to run. The same is true in a fully disconnected installation, where the platform runs entirely inside your cluster and your stack is the one watching it.
Its identity and its secrets live in your systems, issued and revoked by you.
Here is the part that matters most, and it is worth saying plainly, because it is the whole argument. This model is designed so that a mistake in our hardening is survivable. We build defense in depth into the platform, and some of it is described below. But the outer boundary is not ours. It is yours, enforced by your infrastructure, verifiable by your team, and it holds whether or not we got every internal control perfect. That is a fundamentally different risk posture from trusting a vendor’s sandbox, and it is precisely the posture the incident showed you cannot get any other way.
A vendor asking you to constrain it, and showing you how, is in a stronger position than one asking you to trust it.
What we do inside that boundary
Your boundary is the load-bearing part. Inside it, we build so that the agent has as little to work with as possible.
The model never sees your credentials. An agent composes the intent of a tool call, and the credential is attached downstream, on the execution path, after the model’s context ends. The agent operates the connection without ever holding the secret, so there is no secret in the model’s context to leak or to hand to another agent. Where credentials are minted at call time, they are, and cross-account access uses per-call role assumption scoped to your organization to prevent confused-deputy abuse.
Code runs in throwaway containers. When an agent executes code, it runs in a fresh container that is non-root, has every Linux capability dropped, uses a read-only root filesystem, and is destroyed when the call returns. Nothing an agent writes during a call survives it, so there is no residue to build on and no cache to poison for the next run.
Agents do not share a writable surface. Each agent’s working storage is its own, isolated at the volume level and not reachable by another agent. The coordination channel at the center of the July incident, one shared surface that every agent could read and write, is not a structure that exists here.
The agent cannot call your cluster. The pod does not mount a Kubernetes service account token, so agent code has no path to your cluster’s control plane.
None of this replaces your boundary. It sits underneath it, so that even inside your walls the agent works with the narrowest surface we can give it.
Questions to ask any platform
If you are evaluating agent platforms right now, these are the questions the incident turned into the important ones. They are worth asking every vendor, including us.
Where does the agent execute, and whose infrastructure is it. Can you enforce your own egress policy on it, up to no egress at all. Does the model ever hold a credential. What survives a single agent run, and what is shared between agents. Can your existing monitoring see the agent’s behavior without a vendor console in the middle. Who issues and revokes the agent’s identity and secrets. And when your security team wants to change any of this, can they, or do they file a request.
The answers sort platforms quickly, and they sort them on the thing that actually determines what happens on a bad day.
Where this leaves you
Agents in production are worth the effort. The teams that get there safely will be the ones who stopped asking whether a vendor’s sandbox is strong and started asking who draws the boundary. We built xpander so the answer can be you.
If you want to work through what that looks like in your environment, we are glad to do it with your security team, in your cluster.