“It can’t mutate prod” isn’t the same as “it can’t hurt us.”
That’s the line I want platform teams to print out and tape somewhere near the Kubernetes dashboard before the next wave of agent demos arrives with a cheerful chat box and suspiciously broad permissions.
CNCF recently promoted a walkthrough for running a Kubernetes-aware AI agent inside a cluster. A small LinkedIn post. Easy to ignore. Except the implication is much bigger than the engagement number.
The walkthrough is CNCF’s Building a Cluster-Aware AI Agent with Kubernetes, Argo CD, and GitOps. It describes a self-hosted agent running inside Kubernetes, using Ollama and Mistral 7B locally, with a FastAPI layer, GitOps deployment, and read access to live cluster state.
This is interesting work. It’s also exactly where the boring security questions start mattering again.
Boring is good (more on that here). Boring keeps prod alive.
The useful version of this idea
Let me steelman the benign case first, because otherwise this becomes another “AI bad, humans good, return to parchment” rant. I have no interest in that.
A read-only, cluster-aware agent can be genuinely useful. Kubernetes is a wonderful system if your hobby is reading YAML while squinting. For everyone else, it’s a layered cake of pods, deployments, replica sets, events, services, logs, controllers, and little gremlins called CrashLoopBackOff.
An agent that can answer, “Why is this pod failing?”, “Which workloads are unhealthy?”, or “What changed in this namespace?” could reduce toil. It could help new engineers navigate a cluster. It could make observability more conversational. It could give tired platform engineers fewer Slack messages that begin with “quick question…” (the most dangerous phrase in engineering).
The CNCF walkthrough is also careful in one important way: It uses a dedicated Kubernetes identity and read-only RBAC. The ClusterRole shown for the agent, ai-devops-api-reader, grants get and list on resources like pods, pods/log, events, services, configmaps, and namespaces, plus apps resources such as deployments, replicasets, statefulsets, and daemonsets. No create. No update. No delete.
Good.
But still not harmless.
Read-only is still access
Here is the analogy I keep coming back to: Mutation rights are the keyring. Read access is CCTV across the building.
The keyring lets you open doors and move things around. The CCTV tells you who is in the building, where the valuables are, which doors are usually propped open, when the guards change shift, and which team keeps writing passwords on Post-it notes.
Different beast. But still a beast.
A Kubernetes-aware agent with read access can see a lot. Pod specs can include container image names and tags, commands, args, service account names, volume mounts, labels, annotations, node placement, and environment variables. If an environment variable is written as a literal value, a pod reader can see it in the pod spec. If it’s referenced via secretKeyRef, the pod spec shows the Secret name and key reference, not the Secret value – the actual Secret data requires Secret permissions. Kubernetes’ own guidance is blunt that Secrets access should be tightly controlled, and that granting list access to Secrets effectively grants access to their contents.
That distinction matters. I’m not saying “pod read equals secret dump.” That would be sloppy, and sloppy security commentary is how we get conference talks with too many skull emojis.
I am saying pod read can still reveal operationally sensitive information:
- Which Secret exists
- Which workload uses it
- What image is deployed
- What namespace it runs in
- Which node it landed on
- What labels encode ownership or tenant identity
- What logs say when something breaks
Logs deserve their own little circle of hell. Kubernetes controls pod logs through the pods/log subresource; in RBAC you grant it as pods/log, separately from pods (the RBAC examples show this exact resource/subresource pattern). If your agent can read logs, your agent can read whatever your applications accidentally print. Tokens, emails, request payloads, internal URLs, customer identifiers, stack traces with config values… The usual parade of “we definitely don’t log that” things that we definitely sometimes log.
There are real examples of sensitive data in logs. The secrets-store CSI driver had a vulnerability where affected versions disclosed service account tokens in logs. KubeOne’s own PII analysis notes that Kubernetes and system logs can contain user names, email addresses, IPs, hostnames, and other contextual data (including custom metadata such as owner labels).
None of this requires the agent to mutate a single object.
It only has to observe.
Kubernetes already has the primitives. Use them.
The mildly absurd part is that Kubernetes has a perfectly serviceable model for this already. We don’t need to invent a fresh acronym with a gradient logo. We need to use the primitives we already have.
A ServiceAccount provides a distinct identity for pods. RBAC has Roles, ClusterRoles, RoleBindings, and ClusterRoleBindings. A Role is namespaced; a ClusterRole is cluster-scoped. A RoleBinding grants permissions within a namespace, while a ClusterRoleBinding grants them cluster-wide.
That means the question is not “Is this agent read-only?”
The better questions are:
- Which ServiceAccount does it run as?
- Is that ServiceAccount dedicated to the agent?
- Is access namespace-scoped, or did we slap a ClusterRoleBinding on it because demos are easier that way?
- Does it need list, or only get?
- Does it need watch, which creates a continuous feed of state changes?
- Does it need pods/log, or just pod status?
- Does it need ConfigMaps, namespaces, services, events, nodes?
- Can it see across tenants?
- Where does the context go after the agent reads it?
That last one is the new bit. The CNCF walkthrough uses a local model in-cluster and says no cloud AI provider is involved. Excellent. But many agent setups use hosted model APIs. Some Kubernetes agent examples use external LLM APIs and environment-provided API keys. If cluster context is being sent to an external provider, you have a data-handling question, not a vibes question. (“Vibes question” isn’t an official security category, but give it six months and someone will surely put it in a maturity model.)
The checklist I would actually use
If a platform team asked me how to deploy one of these without being silly, I would start here:
- Give the agent its own ServiceAccount. Never the default one.
- Prefer namespace-scoped Roles. Use ClusterRoles only when the use case truly needs cluster-wide visibility.
- Avoid cluster-admin, wildcards, and broad ClusterRoleBindings. OWASP’s Kubernetes Top 10 calls out overly permissive RBAC for a reason.
- Split permissions by task. A pod-status agent doesn’t automatically need ConfigMaps, nodes, or every namespace.
- Treat pods/log as sensitive. Logs are not “just observability.” They are often accidental data exhaust.
- Be careful with list and watch. get retrieves a named object; list inventories a collection; watch turns that into a stream.
- Do not grant Secrets access unless the agent’s job specifically requires it. And then ask again, slower.
- Audit the ServiceAccount. Kubernetes audit logs can show the requesting user, verb, resource, namespace, and response status depending on policy; examples include service account identities like system:serviceaccount:<namespace>:<name> in audit output (SigNoz has a useful walkthrough).
- Put egress controls around the agent. If it can read sensitive state and phone home, you’ve built a very polite exfiltration path.
- Have revocation ready. At 2am, you should know which binding to remove and which token to rotate.
This is just least privilege. The NSA/CISA Kubernetes hardening guidance says privileges for users, groups, and service accounts should follow the principle of least privilege. Kubernetes multi-tenancy guidance also ties namespace isolation and RBAC back to least privilege between tenants.
Very unsexy. Very necessary.
This is API governance
I run an API management company, so yes, I see APIs everywhere. Occupational hazard. But Kubernetes API access is literally API access. Agents don’t magically escape the old rules because they answer in complete sentences.
Identity. Scopes. Policy. Audit. Rate limits where relevant. Ownership. Lifecycle. Revocation.
That is agent governance once you scrape off the glitter.
The industry is currently very excited about whether agents can take action: Deploy, roll back, resize, patch, approve, merge. Fair enough. Mutation is scary. The keyring matters.
But the first governance failure may be quieter. A helpful read-only agent gets deployed with broad cluster visibility. It summarizes logs into a hosted model. It sees tenant names in namespaces, customer hints in labels, internal architecture in service names, image tags that reveal rollout cadence, events that reveal failures, and ConfigMaps someone used as a discount Secret store.
Nothing was mutated. Plenty was learned.
Caveat: Read-only Kubernetes access doesn’t automatically leak secrets. The risk depends on RBAC scope, cluster hygiene, logging practices, tenancy model, and where the agent sends context. A narrow, local, namespace-scoped pod-status helper is a very different animal from a cluster-wide agent with logs, ConfigMaps, and external model calls.
That is precisely the point: “Read-only” reduces one class of risk, but it preserves another.
So, before deploying the cheerful cluster bot, ask the boring questions: What could this agent learn, where could it send that information, and who can revoke it at 2am?
Read-only is a permission, not a blessing.
