Shadow Agents: Half Your AI Fleet Runs Unmonitored and Nobody Owns It

In December 2025, the analyst firm Opinion Matters surveyed 750 IT executives and practitioners across the US and UK on behalf of Gravitee. The number that traveled fastest was the alarming one: a mean of 53% of corporate AI agents were not actively monitored or secured, which the study extrapolated to roughly 1.5 million agents "at risk of going rogue" out of more than three million deployed (CSO Online). The same survey reported that 88% of firms had experienced or suspected an AI agent-related security or data privacy incident in the past twelve months.

Those headlines got filed under security. They belong under a different heading. The survey measures monitoring and security posture, not instrumentation directly, but the most likely root cause of an agent nobody is watching is an agent nobody wired a trace around. You cannot secure, govern, debug, or cost-control a system that emits no signal, and half the fleet shipped without one. The word "rogue" implies the agents are doing something; the more accurate reading is that they are doing it in the dark.

The shadow-agent problem

Shadow IT was the SaaS-era pattern: a team expensed a tool, wired it into a workflow, and IT found out during the breach postmortem. Shadow agents are the agentic version of the same habit, except the thing running unsupervised is autonomous. It calls tools, spends tokens on every loop, and produces a different execution path each time you run it.

The deployment math explains the gap, and it deserves a closer look than the headline gives it. Gravitee's three-million figure came from a mean of 36.9 agents per business applied to government counts of larger firms. Salesforce's 2026 Connectivity Benchmark, a Vanson Bourne survey of 1,050 IT leaders, put the average far lower, at 12 agents per enterprise today, rising to roughly 20 by 2027 (Digital Commerce 360).

That is nearly a 3x spread, and the sampling frames explain most of it: Gravitee polled 750 executives and extrapolated against government tallies of large businesses, while Salesforce surveyed IT leaders directly. Treat the absolute counts as soft. The trend underneath both is the part that holds: agents arrived faster than the instrumentation, and the count is climbing.

The same Salesforce benchmark found that only 54% of organizations have a centralized governance framework for agentic capabilities, that 27% of the APIs connecting agents lack governance oversight, and that 50% of agents run in isolated silos rather than as part of a coordinated system (Kiteworks).

Read those three numbers together and a governance story turns into a telemetry story. The APIs have no audit trail. The silos do not emit a shared trace. Nobody can name the owner because there is no dashboard with a name on it.

An observability failure, not a policy failure

Governance frameworks assume you can answer basic questions on demand: which agents are live, what model each one calls, what tools it can invoke, how many tokens it burned last night and against whose budget. The security press prescribes policy for agents that, in roughly half of cases, emit nothing for a policy to attach to.

The reason half your fleet is ungoverned is rarely that someone skipped the policy meeting. It is that the agents were never instrumented, so there is nothing for a policy to bind to.

A security framing sends you to access controls and incident response. An observability framing sends you to traces, token logs, and evals, which are the foundation every later control sits on top of. You cannot alert on a token budget you are not measuring, and you cannot quarantine an agent you cannot identify. Build the telemetry first; the policy layer has nothing to grab onto without it.

What a trace actually captures

The vendor-neutral answer arrived in 2026 with the OpenTelemetry GenAI semantic conventions, which standardized what an agent run should emit. The span hierarchy mirrors how an agent executes: a top-level invoke_agent span wraps the run, chat spans wrap each LLM call, and execute_tool spans wrap each tool invocation (OpenTelemetry).

Picture a single support agent answering one ticket. The invoke_agent span opens at the top. Underneath it, a first chat span shows the model deciding to look something up. An execute_tool span records a vector search against the knowledge base. A second chat span shows the model reading those results and deciding to escalate. A second execute_tool span fires a ticketing-system API call. One trace, one ticket, the full causal tree underneath it, time-ordered and nested.

Every span captures the model name, token counts, and operation duration by default. When you turn it on, a span also captures full prompt messages, system instructions, tool schemas, tool arguments, and tool results. The defined attribute names are the part almost nobody in the governance camp cites, and they are exactly what makes the output auditable instead of proprietary:

gen_ai.request.model
gen_ai.usage.input_tokens
gen_ai.usage.output_tokens
gen_ai.response.finish_reasons
gen_ai.system_instructions
gen_ai.input.messages
gen_ai.output.messages

If your agents emit those attributes, any OTel-native backend can ingest them, and you are not married to a single vendor's schema. Framework choice matters here. Langchain and its agent runtimes natively emit GenAI-compliant spans, so for teams already on that stack the instrumentation is closer to a configuration switch than a rewrite. The hard part is not the framework. It is deciding to flip the switch on for the agents already in production, where nobody wants to touch a working system.

Why prompt and completion tokens have to be split

The most useful thing those spans give you is honest cost accounting, and the discipline that makes it work is splitting prompt tokens from completion tokens before you roll anything up. gen_ai.usage.input_tokens and gen_ai.usage.output_tokens are separate attributes for a reason. Input and output price differently on nearly every model, and an agent that re-stuffs a giant context window on every loop has a cost profile that a blended number hides completely.

The recommended pattern is concrete. Attach token usage to every call, link it to its trace, split prompt from completion, then build per-user, per-team, and per-project rollups on top. Set alerts at 50%, 80%, and 100% of budget, because one runaway prompt can burn thousands of dollars overnight before a human notices (TokenMix). The 50% alert is the one that actually saves money. By the time the 100% alert fires, the spend already happened.

The silo finding bites hardest here. If half your agents run in isolation with no shared trace, you do not have one cost dashboard with a runaway-prompt alarm. You have a few dozen invoices arriving at the end of the month with no way to attribute a spike to the agent that caused it.

Two retrofit paths exist, and they trade off against each other. A proxy like Helicone sits in front of an OpenAI-compatible API and logs tokens and cost automatically, with no change to the agent's code, which is the fastest way to put accounting on a running shadow agent. The cost of that convenience is that a proxy sees the wire traffic but not the agent's internal reasoning steps, so it is weaker at reconstructing the full tool-call tree. SDK-level instrumentation captures the deeper span hierarchy but requires touching each agent. The proxy is the quick win on cost; the SDK is the durable answer on debugging. Most fleets need both, in that order.

Why non-determinism breaks old debugging

Traditional debugging assumes the same input produces the same output, so you reproduce the bug and step through it. Agents do not honor that contract. The same input can trigger a different tool sequence, retrieve different documents, and generate a different response on each run, which is precisely why input-output debugging fails and end-to-end tracing of the agent-LLM-tool path becomes mandatory (OpenTelemetry).

Consider a returns-processing agent that approved a refund it should have flagged. The output log says "refund approved" and nothing more. The trace says something useful instead: the chat span shows the model received a customer message, the execute_tool span shows it queried the orders API with the wrong order ID, a second tool span shows it pulled a different customer's purchase history, and the final chat span shows it reasoned correctly over the wrong data. The bug is not in the prompt. It is a tool argument two steps upstream. Without the tool spans and the optional message capture, you are reconstructing that path from a single log line and a guess.

Arize Phoenix is built around this use case. It pairs open-source agent tracing with evaluation and embedding-drift monitoring, so the path that produced a bad answer stays visible and you can measure whether retrieval quality is degrading underneath it. For shadow agents that fail intermittently, the drift view is often the first thing that explains a regression nobody changed any code to cause: the prompts held steady, the model held steady, and the embedding space the retrieval depends on quietly moved.

Closing the loop with evals and prompt versioning

Tracing tells you what happened. It does not tell you whether what happened was good. That is the job of evals, the part of observability that greenfield engineering posts treat as optional and governance posts ignore entirely.

Practitioners describe four pillars of LLM observability: prompt-level tracing with correlation IDs across multi-step workflows, performance and token metrics, quality and guardrails covering bias and toxicity and hallucination and embedding drift, and cost attribution with budget enforcement (TrueFoundry). Evals live in the quality pillar. Prompt versioning is what makes the loop close: change a prompt, run the eval suite against the trace history, and see whether quality regressed before the change reaches the fleet.

Braintrust is organized around exactly this work, built for regression testing, scoring, and prompt iteration rather than live dashboards. For a security team the shadow-agent question is whether an agent is dangerous. For the engineers who maintain it, the question is whether last Tuesday's prompt edit quietly made it worse. Both questions read the same trace history, and an eval-first tool is what turns that history into a pass/fail signal an engineer can act on before shipping.

Mapping the tooling to the need

The category is crowded enough that "pick an observability tool" is not actionable advice. The useful move is to match a tool to the specific pillar your shadow fleet is weakest on. A team drowning in untracked cost has a different first step than a team that cannot debug a non-deterministic failure.

Tool	Strongest pillar	Best first move when
Langfuse	Tracing + evals + prompt versioning, self-hostable	You want OTel-native instrumentation with no license gate
LangSmith	Trace/eval backend for LangChain agents	Your agents already run on LangChain or LangGraph
Arize Phoenix	Agent tracing + drift and quality monitoring	Intermittent failures and silent retrieval regressions
Helicone	Proxy-based token and cost capture	You need accounting on running agents without code changes
Braintrust	Eval-first regression testing and scoring	Prompt edits keep degrading quality undetected

Langfuse earns the top row because it is the closest thing to a default answer for instrumenting a shadow fleet without a procurement fight. ClickHouse acquired it on January 16, 2026 and kept the core 100% open-source under its existing MIT license, which means production-scale self-hosting stays on the table (ClickHouse). The MIT commitment is the load-bearing detail for the silo'd-agent retrofit case, because it de-risks self-hosting at scale without a per-seat license negotiation for every team that owns a stray agent.

For teams already standardized on LangChain, LangSmith is the path of least resistance, because the framework's spans flow straight into it without a separate instrumentation step. These tools differ on storage model, eval depth, and pricing, but they share the trait that matters: each ingests OTel-compatible spans. That makes the schema decision more durable than the vendor decision. Standardize on the conventions and you can swap backends later without re-instrumenting the fleet.

The retrofit tax

Here is the timing argument, and it is the whole reason to act this quarter instead of next year. Salesforce's benchmark puts the average enterprise at roughly 12 agents now and heading toward 20 by 2027. Instrumenting 12 agents is a project. Retrofitting tracing, token splitting, and evals onto 20 sprawling, half-undocumented agents, with half of them already in silos and a quarter of their connecting APIs carrying no audit trail, is an excavation.

The cost is not linear in agent count. Each uninstrumented agent that ships adds a dependency on an undocumented prompt, a tool wired in by someone who has since changed teams, and a spend pattern nobody is watching. Standardize on the OTel GenAI conventions now and every new agent inherits the trace schema for free. Wait, and every new agent becomes one more thing to reverse-engineer later, usually during an incident, usually at the worst possible time.

Pick the owner, then the backend

The Gravitee study points at an org-chart problem the policy frameworks never name. There is a population of agents at risk and no role accountable for whether any given one emits a trace. Governance frameworks fail not because the policy is wrong but because there is no instrumented surface to enforce them against and no name attached to the dashboard.

So the next move is two decisions made in one meeting, before the agent count climbs. Name one owner for agent observability, a person rather than a committee, who is accountable for whether every production agent emits invoke_agent, chat, and execute_tool spans with split token counts. Then pick one OTel-native trace backend and require new agents to ship into it on day one.

Wire the 50% budget alert that same week, because it is the single cheapest control to stand up and the one most likely to catch a runaway loop early. In week two, the owner's job is concrete: audit the existing fleet against a single checklist. Does each agent emit the three span types? Are its tokens split prompt-from-completion and attributed to a team? Does a prompt change trigger an eval run before it reaches production? Every "no" on that list is a shadow agent with a name now attached to fixing it, which is the difference between a governance slide and a governed fleet.