All posts
Reliability Engineering

Silently Healthy: When Agent Infrastructure Reports Health and Does Nothing

A bot that logs 'listening' while unauthenticated. A token rejected with a swallowed 404. A network that looks fine from the host. Three failures, one class: components that look healthy while doing nothing.

Tested against C2C Telegram bot · Docker Compose · Claude CLI 2.1.251 · August 2026

The failure class: components that report health and do nothing

Every long-running agent stack eventually produces the same failure class: a component that logs “I’m fine” while silently doing nothing that matters. The process is up. The health endpoint returns 200. The logs are quiet. And the work is not happening.

While wiring up a Telegram demo for our agent stack, three different failures hit us in a single afternoon, and all three wore the same disguise: a component that looked healthy, was trusted on that basis, and was doing nothing.[1]

ComponentLooked likeWas actually
The bot that logged “listening” and never repliedContainer Up, logs say “C2C is listening on Telegram”, every poll appears to succeed — and no passenger message ever gets an answer.The bot service was missing the ~/.claude OAuth mount the api service had. Every intake model call ran unauthenticated, failed after five retries, and was swallowed as “poll failed, retrying” — a line that looks like transient noise.
The token that Telegram rejected with a swallowed 404Bot authenticates to start, polls happily, and receives nothing. Logs are clean.The bot token in .env had extra characters pasted onto it. Telegram returns HTTP 404 for a malformed token — and getUpdates parses the 404 body as “no updates”, so the loop never raises. The bot is healthy and deaf at the same time.
The network that looked fine from the hostEvery container Up and healthy, host egress fast, docker0 egress fast — and the compose network silently unable to reach anything.A stale legacy iptables ruleset with FORWARD policy DROP coexisted with Docker’s correct nftables rules and only knew about docker0. Traffic was accepted, then dropped a millisecond later by the other ruleset.

Case 1: the bot that logged “listening”

Our Telegram bot runs in a container. It starts, prints C2C is listening on Telegram, and begins long-polling getUpdates. It is the system's public face — a passenger describes a disruption, the bot answers, a case is opened.

The intake step that understands a passenger's message calls the model. In the container, that call shells out to claude -p — which needs the OAuth login the api container had mounted in, but the bot container did not. Unauthenticated, the call failed five times, and the failure was caught by the poll loop's blanket except and printed as poll failed, retrying — a line you read as transient network noise and move past.

# the one-line fix: give the bot the same login the api service has
volumes:
  - ${HOME}/.claude:/root/.claude
  - ${HOME}/.claude.json:/root/.claude.json

The interesting part is not the missing mount. It is that the bot had no way to be wrong loudly: the failure was caught, retried, and reduced to a log line indistinguishable from a slow network. The system kept every promise its process could make — and delivered nothing to the passenger.

Case 2: the swallowed 404

Then the token broke. A token pasted into .env had extra characters on the end — the sort of edit that happens at 11pm before a demo. Telegram is strict: any token that is not byte-exact gets 404 Not Found on every call.

And here is the trap: getUpdates does not raise on that 404. It parses the response, sees ok: false, and returns an empty update list. The poll loop sees an empty list and keeps polling forever. The bot authenticates at startup, polls happily, and is permanently deaf — with clean logs. Only getMe, which asks Telegram to identify the token, exposed it.

Case 3: the network that lied from the host

The third failure we have written up separately, because it deserves the space: a stale legacy iptables ruleset dropped the compose network's outbound traffic while Docker's own rules and the host's routing both looked perfect. The container could talk to its siblings and the host had full egress — only the path that mattered, container to the internet, was dead.

That one is the purest example of the pattern: every standard diagnostic returned healthy, and the system was doing nothing.[2]

The fix: check the remote end, not the local report

All three failures share one remedy, and it is a habit rather than a tool: ask the remote end what it sees. Self-reported state — a process that started, a log line, a health endpoint the component writes itself — tells you the component is alive. It says nothing about whether the work is happening.

  • For the token: getMe. Telegram itself says whether the token is valid. getUpdates swallows the answer.
  • For the network: a curl from inside the container to the actual endpoint. It tests the whole path, not the config.
  • For the model: the PONG probe — a one-line model call that must return before you trust anything downstream of the model.

A pre-demo checklist that would have caught all three

StepCommandWhat it proves
Token validcurl -s https://api.telegram.org/bot$TOKEN/getMeTelegram accepts the token (200, not 404)
Container egressdocker compose exec api curl -sS https://api.anthropic.comThe whole network path works from where the code runs
Model reachableclaude -p "Reply with exactly: PONG"The model answers before anything else is trusted

Sources

  1. getUpdates — Telegram Bot API
  2. Docker Compose egress and the stale legacy iptables ruleset

Frequently Asked Questions

Why does getUpdates swallow a 404 from a bad bot token?

getUpdates parses the response and returns an empty update list when ok is false, so a malformed token makes the bot poll happily while receiving nothing — with clean logs. getMe, which asks Telegram to identify the token, is the check that exposes it.

What is the difference between 'up' and 'working'?

Up means the process started and its own health signals look fine. Working means the delivery path — token, network, model, approvals — is functioning end to end. The first is reported by the component; only the second is verified by asking the remote end.

What three checks catch these silent failures before a demo?

getMe to prove the token, a curl from inside the container to prove egress, and a one-line model PONG probe to prove the model is reachable from where the code runs.