Blog

Notes From Production

What we learn building, deploying, and operating AI Agents for real workloads — written up while the detail is still fresh.

Agent Architecture12 min read

Hermes Agent Architecture: One Core, Many Surfaces

One agent core serves a CLI, a messaging gateway, a terminal UI, and a desktop app. How the five layers fit, what a single turn does, and where new capability belongs.

A visual map of the Hermes agent: five layers from surfaces to state, the guarded turn loop, and the footprint ladder that keeps the core narrow.

Reliability Engineering6 min read

Docker Compose Egress and the Stale Legacy iptables Ruleset

Your Compose stack is healthy, your host reaches the internet, and your containers have none. A second firewall ruleset was dropping everything before NAT — here is how to find it.

Docker's nftables rules looked perfect while a stale iptables-legacy ruleset silently dropped the compose network's outbound traffic. Counters, the two-ruleset trap, and the three-command fix.

Reliability Engineering7 min read

Silently Healthy: When Agent Infrastructure Reports Health and Does Nothing

A bot that logs 'listening' while unauthenticated. A token rejected with a swallowed 404. A network that looks fine from the host. Three failures, one class: components that look healthy while doing nothing.

Up and working are different claims. Three production failures where the process was healthy, the logs were clean, and nothing was happening — and the one habit that catches all of them.

Reliability Engineering10 min read

Durable Agent Tasks: Making Hermes More Powerful with Restate

A request-scoped agent dies when its process does. Wrap Hermes-driven workflows in durable execution and they survive crashes, waits, approvals, and schedules.

A request-scoped agent dies when its process does. Journaling every step makes LLM calls, MCP tool runs, and human-approval waits resumable.

Discovery Engineering4 min read

SearXNG: Independent Meta‑Search for Agent‑Ready Discovery

Giving AI agents a broader, independently aggregated view of the web — without permanently binding discovery to a single provider.

Meta‑search cuts dependence on any single ranking engine. Setup, independent verification, and an architecture separating discovery, retrieval, and reasoning.

Reliability Engineering9 min read

Hermes Provider Fallbacks

Designing a failover chain that actually survives a provider outage — a practical guide for teams running production AI agents.

One failed LLM provider shouldn't take your AI agent offline. Here's how to build a fallback chain with genuinely different failure domains.