Hermes Team Hub: coordinating AI agents

A year ago it would have seemed like science fiction: two AI agents — mine, Lupe and Gizmo, programs that work on their own — doing their job on my computer, and a human coordinating them like someone running a small team. The problem I ran into is that they're excellent working alone and terrible at coordinating with each other: each on its own side, with no shared memory and no way of knowing what the other is doing. Hermes Team Hub is the panel I built to solve it. The way I work with agents I explained in How I work with AI agents.

How it's built

It's a monorepo (a single repository with the whole project) with Docker Compose: FastAPI backend with PostgreSQL, React frontend with TypeScript, and a Python client that each agent runs on its machine. The central piece is a WebSocket (a channel that stays open in case the server has something to say on the spot): the agent connects, stays listening, and responds when it's mentioned, when the message is addressed to it, or when it goes to everyone. Each agent has its identifier and its token — an access key with specific permissions: connect, send messages, publish metrics — and the tokens are managed from the panel itself. Everything travels over the private Tailscale network (the one that connects my devices as if they were in the same house); no service exposed to the internet.

The anti-loop rules

The first week of testing I discovered the classic multi-agent problem: two agents greet each other, reply to each other, and after a while they do it hundreds of times per minute. Two parrots tossing the ball back and forth without rest. The solution was to write six anti-loop rules and treat them as part of the protocol, not as advice: who has to respond to what, when to be quiet, and how to prevent a conversation between two agents from becoming an infinite loop. I learned that with agents the problem isn't that they don't know how to do things: it's that if you let them, they'll do them all at once and in duplicate.

Splitting the work

The final flow is simple: I explain a task on the panel, assign it to an agent or mark it as open to any, the agents publish progress and metrics — tokens used, time — and I decide whether it passes to another pair of hands. (The log of what I do each day I keep separately, with DailyRapport, the daily log app.) The part that has surprised me most is that human coordination doesn't disappear: it changes from doing the work to deciding who does it.

What I'd do differently today

I would have started with authentication and the anti-loop rules, instead of adding them when there were already two agents inside. And I would have defined from day one how a task is closed: what its final state is, who validates it, and what happens if nobody finishes it. Infrastructure goes up fast; the conventions of coexistence don't.