The Event Bus Had to Earn Its Restart Button

The first live event I trusted through my self-hosted operations pipeline was deliberately boring. An informational Unraid notification crossed a GraphQL WebSocket, a small Python collector, an n8n intake workflow, and a Hermes webhook. The collector logged the send, and n8n recorded a successful execution at the same time.

That proved the route. It did not prove I should let the route restart anything.

I built the collector with read-only credentials and gave it a small SQLite database. Its job was to remember what it had already seen: container health, runtime state, disk temperature bands, and array state. A snapshot by itself says what is true now. The stored previous value turns that snapshot into an event.

Observation needed memory

The container logic is intentionally uneventful:

healthy   -> unhealthy  = emit warning
unhealthy -> unhealthy  = emit nothing
unhealthy -> healthy    = emit recovery

An unhealthy container on the first observation still produces a warning, so restarting the collector does not create a blind spot. An unexpected-exit event requires a transition from running to exited, plus an autostart flag. A container that was intentionally stopped and left that way should not become a recurring emergency.

Disk temperatures follow the same pattern. Crossing the configured warning or critical threshold creates an event. Remaining above it does not create another one every polling interval. Returning to normal creates a recovery event. Array state changes work the same way.

This distinction cut noise before any alert policy had to guess which duplicates mattered. It also kept the collector narrow. It observes raw server state, normalizes a few transitions, and sends structured events. It does not decide how expensive an outage is, whether someone is using a service, or whether repair is permitted.

The n8n intake adds a second boundary. It rejects events missing their ID, type, source, server, severity, or observation time. Accepted events get a consistent shape before they reach the agent. A deterministic event ID lets the policy recognize the same incident again without relying on natural-language similarity.

I wanted this bookkeeping in ordinary code, not a prompt. Previous state and required fields are mechanical facts. A model should not spend tokens rediscovering them, and a restart decision should not depend on whether today’s wording sounds more urgent than yesterday’s.

Permission came later

The first version could only report. Once that path had real evidence behind it, I added a separate restart policy with a deny-by-default contract.

A container name appearing in an allowlist is only the first gate. The event must describe an unhealthy container or an unexpected exit. Autostart must be enabled. The event ID must be new. The same container must be outside a 30-minute cooldown. The contract permits one attempt, requires evidence before the attempt, requires a health check afterward, and always requires a notification.

Everything outside that path remains notify-and-ask. Interactive services and databases are excluded. So are control-plane components, containers without autostart, and names the policy does not recognize. A new container gets monitoring, not repair authority.

I duplicated the policy in Python tests and the n8n normalization step. Duplication usually makes me suspicious, but here the two copies guard different boundaries. n8n should reject an event that cannot authorize action before it asks Hermes to reason about it. Hermes should still recheck the contract before calling a server tool. Either layer can say no.

There is a cost: the copies can drift. The allowlist and denial reasons need comparison whenever one side changes. A better long-term shape may be one versioned policy artifact consumed by both runtimes. Until then, explicit duplication is easier to audit than an invisible assumption that one layer already checked everything.

For this post, I reran 19 focused tests covering the collector and restart policy. All 19 passed. The broader repository discovery also found three failing synchronization checks in adjacent publishing automation, so I cannot call the entire repository green. Those failures do not change the focused result, and I am keeping both results in the record.

A read-only live check also confirmed that the current automation stack was reachable and the monitored arrays were healthy. It did not prove the collector loop was actively delivering new events at that moment. A healthy server inventory and an exercised event route are different claims, and I do not want a green status from one to impersonate the other.

The restart button exists now, after the state store and the policy checks. It gets one attempt and owes me a health check afterward. Without that evidence, the event bus goes back to reporting what it saw.