fix(hetzner/server-1): make the crow agent's podman API socket survive node loss #6
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "Alpers-Ai-Agent/MyServers:agent/crow-agent-socket-fix"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What
Restructures the Crow local-agent podman API socket wiring with systemd-native mechanisms, after the Oct 31 outage: the agent unit wedge-looped (
start-limit-hit, podman exit 125statfs /run/podman-crow-ci.sock: no such file or directory) because the socket's filesystem node was unlinked while the socket unit stayed active — an orphaned listener that every probe saw as healthy and nothing could connect to.Root cause in one line
A bind-mounted socket file pins the inode: the node vanished (unlinked by an unidentified process during the Oct 31 deploy window — unit files, tmpfiles, and the socket/service timestamps rule out config and systemd itself), the listener survived invisibly, and the agent's next
podman runwas the only thing on the box that ever re-validates the path.The fix — three native mechanisms, no script logic
ListenStream=/run/podman-ci/podman.sock(dir created via a declarativesystemd.tmpfiles.settingsrule, 0750 root:crow-podman); the agent mounts/run/podman-ci:/run/podmaninstead of pinning the socket file. Node unlink + recreate becomes transparent to the container; the in-container path stays/run/podman/podman.sock, so the agent's socket auto-detection is untouched.PartOf/Requires/After.podman-ci.serviceisPartOf=podman-ci.socket(a socket restart mints a fresh listener, so the service must restart to serve it), and the agent unit isRequires+After=podman-ci.socket,PartOf=podman-ci.service— restarts propagate socket → service → agent instead of leaving a container holding a stale mount. This wiring was entirely missing before (the socket floated onsockets.targetwith no consumer relation)..pathunit + conditioned recovery oneshot — the systemd-native detector for the failure class systemd otherwise cannot see (active unit, missing node).PathChanged=/run/podman-citriggerspodman-ci-socket-recover.service, a oneshot whoseConditionPathIsSocket=!</run/podman-ci/podman.sock>makes it a silent no-op when the node exists — which also breaks the restart-fires-event-fires-restart loop, since the node its own recovery creates satisfies the condition and skips the next trigger. Recovery =reset-failedon the agent (clears start-limit) +try-restartsocket and service.Deploy behavior
No manual root commands needed: the deploy restarts the changed socket/service/agent units (after
reset-failed), which creates the new dir + node and heals the currently-failed agent; the old/run/podman-crow-ci.sockorphan listener is closed/unlinked by the socket unit restart.Validation
nix-instantiate --parse+nixfmt --checkcleanListenStream=/run/podman-ci/podman.sock), service (PartOf=podman-ci.socket), agent (Requires/After podman-ci.socket,PartOf=podman-ci.service), path unit (PathChanged=/run/podman-ci), recovery (ConditionPathIsSocket=!…, two orderedExecStarts)volumesevals to["/run/podman-ci:/run/podman"]systemd-analyze verifyclean on socket/service/path/recovery unitsView command line instructions
Checkout
From your project repository, check out a new branch and test the changes.Merge
Merge the changes and update on Forgejo.Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.