fix(hermes): host-side chown PVC backing dirs after sync (#475)

[bussyjd]

Summary

Closes #475.

The in-pod init-hermes-perms chown added in #446 (c066baa) silently no-ops on Linux k3d because the embedded k3d config sets KubeletInUserNamespace=true (internal/embed/k3d-config.yaml:31-37). With user-namespacing the pod's "root" maps to a host subuid that lacks chown authority over the host bind-mount path; the chown succeeds inside the namespace but has no effect on host-side ownership. The next init container, init-hermes-data, runs as UID 10000 and fails with mkdir /data/.hermes/home: Permission denied against a directory that local-path-provisioner created as 1000:1000.

This PR adds ensureHermesPVCOwnership, called from hermes.Sync after helmfile sync. It chowns the PVC backing dirs from outside the user namespace via docker exec into the k3d server container — running at the Docker daemon's real root, not the namespaced kubelet's. Reuses the existing fixRuntimeVolumeOwnership helper (already used for wallet keystores) plus a small kubectl wait for PVC binding and a conditional restart for stuck pods.

Why not just patch local-path-provisioner?

We considered changing internal/embed/infrastructure/base/templates/local-path.yaml's setup script to chown to 10000:10000 instead of 1000:1000. That breaks OpenClaw, which runs as 1000:1000 (internal/openclaw/openclaw.go:801 already chowns to that). Two workloads with different runtime UIDs share the same StorageClass, so the chown has to live in each workload's deploy path, not in the provisioner.

Behavior

• First sync after fresh bootstrap: PVCs go from "Pending → Bound" during helmfile sync; helper-pod sets ownership to 1000:1000; our chown immediately overwrites to 10000:10000; pod's init containers run cleanly on first try.
• Subsequent syncs (e.g. obol model sync after obol model prefer): chown is idempotent. The conditional restart only fires if a pod is actually in Init:CrashLoopBackOff or Error — obol model sync against a healthy stack does not gratuitously restart the agent.
• Stuck-pod recovery: if a fresh sync races with kubelet (helper-pod runs, init-perms no-ops, init-data fails before our chown lands), we detect Init:CrashLoopBackOff via kubectl get pods … jsonpath=…state.waiting.reason and force a single pod delete. Kubelet recreates immediately rather than waiting up to ~5 min for exponential backoff.

Live validation on spark2 (Linux ARM64, Ubuntu 24.04, NVIDIA GB10)

$ sudo chown -R 1000:1000 .workspace/data/hermes-obol-agent
$ ls -la .workspace/data/hermes-obol-agent
drwxr-xr-x 3 claude claude  ...  hermes-data
drwxrwx--- 2 claude claude  ...  remote-signer-keystores

$ obol kubectl -n hermes-obol-agent delete pod -l app.kubernetes.io/name=hermes
$ obol model sync
==> Reading model list from LiteLLM...
==> Syncing Hermes: hermes/obol-agent
  ✓ Running helmfile sync (2s)
  ✓ Hermes installed successfully!

$ ls -la .workspace/data/hermes-obol-agent
drwxr-xr-x 3  10000  10000 ...  hermes-data
drwxrwx--- 2  10000  10000 ...  remote-signer-keystores

$ obol kubectl -n hermes-obol-agent get pods
NAME                            READY   STATUS    RESTARTS   AGE
hermes-69f56c9f75-9dwh8         2/2     Running   0          28s

Before this PR the exact same teardown left hermes-* in Init:CrashLoopBackOff for ~16 minutes until manual sudo chown intervention.

Test plan

• go test ./internal/hermes -count=1 — all existing tests pass, plus the new TestHermesPVCPaths pins the two host paths the helper chowns so a future rename of the PVCs or the namespace layout can't silently regress the fix.
• go build ./... clean.
• End-to-end on spark2 — see "Live validation" above. Pod reaches Running 2/2 in 28s with zero CrashLoopBackOff cycles, where the prior behavior was a 5-15 min stuck state.

Notes

• Issue hermes init PVC perm-denied regression survives #446 (k3d local-path on Linux, both fresh and dev-mode) #475 also raised the option of wiring fsGroupChangePolicy: OnRootMismatch onto the pod spec. We did not take that path because fsGroup-style chowns are themselves performed by kubelet, which on this stack runs with KubeletInUserNamespace=true — they're affected by the same namespacing the in-pod chown is. docker exec is the only path that bypasses it cleanly.
• obol model sync is now a few-hundred-ms slower because of the kubectl wait calls (timeout 60s but typically finish in ~1s since the PVC is already Bound on subsequent syncs). If this matters to anyone, we can short-circuit the wait when both PVCs are already Bound. Left as a follow-up.

Generated with Claude Code