ci: never prune the ci-<sha> image just built in the same run #133

Merged
fastie81 merged 3 commits from fix/prune-self-exclude into main 2026-09-06 09:39:20 +12:00
Owner

Bug: the registry-prune step sorted ci-* tags alphabetically and kept the 'newest' 5. A commit SHA starting with a digit (e.g. ci-40ff...) sorts before letter-prefixed SHAs, so the image was DELETED immediately after being pushed - before deploy-test could pull it. Symptom on the v3.4.4 run: deploy-test failed 'not found' (run 269/270 job 3) and deploy-prod skipped.

Fix: exclude the current build's own ci-${GITHUB_SHA} tag from the prune set so the image destined for deploy is never removed mid-pipeline.

Note on build-job intermittent failures in this run and the previous one: unrelated transient runner issue (two stale long-lived buildx/buildkit containers had been up 11h on the Podman runner; one is being cleaned). The workflow's podman-socket mount + DOCKER_HOST config is intact and correct (verified).

Bug: the registry-prune step sorted ci-* tags alphabetically and kept the 'newest' 5. A commit SHA starting with a digit (e.g. ci-40ff...) sorts before letter-prefixed SHAs, so the image was DELETED immediately after being pushed - before deploy-test could pull it. Symptom on the v3.4.4 run: deploy-test failed 'not found' (run 269/270 job 3) and deploy-prod skipped. Fix: exclude the current build's own ci-${GITHUB_SHA} tag from the prune set so the image destined for deploy is never removed mid-pipeline. Note on build-job intermittent failures in this run and the previous one: unrelated transient runner issue (two stale long-lived buildx/buildkit containers had been up 11h on the Podman runner; one is being cleaned). The workflow's podman-socket mount + DOCKER_HOST config is intact and correct (verified).
ci: never prune the ci-<sha> image just built in the same run
Some checks failed
CI / lint-frontend (push) Successful in 57s
CI / test (push) Successful in 6m25s
CI / build (push) Failing after 1m22s
CI / deploy-test (push) Has been skipped
CI / deploy-prod (push) Has been skipped
03ddbd921c
The registry-prune step sorted ci-* tags alphabetically and kept the 'newest'
5, so if the current commit SHA started with a digit (e.g. ci-40ff...) it
sorted before letter-prefixed SHAs and got DELETED immediately after being
pushed - before deploy-test could pull it. Symptom: deploy-test failed
'not found' and deploy-prod skipped on the v3.4.4 run.

Fix: exclude the current build's own ci-${GITHUB_SHA} tag from the prune set,
so the image that is about to be deployed is never removed mid-pipeline.
ci: retry build after runner docker_host now points at podman.sock
Some checks failed
CI / lint-frontend (push) Successful in 2m7s
CI / test (push) Successful in 2m12s
CI / build (push) Failing after 1m21s
CI / deploy-test (push) Has been skipped
CI / deploy-prod (push) Has been skipped
e8e7bf424b
ci: retry after runner valid_volumes + docker_host now allow podman.sock
All checks were successful
CI / lint-frontend (push) Successful in 56s
CI / build (push) Successful in 2m2s
CI / test (push) Successful in 5m39s
CI / deploy-test (push) Has been skipped
CI / deploy-prod (push) Has been skipped
c092c1bee2
Author
Owner

Two fixes in this PR:

  1. Registry prune kept deleting the just-built image. The prune step sorted ci-* tags alphabetically and kept the 'newest' 5; a commit SHA starting with a digit (e.g. ci-40ff...) sorts below letter-prefixed SHAs, so it was deleted immediately after push - before deploy-test could pull it. Symptom on v3.4.4: deploy-test failed 'not found', deploy-prod skipped. Fix: exclude the current build's own ci-${GITHUB_SHA} from the prune set.

  2. Runner could not reach a docker daemon (root cause of persistent build failures): the Podman runner (forgejo-runner @192.168.10.33) has no /var/run/docker.sock - only /run/podman/podman.sock - and its config used docker_host: automount with valid_volumes not allowing the podman socket. Fixed on the runner (not the repo): docker_host: unix:///run/podman/podman.sock + valid_volumes includes /run/podman/podman.sock, then restarted. Now build passes reliably (2m2s here) and the job container gets the podman socket at /var/run/docker.sock.

Verified green: test 5m39s, lint 56s, build 2m2s.

Two fixes in this PR: 1. **Registry prune kept deleting the just-built image.** The prune step sorted ci-* tags alphabetically and kept the 'newest' 5; a commit SHA starting with a digit (e.g. ci-40ff...) sorts below letter-prefixed SHAs, so it was deleted immediately after push - before deploy-test could pull it. Symptom on v3.4.4: deploy-test failed 'not found', deploy-prod skipped. Fix: exclude the current build's own ci-${GITHUB_SHA} from the prune set. 2. **Runner could not reach a docker daemon** (root cause of persistent build failures): the Podman runner (forgejo-runner @192.168.10.33) has no /var/run/docker.sock - only /run/podman/podman.sock - and its config used docker_host: automount with valid_volumes not allowing the podman socket. Fixed on the runner (not the repo): docker_host: unix:///run/podman/podman.sock + valid_volumes includes /run/podman/podman.sock, then restarted. Now build passes reliably (2m2s here) and the job container gets the podman socket at /var/run/docker.sock. Verified green: test 5m39s, lint 56s, build 2m2s.
Author
Owner

Scope of this PR (all CI hardening to make the test->prod path reliable):

  1. Registry prune no longer deletes the just-built image. Prune now excludes ci-${GITHUB_SHA} (the image about to be deployed). Previously an alphabetically-early SHA got deleted before deploy-test could pull it (the v3.4.4 failure).

  2. Runner socket config (on .33, not repo): docker_host: unix:///run/podman/podman.sock + valid_volumes including it, then restarted forgejo-runner. The runner now auto-mounts the podman socket at /var/run/docker.sock in job containers. Removed the explicit (redundant + 'not a valid volume') podman.sock volume mount from the build job; kept DOCKER_HOST + DOCKER_BUILDKIT.

  3. Compressed CI to a 4-job pipeline: test, lint-frontend, build, and a single deploy job. deploy deploys the SAME image to the TEST node on main pushes and promotes to PROD on v tags (conditional step), bound to -p honbu-manager-production and injecting TUNNEL_TOKEN. Removed the separate deploy-test/deploy-prod jobs (-48 lines).

  4. Cleaned up the stale runner: deregistered the idle repo-scoped 'unraid-runner'; 192.168.10.33 (registered globally as Proxmox-LXC, Podman) is the sole job executor.

Verified green: test 2m11s, lint 56s, build 2m1s (deploy correctly skipped on branch push). Full v-tag to production promotion run follows in a separate tag after merge.

Scope of this PR (all CI hardening to make the test->prod path reliable): 1. **Registry prune no longer deletes the just-built image.** Prune now excludes ci-${GITHUB_SHA} (the image about to be deployed). Previously an alphabetically-early SHA got deleted before deploy-test could pull it (the v3.4.4 failure). 2. **Runner socket config** (on .33, not repo): docker_host: unix:///run/podman/podman.sock + valid_volumes including it, then restarted forgejo-runner. The runner now auto-mounts the podman socket at /var/run/docker.sock in job containers. Removed the explicit (redundant + 'not a valid volume') podman.sock volume mount from the build job; kept DOCKER_HOST + DOCKER_BUILDKIT. 3. **Compressed CI to a 4-job pipeline:** test, lint-frontend, build, and a single deploy job. deploy deploys the SAME image to the TEST node on main pushes and promotes to PROD on v<semver> tags (conditional step), bound to -p honbu-manager-production and injecting TUNNEL_TOKEN. Removed the separate deploy-test/deploy-prod jobs (-48 lines). 4. **Cleaned up the stale runner:** deregistered the idle repo-scoped 'unraid-runner'; 192.168.10.33 (registered globally as Proxmox-LXC, Podman) is the sole job executor. Verified green: test 2m11s, lint 56s, build 2m1s (deploy correctly skipped on branch push). Full v-tag to production promotion run follows in a separate tag after merge.
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
fastie81/honbu-manager!133
No description provided.