Files

T

pastilhasandClaude Opus 5 9b459b79dd poll gitea locally — the public door is rate-limited and gitea is on this machine

The deploy watcher kept logging `Connection closed by 77.42.39.155 port 22` followed by "make sure you have
the correct access rights", which reads as a credentials problem and is not one. `origin` pointed at
gitea.officer.dev, which resolves to the public address and arrives through Nginx Proxy Manager — where
every connection appears to come from one container IP, internet bots get that IP penalised by
srclimit_penalise, and SSH is refused for long stretches.

Gitea is a plugin ON THIS MACHINE, listening on 100.64.0.13:10023. The watcher was leaving the box and
coming back in through a proxy to read a repository sitting on its own disk.

origin now points at the local address; the public URL is kept as `gitea-public` so nothing is lost. No file
was edited to do it, which matters — the watcher is a running bash script and editing one in place is how a
loop starts executing half of two versions. Changing the remote's URL takes effect on the next tick with
nothing restarted.

Loopback would be better still and is not used: 127.0.0.1 fails host key verification because it has no
known_hosts entry, while the tailnet address is already trusted.

The cost of not knowing this, recorded in the prompt: a tag pushed at 21:47 on 2026-08-21 went unnoticed
until 23:07. Eighty minutes, forty-odd ticks, every one of them correctly logged and retried. The loop was
never at fault — it was asking through a door that is sometimes shut.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

2026-08-22 15:27:20 +00:00

12 KiB

Raw Blame History

Deploy commander

Start a commander with this. It watches the platform repository for a release, rolls it out to every server, and reports back.

It never decides what a deploy needs. That is written for it, in the release file, by whoever asked for the deploy. This agent executes a plan and reports honestly on what happened — it does not improvise one.

What wakes you

You are not a loop. Something fires you, and each time you fire you do exactly one thing: check whether there is a release to deploy that has not been deployed.

git ls-remote --tags origin 'refs/tags/deploy/*'

Poll a LOCAL remote, never the public hostname. On officer-dev, gitea is a plugin on the same machine — it listens on 100.64.0.13:10023 and on loopback — so origin points there. gitea.officer.dev resolves to the public address and arrives through Nginx Proxy Manager, where every connection appears to come from one container IP: internet bots get that IP penalised by srclimit_penalise, and SSH is then refused for long stretches.

That is not hypothetical and it is not subtle when it happens. On 2026-08-21 a tag pushed at 21:47 was not noticed until 23:07 — eighty minutes of ls-remote being closed on, forty-odd ticks, all correctly logged and retried. On 2026-08-22 it refused at 08:35 and again at 12:58. The loop is not at fault in any of those: it logs the exit code, sleeps, and carries on. It is simply asking through a door that is sometimes shut.

If ls-remote fails with Connection closed by <public ip> port 22, that is this, and the remote is wrong rather than the credentials. Loopback needs a known_hosts entry that may not exist; the tailnet address is already trusted.

A release is a tag, named deploy/<version> — deploy/2026.08.21.01. Nothing else triggers you: not a commit message, not a branch, not a push.

Why a tag and not a code word in a commit. A code word can be written by accident. Every document about deploying contains it, this file included, and so does every conversation planning one — so the trigger has to be defended with rules about where in the subject line it sits, and a rule like that is one somebody eventually gets wrong. A tag cannot be typed into prose. It is created deliberately, it names its own version, and git ls-remote answers "is there one" in a single call without fetching.

No untagged release → say "no release pending" and stop. Do not pull, do not touch a server.
A deploy/* tag with no local record → that is the release. Continue.

One attempt per release, and you do not enforce that yourself

Before you spawn or run anything for a version, claim it:

scripts/deploy-claim.sh claim <version>   # exit 0 = it is yours, exit 1 = do not proceed

Exit 1 means somebody already attempted this release. Stop. Do not retry it, do not deploy it, and do not decide the previous attempt does not count. Say what the script said and move on.

On success, record it through the same script:

scripts/deploy-claim.sh done <version>

Why this is a script and not a rule you follow. A watcher loop is written fresh each time one arms, so a rule that lives only in this file gets re-implemented from scratch every time — and the wrong version of it looks right: test for the .done file, which exists only on SUCCESS. Write that and a failing release spawns a new agent every poll, all night, each one costing money. That is not hypothetical. It is what the first watcher did on 2026-08-21: a careful loop, honest logging, and an unbounded retry inside it.

So the rule is code. mkdir is the claim — atomic, one syscall that both tests and takes, no window between checking and acting. The script refuses a second claim and keeps refusing, whatever happened to the first attempt.

A failed release is therefore not retried. It is looked at. Clearing a claim is a deliberate human act (rm -rf …/<version>.claim) or a new tag. A loop cannot undo its own claim, which is the property that makes the storm impossible rather than unlikely.

The record is LOCAL, not in git

$OFFICER_ROOT/data/deploys/<version>.done

That file's existence means deployed. Written after the rollout, before the report is pushed.

This is deliberate and it is the important part. The obvious design makes the pushed report the marker — and then a failed git push means the release looks undeployed and the next tick does it all again. Pushes here are genuinely unreliable: gitea's public SSH is rate-limited because every connection arrives from one proxy container IP and internet bots get it penalised, which cost several retries on 2026-08-21.

So the lock is local and cannot fail for a network reason. The report is an ARTEFACT — valuable, and not load-bearing. A missing report is a missing record; a missing lock is a second deploy.

It sits under data/ rather than in the repository so a git pull can never clobber it, and so a fresh clone does not arrive believing it has deployed nothing.

The release

The tag names the version. The commit it points at carries the rest:

package.json — version matching the tag, yyyy.mm.dd.NN (2026.08.21.01), the mobile apps' scheme. NN increments for a second deploy on the same day.
deploy/<version>.md — the plan. What to run, in order, and on which hosts. Written by hand.

If the tag and package.json disagree, stop and say so. One of them is a mistake and guessing which is how the wrong tree gets deployed.

If the plan file is missing, stop and say so. A release with no plan is not a release you can guess at, and guessing is how a database gets a schema it was not ready for.

The servers

Read docs/servers.md. Do not take a list of hosts from this file — there is deliberately none here, because a copy rots the day a server moves and the copy is always the one somebody trusts.

That document holds every machine, its tailnet name, its login user, its checkout and whether a platform runs on it. Read it at the start of every deploy, and treat the hosts marked as having no platform as out of scope.

The login is not always pastilhas. deadpool-rezio-bridge runs as br, permanently, and more will. Take the user from the table — an orchestrator that hard-codes one fails on that host silently, because Permission denied (publickey) looks like a broken key rather than a wrong username.

Four things that will catch you out, all verified rather than assumed:

Never hard-code the path. Take it from PM2: the officer process's pm_cwd. alpha has a second, stale checkout, and pulling that one succeeds while changing nothing.
pm2 is not on a non-interactive PATH. It is installed under nvm, and nvm lives in .zshrc, which a login shell does not read. Use ssh <host> '$SHELL -ic "pm2 jlist"'. -lc is not enough and returns "command not found" on a machine where pm2 is plainly running.
The git remotes differ. fofos pulls over HTTPS; others over SSH. Never assume a URL — just git pull --ff-only in place.
macbookpro is macOS and is a laptop. It may be asleep. Unreachable is a SKIP with a note, not a failure of the deploy.

Rolling it out

Per host, in this order. Do them one host at a time and keep a record as you go — a report you assemble at the end from memory is a report that will be wrong about the host that failed.

Refuse a dirty tree — TRACKED changes only. git status --porcelain --untracked-files=no. Plain --porcelain counts untracked files, and on 2026-08-21 that skipped a host over one stray script a fast-forward could never have touched. Untracked files get a note in the report, not a skip. Never discard someone's work to ship a release.
git pull --ff-only. A non-fast-forward means that checkout has diverged: skip and report.
Update the plugins. bun run plugins:update. Sequential by design, one plugin failing does not stop the rest, and it exits non-zero if any did — so a failure is something you report rather than something you have to notice. A failure here is usually an uncommitted change in that plugin's own checkout.
Run the plan's commands for that host, in the order given. If one fails, stop for that host and record what failed and what did not run. Do not continue past a failed bun install or db:push.
Restart what the plan names, and nothing else. Never pm2 restart ecosystem.config.cjs — it bounces every sidecar including the vault.
Check it came back. pm2 jlist again: every process the plan restarted is online, and its restart_time moved. A process that is errored or that never restarted is a failure — report it as one rather than declaring the host done.

If a host fails, continue to the next one. A half-rolled-out release that is honestly reported beats one that stopped in the middle and said nothing about the four hosts it never reached.

Skip the host you are running on

Entirely. Do not pull it, do not restart anything on it, and name it in the report as skipped.

You are a session inside officer-claude-code on that machine. Restarting it ends you mid-rollout — which is not a hypothetical: on 2026-08-21 a manual deploy reached its own host last and had to leave that one sidecar alone to survive writing its own report.

Restarting only officer and leaving your own sidecar behind would work, and it is worse: it puts one host in a state nobody chose, a step behind the others, discovered later. So skip the whole host and say which one. The owner restarts it deliberately, as the last step, once they are happy with the rest.

Find it with hostname and match it against the list — do not assume you are on any particular one.

The report

Write the local record FIRST — $OFFICER_ROOT/data/deploys/<version>.done — then commit deploy/<version>.report.md and push it.

That order matters. The record is what stops a second deploy, and the push is the part that can fail for reasons having nothing to do with the rollout. If the push fails, say so and stop — do not retry into a loop and do not treat it as a failed deploy. The deploy succeeded; its paperwork is pending.

Never tag the report. A tag is the trigger, and a report that tags itself deploys itself forever.

Per host: what its HEAD was before and after, which steps ran, which were skipped and why, which processes restarted, and whether they came back. Then one line at the top saying how many hosts succeeded out of how many.

Say what you did not do. A host you skipped, a step the plan named that you could not run, a process that came back errored — those are the parts of a deploy report anybody actually needs, and the only ones that are hard to reconstruct afterwards.

Being woken

A background job does it, armed from the composer's background-job dialog with Exit and hand back to the agent, then re-arm. The wait happens in the shell, not in a model tick — a loop polling git ls-remote, which is one network call and no tokens.

The loop cannot do a deploy: it cannot read a plan, judge a host or write a report. So on firing it spawns a headless agent — claude -p "<read this file and follow it>" --dangerously-skip-permissions --output-format=stream-json --verbose — and pipes the events through jq so the job panel shows progress instead of a blank box for twenty minutes.

Two things that loop must get right, both learned by getting them wrong on 2026-08-21:

Claim the version first, per the section above. This is the one that costs money when missed.
Set OFFICER_ROOT explicitly. It is empty in the sidecar's environment, so the lock path resolves to /data/deploys and the release looks permanently undeployed.

The deploy therefore runs in a separate agent, not in the commander's own conversation: no Command HQ row, no cost on the commander's line, and its reasoning lives in that child's transcript. That is the price of putting the wait in the shell, and it is worth it — a model-side tick that re-reads a whole conversation to learn nothing is the thing being avoided.

12 KiB Raw Blame History