fix(cluster): no second cluster from a former member, and no restart of a retired server

.github/workflows/mesh-integration.yml

@@ -1,6 +1,6 @@
 name: Mesh integration
 
-# Brings up three k3s nodes in Docker and joins them through Headscale with the cluster role (tests/mesh/run.sh), once per scenario: headscale_hosted, headscale_in_cluster, and the headscale_existing policy check, plus the role's refusal of a Docker Compose release that recreates the containers it has just built. headscale_in_cluster also runs with github_runner_cluster_docker_context pinned while the current Docker context points at nothing, which covers the cluster role's own docker calls and the recovery playbook's, and context_checks checks the role refuses a pin it cannot honour. regrade turns memory-pressure regrading on and steps a server down to etcd-only under (simulated) pressure and back up, checking that it keeps its etcd seat and vote and that no other server steps meanwhile. native_server joins a native host (a systemd container) as an agent beside three Docker servers, then promotes it to a server in place: it must join etcd as a learner, be promoted to a voter, and keep the cluster writable throughout. role_controller adds a standby agent, runs the role controller's probe and controller from the image built from this checkout, makes a voter slow with tc netem and pauses another, and checks every probe reports the link medium the cluster role labelled its node with, the controller reports the voter unfit with the swap named, reports that it would not act with a voter paused, and changes nothing. Only on changes that can affect how nodes join the mesh.
+# Brings up three k3s nodes in Docker and joins them through Headscale with the cluster role (tests/mesh/run.sh), once per scenario: headscale_hosted, headscale_in_cluster, and the headscale_existing policy check, plus the role's refusal of a Docker Compose release that recreates the containers it has just built. headscale_in_cluster also runs with github_runner_cluster_docker_context pinned while the current Docker context points at nothing, which covers the cluster role's own docker calls and the recovery playbook's, and context_checks checks the role refuses a pin it cannot honour. regrade turns memory-pressure regrading on and steps a server down to etcd-only under (simulated) pressure and back up, checking that it keeps its etcd seat and vote and that no other server steps meanwhile. native_server joins a native host (a systemd container) as an agent beside three Docker servers, then promotes it to a server in place: it must join etcd as a learner, be promoted to a voter, and keep the cluster writable throughout. rejoin removes the bootstrap Docker server's etcd member and checks the server idles on its tombstone, refuses to start a second cluster from its moved-aside datastore with no server to join, joins the existing cluster from a join URL in its role directory with the same cluster ID, and idles as a server whose role file names the agent. role_controller adds a standby agent, runs the role controller's probe and controller from the image built from this checkout, makes a voter slow with tc netem and pauses another, and checks every probe reports the link medium the cluster role labelled its node with, the controller reports the voter unfit with the swap named, reports that it would not act with a voter paused, and changes nothing. Only on changes that can affect how nodes join the mesh.
 #
 # Runs on GitHub-hosted ubuntu-latest, whose Docker daemon runs the privileged containers the test needs (k3s, and Tailscale inside it). The work directory sits under runner.temp, which the job always has write access to.
 
@@ -44,6 +44,7 @@ jobs:
           - {scenario: regrade, compose: v5.5.1}
           - {scenario: native_server, compose: v5.5.1}
           - {scenario: role_controller, compose: v5.5.1}
+          - {scenario: rejoin, compose: v5.5.1}
           - {scenario: hosted, compose: v2.39.0}
           - {scenario: compose_faulty, compose: v2.38.2}
           - {scenario: in_cluster, compose: v5.5.1, pin: true}

README.md

@@ -246,7 +246,7 @@ Commits follow Conventional Commits (`fix:`, `feat:`, `tune:`, and so on).
 - **etcd's own peer traffic does not follow vpn-auth's `--node-ip`/`--advertise-address` the way every other kind of cluster traffic does - it stays on the container's own internal, cross-machine-unroutable Docker-bridge IP by default.** k3s logs this outright ("Etcd IP (PrivateIP) remains the local IP. Running etcd traffic over VPN is not recommended due to performance issues") and deliberately avoids the VPN interface for etcd specifically, assuming a real local network is the better alternative - an assumption that doesn't hold here, since a container's own bridge IP is never reachable from another machine at all. Confirmed live: a second server's join failed outright with `MemberAdd request timed out`, because it was trying to reach the first server's own unroutable bridge IP. An `--etcd-arg="initial-advertise-peer-urls=..."` passthrough looks like the fix but isn't: it only overrides etcd's own raw flag, while k3s independently builds its own `--initial-cluster` string from `config.PrivateIP`, computed separately and earlier - the two then disagree and etcd's bootstrap validation rejects the mismatch outright (`"--initial-cluster has ...=<bridge-ip> but missing from --initial-advertise-peer-urls=<tailscale-ip>"`). The actual fix, traced through k3s's own source (`pkg/cli/server/server.go`, `pkg/executor/embed/embed.go`): `config.PrivateIP` is populated from `--node-ip`'s own value if set, and this happens *before* `--vpn-auth`'s own executor-layer code later rebuilds `NodeIP` from its own Tailscale detection - so passing `--node-ip` explicitly ourselves, to the same real Tailscale IP vpn-auth would itself have detected, is what actually makes `PrivateIP` (and therefore etcd's peer-advertise address) correct, with no second, disagreeing source of truth for the same value. `docker-compose.yml`'s `k3s` service entrypoint brings Tailscale up itself and reads `tailscale ip -4` before exec'ing k3s specifically to learn this IP first, since `exec` replaces the script's own process and nothing can react afterward to what `--vpn-auth`'s own internal call would have produced.
 - **A global `--etcd-arg` override is not scoped to the "real" server's own etcd instance - it also applies to k3s's separate, internal "temporary etcd", spun up on every *restart* (never on first boot) to reconcile with an already-initialized on-disk datastore before the real instance starts.** An earlier `--etcd-arg="listen-peer-urls=https://0.0.0.0:2380"` (kept as a "harmless" belt-and-braces bind override after fixing the peer-URL issue above) forced that temporary instance onto the same TLS `0.0.0.0:2380` bind it has no certificates for by design (it normally listens on a local, ephemeral, non-TLS port). Confirmed live: this only broke on a restart against an already-initialized datastore, never on a fresh `--cluster-init`, which is exactly why it wasn't caught until well after the first working deploy - fatal: `"cannot listen on TLS for [::]:2380: KeyFile and CertFile are not presented"`. `--node-ip` alone (above) is sufficient; etcd's own default `listen-peer-urls` already derives from the same `PrivateIP` correctly, so there's nothing left for an explicit override to add.
 - **Calling `tailscale up --authkey=...` unconditionally on every container restart re-registers a brand new tailnet device each time, instead of reconnecting the persisted one - even with the join key's own state volume intact.** A reusable Tailscale auth key is designed to authorize provisioning *new* devices, and a naive `tailscale up` call on every restart (needed here to learn this node's own Tailscale IP before `exec`ing k3s - see above) doesn't check whether the daemon is already authenticated first, so it keeps minting new devices under names like `node1`, `node1-1`, `node1-2`... k3s's own `--vpn-auth` integration avoids exactly this from v1.35.0 on, by checking the Tailscale backend state first and skipping its own `tailscale up` call entirely once already `"Running"` (`StartVPN()` in `pkg/vpn/vpn.go`; the pinned v1.34 releases run `tailscale up --authkey ... --reset` on every start whatever the state) - the manual pre-empt call needs the identical guard: check `tailscale ip -4` succeeds *before* calling `tailscale up` at all, only calling it when not already authenticated.
-- **Tailscale assigns an IP and wires up its own MagicDNS resolver as two separate steps, not atomically - a node having a real IP is not proof DNS resolution against another tailnet member is working yet.** A joining server needs to resolve the bootstrap host's own MagicDNS hostname (`K3S_JOIN_SERVER_URL`) before it can bootstrap against it at all. Confirmed live: k3s failed outright with `"dial tcp: lookup k3s-server.example.ts.net: no such host"` immediately after `tailscale ip -4` had already returned a real address for the *local* node. Fix: wait for a real `nslookup` success against the join server's own hostname specifically, not just local Tailscale readiness, before invoking k3s (the `k3s` service's entrypoint only does this when `K3S_JOIN_SERVER_URL` is actually set - the bootstrap host has nothing else to resolve).
+- **Tailscale assigns an IP and wires up its own MagicDNS resolver as two separate steps, not atomically - a node having a real IP is not proof DNS resolution against another tailnet member is working yet.** A joining server needs to resolve the bootstrap host's own MagicDNS hostname (`K3S_JOIN_SERVER_URL`) before it can bootstrap against it at all. Confirmed live: k3s failed outright with `"dial tcp: lookup k3s-server.example.ts.net: no such host"` immediately after `tailscale ip -4` had already returned a real address for the *local* node. Fix: wait for a real `nslookup` success against the join server's own hostname specifically, not just local Tailscale readiness, before invoking k3s (the `k3s` service's entrypoint only does this when it has a server to join, `K3S_JOIN_SERVER_URL` or the role directory's `join-url` - the bootstrap host has nothing else to resolve).
 - **A server that memory-pressure regrading has stepped to `etcd-only` is `NotReady` by design, and must stay registered.** With `github_runner_cluster_regrade` on, a server under sustained memory pressure steps down (`full`, `control-plane`, `etcd-only`) while it keeps its etcd member and vote; at `etcd-only` k3s runs with `--disable-apiserver --disable-controller-manager --disable-scheduler --disable-agent`, so there is no kubelet to keep its Node Ready, and the Node goes `NotReady` with `Unknown` conditions and stays so. That is not a failure to clear. Its Node object must not be deleted (k3s removes that server's etcd member with it), so anything that culls long-`NotReady` nodes must leave a Node carrying the `github-runner.exadev.com/regrade-level` label alone, and the role's wait for every node to be Ready already does. `kubectl` on such a server's own container is unreliable (its API server is off); the regrade service always reaches the cluster through another server. See the cluster role's README, Memory-pressure regrading. Confirmed against a three-server k3s cluster: the same etcd member ids before, during and after, the cluster taking writes with the stepped-down server as one of the two live voters, and the same Node and node password after stepping back up.
 - **A single remaining healthy etcd server can get stuck indefinitely if another member is registered but permanently unreachable (e.g. a partial join that never completed), even though that member is a non-voting learner and shouldn't block quorum.** Symptom: `"failed to publish local member to cluster through raft"` with `context deadline exceeded`, repeating forever, alongside `kubectl` returning `503 the server is currently unable to handle the request`. The fix is k3s's own documented recovery mechanism, run once against the affected node: stop the container, then `k3s server --cluster-reset` (with the SAME flags the real service uses, including `--node-ip` set to the node's own persisted Tailscale IP - getting this wrong re-registers the single remaining member under the *wrong* address, e.g. the invoking container's own ad-hoc bridge IP, and the real service then fails to start with `"this server is a not a member of the etcd cluster"`) - this resets *etcd cluster membership* back to a single member using the *existing local data*, it does not wipe anything. It's a one-shot operation: k3s writes a guard file after a successful reset and refuses to run `--cluster-reset` again until it's manually deleted (`rm /var/lib/rancher/k3s/server/db/reset-flag`) if a second reset is genuinely needed. After a reset, restart the service normally (without the flag) and clean up the stale member's own now-orphaned Kubernetes `Node` object and `node-password` Secret before it rejoins fresh.