[bhouse-nexthop]

Description

kubeadm issues the control plane certificates of a CKS cluster for one year. CKS never renewed them and never checked their expiry, so a cluster that isn't upgraded within a year stops working when they expire. And because CKS only read the kubeconfig that getKubernetesClusterConfig returns when the cluster started, even an upgrade, which renews the certificates, left it handing out the old admin certificate until that expired.

This PR checks the expiry, alerts ahead of it, adds an API to renew the certificates one control node at a time, renews them automatically before they expire, and refreshes the stored kubeconfig after every upgrade and renewal.

Fixes #14650

What changed

1. Stored kubeconfig refresh. kubeConfigData is read again from the control node:

   • after upgradeKubernetesCluster (a failure there is logged and doesn't fail the upgrade);
   • after every certificate rotation;
   • whenever the expiry check finds the stored copy's admin certificate expiring before the one on the cluster, e.g. after a renewal by hand.

   The read-and-rewrite code that StartWorker had is now KubernetesClusterActionWorker.refreshKubeConfigData(), shared by all three. KubernetesClusterUtil.getKubernetesClusterConfig now waits 5 seconds between attempts instead of retrying a refused SSH connection in a tight loop until its timeout.

2. Expiry check. A new scanner, on its own thread next to KubernetesClusterStatusScanner, runs kubeadm certs check-expiration over SSH on every control node of each Running or Alert cluster, at most every cloud.kubernetes.cluster.certificate.check.interval hours. It also reads the client certificate of /root/.kube/config, which CKS uses for its own kubectl calls. The scanner wakes every 15 minutes, but the per-cluster interval is tracked in a cluster detail, so a cluster is checked once per interval whichever management server runs it, and an unreachable cluster isn't retried until the next interval.

   • When the first leaf certificate reaches one of the cloud.kubernetes.cluster.certificate.expiry.alert.days thresholds, CKS raises an alert (new type ALERT.KUBERNETES.CLUSTER.CERTIFICATE) and records a KUBERNETES.CLUSTER.CERTIFICATES.EXPIRY event. Each threshold fires once per expiry date, and expiry itself alerts once more. A renewal moves the date, so the thresholds start over.
   • The first certificate authority to expire is tracked the same way, with its own alert. CAs are reported, never rotated.
   • When none of a cluster's control nodes can be checked, or the cluster has no address to reach it on, an alert says so, at most once a day, so an unreachable cluster can't expire unnoticed. Each cluster's state is read again just before it is checked, so a cluster stopped during a pass is skipped.

3. rotateKubernetesClusterCertificates API (async, same roles as upgradeKubernetesCluster). For each control node, one at a time, it copies rotate-kubernetes-certificates.sh to the node and runs it. The script:

   • puts back any static pod manifest left aside by an interrupted earlier run;
   • fails before touching anything if crictl can't list the containers, since it needs it to confirm the static pods restarted;
   • waits up to 3 minutes for the local API server to be ready, because kubeadm certs renew silently falls back to its default configuration when it can't read the cluster's. If it still isn't ready, the script warns and renews anyway: once the certificates have expired the API server never becomes ready, and renewing is the way out;
   • runs kubeadm certs renew all, and fails if apiserver.crt or admin.conf did not change or kubeadm says a certificate "can't be renewed" (external CA mode, where kubeadm exits 0 without renewing anything);
   • copies admin.conf to /root/.kube/config. If that file was something else before the renewal, it is kept as config.cks-backup, and an existing backup is never overwritten;
   • moves the kube-apiserver, kube-controller-manager, kube-scheduler and etcd manifests out, waits until the kubelet has stopped those containers, then moves the manifests back. An exit trap moves them back on any failure too;
   • waits for /readyz on the API server, /healthz on the controller manager and scheduler, and for the API server to serve the renewed certificate.

   Once the script is done on a node, the worker waits for the cluster API endpoint to answer and the node to be Ready before it moves to the next node. The shared readiness check now also accepts Ready,SchedulingDisabled, so a cordoned control node doesn't fail the rotation. It stops at the first node that fails, moves the cluster to Alert and raises an alert. After the last node it refreshes the stored kubeconfig.

   A failed rotation leaves the cluster in Alert, and the API accepts Alert as well as Running, so it can be retried right away. A rotation started from Alert goes back to Alert when it succeeds. Whatever else put the cluster there hasn't been checked, so the status scanner moves it to Running once its nodes are ready.

   The API also rescues a cluster whose certificates have already expired. Its API server no longer answers, so it is in Alert. With stacked etcd and several control nodes, one renewed node can't get etcd quorum until the others are renewed too, so waiting for each node in turn would never get past the first. When kubeadm certs check-expiration on any control node shows an expired certificate that the control plane runs on, the worker switches to rescue mode. An expired admin.conf or super-admin.conf breaks no component and doesn't count. Checking every node matters: after an interrupted rescue the first node is renewed while the others still block quorum. The decision goes by expiry, not by whether the API answers. An expired API server that is still running keeps answering /version, while a healthy cluster that is briefly unreachable must keep the node-by-node path, since restarting every control node at once would cost it etcd quorum. In rescue mode:

   • it renews every control node first, and on each node the script only waits for etcd (when the node runs one) to run again and for new sandboxes of the other control plane pods. The API server keeps restarting until etcd has quorum, so it can't wait for that;
   • then it waits for the cluster API and for every control node to be Ready.

   Rerunning is safe. kubeadm certs renew issues fresh certificates every time, and the interrupted-run recovery above makes the script pick up where a killed run left off.

4. Automatic renewal. When the first renewable leaf certificate is within cloud.kubernetes.cluster.certificate.auto.renew.days and cloud.kubernetes.cluster.certificate.auto.renew.enabled is true for the cluster's account, the scanner hands the same rotation to a small thread pool, so a renewal that takes minutes doesn't hold up the checks of other clusters. When it is done, an event tells users to download the kubeconfig again. Values above 180 days are treated as 180, so a mistyped setting can't renew every cluster on every check.

   • Certificates kubeadm reports as externally managed are left out of this decision, since renewing can't fix them and every check would renew again.
   • After a failed rotation, automatic renewal waits 24 hours before trying again; the failure has already raised an alert.
   • When a control node couldn't be checked, automatic renewal waits too: it would renew the reachable nodes and then fail on that one. The expiry alert then says renewal is on hold and names the nodes.

5. Guards.

   • Rotation and the check need Kubernetes 1.20 or later, where kubeadm certs replaced kubeadm alpha certs. The API refuses older clusters and the scanner skips them.
   • The API refuses clusters that aren't Running or Alert, and externally managed clusters. Clusters in Alert are checked and alerted on, since one can sit there for long (a stopped worker is enough), but automatic renewal only runs on Running clusters.
   • A queued automatic renewal checks again when it gets a thread, and skips the cluster if it is no longer Running or was rotated meanwhile, successfully or not.
   • Clusters created before 4.16 have no control node flag in kubernetes_cluster_vm_map. For those, the control nodes are taken by position, after any etcd nodes, as fetchControlVmIfMissing already does.
   • startKubernetesCluster and stopKubernetesCluster refuse a cluster in RotatingCertificates, whose control plane manifests may be moved aside at that moment. Upgrade already requires Running, and scale Created, Running or Stopped.
   • A new transient state, RotatingCertificates, is entered from Running or Alert through the new CertificateRotationRequested event. It goes back to Running on success and to Alert on failure. As with upgrade and scale, the state transition is what keeps other operations out: none of them can start from RotatingCertificates, and a rotation can't start while another operation runs.
   • The status scanner's Alert reconciliation could move a cluster to Running, or to Stopped, while a rotation from Alert had just started. reconcileAlertCluster and the all-VMs-stopped branch now stop when their own first transition fails, instead of applying OperationSucceeded regardless.
   • On management server shutdown, automatic renewals still waiting for a thread are dropped rather than started; the next check after the restart schedules them again.
   • If the management server running a rotation goes away, the cluster would stay in RotatingCertificates. The rotation records which management server runs it. When that server comes back, its scanner moves the rotation to Alert straight away; any other server does so once it is well past cloud.kubernetes.cluster.certificate.rotation.timeout (twice the timeout plus 10 minutes). Either way an alert is raised.

6. upgrade-kubernetes.sh now also copies the renewed admin.conf to /root/.kube/config on control nodes, so CKS's own kubectl keeps working after the old certificate expires. As in the rotation script, a /root/.kube/config that differed from admin.conf before the upgrade is kept as a backup first.

No schema changes. The state column is char(32), and the bookkeeping (last check, alerted thresholds, rotation start and last failure) is kept in kubernetes_cluster_details.

Settings

Setting	Default	Scope	Meaning
cloud.kubernetes.cluster.certificate.check.interval	6	Global	Hours between expiry checks of a cluster. 0 disables the check and automatic renewal.
cloud.kubernetes.cluster.certificate.expiry.alert.days	60,30	Global	Days before expiry at which to alert.
cloud.kubernetes.cluster.certificate.auto.renew.enabled	true	Account	Renew automatically before expiry.
cloud.kubernetes.cluster.certificate.auto.renew.days	30	Account	How many days before expiry to renew automatically. 0 disables it; values above 180 are treated as 180.
cloud.kubernetes.cluster.certificate.rotation.timeout	3600	Global	Timeout of a rotation, in seconds.

All of them are dynamic.

Automatic renewal is on by default. A plain kubeadm cluster with no upgrades and no renewal timer is the one setup where the certificates simply run out. RKE2 and k3s renew on restart, and managed services rotate them unseen. Renewing 30 days ahead leaves a month for people and CI that downloaded the old kubeconfig to fetch it again, and the event says so. Anyone who wants to schedule the restarts themselves can turn it off globally or per account and use the API.

Restart impact

Static pods don't reload certificates, so each control node's API server, controller manager, scheduler and stacked etcd restart. One node takes about 40 seconds.

• Several control nodes: the restart rolls one node at a time and etcd keeps quorum. The API as a whole stays up behind the load balancer, but requests the load balancer sends to the restarting node fail until its API server is back; clients retry them.
  • How many fail depends on the load balancer. With kind's load balancer, 6 of 187 polls failed, two per node, because its health check takes the node out quickly.
  • With the CloudStack network load balancer in front of a 3-control-node CKS cluster, about a third of requests failed for roughly 20 s per node, because it kept sending to the restarting node: 42 of 491 polls across two rotations.
  • The cloud controller manager pod may restart once per rotation when it loses its leader lease. It recovers on its own.
• One control node: the API is unavailable for roughly 30–40 seconds.
• Workloads keep running in both cases. Only the control plane restarts; kubelets and pods don't.
• Kubeconfigs downloaded before the renewal keep working until their old certificate expires. Renewal doesn't revoke the old certificate.

External etcd

kubeadm certs renew doesn't touch separate etcd nodes, but the etcd nodes CKS creates (etcd-node.yml) serve etcd over plain HTTP: --listen-client-urls http://..., peer URLs http://..., no --cert-file. They hold no certificates, so there is nothing to renew or check on them, and the rotation only touches control nodes. kubeadm sees those clusters as having external etcd, so check-expiration and renew leave out the etcd certificates there and handle the rest normally.

If etcd nodes ever get TLS, they'll need their own renewal step. Unencrypted etcd traffic is a separate concern and out of scope here.

Not in this PR

• CA rotation; CAs are only reported.
• A UI button for the new API.
• /etc/kubernetes/user.conf. CKS returns it instead of admin.conf when an operator has put one on the control node. It is the operator's file, possibly signed by something CKS doesn't manage, so CKS neither renews nor rewrites it. The stale-kubeconfig check compares the stored copy with it rather than with admin.conf.
• The cluster state update is not a compare-and-set (KubernetesClusterDaoImpl.updateState writes unconditionally). That's how every CKS operation works today, and this PR uses the same guard. Likewise, the existing workers carry on when their own first transition fails. An upgrade validated as Running just before an automatic renewal takes the cluster could therefore still run alongside it; the window is the time between the API's state check and the worker's transition. Making updateState conditional, and having every worker stop when its first transition fails, would close this race for all operations. That's a separate change.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• Build/CI
• Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

How Has This Been Tested?

Unit tests

All 189 tests of the CKS plugin pass, including these new ones.

• KubernetesClusterCertificateUtilTest:
  • parsing real kubeadm certs check-expiration output (v1.31): both tables, missing, expired and externally managed certificates, and error output;
  • reading a kubeconfig's client certificate expiry;
  • the threshold parsing and alert decision (one alert per threshold per expiry date; a cluster first seen inside both windows alerts once; expiry alerts once more);
  • the auto-renew decision.
• KubernetesClusterCertificateWorkerTest:
  • the rolling sequence: node by node, renew then wait for ready, then the kubeconfig refresh and Running;
  • stopping at the first node that fails to renew or doesn't become ready, with the remaining nodes untouched, Alert and an alert;
  • rescue mode when a certificate on any control node has expired (every node renewed first, then each waited for), and the node-by-node path otherwise;
  • a rotation from Alert going back to Alert;
  • a second rotation of the same cluster refused without disturbing the first, and a cluster another server moved out of this rotation meanwhile left in its state;
  • /root/.kube/config treated as externally managed when admin.conf is (external CA mode), so it doesn't trigger renewal;
  • failure events recorded only for automatic renewals, since the API's async job records its own;
  • a failed kubeconfig refresh;
  • refusal when the state transition fails and before 1.20;
  • the per-node SSH endpoint (direct network, the cluster IP's port forwarding rule rather than a user's rule on another IP, and by position);
  • the check across control nodes with one unreachable;
  • refreshing a stale stored kubeconfig, and comparing it with user.conf when there is one.
• KubernetesClusterCertificateManagementTest:
  • the scanner: at most one check per interval, the 60 and 30 day alerts firing once each as expiry approaches, thresholds starting over after a renewal, CA alerts without rotation, Alert clusters checked but not renewed, no automatic renewal while a control node is unreachable (and the alert saying so), the daily alert when no node can be checked or the cluster can't be reached at all, the alert advice using the effective renewal days, auto-renew on and off and handed to the rotation thread pool (queued once, requeued after a rejection, skipped when renewed meanwhile), the 24 hour back-off after a failure, externally managed certificates not triggering renewal, and the version gate;
  • moving an interrupted rotation to Alert: at once when it belongs to this management server, otherwise only once it is stale, and never a rotation that just finished;
  • the API validation (Running and Alert accepted) and the new state machine transitions.
• KubernetesClusterUpgradeWorkerTest: the stored kubeconfig is refreshed after a successful upgrade, before the cluster goes back to Running; a failed refresh doesn't fail the upgrade; a failed upgrade doesn't refresh.

The renewal script on real kubeadm control planes

I ran rotate-kubernetes-certificates.sh on throwaway kind clusters (Kubernetes v1.31, kubeadm with stacked etcd):

• Single control node:
  • certificates renewed, all four static pods restarted with new container IDs, the API server serving the renewed certificate, /root/.kube/config identical to the new admin.conf, the node Ready. About 40 seconds.
  • ca.key moved away (external CA mode): kubeadm exits 0 and renews nothing it needs the CA key for. The script fails before restarting anything, and the API server container is unchanged.
  • Killed half-way, with the kube-apiserver and etcd manifests left aside and the API server down: the rerun puts them back, waits for the API server, renews with the cluster's own kubeadm configuration rather than the fallback defaults, and completes.
  • ca.crt hidden so kubeadm certs renew all itself fails: the script prints kubeadm's error, exits 1, and leaves all four manifests in place.
  • API server unreachable (port 6443 rejected inside the node, standing in for expired certificates): the script waits 3 minutes, warns, renews, restarts the static pods and completes once the API server is reachable again.
• Rescue of a three-node cluster with no etcd quorum. I rejected the etcd peer port on all three nodes, standing in for expired peer certificates. That leaves etcd without quorum and the API servers restarting. This confirmed the review's point: a node-by-node wait can't get past the first node.
  • In rescue mode, each node took about 40 seconds, and the script exited 0.
  • Once the peer port was open again, /readyz returned ok and all three nodes were Ready. Each API server served its renewed certificate, and etcd reported all three members healthy.
  • Without the cluster's API, kubeadm renews with its default configuration. It takes the certificate attributes (CN, SANs) from the existing certificates, so they come out the same.
• A custom /root/.kube/config survived two renewals in a row as config.cks-backup, and the readiness check matched a cordoned control node.
• Three control nodes behind kind's load balancer: renewed node by node, waiting for each node to be Ready with the same kubectl get nodes check the worker uses, while polling /readyz through the load balancer every 0.5 s.
  • Every node served its renewed certificate afterwards, and etcd reported all three members healthy.
  • 6 of 187 load balancer polls failed, two per node while its API server restarted.

I also fed the check-expiration output and the /root/.kube/config certificate line from those nodes through the Java parser, and the expiry dates matched.

I haven't run a full CKS cluster through the new API and the scanner end to end. The Java orchestration around the script is covered by the unit tests above.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 3.87%. Comparing base (bc5522e) to head (f93fff6).
⚠️ Report is 5 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff            @@
##            main   #14651      +/-   ##
=========================================
- Coverage   3.88%    3.87%   -0.01%     
=========================================
  Files        493      493              
  Lines      42580    42609      +29     
  Branches    8072     8076       +4     
=========================================
  Hits        1653     1653              
- Misses     40696    40725      +29     
  Partials     231      231              

Flag	Coverage Δ
uitests	3.87% <ø> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[bhouse-nexthop]

Test report: CKS control plane certificate check, rotation and automatic renewal, on live clusters

Summary:

• Expiry check. After the upgrade, the scanner checked every cluster and raised no alerts at the default thresholds. It found two clusters whose stored kubeconfig was older than admin.conf and refreshed both. With the threshold raised above the remaining validity, each cluster alerted once, with one event, and not again on the next pass.
• Rotation through the API. rotateKubernetesClusterCertificates renewed a 3-control-node cluster one node at a time, twice in a row. The API stayed up behind the load balancer, workloads kept running, and the stored kubeconfig was refreshed and worked. On a single control node the API was down for about 38 s.
• Guards. Start and stop were refused during a rotation, and a rotation of a stopped cluster was refused.
• Upgrade. An upgrade refreshed the stored kubeconfig.
• Automatic renewal. It ran on its own for a cluster whose certificates were within the renewal window, with no change to the clock.

Environment

• A CloudStack 24.0 (main) build with this PR, alongside some of my other open PRs. The management server and all three agents were upgraded in place to this build.

• The PR targets main; this is a main-based build.

• Three KVM hosts, Ceph RBD primary storage, one management server.

• CKS clusters on isolated networks:

  • A: 3 control nodes and 2 workers, Kubernetes v1.33.1, with the API behind the network's load balancer;
  • B: 1 control node and 1 worker, v1.33.13;
  • C: 1 control node and 2 workers, v1.33.1;
  • D: a fresh cluster with 1 control node and 1-3 workers, v1.32.13 upgraded to v1.33.13, then deleted.

  A, B and C were created earlier. A and C have since had their certificates renewed by an upgrade, and B by an earlier upgrade made before this build.

• Settings changed for the test and reset afterwards:

  • cloud.kubernetes.cluster.certificate.check.interval=1;
  • cloud.kubernetes.cluster.certificate.expiry.alert.days=400;
  • cloud.kubernetes.cluster.certificate.auto.renew.days=180, for the owner account only.

• API polling. I polled the API through each cluster's public endpoint every 0.5 s with GET /readyz, using a kubeconfig downloaded before the rotation, with a 2 s timeout.

Results

Scenario	Result
First check after the upgrade (defaults)	pass. All 4 clusters checked, with no alerts or events. The stale stored kubeconfig of B and C was refreshed
Threshold above the remaining validity (400)	pass. One alert and one event per cluster; none on the next pass an hour later
Rotation of A (3 control nodes)	pass, 132 s. Node by node, every certificate renewed, back to Running
API during A's rotation	20 of 205 polls failed, in a ~20 s window per node; it was never fully down
Immediate second rotation of A	pass, 134 s. Same pattern, fresh certificates again
Workloads on A	pass. No pod restarted or was rescheduled apart from the CCM (see below); scheduling still worked
Stored kubeconfig after each rotation	pass. Matches the first control node's new admin.conf, and kubectl get nodes works with it
Rotation of B (1 control node)	pass, 45 s; API outage 38.1 s
Start during a rotation (A, twice; B)	pass: refused, "Kubernetes cluster ... is renewing its certificates"
Stop during a rotation (B)	pass: refused, "... is renewing its certificates, stop it once that is done"
Rotation of a Stopped cluster (D)	pass: refused, "Kubernetes cluster ... is not in Running or Alert state"
Upgrade of D, v1.32.13 -> v1.33.13	pass. The stored kubeconfig after the upgrade is the new admin.conf
Automatic renewal (B with real 99-day certificates)	pass. Renewed on the scanner's pass, back to 1-year certificates, event recorded; API outage 37.2 s
Rescue mode (expired certificates)	not covered

First check after the upgrade

• The scanner's first pass ran 10 minutes after the management server started. It recorded certificateExpiryLastCheck for all four clusters. It raised no alert and no KUBERNETES.CLUSTER.CERTIFICATES.EXPIRY event, since everything had more than 360 days left.
• Log, trimmed:

  The stored kubeconfig of Kubernetes cluster B expires on 2027-11-09T15:39:46, before the kubeconfig on the cluster (2027-11-09T20:32:00), refreshing it
  Kubernetes cluster B: first control plane certificate to expire is /root/.kube/config on B-control: Mon Nov 08 18:31:52 UTC 2027, first certificate authority is ca ...: Wed Nov 05 18:31:00 UTC 2036
  Kubernetes cluster A: first control plane certificate to expire is /root/.kube/config on A-control-1: Mon Nov 08 18:36:47 UTC 2027, ...
  The stored kubeconfig of Kubernetes cluster C expires on 2027-11-08T18:34:16, before the kubeconfig on the cluster (2027-11-09T15:59:00), refreshing it
  Kubernetes cluster C: first control plane certificate to expire is /root/.kube/config on C-control: Mon Nov 08 18:34:16 UTC 2027, ...
  

• B and C were upgraded before this build. Their stored kubeconfig still had the admin certificate from before the upgrade. After this pass, getKubernetesClusterConfig returned each cluster's current admin.conf (serials checked against the control node). A's copy was already current and was left alone.
• The first entry to expire on each node was /root/.kube/config, which those older upgrades hadn't refreshed. It's checked and reported as the PR describes.

Alert thresholds

• I set expiry.alert.days=400, above the ~364 days left, and check.interval=1. On the next check of each cluster:

  alert 38  A control plane certificate of Kubernetes cluster B expires in 364 days
  alert 38  A control plane certificate of Kubernetes cluster A expires in 364 days
  alert 38  A control plane certificate of Kubernetes cluster C expires in 363 days
  event KUBERNETES.CLUSTER.CERTIFICATES.EXPIRY WARN  "... admin.conf on A-control-1 expires on Tue Nov 09 23:53:00 UTC 2027.
        They are renewed automatically 30 days before they expire while the cluster is Running, or run rotateKubernetesClusterCertificates."
  

  There was one alert and one event per cluster. D, created meanwhile, alerted once on its own first check. The CAs, 9-10 years out, didn't alert.
• An hour later, the next pass checked A and C again (certificateExpiryLastCheck moved) and raised nothing. B alerted again because its first expiry had moved to a new date (see Automatic renewal), which is the documented "a renewal starts the thresholds over".
• D was upgraded between its two checks, which moved its first expiry by 53 minutes. It didn't alert again, as dates within 7 days count as the same expiry.
• Afterwards I reset both settings to their defaults (6, 60,30).

Rotation of A, with the API polled

• Each run went Running -> RotatingCertificates -> Running, with one control node at a time:

  23:51:11 Rotating the control plane certificates of Kubernetes cluster A
  23:51:55 Renewed the certificates on control node A-control-1
  23:52:39 Renewed the certificates on control node A-control-2
  23:53:22 Renewed the certificates on control node A-control-3
  23:53:23 Rotated the control plane certificates of Kubernetes cluster A
  

• The static pod containers were recreated node by node: etcd, kube-apiserver, controller-manager and scheduler, at 23:51:48, 23:52:31 and 23:53:15.
• Every certificate moved on every node: apiserver.crt, etcd/peer.crt and admin.conf, each with a new serial and a notAfter one year out. /root/.kube/config on each node now matches that node's admin.conf.
• Second run, started right away. It took 134 s and showed the same sequence. All three nodes got fresh certificates again, with new serials and notAfter moved by about 2.5 minutes. It ended Running, with all 5 nodes Ready.
• Polls through the load balancer:

  run 1: 205 polls, 20 failed     run 2: 286 polls, 22 failed     (36 timeouts/connection failures, 6 HTTP 500)
  failures came every ~3.5 s in a ~20 s window per node (23:51:17-23:51:38, 23:52:00-23:52:20, 23:52:43-23:53:04, ...),
  plus one at each node's end
  

  • The API never went fully down. About 1 poll in 3 failed while one node restarted, which is the share the load balancer sends to that node.
  • The PR description says "a few requests ... before its health check takes the node out", with 2 per node in its kind test. Here it was about 6-7 per node, because this load balancer kept sending requests to the restarting node for the whole restart.
• Workloads:
  • A 2-replica test Deployment and every pod in kube-system and kubernetes-dashboard kept their UID and restart count, apart from the static control plane pods.
  • The exception was cloud-controller-manager. Its restart count went from 2 to 4, i.e. it restarted once per run, most likely on losing its leader lease during an API restart.
  • Scaling the test Deployment to 3 afterwards scheduled the new pod at once.
• Stored kubeconfig: after each run its client certificate was the first control node's new admin.conf (77D29D... then 2EDE16...), and kubectl get nodes with it showed 5 Ready.

Rotation of B, with the API polled

• It went RotatingCertificates at 23:57:29 and was back to Running at 23:58:13. The node was renewed at 23:58:12.
• The API was unavailable from 23:57:31.9 to 23:58:10.1, which is 38.1 s. There were 29 timeouts, then 2 HTTP 500s and 1 HTTP 403 while the API server started.
• The certificates moved from the earlier upgrade's (notAfter Nov 9 2027, 20:32) to Nov 9 2027, 23:57. The stored kubeconfig was refreshed and worked.
• Workloads: the same picture as A. The CCM restarted twice during this rotation; nothing else restarted.

Guards

• During A's rotations, startKubernetesCluster failed with Kubernetes cluster A is renewing its certificates.
• During B's rotation:
  • stopKubernetesCluster failed with ... is renewing its certificates, stop it once that is done;
  • startKubernetesCluster failed with ... is renewing its certificates;
  • the cluster stayed RotatingCertificates and finished normally.
• On cluster D, Stopped, rotateKubernetesClusterCertificates failed with Kubernetes cluster D is not in Running or Alert state.

Kubeconfig after an upgrade

• D was upgraded from v1.32.13 to v1.33.13 with upgradeKubernetesCluster, which took 323 s. I read the kubeconfig as soon as the job returned:

  before:  serial=0784099664F87964 notAfter=Nov  9 23:55:49 2027   (= admin.conf before)
  after:   serial=6AC793C3F84605D1 notAfter=Nov 10 00:50:59 2027   (= admin.conf and /root/.kube/config after the upgrade)
  

• kubectl with the new kubeconfig showed both nodes Ready on v1.33.13.

Automatic renewal, without changing the clock

• auto.renew.days is capped at 180, and every cluster had about 364 days left. So I gave B real certificates with a 100-day validity.
• I ran kubeadm certs renew all on B's control node, with the cluster's own ClusterConfiguration and certificateValidityPeriod: 2400h0m0s added. This went to a temporary file; the cluster's kubeadm-config ConfigMap was left unchanged. check-expiration then showed 99 days for every leaf certificate.
• I set auto.renew.days=180 for the owner account. On B's next check:

  01:47:43 Renewing the certificates of Kubernetes cluster B automatically, admin.conf on B-control expires on 2027-02-18T00:55:00
  01:47:43 Rotating the control plane certificates of Kubernetes cluster B          [Kubernetes-Cluster-Certificate-Rotation thread]
  01:48:24 Renewed the certificates on control node B-control
  01:48:25 Rotated the control plane certificates of Kubernetes cluster B
  event KUBERNETES.CLUSTER.CERTIFICATES.ROTATE INFO "Automatically renewed the control plane certificates of Kubernetes cluster B.
        Kubeconfigs downloaded before keep working only until their old certificate expires; download it again ..."
  

• The certificates were valid for a year again (Nov 10, 2027). The stored kubeconfig was the new admin.conf, and the cluster went RotatingCertificates -> Running.
• The API outage was 37.2 s. The same pass raised the 99-day expiry alert, as the 400-day threshold was still set.
• One more API outage during this wait isn't from the PR. It lasted 66 s, at 01:01, about 45 minutes before the renewal. Nothing was rotating then; the control node's kubelet logged etcdserver: request timed out, and Ceph reported BlueStore slow operations at the time.
• Afterwards I reset the account setting.

Result: pass.

• The check finds and reports expiry on every control node, alerts once per threshold, and refreshes stale stored kubeconfigs.
• The rotation renews the control nodes one at a time and keeps the cluster usable.
• The guards hold, the upgrade refreshes the kubeconfig, and the automatic renewal runs on its own.

Not covered

• Rescue mode (certificates already expired, etcd without quorum). Testing it live would have meant letting a cluster's certificates expire and leaving it down for that time, so I didn't run it here.
• A failed rotation, interrupted rotations, and a management server restart during one.
• External etcd clusters, and clusters older than 1.20.
• Several management servers.

Notes

• With a 3-node control plane behind the CloudStack network load balancer, expect about a third of API requests to fail for ~20 s per node. That's longer than the description's "a few requests". It might be worth saying so in the restart impact section, or checking whether the load balancer can health-check port 6443.
• The CCM restarts once or twice per rotation (on losing its leader lease). It recovers on its own.

[DaanHoogland]

@bhouse-nexthop , you reported this one to be present on 4.22. care to rebase?

[bhouse-nexthop]

@DaanHoogland

@bhouse-nexthop , you reported this one to be present on 4.22. care to rebase?

I could but I thought 4.22 didn't take new features. This adds a new API for this and whatnot so I was just targeting 4.24 for this. I've actually got a huge queue of CKS fixes for 4.24.