CKS: node drain on scale-down and upgrade fails too easily, and a failed upgrade can't be resumed #14610

problem

Both scale-down and upgrade of a CloudStack-managed Kubernetes cluster drain a node before they remove or upgrade it, and both handle a drain that fails, or doesn't finish quickly, badly.

Scale-down (KubernetesClusterScaleWorker.removeKubernetesClusterNode):

• When kubectl drain exits non-zero, for example because a PodDisruptionBudget doesn't allow an eviction yet, it gives up at once with no retry.
• When it does retry, after an exception, each attempt is counted twice.
• The node is left cordoned.
• The cluster goes to Alert, although no node was removed and the cluster still works.

Upgrade (KubernetesClusterUpgradeWorker):

• A failed drain is retried at once, with no wait for the eviction to become possible, and on the last failure the node is left cordoned.
• Only the first control node is waited on to be Ready. The next node can be drained while the previous one is still coming back.
• The script for node i is run over SSH on port sshPort + i. The port-forwarding rules aren't created in the order of the upgrade's VM list: manually upgraded nodes are left out of that list, and nodes added later get the next free port. So the script can run on the wrong node.

Both: kubectl drain runs without --timeout, so a fixed 60s SSH timeout cuts it off rather than a deadline of its own.

After a failed upgrade the cluster is left in Alert. upgradeKubernetesCluster only accepts a Running cluster, so the upgrade can't be retried or finished, although the nodes already upgraded are known (kubernetesnodeversion in the cluster's VM list).

versions

4.22 (the code is the same on main).

The steps to reproduce the bug

1. Create a CKS cluster with 2 or more worker nodes.
2. Deploy a workload with a PodDisruptionBudget that allows no disruption (for example minAvailable equal to the replica count) on the worker that a scale-down would remove.
3. Scale the cluster down by one worker.
4. The drain fails at once, the cluster goes to Alert, and the node stays cordoned. No node is removed.

For the upgrade: let the drain or the upgrade script fail on any node after the first one. The cluster goes to Alert with some nodes at the new version, and upgradeKubernetesCluster refuses to run again because the cluster isn't Running.

What to do about it?

• Use one drain helper for both paths:
  • each attempt counts once, and a non-zero exit or an exception is retried after a wait;
  • kubectl drain --timeout comes from a setting, and the SSH wait is derived from it;
  • on the last failure the node is uncordoned and the drain output is reported.
• On scale-down, cordon and drain every node to remove before removing any. If one can't be drained, uncordon them all and leave the cluster as it was rather than in Alert.
• On upgrade, wait for every node to be Ready, and reach each node through its own SSH port-forwarding rule.
• Let upgradeKubernetesCluster be retried on a cluster in Alert after an upgrade to the same version failed part way, skipping the nodes already upgraded.