That of course is easier said than done but it's one of those things that you'll pay of the hard work today for an easier tomorrow

But you replacing node is not an option, I have some scripts for doing rolling OS upgrades.

Note: I built this for my lab so it does things like sleep between nodes for 900 seconds to avoid pods moving around like crazy during patching and to allow longhorn to fully rebuild between nodes.

#!/bin/bash

while getopts "c:h" opt; do
  case $opt in
    c)
      CLUSTER="${OPTARG}"
      ;;
    h)
      help && exit 0
      ;;
    :)
      echo "Option -$OPTARG requires an argument."
      exit 1
      ;;
    *)
      help && exit 0
  esac
done

if [[ -z "${CLUSTER}" ]]; then
  echo "Please specify a cluster name."
  exit 1
fi

export KUBECONFIG=~/.kube/${CLUSTER}
kubeconfig=~/.kube/${CLUSTER}
cd ~/scripts/rolling-patching/

check_ssh() {
  echo "Checking ${server}"
  until ssh -q -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@${server} 'uptime' > /dev/null
  do
    echo "Trying again..."
    sleep 1
  done
}

echo "Starting patching..."
for server in `kubectl get nodes -o name | awk -F '/' '{print $2}'`
do
  i=0
  if ping -c 1 $server
  then
    echo "Server is pingable..."
    scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no ./90forceyes root@${server}:/etc/apt/apt.conf.d/
    scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no ./release-upgrades root@${server}:/etc/update-manager/release-upgrades
    echo "Draining node..."
    kubectl --kubeconfig ${kubeconfig} drain --delete-emptydir-data --ignore-daemonsets ${server}
    echo "Running apt update and upgrade"
    ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@${server} 'export DEBIAN_FRONTEND=noninteractive; apt-get update && apt-get -o Dpkg::Options::="--force-confold" -o Dpkg::Options::=--force-confdef upgrade -q -y --allow-downgrades --allow-remove-essential --allow-change-held-packages && apt-get -o Dpkg::Options::="--force-confold" -o Dpkg::Options::=--force-confdef dist-upgrade -q -y --allow-downgrades --allow-remove-essential --allow-change-held-packages && reboot'
    sleep 60
    check_ssh
    echo "Running do-release-upgrade"
    ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@${server} 'export DEBIAN_FRONTEND=noninteractive; do-release-upgrade -f DistUpgradeViewNonInteractive; reboot'
    sleep 450
    echo "Uncordon node..,"
    kubectl --kubeconfig ${kubeconfig} uncordon ${server}
    echo "Sleeping..."
    sleep 900
  else
    echo "Skipping..."
  fi
done

@happy-wire-88980 why is replacing node in my case is not an option? Is it because I only have 3 nodes in my admin cluster? If that's the issue here, what's the correct admin cluster setup to use the node rehydration? Since I'm working with the rke terraform provider and VM templates, I'm pretty flexible and have the opportunity to deploy any number of nodes to integrate them to my admin cluster.

We don't want to remove all 3 old nodes at once because that can lead to a split brain in etcd