https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# harvester
a
This message was deleted.
b
I restarted the harvester pods, but that didn't generate a promotion job. I checked that there wasn't a worker only label and I even tried to add a management-node:true label (not accurate I followed the docs on exact syntax) and re-rolled out pods, but didn't see any change or promotion.
No stale records in machines.cluster capi:
Copy code
$ kc get <http://machines.cluster.x-k8s.io|machines.cluster.x-k8s.io> -n fleet-local
NAME                  CLUSTER   NODENAME   PROVIDERID    PHASE     AGE    VERSION
custom-48909be85546   local     gpu1       <rke2://gpu1>   Running   195d   
custom-561bc75fdd0f   local     gpu3       <rke2://gpu3>   Running   195d   
custom-6f8b0ffd9f65   local     gpu2       <rke2://gpu2>   Running   195d
Ugh... this has been cursed. I think I'm gonna have to give up and just re-install
ugh... still not right.
Copy code
NAME   STATUS   ROLES                       AGE    VERSION
gpu1   Ready    <none>                      41m    v1.34.3+rke2r3
gpu2   Ready    control-plane,etcd,master   195d   v1.34.3+rke2r3
gpu3   Ready    control-plane,etcd,master   196d   v1.34.3+rke2r3
I still can't comprehend why it's not working. I thought maybe the fact that lsblk was showing the root device as a spinner (it's a virtual disk on a raid with SSDs), but it doesn't seem to be that...
I tried a bunch of things, including removing nodeconfigs that seemed stale, custommachine objects, machines, and restarting most of the ds and deployments. After this second node re-install, I'm still at a loss. I marked the node as a Management Node in the 
1.7.1
installer but it didn't get promoted. Here's a new support bundle.
supportbundle_be68a194-1d47-4363-ba75-8ed2b9f25cbd_2026-04-22T15-20-01Z.zip
I read somewhere that the promotion was triggered by node removal, so I had another node join, then I removed it (so there was still 3 in the cluster at the time) but that didn't trigger the promotion job either.
Ah that was a nightmare to figure out....
I'll document this for other in case they're digging through logs. The biggest thing was finding the 
promote.sh
that's used for node promotion. It was in a configmap that I think got created for something else: 
kubectl get configmap harvester-helpers -n harvester-system -oyaml
Copy code
promote.sh: |-
    LONGHORN_NAMESPACE="longhorn-system"
    KUBECTL="/host/$(readlink /host/var/lib/rancher/rke2/bin)/kubectl"
    YQ="/host/usr/bin/yq"
    ROLE_LABELS="<http://rke.cattle.io/control-plane-role=true|rke.cattle.io/control-plane-role=true> <http://rke.cattle.io/etcd-role=true|rke.cattle.io/etcd-role=true>"
    ETCD_ONLY=false
    if [[ -n "$1" && $1 == "<http://rke.cattle.io/etcd-role=true|rke.cattle.io/etcd-role=true>" ]]; then
      ETCD_ONLY=true
      ROLE_LABELS=$1
    fi

    case $ROLE_LABELS in
      "<http://rke.cattle.io/control-plane-role=true|rke.cattle.io/control-plane-role=true> <http://rke.cattle.io/etcd-role=true|rke.cattle.io/etcd-role=true>" | "<http://rke.cattle.io/etcd-role=true|rke.cattle.io/etcd-role=true>")
    ;;
      *)
      echo "ROLE $ROLE_LABELS is not supported."
      exit 1
    ;;
    esac

    get_machine_from_node() {
      $KUBECTL get node $HARVESTER_PROMOTE_NODE_NAME -o jsonpath='{.metadata.annotations.cluster\.x-k8s\.io/machine}'
    }

    # Wait for rancher-webhook ready. It's default to one replica.
    # Otherwise lebeling capi resources later might fail.
    $KUBECTL rollout status --watch=true deployment rancher-webhook -n cattle-system
    # <https://github.com/rancher/webhook/blob/436e359b136b633cb1a6fa7cdedbed4d74821bdb/pkg/server/server.go#L114>
    sleep 20

    CUSTOM_MACHINE=$(get_machine_from_node)
    until [ -n "$CUSTOM_MACHINE" ]
    do
      echo Waiting for custom machine label of $HARVESTER_PROMOTE_NODE_NAME ...
      sleep 2
      CUSTOM_MACHINE=$(get_machine_from_node)
    done

    until $KUBECTL get <http://machines.cluster.x-k8s.io|machines.cluster.x-k8s.io> $CUSTOM_MACHINE -n fleet-local &> /dev/null
    do
      echo Waiting for custom machine $CUSTOM_MACHINE...
      sleep 2
    done

    PLAN_SECRET="${CUSTOM_MACHINE}-machine-plan"
    until $KUBECTL get secret $PLAN_SECRET -n fleet-local &> /dev/null
    do
      echo Waiting for machine plan of $CUSTOM_MACHINE...
      sleep 2
    done

    until $KUBECTL get <http://rkebootstraps.rke.cattle.io|rkebootstraps.rke.cattle.io> "${CUSTOM_MACHINE}" -n fleet-local &> /dev/null
    do
      echo Waiting for bootstrap object of $CUSTOM_MACHINE...
      sleep 2
    done


    VIP=$($KUBECTL get configmap vip -n harvester-system -o=jsonpath='{.data.ip}')
    cat > /host/etc/rancher/rke2/config.yaml.d/90-harvester-server.yaml <<EOF
    cni: multus,canal
    cluster-cidr: 10.52.0.0/16
    service-cidr: 10.53.0.0/16
    cluster-dns: 10.53.0.10
    tls-san:
      - $VIP
    audit-policy-file: /etc/rancher/rke2/config.yaml.d/92-harvester-kube-audit-policy.yaml
    EOF


    # Disable snapshot-controller related charts because we manage them in Harvester.
    # RKE2 enables these charts by default after v1.25.7 (<https://github.com/rancher/rke2/releases/tag/v1.25.7%2Brke2r1>)
    cat > /host/etc/rancher/rke2/config.yaml.d/40-disable-charts.yaml <<EOF
    disable:
    - rke2-snapshot-controller
    - rke2-snapshot-controller-crd
    - rke2-snapshot-validation-webhook
    EOF

    # make sure we should not have any related label/taint on the node
    if [[ $ETCD_ONLY == false ]]; then
      found=$($KUBECTL get node $HOSTNAME -o yaml | $YQ '.spec.taints[] | select (.effect == "NoSchedule" and .key == "<http://node-role.kubernetes.io/etcd=true|node-role.kubernetes.io/etcd=true>") | .effect')
      if [[ -n $found ]]
      then
        $KUBECTL taint nodes $HOSTNAME <http://node-role.kubernetes.io/etcd=true:NoExecute-|node-role.kubernetes.io/etcd=true:NoExecute->
      fi
      $KUBECTL label --overwrite nodes $HOSTNAME <http://node-role.harvesterhci.io/witness-|node-role.harvesterhci.io/witness->
    fi

    # For how to promote nodes, see: <https://github.com/rancher/rancher/issues/36480#issuecomment-1039253499>
    $KUBECTL label --overwrite -n fleet-local <http://machines.cluster.x-k8s.io|machines.cluster.x-k8s.io> $CUSTOM_MACHINE $ROLE_LABELS
    $KUBECTL label --overwrite -n fleet-local <http://machines.cluster.x-k8s.io|machines.cluster.x-k8s.io> $CUSTOM_MACHINE <http://cluster.x-k8s.io/control-plane=true|cluster.x-k8s.io/control-plane=true>
    $KUBECTL label --overwrite -n fleet-local secret $PLAN_SECRET $ROLE_LABELS
    $KUBECTL label --overwrite -n fleet-local <http://rkebootstraps.rke.cattle.io|rkebootstraps.rke.cattle.io> $CUSTOM_MACHINE $ROLE_LABELS

    kickout_longhorn_node()
    {
      target=$1
      found=$($KUBECTL get <http://nodes.longhorn.io|nodes.longhorn.io> -n $LONGHORN_NAMESPACE |grep -q $target && echo true || echo false)
      if [[ $found == true ]]; then
        echo "Found longhorn node $target, kicking it out..."
        $KUBECTL delete <http://nodes.longhorn.io|nodes.longhorn.io> $target -n $LONGHORN_NAMESPACE
      fi
    }

    while true
    do
      if [[ $ETCD_ONLY == true ]]; then
        ETCD_STATE=$($KUBECTL get node $HOSTNAME -o go-template=$'{{index .metadata.labels "<http://node-role.kubernetes.io/etcd|node-role.kubernetes.io/etcd>"}}\n' || true)

        if [ "$ETCD_STATE" = "true" ]; then
          $KUBECTL taint nodes $HOSTNAME <http://node-role.kubernetes.io/etcd=true:NoExecute|node-role.kubernetes.io/etcd=true:NoExecute> --overwrite
          $KUBECTL patch managedchart harvester -n fleet-local --type=json -p='[{"op":"replace", "path":"/spec/values/replicas", "value": 2}]'
          $KUBECTL patch managedchart harvester -n fleet-local --type=json -p='[{"op":"replace", "path":"/spec/values/webhook/replicas", "value": 2}]'
          $KUBECTL annotate --overwrite deployment rancher -n cattle-system <http://management.cattle.io/scale-available=%222%22|management.cattle.io/scale-available="2">
          kickout_longhorn_node $HOSTNAME
          break
        fi

      else
        CONTROL_PLANE=$($KUBECTL get node $HOSTNAME -o go-template=$'{{index .metadata.labels "<http://node-role.kubernetes.io/control-plane|node-role.kubernetes.io/control-plane>"}}\n' || true)

        if [ "$CONTROL_PLANE" = "true" ]; then
          break
        fi
      fi
      echo Waiting for promotion...
      sleep 2
    done
Some dummy down'd steps I took to get things back: • Get the machine ID for the node you're trying to promote (gpu1 in this example) : 
MACHINE_ID=$(kubectl get node gpu1 -o jsonpath='{.metadata.annotations.cluster\.x-k8s\.io/machine}') ; echo $MACHINE_ID
• There's some files you need to create on that node. Copy over them from an existing good node: 
/etc/rancher/rke2/config.yaml.d/90-harvester-server.yaml
and 
/etc/rancher/rke2/config.yaml.d/40-disable-charts.yaml
• Label the Machine object 
kubectl label --overwrite -n fleet-local <http://machines.cluster.x-k8s.io|machines.cluster.x-k8s.io> $MACHINE_ID   <http://rke.cattle.io/control-plane-role=true|rke.cattle.io/control-plane-role=true>
<http://rke.cattle.io/etcd-role=true|rke.cattle.io/etcd-role=true>
<http://cluster.x-k8s.io/control-plane=true|cluster.x-k8s.io/control-plane=true>
• Label the Machine Plan Secret 
kubectl label --overwrite -n fleet-local secret ${MACHINE_ID}-machine-plan  <http://rke.cattle.io/control-plane-role=true|rke.cattle.io/control-plane-role=true>
<http://rke.cattle.io/etcd-role=true|rke.cattle.io/etcd-role=true>
• Label the RKE Bootstrap object 
kubectl label --overwrite -n fleet-local <http://rkebootstraps.rke.cattle.io|rkebootstraps.rke.cattle.io> $MACHINE_ID <http://rke.cattle.io/control-plane-role=true|rke.cattle.io/control-plane-role=true>
<http://rke.cattle.io/etcd-role=true|rke.cattle.io/etcd-role=true>
• Wait a few minutes and then add the 
master
role to the node and verify they all look the same: 
kubectl label node gpu1 <http://node-role.kubernetes.io/master=true|node-role.kubernetes.io/master=true> ; kubectl get nodes
• profit.
🎉 1
10 Views
Open in Slack
PreviousNext
Powered by