I ve been trying to get the automated upgrade to work but so Rancher Users #k3s

I’ve been trying to get the automated upgrade to w...

thankful-policeman-29475

10/23/2023, 7:02 PM

I’ve been trying to get the automated upgrade to work, but so far no joy. My master node updated, but with the worker nodes I end up with a job that looks like this:

Copy code

$ kubectl describe jobs -n system-upgrade 
Name:                     apply-agent-plan-on-k3s-sd-03-with-0e4e3f4e3f8b1e811d8410-45090
Namespace:                system-upgrade
Selector:                 batch.kubernetes.io/controller-uid=6bcc362a-6613-48d0-9d1a-d12611ba8701
Labels:                   objectset.rio.cattle.io/hash=a7f25c30b32cf9ae48bffb9806c47e347a1b64f3
                          plan.upgrade.cattle.io/agent-plan=0e4e3f4e3f8b1e811d841099cb49e4712b93833bee0604115b9a141c
                          upgrade.cattle.io/controller=system-upgrade-controller
                          upgrade.cattle.io/node=k3s-sd-03
                          upgrade.cattle.io/plan=agent-plan
                          upgrade.cattle.io/version=v1.27.6-k3s1
Annotations:              batch.kubernetes.io/job-tracking: 
                          objectset.rio.cattle.io/applied:
                            H4sIAAAAAAAA/+xYUW/iOBD+Kyc/JzQpaUuQ7oEr7C3aLaDS3dNqhSrHmYAPx87ZDi1C/PfTOAHClna7e/ewD1XVFif2+PPM941n2JAcLE2ppaS7IVRKZanlShocquRvYNaAbWmuWo...
                          objectset.rio.cattle.io/id: system-upgrade-controller
                          objectset.rio.cattle.io/owner-gvk: upgrade.cattle.io/v1, Kind=Plan
                          objectset.rio.cattle.io/owner-name: agent-plan
                          objectset.rio.cattle.io/owner-namespace: system-upgrade
                          upgrade.cattle.io/ttl-seconds-after-finished: 900
Controlled By:            Plan/agent-plan
Parallelism:              1
Completions:              1
Completion Mode:          NonIndexed
Start Time:               Mon, 23 Oct 2023 12:55:55 -0500
Active Deadline Seconds:  900s
Pods Statuses:            0 Active (0 Ready) / 0 Succeeded / 1 Failed
Pod Template:
  Labels:           batch.kubernetes.io/controller-uid=6bcc362a-6613-48d0-9d1a-d12611ba8701
                    batch.kubernetes.io/job-name=apply-agent-plan-on-k3s-sd-03-with-0e4e3f4e3f8b1e811d8410-45090
                    controller-uid=6bcc362a-6613-48d0-9d1a-d12611ba8701
                    job-name=apply-agent-plan-on-k3s-sd-03-with-0e4e3f4e3f8b1e811d8410-45090
                    plan.upgrade.cattle.io/agent-plan=0e4e3f4e3f8b1e811d841099cb49e4712b93833bee0604115b9a141c
                    upgrade.cattle.io/controller=system-upgrade-controller
                    upgrade.cattle.io/node=k3s-sd-03
                    upgrade.cattle.io/plan=agent-plan
                    upgrade.cattle.io/version=v1.27.6-k3s1
  Service Account:  system-upgrade
  Init Containers:
   prepare:
    Image:      rancher/k3s-upgrade:v1.27.6-k3s1
    Port:       <none>
    Host Port:  <none>
    Args:
      prepare
      server-plan
    Environment:
      SYSTEM_UPGRADE_NODE_NAME:             (v1:spec.nodeName)
      SYSTEM_UPGRADE_POD_NAME:              (v1:metadata.name)
      SYSTEM_UPGRADE_POD_UID:               (v1:metadata.uid)
      SYSTEM_UPGRADE_PLAN_NAME:            agent-plan
      SYSTEM_UPGRADE_PLAN_LATEST_HASH:     0e4e3f4e3f8b1e811d841099cb49e4712b93833bee0604115b9a141c
      SYSTEM_UPGRADE_PLAN_LATEST_VERSION:  v1.27.6-k3s1
    Mounts:
      /host from host-root (rw)
      /run/system-upgrade/pod from pod-info (ro)
   cordon:
    Image:      rancher/kubectl:v1.25.4
    Port:       <none>
    Host Port:  <none>
    Args:
      cordon
      k3s-sd-03
    Environment:
      SYSTEM_UPGRADE_NODE_NAME:             (v1:spec.nodeName)
      SYSTEM_UPGRADE_POD_NAME:              (v1:metadata.name)
      SYSTEM_UPGRADE_POD_UID:               (v1:metadata.uid)
      SYSTEM_UPGRADE_PLAN_NAME:            agent-plan
      SYSTEM_UPGRADE_PLAN_LATEST_HASH:     0e4e3f4e3f8b1e811d841099cb49e4712b93833bee0604115b9a141c
      SYSTEM_UPGRADE_PLAN_LATEST_VERSION:  v1.27.6-k3s1
    Mounts:
      /host from host-root (rw)
      /run/system-upgrade/pod from pod-info (ro)
  Containers:
   upgrade:
    Image:      rancher/k3s-upgrade:v1.27.6-k3s1
    Port:       <none>
    Host Port:  <none>
    Environment:
      SYSTEM_UPGRADE_NODE_NAME:             (v1:spec.nodeName)
      SYSTEM_UPGRADE_POD_NAME:              (v1:metadata.name)
      SYSTEM_UPGRADE_POD_UID:               (v1:metadata.uid)
      SYSTEM_UPGRADE_PLAN_NAME:            agent-plan
      SYSTEM_UPGRADE_PLAN_LATEST_HASH:     0e4e3f4e3f8b1e811d841099cb49e4712b93833bee0604115b9a141c
      SYSTEM_UPGRADE_PLAN_LATEST_VERSION:  v1.27.6-k3s1
    Mounts:
      /host from host-root (rw)
      /run/system-upgrade/pod from pod-info (ro)
  Volumes:
   host-root:
    Type:          HostPath (bare host directory volume)
    Path:          /
    HostPathType:  Directory
   pod-info:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.labels -> labels
      metadata.annotations -> annotations
Events:
  Type     Reason            Age   From            Message
  ----     ------            ----  ----            -------
  Normal   SuccessfulCreate  16m   job-controller  Created pod: apply-agent-plan-on-k3s-sd-03-with-0e4e3f4e3f8b1e811d8410-gswd7
  Normal   SuccessfulDelete  74s   job-controller  Deleted pod: apply-agent-plan-on-k3s-sd-03-with-0e4e3f4e3f8b1e811d8410-gswd7
  Warning  DeadlineExceeded  74s   job-controller  Job was active longer than specified deadline

The pod isn’t any help either:

Copy code

$ kubectl describe pod -n system-upgrade apply-agent-plan-on-k3s-sd-03-with-0e4e3f4e3f8b1e811d8410-gswd7 
Name:             apply-agent-plan-on-k3s-sd-03-with-0e4e3f4e3f8b1e811d8410-gswd7
Namespace:        system-upgrade
Priority:         0
Service Account:  system-upgrade
Node:             k3s-sd-03/23.29.118.37
Start Time:       Mon, 23 Oct 2023 12:55:55 -0500
Labels:           batch.kubernetes.io/controller-uid=6bcc362a-6613-48d0-9d1a-d12611ba8701
                  batch.kubernetes.io/job-name=apply-agent-plan-on-k3s-sd-03-with-0e4e3f4e3f8b1e811d8410-45090
                  controller-uid=6bcc362a-6613-48d0-9d1a-d12611ba8701
                  job-name=apply-agent-plan-on-k3s-sd-03-with-0e4e3f4e3f8b1e811d8410-45090
                  plan.upgrade.cattle.io/agent-plan=0e4e3f4e3f8b1e811d841099cb49e4712b93833bee0604115b9a141c
                  upgrade.cattle.io/controller=system-upgrade-controller
                  upgrade.cattle.io/node=k3s-sd-03
                  upgrade.cattle.io/plan=agent-plan
                  upgrade.cattle.io/version=v1.27.6-k3s1
Annotations:      <none>
Status:           Pending
IP:               23.29.118.37
IPs:
  IP:           23.29.118.37
  IP:           2602:ff16:3:0:1:384:0:1
Controlled By:  Job/apply-agent-plan-on-k3s-sd-03-with-0e4e3f4e3f8b1e811d8410-45090
Init Containers:
  prepare:
    Container ID:  <containerd://69e45fa8c760ae7f8387f306cbeabb4a9ef231886655c7835f11051f1b5a549>6
    Image:         rancher/k3s-upgrade:v1.27.6-k3s1
    Image ID:      docker.io/rancher/k3s-upgrade@sha256:6c4543ecde336df20a21f88e5e84399f923bdb3f9bbdc7e815cfdbca643ec50a
    Port:          <none>
    Host Port:     <none>
    Args:
      prepare
      server-plan
    State:          Running
      Started:      Mon, 23 Oct 2023 12:55:57 -0500
    Ready:          False
    Restart Count:  0
    Environment:
      SYSTEM_UPGRADE_NODE_NAME:             (v1:spec.nodeName)
      SYSTEM_UPGRADE_POD_NAME:             apply-agent-plan-on-k3s-sd-03-with-0e4e3f4e3f8b1e811d8410-gswd7 (v1:metadata.name)
      SYSTEM_UPGRADE_POD_UID:               (v1:metadata.uid)
      SYSTEM_UPGRADE_PLAN_NAME:            agent-plan
      SYSTEM_UPGRADE_PLAN_LATEST_HASH:     0e4e3f4e3f8b1e811d841099cb49e4712b93833bee0604115b9a141c
      SYSTEM_UPGRADE_PLAN_LATEST_VERSION:  v1.27.6-k3s1
    Mounts:
      /host from host-root (rw)
      /run/system-upgrade/pod from pod-info (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-c42lp (ro)
  cordon:
    Container ID:  
    Image:         rancher/kubectl:v1.25.4
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Args:
      cordon
      k3s-sd-03
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:
      SYSTEM_UPGRADE_NODE_NAME:             (v1:spec.nodeName)
      SYSTEM_UPGRADE_POD_NAME:             apply-agent-plan-on-k3s-sd-03-with-0e4e3f4e3f8b1e811d8410-gswd7 (v1:metadata.name)
      SYSTEM_UPGRADE_POD_UID:               (v1:metadata.uid)
      SYSTEM_UPGRADE_PLAN_NAME:            agent-plan
      SYSTEM_UPGRADE_PLAN_LATEST_HASH:     0e4e3f4e3f8b1e811d841099cb49e4712b93833bee0604115b9a141c
      SYSTEM_UPGRADE_PLAN_LATEST_VERSION:  v1.27.6-k3s1
    Mounts:
      /host from host-root (rw)
      /run/system-upgrade/pod from pod-info (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-c42lp (ro)
Containers:
  upgrade:
    Container ID:   
    Image:          rancher/k3s-upgrade:v1.27.6-k3s1
    Image ID:       
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:
      SYSTEM_UPGRADE_NODE_NAME:             (v1:spec.nodeName)
      SYSTEM_UPGRADE_POD_NAME:             apply-agent-plan-on-k3s-sd-03-with-0e4e3f4e3f8b1e811d8410-gswd7 (v1:metadata.name)
      SYSTEM_UPGRADE_POD_UID:               (v1:metadata.uid)
      SYSTEM_UPGRADE_PLAN_NAME:            agent-plan
      SYSTEM_UPGRADE_PLAN_LATEST_HASH:     0e4e3f4e3f8b1e811d841099cb49e4712b93833bee0604115b9a141c
      SYSTEM_UPGRADE_PLAN_LATEST_VERSION:  v1.27.6-k3s1
    Mounts:
      /host from host-root (rw)
      /run/system-upgrade/pod from pod-info (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-c42lp (ro)
Conditions:
  Type              Status
  Initialized       False 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  host-root:
    Type:          HostPath (bare host directory volume)
    Path:          /
    HostPathType:  Directory
  pod-info:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.labels -> labels
      metadata.annotations -> annotations
  kube-api-access-c42lp:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  14m   default-scheduler  Successfully assigned system-upgrade/apply-agent-plan-on-k3s-sd-03-with-0e4e3f4e3f8b1e811d8410-gswd7 to k3s-sd-03
  Normal  Pulling    14m   kubelet            Pulling image "rancher/k3s-upgrade:v1.27.6-k3s1"
  Normal  Pulled     14m   kubelet            Successfully pulled image "rancher/k3s-upgrade:v1.27.6-k3s1" in 272.726271ms (272.760393ms including waiting)
  Normal  Created    14m   kubelet            Created container prepare
  Normal  Started    14m   kubelet            Started container prepare

6 Views

Open in Slack

Previous Next