https://rancher.com/ logo
Join Slack
Powered by
Sadly, my 2 node v1.4.2 to v1.5.0 upgrade has been...
# harvester
s
Sadly, my 2 node v1.4.2 to v1.5.0 upgrade has been stuck for >24 hours.
Following the troubleshooting page.
Copy code
# kubectl get jobs -n harvester-system -l <http://harvesterhci.io/upgradeComponent=manifest|harvesterhci.io/upgradeComponent=manifest>
NAME                                 STATUS    COMPLETIONS   DURATION   AGE
hvst-upgrade-bckj7-apply-manifests   Running   0/1           44h        44h
and
Copy code
...
longhorn nodeDrainPolicy has been set in managedchart, do not patch again
true
longhorn detachManuallyAttachedVolumesWhenCordoned has been set in managedchart, do not patch again
longhorn related config
defaultSettings:
  defaultDataPath: /var/lib/harvester/defaultdisk
  detachManuallyAttachedVolumesWhenCordoned: true
  nodeDrainPolicy: allow-if-replica-is-stopped
  taintToleration: <http://kubevirt.io/drain:NoSchedule|kubevirt.io/drain:NoSchedule>
enabled: true
<http://managedchart.management.cattle.io/harvester|managedchart.management.cattle.io/harvester> configured
<http://managedchart.management.cattle.io/harvester|managedchart.management.cattle.io/harvester> patched
<http://managedchart.management.cattle.io/harvester-crd|managedchart.management.cattle.io/harvester-crd> patched
Waiting for ManagedChart fleet-local/harvester from generation 8
Target version: 1.5.0, Target state: ready
Current version: 1.5.0, Current state: OutOfSync, Current generation: 10
Sleep for 5 seconds to retry
Current version: 1.5.0, Current state: WaitApplied, Current generation: 10
Sleep for 5 seconds to retry
Current version: 1.5.0, Current state: WaitApplied, Current generation: 10
Sleep for 5 seconds to retry
Current version: 1.5.0, Current state: WaitApplied, Current generation: 10
Sleep for 5 seconds to retry
Current version: 1.5.0, Current state: WaitApplied, Current generation: 10
Sleep for 5 seconds to retry
... (repeat ad-nauseam)
There doesn't seem to be any remediation described for 
Phase 3: Upgrade the system services
f
s
Hi @sticky-summer-13450, Could you generate the SB for investigation? Also, could you get the following command result?
Copy code
$ kubectl get bundle -A
s
Thank you @salmon-city-57654.
Copy code
# kubectl get bundle -A
NAMESPACE     NAME                                          BUNDLEDEPLOYMENTS-READY   STATUS
fleet-local   fleet-agent-local                             1/1                       
fleet-local   local-managed-system-agent                    1/1                       
fleet-local   mcc-harvester                                 0/1                       Modified(1) [Cluster fleet-local/local]; <http://cdi.cdi.kubevirt.io|cdi.cdi.kubevirt.io> cdi missing
fleet-local   mcc-harvester-crd                             1/1                       
fleet-local   mcc-hvst-upgrade-bckj7-upgradelog-operator    0/1                       ErrApplied(1) [Cluster fleet-local/local: execution error at (rancher-logging/templates/validate-install-crd.yaml:26:7): Required CRDs are missing. Please install the corresponding CRD chart before installing this chart.]
fleet-local   mcc-local-managed-system-upgrade-controller   1/1                       
fleet-local   mcc-rancher-logging-crd                       1/1                       
fleet-local   mcc-rancher-monitoring-crd                    1/1
supportbundle_5351197f-4e80-4b39-99a7-44607b6fdbd8_2025-04-28T17-24-58Z.zip
Did these details help in discovering why this upgrade has stalled?
s
Looks like the cdi CRD is already added, but I am investigating why the fleet is still showing the warning message 
<http://cdi.cdi.kubevirt.io|cdi.cdi.kubevirt.io> cdi missing
👍 1
cc @ancient-pizza-13099, did you ever saw this similar situation? (I mean, the fleet-agent did not update to date after we applied the new CRDs?
Hi @sticky-summer-13450, Could you help to show the following command result?
Copy code
$ helm history harvester -n harvester-system
s
Sure:
Copy code
# helm history harvester -n harvester-system
REVISION	UPDATED                 	STATUS    	CHART          	APP VERSION	DESCRIPTION     
5       	Sat Apr 26 13:02:24 2025	superseded	harvester-1.4.2	v1.4.2     	Rollback to 3   
6       	Sat Apr 26 13:03:11 2025	deployed  	harvester-1.5.0	v1.5.0     	Upgrade complete
s
Hmm, could you try to delete (restart) the fleet-agent pod to let fleet recheck? It used the namespace 
cattle-fleet-local-system
s
sure:
Copy code
# kubectl get pod -n cattle-fleet-local-system
NAME                           READY   STATUS    RESTARTS   AGE
fleet-agent-79c87b85c5-9lcmk   1/1     Running   0          9d

# kubectl delete pod fleet-agent-79c87b85c5-9lcmk -n cattle-fleet-local-system
pod "fleet-agent-79c87b85c5-9lcmk" deleted
The logs from the 
hvst-upgrade-bckj7-apply-manifests
job are continuing to output:
Copy code
Sleep for 5 seconds to retry
Current version: 1.5.0, Current state: Modified, Current generation: 10
Sleep for 5 seconds to retry
Current version: 1.5.0, Current state: Modified, Current generation: 10
The logs of the new fleet-agent are repeating this error:
Copy code
{"level":"info","ts":"2025-05-06T09:19:10Z","logger":"bundledeployment.helm-deployer.install","msg":"Upgrading helm release","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"mcc-hvst-upgrade-bckj7-upgradelog-operator","namespace":"cluster-fleet-local-local-1a3d67d0a899"},"namespace":"cluster-fleet-local-local-1a3d67d0a899","name":"mcc-hvst-upgrade-bckj7-upgradelog-operator","reconcileID":"5168e9e0-3168-405e-8b2c-8e7ec021173c","commit":"","dryRun":false}
{"level":"error","ts":"2025-05-06T09:19:10Z","msg":"Reconciler error","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"mcc-hvst-upgrade-bckj7-upgradelog-operator","namespace":"cluster-fleet-local-local-1a3d67d0a899"},"namespace":"cluster-fleet-local-local-1a3d67d0a899","name":"mcc-hvst-upgrade-bckj7-upgradelog-operator","reconcileID":"5168e9e0-3168-405e-8b2c-8e7ec021173c","error":"failed deploying bundle: execution error at (rancher-logging/templates/validate-install-crd.yaml:26:7): Required CRDs are missing. Please install the corresponding CRD chart before installing this chart.","errorCauses":[{"error":"failed deploying bundle: execution error at (rancher-logging/templates/validate-install-crd.yaml:26:7): Required CRDs are missing. Please install the corresponding CRD chart before installing this chart."}],"stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.2/pkg/internal/controller/controller.go:341\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.2/pkg/internal/controller/controller.go:288\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.2/pkg/internal/controller/controller.go:249"}
s
What is the result of the following command?
Copy code
$ kubectl get bundle -A
s
Copy code
# kubectl get bundle -A
NAMESPACE     NAME                                          BUNDLEDEPLOYMENTS-READY   STATUS
fleet-local   fleet-agent-local                             1/1                       
fleet-local   local-managed-system-agent                    1/1                       
fleet-local   mcc-harvester                                 0/1                       Modified(1) [Cluster fleet-local/local]; <http://cdi.cdi.kubevirt.io|cdi.cdi.kubevirt.io> cdi missing
fleet-local   mcc-harvester-crd                             1/1                       
fleet-local   mcc-hvst-upgrade-bckj7-upgradelog-operator    0/1                       ErrApplied(1) [Cluster fleet-local/local: execution error at (rancher-logging/templates/validate-install-crd.yaml:26:7): Required CRDs are missing. Please install the corresponding CRD chart before installing this chart.]
fleet-local   mcc-local-managed-system-upgrade-controller   1/1                       
fleet-local   mcc-rancher-logging-crd                       1/1                       
fleet-local   mcc-rancher-monitoring-crd                    1/1
s
I am wondering if the Error from the rancher-logging might be a false alarm. But I need some time to investigate… @ancient-pizza-13099, did you ever see this message for the rancher logging?
cc @great-bear-19718
Hi @sticky-summer-13450, Could you try to patch the mcc-harvester bundle? To add this field: 
spec.forceSyncGeneration: 1
It should trigger the redeployment.
s
Hi. I should have time this evening (UK time), I'm on a family hike at the moment 😊
s
Sure, no worries, just update some investigation here
BTW, could you create a GH issue for it? We would like to have the note for your case.
a
The harvester managedchart seems to be due to 
Modified(1) [Cluster fleet-local/local]; <http://cdi.cdi.kubevirt.io|cdi.cdi.kubevirt.io> cdi missing
the 
mcc-hvst-upgrade-bckj7-upgradelog-operator
is due to others
the normal CRD has such labels and annotations:
Copy code
- apiVersion: <http://apiextensions.k8s.io/v1|apiextensions.k8s.io/v1>
  kind: CustomResourceDefinition
  metadata:
    annotations:
      <http://helm.sh/resource-policy|helm.sh/resource-policy>: keep
      <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: harvester-crd
      <http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: harvester-system
      <http://objectset.rio.cattle.io/id|objectset.rio.cattle.io/id>: default-mcc-harvester-crd-cattle-fleet-local-system
    creationTimestamp: "2024-12-01T12:57:22Z"
    generation: 1
    labels:
      <http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: Helm
      <http://objectset.rio.cattle.io/hash|objectset.rio.cattle.io/hash>: d4a83267ddde6a8769c04362d4a0e5605db9baa7
    managedFields:
but 
kubectl create -f
created 
<http://cdi.cdi.kubevirt.io|cdi.cdi.kubevirt.io>
has no such
Copy code
creationTimestamp: "2025-04-26T12:58:06Z"
    generation: 3
    managedFields:
    - apiVersion: <http://apiextensions.k8s.io/v1|apiextensions.k8s.io/v1>
,,
      manager: kube-apiserver
      operation: Update
      subresource: status
      time: "2025-04-26T12:58:06Z"
    - apiVersion: <http://apiextensions.k8s.io/v1|apiextensions.k8s.io/v1>
,,
      manager: kubectl-client-side-apply
      operation: Update
      time: "2025-04-26T13:03:46Z"
    name: <http://cdis.cdi.kubevirt.io|cdis.cdi.kubevirt.io>
    resourceVersion: "147939263"
    uid: 22c13f71-f4b3-4094-b82f-3e953587d010
  spec:
    conversion:
      strategy: None
    group: <http://cdi.kubevirt.io|cdi.kubevirt.io>
    names:
      kind: CDI
      listKind: CDIList
      plural: cdis
      shortNames:
      - cdi
      - cdis
      singular: cdi
    scope: Cluster
@sticky-summer-13450 Did you ever install third party 
logging
on your cluster? if you installed and revmoed it, which may cause the 
rancher-logging-crd
installed CRDs were removed by them accidentally, and the 
mcc-hvst-upgrade-bckj7-upgradelog-operator
is reporting those errors, but which did not block the upgrade
@sticky-summer-13450 From the support-bundle, the 
cdi.cdi...
resource has been in the CRDs; you may try below: 
kubectl edit managedchart -n fleet-local harvester
, set the 
paused
to `true`; wait 5 minutes, set it it 
false
again, wait another 5 mintues, check the 
managedchart
and 
bundle
, if the 
harvester
managedchart has changes
I suspect this is a side-effect of https://github.com/harvester/harvester/issues/8239
s
BTW, could you create a GH issue for it?
I have created https://github.com/harvester/harvester/issues/8240. I’ll try to add more notes to it as I get time during my vacation (when there is enough bandwidth for a VPN to the cluster)
> Did you ever install third party 
logging
on your cluster? No, I have not installed anything directly into the cluster. This cluster was installed as v1.4.0 and upgraded successfully through v1.4.1 and v1.4.2.
> Could you try to patch the mcc-harvester bundle? > To add this field: > 
spec.forceSyncGeneration: 1
 I think this suggestion has been superseded in the time I’ve managed to reply. Please tell me if I’ve got that wrong. Edit: See below.
kubectl edit managedchart -n fleet-local harvester
, set the 
paused
to `true`; wait 5 minutes, set it it 
false
again, wait another 5 mintues, check the 
managedchart
and 
bundle
, if the 
harvester
managedchart has changes
Did this and nothing has changed in the harvester ManagedChart or to the errors listed in bundle resources
Could you try to patch the mcc-harvester bundle?
To add this field:
spec.forceSyncGeneration: 1
I did this and after a couple of minutes the status cleared of notifications/errors and the deployed 
mcc-harvester
bundle became ready, then 5 minutes later the 
mcc-hvst-upgrade-bckj7-upgradelog-operator
bundle became ready too. The ‘Upgrading System Service’ is now at 100%, one node has moved into the ‘Pre-draining’ phase, and the VMs are being migrated to the other node. Thanks all 👏 - looks like this upgrade might be un-stuck now 🎉
👍 3
(GH issue updated with the above thread)
👍 1
a
Sounds great.
s
The fields 
spec.forceSyncGeneration: 1
will help the fleet re-deploy the chart. We want to do that because somehow the CDI CR is deleted. Happy to know your upgrade is getting back!
s
Just a sad continuation… The upgrade of the nodes stalled on the pre-draining stage. I stopped all of the running VMs and rebooted of each node individually, as I have in the past. That caused the upgrade to continue to completion on each node. Sadly, none of the VMs will start and I guess from looking at the Longhorn UI, all the data is lost.
I’ve tried creating a new VM from one of the backups but that errors saying the VLANs are not available. 
admission webhook "<http://validator.harvesterhci.io|validator.harvesterhci.io>" denied the request: Failed to get network attachment definition vlan648, err: <http://network-attachment-definitions.k8s.cni.cncf.io|network-attachment-definitions.k8s.cni.cncf.io> "vlan648" not found
Which, as far as I can see, they do.
So, all in all, a disappointing upgrade for me, unless something unexpected happens.
s
Hi @sticky-summer-13450, sorry to hear that. 🫠 Could you generate the support bundle again? I would like to check your current status.
s
I think the most disturbing thing is that “Restore New” from a Virtual Machine Backup is erroring as I described above. I’m quite worried that the VMs did not survive the upgrade, but not being able to restore from the backup is really devastating.
g
the support bundle is incomplete..
s
yeah, @sticky-summer-13450, sorry, I just checked your SB. It lacks the 
yamls
folder. Could you generate SB again and check it should contain the 
yamls
folder.
s
Just created a new SB. It does not contain a 
yamls
folder.
s
Did you have any cert expired?
We have a similar case on another community user, and he found that his cert has expired.
s
Which cert? I add the UI cert which is valid until July. Aren’t all the internal certs managed by Harvester?
g
likely scheduler cert etc
lol.. that was you
s
Haha. I'll do that again. This cluster has only been running for 161 days. And that bug was filled over 2 years ago 😉
I did what I wrote that I’d previously done, as you linked to:
Copy code
rancher@harvester003:~> echo "Rotating kube-controller-manager certificate"
Rotating kube-controller-manager certificate
rancher@harvester003:~> sudo rm /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.{crt,key}
rancher@harvester003:~> sudo /var/lib/rancher/rke2/bin/crictl -r unix:///var/run/k3s/containerd/containerd.sock rm -f $(sudo /var/lib/rancher/rke2/bin/crictl -r unix:///var/run/k3s/containerd/containerd.sock ps -q --name kube-controller-manager)
WARN[0000] Config "/etc/crictl.yaml" does not exist, trying next: "/var/lib/rancher/rke2/data/v1.32.3-rke2r1-543f84e7e830/bin/crictl.yaml" 
WARN[0000] Config "/etc/crictl.yaml" does not exist, trying next: "/var/lib/rancher/rke2/data/v1.32.3-rke2r1-543f84e7e830/bin/crictl.yaml" 
9972fb861a1d4ffd8ca8bd577a3c2fc5c50895b4aa04425576c71a7037f04c19
9972fb861a1d4ffd8ca8bd577a3c2fc5c50895b4aa04425576c71a7037f04c19

rancher@harvester003:~> echo "Rotating kube-scheduler certificate"
Rotating kube-scheduler certificate
rancher@harvester003:~> sudo rm /var/lib/rancher/rke2/server/tls/kube-scheduler/kube-scheduler.{crt,key}
rancher@harvester003:~> sudo /var/lib/rancher/rke2/bin/crictl -r unix:///var/run/k3s/containerd/containerd.sock rm -f $(sudo /var/lib/rancher/rke2/bin/crictl -r unix:///var/run/k3s/containerd/containerd.sock ps -q --name kube-scheduler)
WARN[0000] Config "/etc/crictl.yaml" does not exist, trying next: "/var/lib/rancher/rke2/data/v1.32.3-rke2r1-543f84e7e830/bin/crictl.yaml" 
WARN[0000] Config "/etc/crictl.yaml" does not exist, trying next: "/var/lib/rancher/rke2/data/v1.32.3-rke2r1-543f84e7e830/bin/crictl.yaml" 
e17947504f0df0b4add0ad466ecb1fcbcb6c28fbe83c2dbdf75928bb8dc442ce
e17947504f0df0b4add0ad466ecb1fcbcb6c28fbe83c2dbdf75928bb8dc442ce
Then I created a new SB but it still does not contain a 
yamls
folder.
Is there a correct way to check all of the certificates?
g
there would be a support-bundle-manager deployment running would be nice to check its logs
s
No errors in there - I wonder if you need the logs from the more transient 
supportbundle-agent-bundle
from each node
support-bundle-manager.log
Curious - there is a 
yamls
folder now. I wonder if I needed to have waited longer for certificate creation. Here’s an SB.
supportbundle_5351197f-4e80-4b39-99a7-44607b6fdbd8_2025-05-14T06-55-37Z.zip
s
Thanks @sticky-summer-13450, I will check this today
👍 1
Hi @sticky-summer-13450, Sorry for the delay, I checked your SB. Looks like the volume and the corresponding replicas are still here. Could you share the original issue after your upgrade? Thanks!
s
Hi Vincent. The problems are: • at upgrade time the VMs would not migrate, so I shut them down • post upgrade the VMs would not start • Longhorn says the volumes are failed (see example screenshot from mobile - I’m on the beach on holiday) • attempting to restore from backup is failing saying that the vlans don’t exist • creating an SB the zip does not contain a 
yamls
folder I have created a couple of new VMs using Terraform which are working fine. Just the old ones that are not. When I get back to a real computer this evening I could attempt to start a VM and grab another SB?
Got to a PC. Selected to start a VM - it’s hanging in the “starting” phase. Selected to create an SB - it doesn’t contain a 
yamls
folder. The logs from the 
supportbundle-manager-bundle
pod:
Copy code
+ exec tini -- /usr/bin/support-bundle-kit manager
time="2025-05-16T15:13:52Z" level=info msg="Running phase init"
time="2025-05-16T15:13:52Z" level=debug msg="Create a local state store. (harvester-system/bundle-5mexo)"
time="2025-05-16T15:13:52Z" level=debug msg="Get supportbundle harvester-system/bundle-5mexo state generating"
time="2025-05-16T15:13:52Z" level=info msg="Succeed to run phase init. Progress (16)."
time="2025-05-16T15:13:52Z" level=info msg="Running phase cluster bundle"
time="2025-05-16T15:13:52Z" level=debug msg="Generating cluster bundle..."
time="2025-05-16T15:13:53Z" level=info msg="Prepare to get all support bundle yamls!"
time="2025-05-16T15:13:53Z" level=info msg="[Harvester] generate YAMLs, yamlsDir: /tmp/support-bundle-kit/bundle/yamls"
time="2025-05-16T15:13:53Z" level=error msg="Unable to fetch namespaced resources" error="unable to retrieve the complete list of server APIs: ext.cattle.io/v1: stale GroupVersion discovery: ext.cattle.io/v1"
time="2025-05-16T15:13:53Z" level=info msg="[Cluster] generate YAMLs, yamlsDir: /tmp/support-bundle-kit/bundle/yamls"
time="2025-05-16T15:13:53Z" level=error msg="Unable to fetch cluster resources" error="unable to retrieve the complete list of server APIs: ext.cattle.io/v1: stale GroupVersion discovery: ext.cattle.io/v1"
time="2025-05-16T15:13:53Z" level=info msg="[Default] generate YAMLs, yamlsDir: /tmp/support-bundle-kit/bundle/yamls"
time="2025-05-16T15:13:53Z" level=error msg="Unable to fetch namespaced resources" error="unable to retrieve the complete list of server APIs: ext.cattle.io/v1: stale GroupVersion discovery: ext.cattle.io/v1"
time="2025-05-16T15:13:53Z" level=error msg="Unable to fetch namespaced resources" error="unable to retrieve the complete list of server APIs: ext.cattle.io/v1: stale GroupVersion discovery: ext.cattle.io/v1"
time="2025-05-16T15:13:53Z" level=error msg="Unable to fetch namespaced resources" error="unable to retrieve the complete list of server APIs: ext.cattle.io/v1: stale GroupVersion discovery: ext.cattle.io/v1"
time="2025-05-16T15:13:53Z" level=error msg="Unable to fetch namespaced resources" error="unable to retrieve the complete list of server APIs: ext.cattle.io/v1: stale GroupVersion discovery: ext.cattle.io/v1"
time="2025-05-16T15:13:53Z" level=error msg="Unable to fetch namespaced resources" error="unable to retrieve the complete list of server APIs: ext.cattle.io/v1: stale GroupVersion discovery: ext.cattle.io/v1"
time="2025-05-16T15:13:53Z" level=error msg="Unable to fetch namespaced resources" error="unable to retrieve the complete list of server APIs: ext.cattle.io/v1: stale GroupVersion discovery: ext.cattle.io/v1"
time="2025-05-16T15:13:53Z" level=error msg="Unable to fetch namespaced resources" error="unable to retrieve the complete list of server APIs: ext.cattle.io/v1: stale GroupVersion discovery: ext.cattle.io/v1"
time="2025-05-16T15:13:53Z" level=error msg="Unable to fetch namespaced resources" error="unable to retrieve the complete list of server APIs: ext.cattle.io/v1: stale GroupVersion discovery: ext.cattle.io/v1"
time="2025-05-16T15:13:53Z" level=error msg="Unable to fetch namespaced resources" error="unable to retrieve the complete list of server APIs: ext.cattle.io/v1: stale GroupVersion discovery: ext.cattle.io/v1"
time="2025-05-16T15:13:53Z" level=error msg="Unable to fetch namespaced resources" error="unable to retrieve the complete list of server APIs: ext.cattle.io/v1: stale GroupVersion discovery: ext.cattle.io/v1"
time="2025-05-16T15:13:53Z" level=error msg="Unable to fetch namespaced resources" error="unable to retrieve the complete list of server APIs: ext.cattle.io/v1: stale GroupVersion discovery: ext.cattle.io/v1"
time="2025-05-16T15:13:53Z" level=error msg="Unable to fetch namespaced resources" error="unable to retrieve the complete list of server APIs: ext.cattle.io/v1: stale GroupVersion discovery: ext.cattle.io/v1"
time="2025-05-16T15:13:53Z" level=error msg="Unable to fetch namespaced resources" error="unable to retrieve the complete list of server APIs: ext.cattle.io/v1: stale GroupVersion discovery: ext.cattle.io/v1"
time="2025-05-16T15:13:53Z" level=error msg="Unable to fetch namespaced resources" error="unable to retrieve the complete list of server APIs: ext.cattle.io/v1: stale GroupVersion discovery: ext.cattle.io/v1"
time="2025-05-16T15:13:53Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/default/virt-launcher-kube004-dqv79/compute.log"
time="2025-05-16T15:13:53Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/default/virt-launcher-kube004-dqv79/guest-console-log.log"
time="2025-05-16T15:13:53Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/kube-system/cloud-controller-manager-harvester003/cloud-controller-manager.log"
time="2025-05-16T15:13:53Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/kube-system/cloud-controller-manager-harvester003/cloud-controller-manager.log.1"
time="2025-05-16T15:13:54Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/kube-system/etcd-harvester003/etcd.log"
time="2025-05-16T15:13:54Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/kube-system/harvester-snapshot-validation-webhook-8594c5f8f8-rb5nx/snapshot-validation-webhook.log"
time="2025-05-16T15:13:55Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/kube-system/harvester-snapshot-validation-webhook-8594c5f8f8-zbfcj/snapshot-validation-webhook.log"
time="2025-05-16T15:13:55Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/kube-system/harvester-whereabouts-6gr2x/whereabouts.log"
time="2025-05-16T15:13:55Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/kube-system/harvester-whereabouts-6gr2x/whereabouts.log.1"
time="2025-05-16T15:13:56Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/kube-system/harvester-whereabouts-6p8fg/whereabouts.log"
time="2025-05-16T15:13:56Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/kube-system/harvester-whereabouts-6p8fg/whereabouts.log.1"
time="2025-05-16T15:13:56Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/kube-system/helm-install-rke2-canal-29t8m/helm.log"
time="2025-05-16T15:13:57Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/kube-system/helm-install-rke2-coredns-slr28/helm.log"
time="2025-05-16T15:13:57Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/kube-system/helm-install-rke2-multus-ncxwq/helm.log"
time="2025-05-16T15:13:57Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/kube-system/kube-apiserver-harvester003/kube-apiserver.log"
time="2025-05-16T15:13:58Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/kube-system/kube-controller-manager-harvester003/kube-controller-manager.log"
time="2025-05-16T15:13:58Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/kube-system/kube-controller-manager-harvester003/kube-controller-manager.log.1"
time="2025-05-16T15:13:59Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/kube-system/kube-proxy-harvester001/kube-proxy.log"
time="2025-05-16T15:13:59Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/kube-system/kube-proxy-harvester003/kube-proxy.log"
time="2025-05-16T15:13:59Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/kube-system/kube-scheduler-harvester003/kube-scheduler.log"
time="2025-05-16T15:14:00Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/kube-system/kube-scheduler-harvester003/kube-scheduler.log.1"
time="2025-05-16T15:14:00Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/kube-system/rke2-canal-qh2cc/calico-node.log"
time="2025-05-16T15:14:00Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/kube-system/rke2-canal-qh2cc/kube-flannel.log"
time="2025-05-16T15:14:01Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/kube-system/rke2-canal-tmwz8/calico-node.log"
time="2025-05-16T15:14:01Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/kube-system/rke2-canal-tmwz8/calico-node.log.1"
time="2025-05-16T15:14:01Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/kube-system/rke2-canal-tmwz8/kube-flannel.log"
time="2025-05-16T15:14:02Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/kube-system/rke2-canal-tmwz8/kube-flannel.log.1"
time="2025-05-16T15:14:02Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/kube-system/rke2-coredns-rke2-coredns-7769c87bd-8klns/coredns.log"
time="2025-05-16T15:14:02Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/kube-system/rke2-coredns-rke2-coredns-7769c87bd-m56rs/coredns.log"
time="2025-05-16T15:14:03Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/kube-system/rke2-coredns-rke2-coredns-autoscaler-5b9b76dbf6-n9kn8/autoscaler.log"
time="2025-05-16T15:14:03Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/kube-system/rke2-ingress-nginx-controller-h2dbz/rke2-ingress-nginx-controller.log"
time="2025-05-16T15:14:04Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/kube-system/rke2-ingress-nginx-controller-h2dbz/rke2-ingress-nginx-controller.log.1"
time="2025-05-16T15:14:04Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/kube-system/rke2-ingress-nginx-controller-hfcpp/rke2-ingress-nginx-controller.log"
time="2025-05-16T15:14:04Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/kube-system/rke2-ingress-nginx-controller-hfcpp/rke2-ingress-nginx-controller.log.1"
time="2025-05-16T15:14:04Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/kube-system/rke2-metrics-server-85479b695c-rqpqz/metrics-server.log"
time="2025-05-16T15:14:05Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/kube-system/rke2-multus-8dllq/kube-rke2-multus.log"
time="2025-05-16T15:14:05Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/kube-system/rke2-multus-8dllq/kube-rke2-multus.log.1"
time="2025-05-16T15:14:05Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/kube-system/rke2-multus-9hw5n/kube-rke2-multus.log"
time="2025-05-16T15:14:06Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/kube-system/snapshot-controller-5fb6d65787-lfjbt/snapshot-controller.log"
time="2025-05-16T15:14:06Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/kube-system/snapshot-controller-5fb6d65787-nmpsc/snapshot-controller.log"
time="2025-05-16T15:14:07Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/cattle-system/apply-hvst-upgrade-bckj7-cleanup-on-harvester001-with-310-9qd62/upgrade.log"
time="2025-05-16T15:14:07Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/cattle-system/apply-hvst-upgrade-bckj7-cleanup-on-harvester003-with-310-twfxs/upgrade.log"
time="2025-05-16T15:14:08Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/cattle-system/harvester-cluster-repo-558c6dd488-xjmt5/httpd.log"
time="2025-05-16T15:14:08Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/cattle-system/rancher-556698cf6f-6cqm8/rancher.log"
time="2025-05-16T15:14:08Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/cattle-system/rancher-556698cf6f-75mcg/rancher.log"
time="2025-05-16T15:14:09Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/cattle-system/rancher-webhook-5d578f98cf-9dtdj/rancher-webhook.log"
time="2025-05-16T15:14:09Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/cattle-system/system-upgrade-controller-6bb66b458b-nqhts/system-upgrade-controller.log"
time="2025-05-16T15:14:10Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/cattle-system/system-upgrade-controller-6bb66b458b-nqhts/system-upgrade-controller.log.1"
time="2025-05-16T15:14:10Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/cattle-fleet-local-system/fleet-agent-566b7845f-2vdwq/fleet-agent.log"
time="2025-05-16T15:14:11Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/cattle-fleet-system/fleet-controller-76fb9c5fbd-j67lp/fleet-controller.log"
time="2025-05-16T15:14:11Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/cattle-fleet-system/fleet-controller-76fb9c5fbd-j67lp/fleet-controller.log.1"
time="2025-05-16T15:14:11Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/cattle-fleet-system/fleet-controller-76fb9c5fbd-j67lp/fleet-cleanup.log"
time="2025-05-16T15:14:12Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/cattle-fleet-system/fleet-controller-76fb9c5fbd-j67lp/fleet-cleanup.log.1"
time="2025-05-16T15:14:12Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/cattle-fleet-system/fleet-controller-76fb9c5fbd-j67lp/fleet-agentmanagement.log"
time="2025-05-16T15:14:12Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/cattle-fleet-system/fleet-controller-76fb9c5fbd-j67lp/fleet-agentmanagement.log.1"
time="2025-05-16T15:14:13Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/cattle-fleet-system/gitjob-7bd6855c9b-jhcmj/gitjob.log"
time="2025-05-16T15:14:13Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/cattle-fleet-system/gitjob-7bd6855c9b-jhcmj/gitjob.log.1"
time="2025-05-16T15:14:14Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/cattle-monitoring-system/alertmanager-rancher-monitoring-alertmanager-0/alertmanager.log"
time="2025-05-16T15:14:14Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/cattle-monitoring-system/alertmanager-rancher-monitoring-alertmanager-0/config-reloader.log"
time="2025-05-16T15:14:14Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/cattle-monitoring-system/prometheus-rancher-monitoring-prometheus-0/prometheus.log"
time="2025-05-16T15:14:15Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/cattle-monitoring-system/prometheus-rancher-monitoring-prometheus-0/config-reloader.log"
time="2025-05-16T15:14:15Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/cattle-monitoring-system/prometheus-rancher-monitoring-prometheus-0/prometheus-proxy.log"
time="2025-05-16T15:14:16Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/cattle-monitoring-system/rancher-monitoring-grafana-7887776f6f-xwbhv/grafana-sc-dashboard.log"
time="2025-05-16T15:14:16Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/cattle-monitoring-system/rancher-monitoring-grafana-7887776f6f-xwbhv/grafana.log"
time="2025-05-16T15:14:16Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/cattle-monitoring-system/rancher-monitoring-grafana-7887776f6f-xwbhv/grafana-proxy.log"
time="2025-05-16T15:14:17Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/cattle-monitoring-system/rancher-monitoring-kube-state-metrics-596b877446-jcw6n/kube-state-metrics.log"
time="2025-05-16T15:14:17Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/cattle-monitoring-system/rancher-monitoring-operator-d5f56bff9-k4brp/rancher-monitoring.log"
time="2025-05-16T15:14:18Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/cattle-monitoring-system/rancher-monitoring-prometheus-adapter-9d8f566c4-9jszm/prometheus-adapter.log"
time="2025-05-16T15:14:18Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/cattle-monitoring-system/rancher-monitoring-prometheus-node-exporter-c8f9b/node-exporter.log"
time="2025-05-16T15:14:18Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/cattle-monitoring-system/rancher-monitoring-prometheus-node-exporter-c8f9b/node-exporter.log.1"
time="2025-05-16T15:14:19Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/cattle-monitoring-system/rancher-monitoring-prometheus-node-exporter-x6kwd/node-exporter.log"
time="2025-05-16T15:14:19Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/cattle-monitoring-system/rancher-monitoring-prometheus-node-exporter-x6kwd/node-exporter.log.1"
time="2025-05-16T15:14:20Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/cdi-apiserver-58bf7b8b74-spgs5/cdi-apiserver.log"
time="2025-05-16T15:14:20Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/cdi-deployment-6cf7d9b445-rnbx5/cdi-deployment.log"
time="2025-05-16T15:14:20Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/cdi-operator-7b6695c56d-fp799/cdi-operator.log"
time="2025-05-16T15:14:21Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/cdi-uploadproxy-68d59ccffb-hqt8c/cdi-uploadproxy.log"
time="2025-05-16T15:14:21Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/harvester-6586b5ff7b-bzvw5/apiserver.log"
time="2025-05-16T15:14:22Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/harvester-6586b5ff7b-c7w8n/apiserver.log"
time="2025-05-16T15:14:23Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/harvester-load-balancer-f6c5654f4-mvj5h/harvester-load-balancer.log"
time="2025-05-16T15:14:23Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/harvester-load-balancer-webhook-66794df8ff-tt9nt/harvester-load-balancer-webhook.log"
time="2025-05-16T15:14:23Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/harvester-network-controller-k5dnr/harvester-network.log"
time="2025-05-16T15:14:23Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/harvester-network-controller-k5dnr/harvester-network.log.1"
time="2025-05-16T15:14:23Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/harvester-network-controller-kkq8q/harvester-network.log"
time="2025-05-16T15:14:24Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/harvester-network-controller-kkq8q/harvester-network.log.1"
time="2025-05-16T15:14:24Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/harvester-network-controller-manager-b65465c9-5xqfq/harvester-network-manager.log"
time="2025-05-16T15:14:24Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/harvester-network-controller-manager-b65465c9-vfbwz/harvester-network-manager.log"
time="2025-05-16T15:14:25Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/harvester-network-webhook-6c55485b58-mdspn/harvester-network-webhook.log"
time="2025-05-16T15:14:25Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/harvester-networkfs-manager-kbzfv/harvester-networkfs-manager.log"
time="2025-05-16T15:14:26Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/harvester-networkfs-manager-kbzfv/harvester-networkfs-manager.log.1"
time="2025-05-16T15:14:26Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/harvester-networkfs-manager-p57q2/harvester-networkfs-manager.log"
time="2025-05-16T15:14:26Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/harvester-networkfs-manager-p57q2/harvester-networkfs-manager.log.1"
time="2025-05-16T15:14:26Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/harvester-node-disk-manager-8vgff/harvester-node-disk-manager.log"
time="2025-05-16T15:14:27Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/harvester-node-disk-manager-8vgff/harvester-node-disk-manager.log.1"
time="2025-05-16T15:14:27Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/harvester-node-disk-manager-m9f62/harvester-node-disk-manager.log"
time="2025-05-16T15:14:27Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/harvester-node-disk-manager-m9f62/harvester-node-disk-manager.log.1"
time="2025-05-16T15:14:28Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/harvester-node-disk-manager-webhook-55b8694747-285jf/harvester-node-disk-manager-webhook.log"
time="2025-05-16T15:14:28Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/harvester-node-manager-cjj97/node-manager.log"
time="2025-05-16T15:14:28Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/harvester-node-manager-cjj97/node-manager.log.1"
time="2025-05-16T15:14:29Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/harvester-node-manager-gr4gt/node-manager.log"
time="2025-05-16T15:14:29Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/harvester-node-manager-gr4gt/node-manager.log.1"
time="2025-05-16T15:14:29Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/harvester-node-manager-webhook-77d466598d-b2hxx/harvester-node-manager-webhook.log"
time="2025-05-16T15:14:30Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/harvester-node-manager-webhook-77d466598d-tjxkh/harvester-node-manager-webhook.log"
time="2025-05-16T15:14:30Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/harvester-vm-import-controller-7485d5dd56-dxnpf/harvester-vm-import-controller.log"
time="2025-05-16T15:14:30Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/harvester-webhook-588b5d84f5-76nq6/harvester-webhook.log"
time="2025-05-16T15:14:31Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/harvester-webhook-588b5d84f5-xkqh7/harvester-webhook.log"
time="2025-05-16T15:14:31Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/hvst-upgrade-bckj7-post-drain-harvester001-s9gs5/apply.log"
time="2025-05-16T15:14:32Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/hvst-upgrade-bckj7-post-drain-harvester003-xxpfq/apply.log"
time="2025-05-16T15:14:32Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/kube-vip-wt5j5/kube-vip.log"
time="2025-05-16T15:14:33Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/kube-vip-wt5j5/kube-vip.log.1"
time="2025-05-16T15:14:33Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/supportbundle-manager-bundle-5mexo-54bf9fd574-fr5z9/manager.log"
time="2025-05-16T15:14:33Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/svmb-c1ef3a33-1307-4c70-aca1-6ea90b4385a6-29119860-lckbl/svmb-c1ef3a33-1307-4c70-aca1-6ea90b4385a6.log"
time="2025-05-16T15:14:34Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/svmb-c1ef3a33-1307-4c70-aca1-6ea90b4385a6-29121300-6nthg/svmb-c1ef3a33-1307-4c70-aca1-6ea90b4385a6.log"
time="2025-05-16T15:14:34Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/svmb-c1ef3a33-1307-4c70-aca1-6ea90b4385a6-29122740-xmj8b/svmb-c1ef3a33-1307-4c70-aca1-6ea90b4385a6.log"
time="2025-05-16T15:14:35Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/virt-api-745c68d9f-dksl4/virt-api.log"
time="2025-05-16T15:14:35Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/virt-api-745c68d9f-mbd4n/virt-api.log"
time="2025-05-16T15:14:35Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/virt-controller-5dd599df-22w9p/virt-controller.log"
time="2025-05-16T15:14:36Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/virt-controller-5dd599df-gl5dw/virt-controller.log"
time="2025-05-16T15:14:36Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/virt-handler-jcrt6/virt-handler.log"
time="2025-05-16T15:14:37Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/virt-handler-jcrt6/virt-handler.log.1"
time="2025-05-16T15:14:37Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/virt-handler-pmrb7/virt-handler.log"
time="2025-05-16T15:14:37Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/virt-handler-pmrb7/virt-handler.log.1"
time="2025-05-16T15:14:37Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/harvester-system/virt-operator-7888b89b5-klgkm/virt-operator.log"
time="2025-05-16T15:14:38Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/backing-image-manager-8494-378e/backing-image-manager.log"
time="2025-05-16T15:14:39Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/backing-image-manager-8494-cddb/backing-image-manager.log"
time="2025-05-16T15:14:39Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/csi-attacher-867869c54d-krjbx/csi-attacher.log"
time="2025-05-16T15:14:39Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/csi-attacher-867869c54d-mnd5n/csi-attacher.log"
time="2025-05-16T15:14:40Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/csi-attacher-867869c54d-xl444/csi-attacher.log"
time="2025-05-16T15:14:40Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/csi-provisioner-8846bb87-9kv5g/csi-provisioner.log"
time="2025-05-16T15:14:41Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/csi-provisioner-8846bb87-ldmhz/csi-provisioner.log"
time="2025-05-16T15:14:41Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/csi-provisioner-8846bb87-zvcn9/csi-provisioner.log"
time="2025-05-16T15:14:41Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/csi-resizer-76679bb7f8-5zd6l/csi-resizer.log"
time="2025-05-16T15:14:42Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/csi-resizer-76679bb7f8-lvcxh/csi-resizer.log"
time="2025-05-16T15:14:42Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/csi-resizer-76679bb7f8-rnt7f/csi-resizer.log"
time="2025-05-16T15:14:43Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/csi-snapshotter-58c98ccf94-9cnr9/csi-snapshotter.log"
time="2025-05-16T15:14:43Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/csi-snapshotter-58c98ccf94-ltjq4/csi-snapshotter.log"
time="2025-05-16T15:14:43Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/csi-snapshotter-58c98ccf94-zkwsb/csi-snapshotter.log"
time="2025-05-16T15:14:44Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/engine-image-ei-51cc7b9c-sn5hl/engine-image-ei-51cc7b9c.log"
time="2025-05-16T15:14:44Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/engine-image-ei-51cc7b9c-sn5hl/engine-image-ei-51cc7b9c.log.1"
time="2025-05-16T15:14:44Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/engine-image-ei-51cc7b9c-tdj6j/engine-image-ei-51cc7b9c.log"
time="2025-05-16T15:14:45Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/engine-image-ei-51cc7b9c-tdj6j/engine-image-ei-51cc7b9c.log.1"
time="2025-05-16T15:14:45Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/engine-image-ei-db6c2b6f-2tnm2/engine-image-ei-db6c2b6f.log"
time="2025-05-16T15:14:45Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/engine-image-ei-db6c2b6f-2tnm2/engine-image-ei-db6c2b6f.log.1"
time="2025-05-16T15:14:46Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/engine-image-ei-db6c2b6f-5wzq6/engine-image-ei-db6c2b6f.log"
time="2025-05-16T15:14:46Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/engine-image-ei-db6c2b6f-5wzq6/engine-image-ei-db6c2b6f.log.1"
time="2025-05-16T15:14:46Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/instance-manager-6cd61b2e5d4b2d4263b4eef12e449c2d/instance-manager.log"
time="2025-05-16T15:14:47Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/instance-manager-ecad7f7d3a2682bd91344371908f4d91/instance-manager.log"
time="2025-05-16T15:14:47Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/longhorn-csi-plugin-29jcz/node-driver-registrar.log"
time="2025-05-16T15:14:47Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/longhorn-csi-plugin-29jcz/node-driver-registrar.log.1"
time="2025-05-16T15:14:48Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/longhorn-csi-plugin-29jcz/longhorn-liveness-probe.log"
time="2025-05-16T15:14:48Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/longhorn-csi-plugin-29jcz/longhorn-liveness-probe.log.1"
time="2025-05-16T15:14:48Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/longhorn-csi-plugin-29jcz/longhorn-csi-plugin.log"
time="2025-05-16T15:14:49Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/longhorn-csi-plugin-29jcz/longhorn-csi-plugin.log.1"
time="2025-05-16T15:14:49Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/longhorn-csi-plugin-2wnxz/node-driver-registrar.log"
time="2025-05-16T15:14:49Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/longhorn-csi-plugin-2wnxz/node-driver-registrar.log.1"
time="2025-05-16T15:14:49Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/longhorn-csi-plugin-2wnxz/longhorn-liveness-probe.log"
time="2025-05-16T15:14:50Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/longhorn-csi-plugin-2wnxz/longhorn-liveness-probe.log.1"
time="2025-05-16T15:14:50Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/longhorn-csi-plugin-2wnxz/longhorn-csi-plugin.log"
time="2025-05-16T15:14:50Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/longhorn-csi-plugin-2wnxz/longhorn-csi-plugin.log.1"
time="2025-05-16T15:14:51Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/longhorn-driver-deployer-597c97c75b-ns72l/longhorn-driver-deployer.log"
time="2025-05-16T15:14:51Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/longhorn-loop-device-cleaner-4bx9f/longhorn-loop-device-cleaner.log"
time="2025-05-16T15:14:52Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/longhorn-loop-device-cleaner-4bx9f/longhorn-loop-device-cleaner.log.1"
time="2025-05-16T15:14:52Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/longhorn-loop-device-cleaner-qmfqw/longhorn-loop-device-cleaner.log"
time="2025-05-16T15:14:53Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/longhorn-loop-device-cleaner-qmfqw/longhorn-loop-device-cleaner.log.1"
time="2025-05-16T15:14:53Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/longhorn-manager-mdkvh/longhorn-manager.log"
time="2025-05-16T15:14:54Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/longhorn-manager-mdkvh/longhorn-manager.log.1"
time="2025-05-16T15:14:54Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/longhorn-manager-mdkvh/pre-pull-share-manager-image.log"
time="2025-05-16T15:14:54Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/longhorn-manager-mdkvh/pre-pull-share-manager-image.log.1"
time="2025-05-16T15:14:54Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/longhorn-manager-pjfgq/longhorn-manager.log"
time="2025-05-16T15:14:54Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/longhorn-manager-pjfgq/longhorn-manager.log.1"
time="2025-05-16T15:14:54Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/longhorn-manager-pjfgq/pre-pull-share-manager-image.log"
time="2025-05-16T15:14:54Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/longhorn-manager-pjfgq/pre-pull-share-manager-image.log.1"
time="2025-05-16T15:14:55Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/longhorn-ui-c55c69896-m5bqk/longhorn-ui.log"
time="2025-05-16T15:14:55Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/longhorn-system/longhorn-ui-c55c69896-zz7lt/longhorn-ui.log"
time="2025-05-16T15:14:56Z" level=debug msg="Prepare to log to file: /tmp/support-bundle-kit/bundle/logs/cattle-provisioning-capi-system/capi-controller-manager-d55f9f8d-c8mf9/manager.log"
time="2025-05-16T15:14:56Z" level=info msg="Succeed to run phase cluster bundle. Progress (33)."
time="2025-05-16T15:14:56Z" level=info msg="Running phase node bundle"
time="2025-05-16T15:14:56Z" level=debug msg="Creating daemonset supportbundle-agent-bundle-5mexo with image rancher/support-bundle-kit:v0.0.46"
time="2025-05-16T15:14:57Z" level=debug msg="Waiting for the creation of agent DaemonSet Pods for scheduled node names collection"
time="2025-05-16T15:14:57Z" level=debug msg="Expecting bundles from nodes: map[harvester001: harvester003:]"
time="2025-05-16T15:15:45Z" level=debug msg="Handle create node bundle for harvester003"
time="2025-05-16T15:15:45Z" level=debug msg="Complete node harvester003"
time="2025-05-16T15:15:53Z" level=debug msg="Handle create node bundle for harvester001"
time="2025-05-16T15:15:53Z" level=debug msg="Complete node harvester001"
time="2025-05-16T15:15:53Z" level=debug msg="All nodes are completed"
time="2025-05-16T15:15:53Z" level=info msg="All node bundles are received."
time="2025-05-16T15:15:53Z" level=info msg="Succeed to run phase node bundle. Progress (50)."
time="2025-05-16T15:15:53Z" level=info msg="Running phase prometheus bundle"
time="2025-05-16T15:15:53Z" level=info msg="Succeed to run phase prometheus bundle. Progress (66)."
time="2025-05-16T15:15:53Z" level=info msg="Running phase package"
time="2025-05-16T15:15:54Z" level=info msg="Succeed to run phase package. Progress (83)."
time="2025-05-16T15:15:54Z" level=info msg="Running phase done"
time="2025-05-16T15:15:54Z" level=info msg="Support bundle /tmp/support-bundle-kit/supportbundle_5351197f-4e80-4b39-99a7-44607b6fdbd8_2025-05-16T15-13-53Z.zip ready to download"
time="2025-05-16T15:15:54Z" level=info msg="Succeed to run phase done. Progress (100)."
s
Hi @sticky-summer-13450, Sorry for the late reply. What is the problem with the VM namespace? Could you try to collect the VM with the corresponding namespace through the support bundle? If you still encounter the issue that the 
yaml
folder is missing, could you try to refresh the certificates again like you do above? Or create a GH issue, that’s weird…
r
regarding the 
yaml
folder missing situation, we’ve created a GH issue under the SBK project see https://github.com/rancher/support-bundle-kit/issues/137
👍 2
s
Hi @salmon-city-57654 > If you still encounter the issue that the 
yaml
folder is missing, could you try to refresh the certificates again like you do above? I have refreshed the certs again, and waited a few hours in case that was needed. • The first time I made an SB it did not have a 
yamls
folder. • The second time I made an SB it did have a 
yamls
folder. • The third time I made an SB it did not have a 
yamls
folder. • The fourth time I made an SB it did not have a 
yamls
folder. This is very odd. It should not be indeterminate, it should work, or not work.
Hi @salmon-city-57654
What is the problem with the VM namespace?
I have no idea - I haven't mentioned there being a problem with the VM namespace.
Could you try to collect the VM with the corresponding namespace through the support bundle?
I don't know what you mean. If I can recap what I know. • I have some VMs which were working - running right up to the upgrade from 1.4.2 to 1.5.0 - which will not start now. One in the starting state in the SB I'm about to attach. • I have daily backups of the VMs which cannot be started, but they can not be restored because the VLAN definition apparently cannot be found 
admission webhook "<http://validator.harvesterhci.io|validator.harvesterhci.io>" denied the request: Failed to get network attachment definition vlan640, err: <http://network-attachment-definitions.k8s.cni.cncf.io|network-attachment-definitions.k8s.cni.cncf.io> "vlan640" not found
even though the VLANs exist perfectly well.
24 Views
Open in Slack
PreviousNext
Powered by