This message was deleted.
# generala
adamant-kite-43734
05/27/2023, 3:56 PMThis message was deleted.
a
acceptable-match-53099
05/28/2023, 2:19 PMLooks like general error. Your K3S is down for unclear reason.
Can you get nodes, pods, ?
acceptable-match-53099
05/28/2023, 2:19 PMCould you write output of next command:
kubectl get nodes && echo && kubectl cluster-info && echo && kubectl get -A cm,secrets,deploy,sts,rs,pod,pvc,svc,ing -o wideh
hundreds-jackal-3140
05/28/2023, 3:56 PMYes, @acceptable-match-53099. I can get output from inside the Docker container.
hundreds-jackal-3140
05/28/2023, 3:56 PMHuge output, I will break it down.
hundreds-jackal-3140
05/28/2023, 3:57 PM Copy code
d196e7fcfd68:/var/lib/rancher # kubectl get nodes
NAME STATUS ROLES AGE VERSION
local-node Ready control-plane,etcd,master 573d v1.25.5+k3s1hundreds-jackal-3140
05/28/2023, 3:57 PM Copy code
d196e7fcfd68:/var/lib/rancher # kubectl cluster-info
Kubernetes control plane is running at <https://127.0.0.1:6443>
CoreDNS is running at <https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy>
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.hundreds-jackal-3140
05/28/2023, 4:04 PM Copy code
d196e7fcfd68:/var/lib/rancher # kubectl get -o wide -A cm
NAMESPACE NAME DATA AGE
c-cvcjb kube-root-ca.crt 1 26d
c-cvcjb provisioning-log 1 26d
c-xv8rd kube-root-ca.crt 1 573d
c-xv8rd provisioning-log 1 428d
cattle-fleet-clusters-system kube-root-ca.crt 1 573d
cattle-fleet-local-system fleet-agent 1 573d
cattle-fleet-local-system fleet-agent-lock 0 573d
cattle-fleet-local-system kube-root-ca.crt 1 573d
cattle-fleet-system fleet-controller 1 573d
cattle-fleet-system fleet-controller-lock 0 573d
cattle-fleet-system gitjob 0 573d
cattle-fleet-system kube-root-ca.crt 1 573d
cattle-global-data kube-root-ca.crt 1 573d
cattle-global-nt kube-root-ca.crt 1 573d
cattle-impersonation-system kube-root-ca.crt 1 573d
cattle-system admincreated 3 573d
cattle-system forcelocalprojectcreation 1 573d
cattle-system forcesystemnamespaceassignment 1 471d
cattle-system forceupgradelogout 1 573d
cattle-system kube-root-ca.crt 1 573d
cattle-system migratedynamicschematomachinepools 1 36d
cattle-system migrateencryptionkeyrotationleadertostatus 1 36d
cattle-system migratefrommachinetoplanesecret 1 419d
cattle-system rancher-charts-0-da331d05-0610-4d50-8335-1c648ad3b65f 1 573d
cattle-system rancher-partner-charts-0-5c10a6cf-7489-4d09-85a7-356e2b73af46 1 573d
cattle-system rancher-partner-charts-1-5c10a6cf-7489-4d09-85a7-356e2b73af46 1 58d
cattle-system rancher-rke2-charts-0-17c4b79c-5212-44fd-9e86-1a2fde8a06d9 1 573d
cattle-system system-library-catalog-image-list 2 573d
cluster-fleet-default-c-cvcjb-b2b3e6f2ad36 kube-root-ca.crt 1 26d
cluster-fleet-local-local-1a3d67d0a899 kube-root-ca.crt 1 573d
default kube-root-ca.crt 1 573d
fleet-default kube-root-ca.crt 1 573d
fleet-local kube-root-ca.crt 1 573d
kube-node-lease kube-root-ca.crt 1 573d
kube-public kube-root-ca.crt 1 573d
kube-system cattle-controllers 0 573d
kube-system cluster-dns 2 573d
kube-system coredns 2 573d
kube-system extension-apiserver-authentication 6 573d
kube-system k3s 0 573d
kube-system k3s-etcd-snapshots 5 471d
kube-system kube-root-ca.crt 1 573d
local kube-root-ca.crt 1 573d
p-22mdv kube-root-ca.crt 1 26d
p-6np7g kube-root-ca.crt 1 26d
p-7f55m kube-root-ca.crt 1 573d
p-7hhxd kube-root-ca.crt 1 573d
p-fb5h2 kube-root-ca.crt 1 573d
p-gr86m kube-root-ca.crt 1 573d
u-4rzu6btj4l kube-root-ca.crt 1 529d
u-6xo76pkfpp kube-root-ca.crt 1 529d
u-bhcttyfs2r kube-root-ca.crt 1 428d
u-bvngnfuomk kube-root-ca.crt 1 570d
u-cx63gq3clv kube-root-ca.crt 1 550d
u-jllvx22khv kube-root-ca.crt 1 230d
u-mppukhbhca kube-root-ca.crt 1 225d
u-z7bgaodeek kube-root-ca.crt 1 570d
user-lfdzw kube-root-ca.crt 1 573dhundreds-jackal-3140
05/28/2023, 4:05 PM Copy code
d196e7fcfd68:/var/lib/rancher # kubectl get -o wide -A cm
NAMESPACE NAME DATA AGE
c-cvcjb kube-root-ca.crt 1 26d
c-cvcjb provisioning-log 1 26d
c-xv8rd kube-root-ca.crt 1 573d
c-xv8rd provisioning-log 1 428d
cattle-fleet-clusters-system kube-root-ca.crt 1 573d
cattle-fleet-local-system fleet-agent 1 573d
cattle-fleet-local-system fleet-agent-lock 0 573d
cattle-fleet-local-system kube-root-ca.crt 1 573d
cattle-fleet-system fleet-controller 1 573d
cattle-fleet-system fleet-controller-lock 0 573d
cattle-fleet-system gitjob 0 573d
cattle-fleet-system kube-root-ca.crt 1 573d
cattle-global-data kube-root-ca.crt 1 573d
cattle-global-nt kube-root-ca.crt 1 573d
cattle-impersonation-system kube-root-ca.crt 1 573d
cattle-system admincreated 3 573d
cattle-system forcelocalprojectcreation 1 573d
cattle-system forcesystemnamespaceassignment 1 471d
cattle-system forceupgradelogout 1 573d
cattle-system kube-root-ca.crt 1 573d
cattle-system migratedynamicschematomachinepools 1 36d
cattle-system migrateencryptionkeyrotationleadertostatus 1 36d
cattle-system migratefrommachinetoplanesecret 1 419d
cattle-system rancher-charts-0-da331d05-0610-4d50-8335-1c648ad3b65f 1 573d
cattle-system rancher-partner-charts-0-5c10a6cf-7489-4d09-85a7-356e2b73af46 1 573d
cattle-system rancher-partner-charts-1-5c10a6cf-7489-4d09-85a7-356e2b73af46 1 58d
cattle-system rancher-rke2-charts-0-17c4b79c-5212-44fd-9e86-1a2fde8a06d9 1 573d
cattle-system system-library-catalog-image-list 2 573d
cluster-fleet-default-c-cvcjb-b2b3e6f2ad36 kube-root-ca.crt 1 26d
cluster-fleet-local-local-1a3d67d0a899 kube-root-ca.crt 1 573d
default kube-root-ca.crt 1 573d
fleet-default kube-root-ca.crt 1 573d
fleet-local kube-root-ca.crt 1 573d
kube-node-lease kube-root-ca.crt 1 573d
kube-public kube-root-ca.crt 1 573d
kube-system cattle-controllers 0 573d
kube-system cluster-dns 2 573d
kube-system coredns 2 573d
kube-system extension-apiserver-authentication 6 573d
kube-system k3s 0 573d
kube-system k3s-etcd-snapshots 5 471d
kube-system kube-root-ca.crt 1 573d
local kube-root-ca.crt 1 573d
p-22mdv kube-root-ca.crt 1 26d
p-6np7g kube-root-ca.crt 1 26d
p-7f55m kube-root-ca.crt 1 573d
p-7hhxd kube-root-ca.crt 1 573d
p-fb5h2 kube-root-ca.crt 1 573d
p-gr86m kube-root-ca.crt 1 573d
u-4rzu6btj4l kube-root-ca.crt 1 529d
u-6xo76pkfpp kube-root-ca.crt 1 529d
u-bhcttyfs2r kube-root-ca.crt 1 428d
u-bvngnfuomk kube-root-ca.crt 1 570d
u-cx63gq3clv kube-root-ca.crt 1 550d
u-jllvx22khv kube-root-ca.crt 1 230d
u-mppukhbhca kube-root-ca.crt 1 225d
u-z7bgaodeek kube-root-ca.crt 1 570d
user-lfdzw kube-root-ca.crt 1 573dhundreds-jackal-3140
05/28/2023, 4:06 PM Copy code
d196e7fcfd68:/var/lib/rancher # kubectl get -o wide -A deploy
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
cattle-fleet-local-system fleet-agent 1/1 1 1 573d fleet-agent rancher/fleet-agent:v0.6.0 app=fleet-agent
cattle-fleet-system fleet-controller 1/1 1 1 573d fleet-controller rancher/fleet:v0.6.0 app=fleet-controller
cattle-fleet-system gitjob 1/1 1 1 573d gitjob rancher/gitjob:v0.1.37 app=gitjob
cattle-system rancher-webhook 1/1 1 1 573d rancher-webhook rancher/rancher-webhook:v0.3.3 app=rancher-webhook
kube-system coredns 1/1 1 1 573d coredns rancher/mirrored-coredns-coredns:1.9.4 k8s-app=kube-dnshundreds-jackal-3140
05/28/2023, 4:06 PM Copy code
d196e7fcfd68:/var/lib/rancher # kubectl get -o wide -A sts
No resources foundhundreds-jackal-3140
05/28/2023, 4:06 PM Copy code
d196e7fcfd68:/var/lib/rancher # kubectl get -o wide -A rs
NAMESPACE NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR
cattle-fleet-local-system fleet-agent-54c474c564 1 1 1 36d fleet-agent rancher/fleet-agent:v0.6.0 app=fleet-agent,pod-template-hash=54c474c564
cattle-fleet-local-system fleet-agent-5744768557 0 0 0 277d fleet-agent rancher/fleet-agent:v0.3.10 app=fleet-agent,pod-template-hash=5744768557
cattle-fleet-local-system fleet-agent-57497ff7dc 0 0 0 419d fleet-agent rancher/fleet-agent:v0.3.9 app=fleet-agent,pod-template-hash=57497ff7dc
cattle-fleet-local-system fleet-agent-59b74595c 0 0 0 573d fleet-agent rancher/fleet-agent:v0.3.7 app=fleet-agent,pod-template-hash=59b74595c
cattle-fleet-local-system fleet-agent-59d884fc56 0 0 0 269d fleet-agent rancher/fleet-agent:v0.3.11 app=fleet-agent,pod-template-hash=59d884fc56
cattle-fleet-local-system fleet-agent-78f694664b 0 0 0 471d fleet-agent rancher/fleet-agent:v0.3.8 app=fleet-agent,pod-template-hash=78f694664b
cattle-fleet-local-system fleet-agent-7bcc7d69cf 0 0 0 169d fleet-agent rancher/fleet-agent:v0.5.0 app=fleet-agent,pod-template-hash=7bcc7d69cf
cattle-fleet-local-system fleet-agent-86bcc7466d 0 0 0 107d fleet-agent rancher/fleet-agent:v0.5.1 app=fleet-agent,pod-template-hash=86bcc7466d
cattle-fleet-local-system fleet-agent-f449d485 0 0 0 573d fleet-agent rancher/fleet-agent:v0.3.7 app=fleet-agent,pod-template-hash=f449d485
cattle-fleet-system fleet-controller-55c547c6d5 1 1 1 36d fleet-controller rancher/fleet:v0.6.0 app=fleet-controller,pod-template-hash=55c547c6d5
cattle-fleet-system fleet-controller-5746685958 0 0 0 419d fleet-controller rancher/fleet:v0.3.9 app=fleet-controller,pod-template-hash=5746685958
cattle-fleet-system fleet-controller-58db4bf695 0 0 0 269d fleet-controller rancher/fleet:v0.3.11 app=fleet-controller,pod-template-hash=58db4bf695
cattle-fleet-system fleet-controller-5cbbd7c4c9 0 0 0 169d fleet-controller rancher/fleet:v0.5.0 app=fleet-controller,pod-template-hash=5cbbd7c4c9
cattle-fleet-system fleet-controller-6666887949 0 0 0 471d fleet-controller rancher/fleet:v0.3.8 app=fleet-controller,pod-template-hash=6666887949
cattle-fleet-system fleet-controller-66cc4c6b5b 0 0 0 573d fleet-controller rancher/fleet:v0.3.7 app=fleet-controller,pod-template-hash=66cc4c6b5b
cattle-fleet-system fleet-controller-76f56b4568 0 0 0 277d fleet-controller rancher/fleet:v0.3.10 app=fleet-controller,pod-template-hash=76f56b4568
cattle-fleet-system fleet-controller-7b4669d55d 0 0 0 107d fleet-controller rancher/fleet:v0.5.1 app=fleet-controller,pod-template-hash=7b4669d55d
cattle-fleet-system gitjob-5778966b7c 0 0 0 573d gitjob rancher/gitjob:v0.1.21 app=gitjob,pod-template-hash=5778966b7c
cattle-fleet-system gitjob-5c5979d844 0 0 0 169d gitjob rancher/gitjob:v0.1.32 app=gitjob,pod-template-hash=5c5979d844
cattle-fleet-system gitjob-74b98dff59 1 1 1 36d gitjob rancher/gitjob:v0.1.37 app=gitjob,pod-template-hash=74b98dff59
cattle-fleet-system gitjob-7b97c9c7fd 0 0 0 471d gitjob rancher/gitjob:v0.1.23 app=gitjob,pod-template-hash=7b97c9c7fd
cattle-fleet-system gitjob-8ccfb5499 0 0 0 277d gitjob rancher/gitjob:v0.1.30 app=gitjob,pod-template-hash=8ccfb5499
cattle-fleet-system gitjob-cc9948fd7 0 0 0 419d gitjob rancher/gitjob:v0.1.26 app=gitjob,pod-template-hash=cc9948fd7
cattle-fleet-system gitjob-ff4f88cfb 0 0 0 107d gitjob rancher/gitjob:v0.1.32-security1 app=gitjob,pod-template-hash=ff4f88cfb
cattle-system rancher-webhook-5444f77b78 0 0 0 471d rancher-webhook rancher/rancher-webhook:v0.2.4 app=rancher-webhook,pod-template-hash=5444f77b78
cattle-system rancher-webhook-576c5b6859 0 0 0 277d rancher-webhook rancher/rancher-webhook:v0.2.6 app=rancher-webhook,pod-template-hash=576c5b6859
cattle-system rancher-webhook-5898d78956 0 0 0 169d rancher-webhook rancher/rancher-webhook:v0.3.0 app=rancher-webhook,pod-template-hash=5898d78956
cattle-system rancher-webhook-6958cfcddf 0 0 0 419d rancher-webhook rancher/rancher-webhook:v0.2.5 app=rancher-webhook,pod-template-hash=6958cfcddf
cattle-system rancher-webhook-6979fbd4bf 0 0 0 573d rancher-webhook rancher/rancher-webhook:v0.2.1 app=rancher-webhook,pod-template-hash=6979fbd4bf
cattle-system rancher-webhook-7b896f7796 1 1 1 36d rancher-webhook rancher/rancher-webhook:v0.3.3 app=rancher-webhook,pod-template-hash=7b896f7796
kube-system coredns-597584b69b 1 1 1 36d coredns rancher/mirrored-coredns-coredns:1.9.4 k8s-app=kube-dns,pod-template-hash=597584b69bhundreds-jackal-3140
05/28/2023, 4:07 PM Copy code
d196e7fcfd68:/var/lib/rancher # kubectl get -o wide -A pod
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
cattle-fleet-local-system fleet-agent-54c474c564-swqrw 1/1 Running 56 (24h ago) 36d 10.42.0.60 local-node <none> <none>
cattle-fleet-system fleet-controller-55c547c6d5-47jjd 1/1 Running 105 (24h ago) 36d 10.42.0.59 local-node <none> <none>
cattle-fleet-system gitjob-74b98dff59-z4wmx 1/1 Running 56 (24h ago) 36d 10.42.0.61 local-node <none> <none>
cattle-system rancher-webhook-7b896f7796-shmtf 1/1 Running 55 (24h ago) 36d 10.42.0.58 local-node <none> <none>
kube-system coredns-597584b69b-dv7b2 1/1 Running 57 (24h ago) 36d 10.42.0.57 local-node <none> <none>hundreds-jackal-3140
05/28/2023, 4:07 PM Copy code
d196e7fcfd68:/var/lib/rancher # kubectl get -o wide -A pvc
No resources foundhundreds-jackal-3140
05/28/2023, 4:07 PM Copy code
d196e7fcfd68:/var/lib/rancher # kubectl get -o wide -A svc
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
cattle-fleet-system gitjob ClusterIP 10.43.57.126 <none> 80/TCP 573d app=gitjob
cattle-system rancher ClusterIP 10.43.56.214 <none> 443/TCP 573d <none>
cattle-system rancher-webhook ClusterIP 10.43.144.180 <none> 443/TCP 573d app=rancher-webhook
cattle-system webhook-service ClusterIP 10.43.222.30 <none> 443/TCP 573d app=rancher-webhook
default kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 573d <none>
kube-system kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP,9153/TCP 573d k8s-app=kube-dnshundreds-jackal-3140
05/28/2023, 4:07 PM Copy code
d196e7fcfd68:/var/lib/rancher # kubectl get -o wide -A ing
No resources foundhundreds-jackal-3140
05/28/2023, 4:11 PMSecrets - Part I
Copy code
d196e7fcfd68:/var/lib/rancher # kubectl get -o wide -A secrets
NAMESPACE NAME TYPE DATA AGE
c-xv8rd default-token-cvcj6 <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
cattle-fleet-clusters-system default-token-8rk2s <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
cattle-fleet-local-system default-token-2t792 <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
cattle-fleet-local-system fleet-agent Opaque 4 573d
cattle-fleet-local-system fleet-agent-token-svtgq <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
cattle-fleet-local-system sh.helm.release.v1.fleet-agent-local.v4946 <http://helm.sh/release.v1|helm.sh/release.v1> 1 38d
cattle-fleet-local-system sh.helm.release.v1.fleet-agent-local.v4947 <http://helm.sh/release.v1|helm.sh/release.v1> 1 37d
cattle-fleet-local-system sh.helm.release.v1.fleet-agent-local.v4948 <http://helm.sh/release.v1|helm.sh/release.v1> 1 37d
cattle-fleet-local-system sh.helm.release.v1.fleet-agent-local.v4949 <http://helm.sh/release.v1|helm.sh/release.v1> 1 36d
cattle-fleet-local-system sh.helm.release.v1.fleet-agent-local.v4950 <http://helm.sh/release.v1|helm.sh/release.v1> 1 36d
cattle-fleet-local-system sh.helm.release.v1.fleet-agent-local.v4951 <http://helm.sh/release.v1|helm.sh/release.v1> 1 36d
cattle-fleet-local-system sh.helm.release.v1.fleet-agent-local.v4952 <http://helm.sh/release.v1|helm.sh/release.v1> 1 36d
cattle-fleet-local-system sh.helm.release.v1.fleet-agent-local.v4953 <http://helm.sh/release.v1|helm.sh/release.v1> 1 36d
cattle-fleet-local-system sh.helm.release.v1.fleet-agent-local.v4954 <http://helm.sh/release.v1|helm.sh/release.v1> 1 36d
cattle-fleet-local-system sh.helm.release.v1.fleet-agent-local.v4955 <http://helm.sh/release.v1|helm.sh/release.v1> 1 36d
cattle-fleet-system default-token-v8m9d <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
cattle-fleet-system fleet-controller-bootstrap-token-tzf5j <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
cattle-fleet-system fleet-controller-token-98d9l <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
cattle-fleet-system gitjob-token-fvll8 <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
cattle-fleet-system sh.helm.release.v1.fleet-crd.v4 <http://helm.sh/release.v1|helm.sh/release.v1> 1 277d
cattle-fleet-system sh.helm.release.v1.fleet-crd.v5 <http://helm.sh/release.v1|helm.sh/release.v1> 1 269d
cattle-fleet-system sh.helm.release.v1.fleet-crd.v6 <http://helm.sh/release.v1|helm.sh/release.v1> 1 169d
cattle-fleet-system sh.helm.release.v1.fleet-crd.v7 <http://helm.sh/release.v1|helm.sh/release.v1> 1 107d
cattle-fleet-system sh.helm.release.v1.fleet-crd.v8 <http://helm.sh/release.v1|helm.sh/release.v1> 1 36d
cattle-fleet-system sh.helm.release.v1.fleet.v10 <http://helm.sh/release.v1|helm.sh/release.v1> 1 36d
cattle-fleet-system sh.helm.release.v1.fleet.v6 <http://helm.sh/release.v1|helm.sh/release.v1> 1 269d
cattle-fleet-system sh.helm.release.v1.fleet.v7 <http://helm.sh/release.v1|helm.sh/release.v1> 1 169d
cattle-fleet-system sh.helm.release.v1.fleet.v8 <http://helm.sh/release.v1|helm.sh/release.v1> 1 107d
cattle-fleet-system sh.helm.release.v1.fleet.v9 <http://helm.sh/release.v1|helm.sh/release.v1> 1 36d
cattle-fleet-system stv-aggregation Opaque 3 277d
cattle-global-data activedirectoryconfig-serviceaccountpassword Opaque 1 573d
cattle-global-data cc-9gdzd Opaque 4 573d
cattle-global-data cluster-serviceaccounttoken-fh9q5 Opaque 1 26d
cattle-global-data cluster-vspherevcenter-wgtbz Opaque 1 26d
cattle-global-data default-token-m22zk <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
cattle-global-nt default-token-t6tg7 <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
cattle-impersonation-system cattle-impersonation-u-b4qkhsnliz-token-kzpf7 <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
cattle-impersonation-system cattle-impersonation-u-bhvr65xak6-token-jxpdn <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
cattle-impersonation-system cattle-impersonation-u-gprcti7v3h-token-g8x27 <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
cattle-impersonation-system cattle-impersonation-u-mo773yttt4-token-2jls7 <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
cattle-impersonation-system default-token-82sx8 <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
cattle-system c-c-cvcjb Opaque 1 26d
cattle-system c-c-xv8rd Opaque 1 573d
cattle-system cattle-webhook-ca <http://kubernetes.io/tls|kubernetes.io/tls> 2 573d
cattle-system cattle-webhook-tls <http://kubernetes.io/tls|kubernetes.io/tls> 2 573d
cattle-system default-token-8hknw <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
cattle-system git-webhook-api-service-token <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 277d
cattle-system git-webhook-api-service-token-f6942 <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
cattle-system mc-m-46ltb Opaque 2 26d
cattle-system mc-m-6xnqc Opaque 2 26d
cattle-system mc-m-722h6 Opaque 2 26d
cattle-system mc-m-8r68q Opaque 2 26d
cattle-system mc-m-9qhxv Opaque 2 26d
cattle-system mc-m-mfcdw Opaque 2 27d
cattle-system mc-m-pd98g Opaque 2 26d
cattle-system mc-m-q9qw2 Opaque 2 25d
cattle-system mc-m-qbc9k Opaque 2 420d
cattle-system mc-m-tgh6r Opaque 2 36d
cattle-system mc-m-vtgls Opaque 2 573d
cattle-system mc-m-w4jhc Opaque 2 169dhundreds-jackal-3140
05/28/2023, 4:12 PMSecrets - Port II
Copy code
d196e7fcfd68:/var/lib/rancher # kubectl get -o wide -A secrets
NAMESPACE NAME TYPE DATA AGE
cattle-system rancher-webhook-tls <http://kubernetes.io/tls|kubernetes.io/tls> 2 573d
cattle-system rancher-webhook-token-qxpkn <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
cattle-system sh.helm.release.v1.rancher-webhook.v2 <http://helm.sh/release.v1|helm.sh/release.v1> 1 471d
cattle-system sh.helm.release.v1.rancher-webhook.v3 <http://helm.sh/release.v1|helm.sh/release.v1> 1 419d
cattle-system sh.helm.release.v1.rancher-webhook.v4 <http://helm.sh/release.v1|helm.sh/release.v1> 1 277d
cattle-system sh.helm.release.v1.rancher-webhook.v5 <http://helm.sh/release.v1|helm.sh/release.v1> 1 169d
cattle-system sh.helm.release.v1.rancher-webhook.v6 <http://helm.sh/release.v1|helm.sh/release.v1> 1 36d
cattle-system tls-rancher-internal <http://kubernetes.io/tls|kubernetes.io/tls> 2 573d
cattle-system tls-rancher-internal-ca <http://kubernetes.io/tls|kubernetes.io/tls> 2 573d
cluster-fleet-default-c-cvcjb-b2b3e6f2ad36 request-m4d62-aaee34d1-74fc-42db-843f-03ef879c9cf1-token <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 26d
cluster-fleet-local-local-1a3d67d0a899 default-token-fpps8 <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
cluster-fleet-local-local-1a3d67d0a899 request-g4hh9-387dc684-b153-4732-aa4c-2f803f8c106c-token-vzwlp <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
default default-token-zg7lz <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
fleet-default c-cvcjb-kubeconfig Opaque 2 26d
fleet-default default-token-rwrnj <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
fleet-local default-token-4qj6x <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
fleet-local local-cluster Opaque 3 573d
fleet-local local-kubeconfig Opaque 2 573d
kube-node-lease default-token-nr9n2 <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
kube-public default-token-8kd7r <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
kube-system attachdetach-controller-token-tx4xq <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
kube-system certificate-controller-token-9jkxq <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
kube-system clusterrole-aggregation-controller-token-nxls5 <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
kube-system coredns-token-68jcr <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
kube-system cronjob-controller-token-245tc <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
kube-system daemon-set-controller-token-7t4px <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
kube-system default-token-l9zsp <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
kube-system deployment-controller-token-n8ccw <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
kube-system disruption-controller-token-rtklm <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
kube-system endpoint-controller-token-xt5k5 <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
kube-system endpointslice-controller-token-xkhss <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
kube-system endpointslicemirroring-controller-token-tgxxn <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
kube-system ephemeral-volume-controller-token-vdg49 <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
kube-system expand-controller-token-kptd9 <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
kube-system generic-garbage-collector-token-nqf5t <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
kube-system horizontal-pod-autoscaler-token-pz7c5 <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
kube-system job-controller-token-9trwf <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
kube-system k3s-serving <http://kubernetes.io/tls|kubernetes.io/tls> 2 573d
kube-system namespace-controller-token-7rtc2 <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
kube-system node-controller-token-29ldg <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
kube-system persistent-volume-binder-token-hqdst <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
kube-system pod-garbage-collector-token-jn7hm <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
kube-system pv-protection-controller-token-4px5z <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
kube-system pvc-protection-controller-token-bgb9r <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
kube-system replicaset-controller-token-8lnkb <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
kube-system replication-controller-token-h4sz2 <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
kube-system resourcequota-controller-token-9dhw5 <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
kube-system root-ca-cert-publisher-token-5sspq <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
kube-system service-account-controller-token-m5gn4 <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
kube-system statefulset-controller-token-nbfgn <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
kube-system ttl-after-finished-controller-token-8wpmw <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
kube-system ttl-controller-token-qp5d6 <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
local default-token-z4nn2 <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
p-7f55m default-token-nzwhx <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
p-7hhxd default-token-f276q <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
p-fb5h2 default-token-nfwqz <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
p-gr86m default-token-r6nmd <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573d
u-4rzu6btj4l default-token-m4pf5 <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 529d
u-6xo76pkfpp default-token-kn5tm <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 529d
u-bhcttyfs2r default-token-p8r55 <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 428d
u-bvngnfuomk default-token-d7fqd <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 570d
u-cx63gq3clv default-token-dxxwd <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 550d
u-z7bgaodeek default-token-sbnd8 <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 570d
user-lfdzw default-token-6k758 <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 573da
acceptable-match-53099
05/30/2023, 5:42 AMHi @hundreds-jackal-3140
It seems like Expired Webhook Certificates issue https://stackoverflow.com/questions/69720179/unable-to-add-cluster-through-rke-https-rancher-webhook-cattle-system-svc443
acceptable-match-53099
05/30/2023, 5:42 AMh
hundreds-jackal-3140
05/30/2023, 11:49 AMIt generated a new cattle-webhook-tls certificate.
But no success yet. No web interface available.
hundreds-jackal-3140
05/30/2023, 11:49 AM Copy code
d196e7fcfd68:/var/lib/rancher # kubectl get -o wide -A secrets | grep webhook
cattle-system cattle-webhook-ca <http://kubernetes.io/tls|kubernetes.io/tls> 2 575d
cattle-system cattle-webhook-tls <http://kubernetes.io/tls|kubernetes.io/tls> 2 21m
cattle-system git-webhook-api-service-token <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 279d
cattle-system git-webhook-api-service-token-f6942 <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 575d
cattle-system rancher-webhook-tls <http://kubernetes.io/tls|kubernetes.io/tls> 2 575d
cattle-system rancher-webhook-token-qxpkn <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token> 3 575d
cattle-system sh.helm.release.v1.rancher-webhook.v2 <http://helm.sh/release.v1|helm.sh/release.v1> 1 473d
cattle-system sh.helm.release.v1.rancher-webhook.v3 <http://helm.sh/release.v1|helm.sh/release.v1> 1 421d
cattle-system sh.helm.release.v1.rancher-webhook.v4 <http://helm.sh/release.v1|helm.sh/release.v1> 1 279d
cattle-system sh.helm.release.v1.rancher-webhook.v5 <http://helm.sh/release.v1|helm.sh/release.v1> 1 171d
cattle-system sh.helm.release.v1.rancher-webhook.v6 <http://helm.sh/release.v1|helm.sh/release.v1> 1 38dhundreds-jackal-3140
06/04/2023, 11:38 PMI was able to fix.
Thanks.
w
witty-secretary-79826
06/28/2023, 8:38 PMGood afternoon - we are seeing a very similar issue in adding "tls-rancher-ingress" secret manually in the cattle-system namespace. We are getting "error: failed to create secret Internal error occurred: failed calling webhook "rancher.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation?timeout=10s": context deadline exceeded" We are just wondering how did you resolve your issue? Also, we are also running Rancher MCM 2.7.1
g
great-advantage-74442
08/10/2023, 11:49 AMwe're facing a similar error too
https://rancher-users.slack.com/archives/C3ASABBD1/p1691406384774209
742 Views