billions-easter-91774
08/18/2022, 9:52 PMhundreds-evening-84071
08/19/2022, 1:58 PMv1.23.7
Have following config:
1 control plane/etcd node (rhel 8)
2 Windows worker nodes. 1 is server 2022 and another is server 2019
For some reason, I see one coredns pod stays in pending state; looking at the log for that pod it does not contain any entry.
So, am trying to figure out why this is?
# kubectl get pods -n kube-system | grep dns
helm-install-rke2-coredns-mn6h6 0/1 Completed 0 31d
rke2-coredns-rke2-coredns-86b4fbd678-brpcm 0/1 Pending 0 4s
rke2-coredns-rke2-coredns-86b4fbd678-csqxh 1/1 Running 0 22m
rke2-coredns-rke2-coredns-autoscaler-65c9bb465d-tcv27 1/1 Running 0 31d
billions-easter-91774
08/20/2022, 12:24 PMambitious-plastic-3551
08/20/2022, 9:53 PMambitious-plastic-3551
08/20/2022, 9:53 PMimportant-umbrella-22006
08/22/2022, 11:50 AMTTLAfterFinished=true
But when i change kube-controller-manager and kube-apiserver pods configuration, these gets restarted and revise configuration file to original. Does it control by fleet?
can someone help me with the process of how to modify kube-controller-manager and kube-apiserver pods configuration?great-photographer-94826
08/22/2022, 1:14 PM--cluster-reset \
--cluster-reset-restore-path=/tmp/etcd-snapshot-2022-08-22 \
--token=mytoken
Versions
• OS: "Ubuntu 20.04"
• rke2 version I use: "1.21.5-rke2r2"
• etcd image I use: "etcd:v3.4.13-k3s1-build20210223"
ETCD does not start due to the following error
2022-08-22 12:40:13.947166 I | etcdmain: Loading server configuration from "/var/lib/rancher/rke2/server/db/etcd/config". Other configuration command line flags and environment variables will be ignored if provided.
2022-08-22 12:40:13.947196 E | etcdmain: error verifying flags, open /var/lib/rancher/rke2/server/db/etcd/config: permission denied. See 'etcd --help'.
ETCD config permissions
-rw------- 1 etcd etcd 1043 Aug 22 15:03 /var/lib/rancher/rke2/server/db/etcd/config
ETCD pod
/var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml
...
securityContext:
runAsGroup: 1001
runAsUser: 1001
...
etcd user/groups
id etcd
uid=1001(etcd) gid=1001(etcd) groups=1001(etcd)
RKE2 config.yaml
profile: cis-1.6
tls-san:
- ******
- ******
disable-cloud-controller: true
etcd-snapshot-schedule-cron: "0 */12 * * *"
etcd-snapshot-retention: 5
secrets-encryption: true
stale-painting-80203
08/23/2022, 5:23 AMkubectl logs etcd-harbor1-pool1-114b4517-k4lh7 -n kube-system | grep rejected
{"level":"warn","ts":"2022-08-23T04:08:54.054Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:51538","server-name":"","error":"EOF"}
Number of pods are in pending state:
NAMESPACE NAME READY STATUS RESTARTS AGE
calico-system pod/calico-kube-controllers-677d488b5f-stpbg 0/1 Pending 0 55m
calico-system pod/calico-node-7hntn 0/1 Running 0 55m
calico-system pod/calico-typha-66d8ff6684-rnlx6 0/1 Pending 0 55m
cattle-system pod/cattle-cluster-agent-8df9b48fd-2dkdm 0/1 Pending 0 56m
kube-system pod/etcd-harbor1-pool1-114b4517-k4lh7 1/1 Running 0 56m
kube-system pod/harvester-cloud-provider-748f954ffb-ml8bd 1/1 Running 0 56m
kube-system pod/harvester-csi-driver-controllers-779c557d47-8h7qb 0/3 Pending 0 56m
kube-system pod/harvester-csi-driver-controllers-779c557d47-gvb9d 0/3 Pending 0 56m
kube-system pod/harvester-csi-driver-controllers-779c557d47-rjvwt 0/3 Pending 0 56m
kube-system pod/helm-install-harvester-cloud-provider-z87qt 0/1 Completed 0 56m
kube-system pod/helm-install-harvester-csi-driver-m9pkj 0/1 Completed 0 56m
kube-system pod/helm-install-rke2-calico-crd-scvcm 0/1 Completed 0 56m
kube-system pod/helm-install-rke2-calico-kmppr 0/1 Completed 1 56m
kube-system pod/helm-install-rke2-coredns-plrx2 0/1 Completed 0 56m
kube-system pod/helm-install-rke2-ingress-nginx-dvrd4 0/1 Pending 0 56m
kube-system pod/helm-install-rke2-metrics-server-bwvwv 0/1 Pending 0 56m
kube-system pod/kube-apiserver-harbor1-pool1-114b4517-k4lh7 1/1 Running 0 56m
kube-system pod/kube-controller-manager-harbor1-pool1-114b4517-k4lh7 1/1 Running 0 55m
kube-system pod/kube-proxy-harbor1-pool1-114b4517-k4lh7 1/1 Running 0 56m
kube-system pod/kube-scheduler-harbor1-pool1-114b4517-k4lh7 1/1 Running 0 55m
kube-system pod/rke2-coredns-rke2-coredns-76cb76d66-vl2wg 0/1 Pending 0 56m
kube-system pod/rke2-coredns-rke2-coredns-autoscaler-58867f8fc5-whhwc 0/1 Pending 0 56m
tigera-operator pod/tigera-operator-6457fc8c7c-s97d9 1/1 Running 0 56m
Any suggestion on how to debug the issue or possible root cause?future-monitor-61871
08/23/2022, 9:41 PMechoing-oxygen-99290
08/23/2022, 9:52 PM/dev/root
file system being at 98% utilization when running df -h
. I attached a new disk and created a new file system with a 30gb ssd. I am not seeing how to get rancher to use this new filesystem. Could someone give me any insight into what I am not understanding? Thanksnarrow-noon-75604
08/24/2022, 5:04 AMip a list $INTERFACE
Output:
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:9b:3a:cb brd ff:ff:ff:ff:ff:ff
inet 192.168.10.71/24 brd 192.168.10.255 scope global ens160
valid_lft forever preferred_lft forever
inet 192.168.10.74/32 scope global ens160
valid_lft forever preferred_lft forever
inet6 fe80::250:56ff:fe9b:3acb/64 scope link
valid_lft forever preferred_lft forever
After sometime VIP is disappearing,
ip a list $INTERFACE
Output:
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:9b:3a:cb brd ff:ff:ff:ff:ff:ff
inet 192.168.10.71/24 brd 192.168.10.255 scope global ens160
valid_lft forever preferred_lft forever
inet6 fe80::250:56ff:fe9b:3acb/64 scope link
valid_lft forever preferred_lft forever
Please let me know if I am missing anything.best-actor-8484
08/24/2022, 4:48 PMable-engineer-22050
08/24/2022, 5:02 PMbored-rain-98291
08/25/2022, 5:49 PMbored-rain-98291
08/25/2022, 5:59 PMbored-rain-98291
08/25/2022, 5:59 PM/tmp/etcd-download-test/etcdctl get secret1
{"level":"warn","ts":"2022-08-25T09:59:39.170-0700","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"<endpoint://client-ddc2ced7-320f-4be9-a25d-68c07645e407/127.0.0.1:2379>","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: all SubConns are in TransientFailure, latest connection error: connection closed"}
billions-easter-91774
08/26/2022, 3:50 PMbillions-easter-91774
08/26/2022, 4:15 PMfuture-monitor-61871
08/26/2022, 7:03 PMambitious-plastic-3551
08/26/2022, 9:18 PMambitious-plastic-3551
08/26/2022, 9:27 PMambitious-plastic-3551
08/26/2022, 9:28 PMambitious-plastic-3551
08/27/2022, 8:42 AMbillions-easter-91774
08/28/2022, 6:08 AMnarrow-cpu-6472
08/29/2022, 11:17 AMnarrow-cpu-6472
08/29/2022, 11:32 AMnarrow-cpu-6472
08/29/2022, 11:32 AMbored-rain-98291
08/30/2022, 6:57 PMbroad-farmer-70498
09/01/2022, 1:52 PMbroad-farmer-70498
09/01/2022, 1:53 PM