creamy-crowd-89310
09/20/2022, 8:51 AMcreamy-crowd-89310
09/20/2022, 8:52 AMbrash-planet-10109
09/20/2022, 8:58 AMcreamy-crowd-89310
09/20/2022, 9:16 AMbrash-planet-10109
09/20/2022, 9:33 AMbitter-tailor-4440
09/20/2022, 9:57 AMcreamy-crowd-89310
09/20/2022, 10:03 AMbitter-tailor-4440
09/20/2022, 10:07 AMbitter-tailor-4440
09/20/2022, 10:08 AMbitter-tailor-4440
09/20/2022, 10:09 AMcreamy-crowd-89310
09/20/2022, 10:10 AMbusy-easter-55684
09/20/2022, 10:36 AMsalmon-carpenter-62625
09/20/2022, 11:20 AMError: template: rancher-monitoring/templates/rancher-monitoring/clusterrole.yaml:116:13: executing "rancher-monitoring/templates/rancher-monitoring/clusterrole.yaml" at <include "call-nested" (list . "grafana" "grafana.fullname")>: error calling include: template: rancher-monitoring/templates/_helpers.tpl:21:4: executing "call-nested" at <include $template (dict "Chart" (dict "Name" (last $subchart)) "Values" $values "Release" $dot.Release "Capabilities" $dot.Capabilities)>: error calling include: template: no template "grafana.fullname" associated with template "gotpl"
better-chef-44669
09/20/2022, 11:49 AMcrooked-cat-21365
09/20/2022, 1:11 PM--address eno3 --internal-address eno1
, but apparently this doesn't affect routing. The pods on the worker node still use eno1 for outgoing traffic only, probably because thats the default route on the worker nodes. The dedicated eno3 interface connected to the external gateway is ignored. I wonder if it is possible to configure networking beyond --address and --internal-address on the rancher-agent command line?
The whole docker command line to configure my worker nodes looks like this
docker run -d \
--privileged \
--restart=unless-stopped \
--net=host \
-v /etc/kubernetes:/etc/kubernetes \
-v /var/run:/var/run \
rancher/rancher-agent:v2.6.8 \
--server <https://rancher01.example.com> \
--address eno3 \
--internal-address eno1 \
--token xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx \
--worker
Do I have to "struggle" with canal to change routing manually? Every helpful hint is highly appreciated.wooden-planet-93714
09/20/2022, 2:34 PMkubernetesVersion: v1.23.10-rancher1-1
)
All nodes are the same hardware-wise and software-wise. All were with ubuntu 20.04. I have upgraded 3 of them to 22.04.
The problem is there is difference in kubelet behavior. On one node - everything looks fine. But the kubelet does not seem verbose enough (maybe it is normal for v=2 )
all the logs are of the type:
I0920 14:08:37.253539 3147 container_manager_linux.go:511] "Discovered runtime cgroup name" cgroupName="/system.slice/docker.service"
The other 2 nodes however had problems after the upgrade. Rancher started to spawn unlimited agent containers bricking the servers due to "too many files opened". I had to remove them from the cluster and add them again and it worked. However now there are those log lines on both nodes:
E0920 13:42:03.850145 2845 summary_sys_containers.go:83] "Failed to get system container stats" err="failed to get cgroup stats for \"/../docker.service\": failed to get container info for \"/../docker.service\": unknown container \"/../docker.service\"" containerName="/../docker.service
I see many more orphaned pods cleanup fails also, so I am hesitant to upgrade the last node.
Can someone give me a hint as to want to look for to fix the cgroup issue? The systemd options are the same on all 3.nutritious-crayon-45180
09/20/2022, 6:09 PMflaky-shampoo-86024
09/20/2022, 11:56 PMbusy-country-65143
09/21/2022, 6:04 AMagreeable-oil-87482
09/21/2022, 7:07 AMmelodic-football-95460
09/21/2022, 8:15 AMlate-vr-98727
09/21/2022, 9:12 AMlocalClusterAuthEndpoint: enabled: true
. And then it did give me kubeconfig with all the master nodes info. But when I use once of the context defined in the kubeconfig. I get the following errorquick-lawyer-7096
09/21/2022, 9:18 AMrough-flower-24438
09/21/2022, 9:32 AMcreamy-crowd-89310
09/21/2022, 9:35 AMrough-flower-24438
09/21/2022, 9:41 AMmany-iron-73190
09/21/2022, 10:03 AMcuddly-jordan-17092
09/21/2022, 11:09 AMopenssl s_client -connect kube1001:6443 2>&1 | grep issuer
verify error:num=20:unable to get local issuer certificate
issuer=/CN=k3s-server-ca@1649270763
openssl s_client -connect kube1001:443 2>&1 | grep issuer
verify error:num=20:unable to get local issuer certificate
issuer=/CN=k3s-server-ca@1649270763
openssl s_client -connect kube1001:80 2>&1 | grep issuer
verify error:num=20:unable to get local issuer certificate
issuer=/CN=k3s-server-ca@1649270763
quiet-football-25667
09/21/2022, 11:16 AMcrooked-cat-21365
09/21/2022, 1:22 PMapiVersion: v1
kind: Namespace
metadata:
name: cert-manager-test
annotations:
<http://cattle.io/status|cattle.io/status>: >-
{"Conditions":[{"Type":"ResourceQuotaInit","Status":"True","Message":"","LastUpdateTime":"2022-09-20T11:12:04Z"},{"Type":"InitialRolesPopulated","Status":"True","Message":"","LastUpdateTime":"2022-09-20T11:12:04Z"}]}
<http://kubectl.kubernetes.io/last-applied-configuration|kubectl.kubernetes.io/last-applied-configuration>: >
{"apiVersion":"v1","kind":"Namespace","metadata":{"annotations":{},"name":"cert-manager-test"}}
<http://lifecycle.cattle.io/create.namespace-auth|lifecycle.cattle.io/create.namespace-auth>: 'true'
<http://field.cattle.io/containerDefaultResourceLimit|field.cattle.io/containerDefaultResourceLimit>: '{}'
# key: string
creationTimestamp: '2022-09-20T11:12:03Z'
deletionTimestamp: '2022-09-20T11:13:07Z'
labels:
<http://kubernetes.io/metadata.name|kubernetes.io/metadata.name>: cert-manager-test
# key: string
resourceVersion: '116945107'
uid: 9918aab8-1f1c-48fe-b8b0-9d64a98a5084
fields:
- cert-manager-test
- Terminating
- 25h
spec:
finalizers:
- kubernetes
# - string
__clone: true
If I try to remove the finalizer and save, then I get an error popup in red
Operation cannot be fulfilled on namespaces "cert-manager-test": StorageError: invalid object, Code: 4, Key: /registry/namespaces/cert-manager-test, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 9918aab8-1f1c-48fe-b8b0-9d64a98a5084, UID in object meta:
Every helpful hint is highly appreciated