rancher-users #general

stocky-article-82001

11/23/2022, 12:55 PM

Or will I have to remove nodes and add them back with Internal IPs

☝️ 1

🧵 1

flat-translator-32204

11/23/2022, 5:02 PM

Hello, Any guide on how to deploy k3s on AWS EC2 instances?

loud-lawyer-79863

11/23/2022, 5:17 PM

Hello everyone, hope you are doing well. I have a rancher-ha server running on GKe. Seems rancher is created automatically a bunch of namespace like p-28qkg but do not seems to be cleaned them. anyone knows what are these for ? I have 70 of them without any ressources in it

damp-painting-69352

11/23/2022, 7:17 PM

@polite-mouse-38756 Are you around for a second to answer some questions?

blue-arm-71737

11/23/2022, 7:50 PM

Has anyone from Rancher looked at supporting k8s 1.26 yet? It's a mess of dependency issues between controller-runtime, lasso, CAPI and client-go.

👀 3

loud-lawyer-79863

11/23/2022, 8:47 PM

hmm so I made some nice mistake. I cleaned some empty namespace that I thought were created by some old CI we had. But I think I deleted a couple of these : https://github.com/rancher/rancher/issues/33119 Now I have a couple of clusters in rancher that are unavailable any idea if there a command to rebuild these or something ?

straight-appointment-92464

11/24/2022, 5:43 AM

I have installed rancher desktop on windows, can any one give me the steps to install airlfow on rancher desktop

polite-king-74071

11/24/2022, 1:01 PM

has anyone installed k8s on Ubuntu 22.04?

full-park-34540

11/24/2022, 2:58 PM

👋 Hi everyone! has anyone here came across or have any idea about

invalid bearer token, service account token has expired

Multiple pods from different namespace are in the ContainerCreating state including calico-kube-controllers and coredns in the kube-system namespace

Copy code

kube-system        calico-kube-controllers-54965c7ccb-rvksv                 0/1     ContainerCreating          0               5h39m
kube-system        coredns-d76bd69b-4dtbl                                   0/1     ContainerCreating          0               5h39m
kube-system        kube-sriov-cni-ds-arm64-5j69h                            0/1     ContainerCreating          0               5h38m
kube-system        kube-sriov-cni-ds-arm64-5ljzh                            0/1     ContainerCreating          0               5h38m
kube-system        kube-sriov-cni-ds-arm64-5mmfz                            0/1     ContainerCreating          0               5h38m
kube-system        kube-sriov-cni-ds-arm64-5zkvh                            0/1     ContainerCreating          0               5h38m
kube-system        kube-sriov-cni-ds-arm64-8z5zc                            0/1     ContainerCreating          0               5h38m
kube-system        kube-sriov-cni-ds-arm64-bpk9q                            0/1     ContainerCreating          0               5h38m
kube-system        kube-sriov-cni-ds-arm64-gqks9                            0/1     ContainerCreating          0               5h38m
kube-system        kube-sriov-cni-ds-arm64-hsx9k                            0/1     ContainerCreating          0               5h38m
kube-system        kube-sriov-cni-ds-arm64-jrlpb                            0/1     ContainerCreating          0               5h38m
kube-system        kube-sriov-cni-ds-arm64-krt9n                            0/1     ContainerCreating          0               5h38m
kube-system        kube-sriov-cni-ds-arm64-m42j4                            0/1     ContainerCreating          0               5h38m
kube-system        kube-sriov-cni-ds-arm64-nnrw2                            0/1     ContainerCreating          0               5h38m
kube-system        kube-sriov-cni-ds-arm64-nqshk                            0/1     ContainerCreating          0               5h38m
kube-system        kube-sriov-cni-ds-arm64-q2stf                            0/1     ContainerCreating          0               5h38m
kube-system        kube-sriov-cni-ds-arm64-r8b9d                            0/1     ContainerCreating          0               5h38m
kube-system        kube-sriov-cni-ds-arm64-vn2gj                            0/1     ContainerCreating          0               5h38m
kube-system        kube-sriov-cni-ds-arm64-vstrn                            0/1     ContainerCreating          0               5h38m
kube-system        kube-sriov-cni-ds-arm64-wdbcj                            0/1     ContainerCreating          0               5h38m
kube-system        kube-sriov-cni-ds-arm64-zxgrl                            0/1     ContainerCreating          0               5h38m
kube-system        metrics-server-7cd5fcb6b7-zbkfj                          0/1     ContainerCreating          0               5

kubectl describe pod throws the following error for all the pods stuck in the ContainerCreating state

Copy code

Warning FailedCreatePodSandBox 55s (x1755 over 6h24m) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "986af2c3af9e173a6f4084fcf73795bccf3b996c98f5a80b9f0a04a554cb8a21": plugin type="multus" name="multus-cni-network" failed (add): [cdi/cdi-apiserver-cdb4566f6-vq2zx/d82b198a-de67-4463-8e76-884e022fdc99:k8s-pod-network]: error adding container to network "k8s-pod-network": plugin type="calico" failed (add): error getting ClusterInformation: connection is unauthorized: Unauthorized

Journal logs flooded with the following error

Copy code

Nov 24 13:00:43 dev3-kv-02 k3s[2648839]: E1124 13:00:43.020907 2648839 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, service account token has expired]"

full-park-34540

11/24/2022, 3:41 PM

to clarify We are using k3s + canal as CNI

loud-lawyer-79863

11/24/2022, 4:00 PM

So I deleted by accident some empty rancher managed namespace on our rancher cluster (so I thought ! but actually super important ns https://github.com/rancher/rancher/issues/33119 ) Trying to restore a backup but it is able to restore everything except what looks like what I deleted ... Anyone ever seen this ?

Copy code

ERRO[2022/11/24 15:38:43] Error restoring resource grb-cn99w of type <http://management.cattle.io/v3|management.cattle.io/v3>, Resource=globalrolebindings: restoreResource: err updating resource <http://GlobalRoleBinding.management.cattle.io|GlobalRoleBinding.management.cattle.io> "grb-cn99w" is invalid: metadata.deletionGracePeriodSeconds: Invalid value: 0: field is immutable 
ERRO[2022/11/24 15:38:45] Error restoring cluster-scoped resources [error restoring grb-lr9qh of type <http://management.cattle.io/v3|management.cattle.io/v3>, Resource=globalrolebindings: restoreResource: err updating resource <http://GlobalRoleBinding.management.cattle.io|GlobalRoleBinding.management.cattle.io> "grb-lr9qh" is invalid: metadata.deletionGracePeriodSeconds: Invalid value: 0: field is immutable error restoring grb-cn99w of type <http://management.cattle.io/v3|management.cattle.io/v3>, Resource=globalrolebindings: restoreResource: err updating resource <http://GlobalRoleBinding.management.cattle.io|GlobalRoleBinding.management.cattle.io> "grb-cn99w" is invalid: metadata.deletionGracePeriodSeconds: Invalid value: 0: field is immutable] 
ERRO[2022/11/24 15:40:03] Error restoring cluster-scoped resources [error restoring grb-lr9qh of type <http://management.cattle.io/v3|management.cattle.io/v3>, Resource=globalrolebindings: restoreResource: err updating resource <http://GlobalRoleBinding.management.cattle.io|GlobalRoleBinding.management.cattle.io> "grb-lr9qh" is invalid: metadata.deletionGracePeriodSeconds: Invalid value: 0: field is immutable error restoring grb-cn99w of type <http://management.cattle.io/v3|management.cattle.io/v3>, Resource=globalrolebindings: restoreResource: err updating resource <http://GlobalRoleBinding.management.cattle.io|GlobalRoleBinding.management.cattle.io> "grb-cn99w" is invalid: metadata.deletionGracePeriodSeconds: Invalid value: 0: field is immutable] 
ERRO[2022/11/24 15:40:03] error syncing 'restore-rzrb6': handler restore: error restoring cluster-scoped resources, check logs for exact error, requeuing

stocky-article-82001

11/24/2022, 5:06 PM

Can you assign a node an internal IP while it is already in a cluster or will I have to remove it and re-add it. We’re moving to use an internal VPN for all node <-> node communications.

rich-sundown-87208

11/24/2022, 6:57 PM

Ello 🙂 my rancher/rancher GUI has suddenly started telling me that it cannot connect to the cluster - fun fun

rich-sundown-87208

11/24/2022, 6:58 PM

mostly timeouts in the logs, so I'm thinking SSL issue of some sort?

rich-sundown-87208

11/24/2022, 6:59 PM

I connected to a shell inside the container and couldn't curl google

rich-sundown-87208

11/24/2022, 7:01 PM

"Cluster health check failed: Failed to communicate with API server during namespace check: Get "https://10.43.0.1:443/api/v1/namespaces/kube-system?timeout=45s": context deadline exceeded"

rich-sundown-87208

11/24/2022, 7:02 PM

Copy code

curl <https://10.43.0.1:443/api/v1/namespaces/kube-system?timeout=45s>
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: <https://curl.haxx.se/docs/sslcerts.html>

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

rich-sundown-87208

11/24/2022, 7:02 PM

Copy code

curl -k <https://10.43.0.1:443/api/v1/namespaces/kube-system?timeout=45s>
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
}

rich-sundown-87208

11/24/2022, 7:03 PM

and the response is almost instant, which is in conflict with "context deadline exceeded"

rich-sundown-87208

11/24/2022, 8:37 PM

okay, so I downloaded the ca pem from here; https://curl.haxx.se/ca/cacert.pem

rich-sundown-87208

11/24/2022, 8:37 PM

and now i can hit google, but I still can't get to 10.43.0.1

elegant-librarian-62192

11/25/2022, 12:34 AM

Hi folks, after quitting and relaunching RD, I attempt to start a container that was previously open, and I'm seeing:

FATA[0000] failed to create shim task: OCI runtime create failed: runc create failed: cannot allocate tty if runc will detach without setting console socket: unknown

The container was open with a Bash console at the point when I quit, which I suspect may be part of the problem. No problem to recreate, but curious if there's a quick way to recover from this failure mode in future.

elegant-librarian-62192

11/25/2022, 2:47 AM

Also noticed that the QEMU process floats between 85%-115% of a core regardless of whether or not there are any containers (other than all the k3s stuff) running. Is this expected behavior?

wonderful-airplane-86139

11/25/2022, 3:47 AM

Hi, running rancher 2.4 cluster on aws ec2 and from yesterday pods on newly added nodes don't resolve dns. Anyone has an idea?

silly-jordan-81965

11/25/2022, 7:22 AM

Good Morning - we have a strange behavior regarding Rancher snapshot handling. In one cluster running cilium and vxlan when we take snapshots we receive status on the snapshots and we can see that Rancher reports Status, Name and size of the snapshot. However we have 2 other clusters that is also running Cilium and Bird. These clusters Doesn’t report backup status of taken snapshot or the size of the snapshot but when we do check the control-plane node we can see the snapshot on that node and we can see that the snapshot contain data. On one of the clusters this has worked but stopped working. What can be causing this? Im think we are missing a port opening somewhere - but having checked with our network team and according to them we have all openings that we need. Anyone have an idea where to start to look?

acoustic-businessperson-24722

11/25/2022, 9:03 AM

Hey, has anyone experience with wrong metrics about used and free ram?

silly-jordan-81965

11/25/2022, 9:15 AM

Anyone know what the below secret is referring to?

Copy code

level=error msg="[K8s] received secret to process that was older than the last secret operated on. (256559510 vs 256559619)"
 level=error msg="error syncing 'fleet-default/bootstrap-template-d4lsj-machine-plan': handler secret-watch: secret received was too old, requeuing"

salmon-portugal-38278

11/25/2022, 1:02 PM

Hello, Anyone around have any experience with the new

rancher-monitoring

? Ive been trying to add a

PrometheusRule

using Rancher UI following the docs here https://docs.ranchermanager.rancher.io/v2.6/how-to-guides/advanced-user-guides/monitoring-v2[…]uration-guides/advanced-configuration/prometheusrules After creating the object, I tried to see it's really there in my prometheus dashboard. Unfort, it's not there I looked in the

rulefile

configmap that watches all the prometheusrules and it's not there too !

rough-jordan-89140

11/25/2022, 2:12 PM

Hey all, We would like to add a custom tolerations in cattle-cluster-agent deployment for a user added taint in node, so we are exactly hitting this issue - https://github.com/rancher/rancher/issues/16050 Saw the work around :

Copy code

An empty key with operator Exists matches all keys, values and effects which means this will tolerate everything.
tolerations:
- operator: "Exists"

and

Copy code

kubectl -n cattle-system patch ds/cattle-node-agent -p '{"spec":{"template":{"spec":{"tolerations":[{"operator":"Exists"}]}}}}'

do you think that the first code snippet can be added to the cattle-cluster-agent deployment manifest or it will work only with kubectl patch? We would like to make this persist with code. Please let us know if we really can add a toleration to cattle-cluster-agent deployment manifest, thank you 🙇

creamy-room-58344

11/25/2022, 3:17 PM

Hello there! I'm trying to configure cloud-config on rancher to use with vsphere Most things work but I can't set my specific network configuration I tried pasting this into the Node Template Cloud-config box and also into an extra parameter called

guestinfo.metadata

base64 encoded with

guestinfo.metadata.encoding: base64

as well as

gzip+base64

Copy code

network:
  version: 2
  ethernets:
    nics:
      match:
        name: ens*
      dhcp4: yes
      critical: true
      dhcp-identifier: mac

Testing it on vSphere directly works, but with Rancher it does not What am I missing?