https://rancher.com/ logo
Join the conversationJoin Slack
Channels
academy
amazon
arm
azure
cabpr
chinese
ci-cd
danish
deutsch
developer
elemental
epinio
espanol
events
extensions
fleet
français
gcp
general
harvester
harvester-dev
hobbyfarm
hypper
japanese
k3d
k3os
k3s
k3s-contributor
kim
kubernetes
kubewarden
lima
logging
longhorn-dev
longhorn-storage
masterclass
mesos
mexico
nederlands
neuvector-security
office-hours
one-point-x
onlinemeetup
onlinetraining
opni
os
ozt
phillydotnet
portugues
rancher-desktop
rancher-extensions
rancher-setup
rancher-wrangler
random
rfed_ara
rio
rke
rke2
russian
s3gw
service-mesh
storage
submariner
supermicro-sixsq
swarm
terraform-controller
terraform-provider-rancher2
terraform-provider-rke
theranchcast
training-0110
training-0124
training-0131
training-0207
training-0214
training-1220
ukranian
v16-v21-migration
vsphere
windows
Powered by Linen
general
  • t

    tall-air-15259

    11/23/2022, 12:25 PM
    Same error here.
  • s

    stocky-article-82001

    11/23/2022, 12:55 PM
    Is it possible to give nodes Internal IPs after the cluster has been created?
    r
    • 2
    • 3
  • s

    stocky-article-82001

    11/23/2022, 12:55 PM
    Or will I have to remove nodes and add them back with Internal IPs
    ☝️ 1
    🧵 1
  • f

    flat-translator-32204

    11/23/2022, 5:02 PM
    Hello, Any guide on how to deploy k3s on AWS EC2 instances?
  • l

    loud-lawyer-79863

    11/23/2022, 5:17 PM
    Hello everyone, hope you are doing well. I have a rancher-ha server running on GKe. Seems rancher is created automatically a bunch of namespace like p-28qkg but do not seems to be cleaned them. anyone knows what are these for ? I have 70 of them without any ressources in it
    r
    • 2
    • 4
  • d

    damp-painting-69352

    11/23/2022, 7:17 PM
    @polite-mouse-38756 Are you around for a second to answer some questions?
  • b

    blue-arm-71737

    11/23/2022, 7:50 PM
    Has anyone from Rancher looked at supporting k8s 1.26 yet? It's a mess of dependency issues between controller-runtime, lasso, CAPI and client-go.
    👀 3
    c
    r
    r
    • 4
    • 16
  • l

    loud-lawyer-79863

    11/23/2022, 8:47 PM
    hmm so I made some nice mistake. I cleaned some empty namespace that I thought were created by some old CI we had. But I think I deleted a couple of these : https://github.com/rancher/rancher/issues/33119 Now I have a couple of clusters in rancher that are unavailable any idea if there a command to rebuild these or something ?
    m
    • 2
    • 2
  • s

    straight-appointment-92464

    11/24/2022, 5:43 AM
    I have installed rancher desktop on windows, can any one give me the steps to install airlfow on rancher desktop
    s
    • 2
    • 2
  • p

    polite-king-74071

    11/24/2022, 1:01 PM
    has anyone installed k8s on Ubuntu 22.04?
    m
    • 2
    • 1
  • f

    full-park-34540

    11/24/2022, 2:58 PM
    👋 Hi everyone! has anyone here came across or have any idea about
    invalid bearer token, service account token has expired
    Multiple pods from different namespace are in the ContainerCreating state including calico-kube-controllers and coredns in the kube-system namespace
    kube-system        calico-kube-controllers-54965c7ccb-rvksv                 0/1     ContainerCreating          0               5h39m
    kube-system        coredns-d76bd69b-4dtbl                                   0/1     ContainerCreating          0               5h39m
    kube-system        kube-sriov-cni-ds-arm64-5j69h                            0/1     ContainerCreating          0               5h38m
    kube-system        kube-sriov-cni-ds-arm64-5ljzh                            0/1     ContainerCreating          0               5h38m
    kube-system        kube-sriov-cni-ds-arm64-5mmfz                            0/1     ContainerCreating          0               5h38m
    kube-system        kube-sriov-cni-ds-arm64-5zkvh                            0/1     ContainerCreating          0               5h38m
    kube-system        kube-sriov-cni-ds-arm64-8z5zc                            0/1     ContainerCreating          0               5h38m
    kube-system        kube-sriov-cni-ds-arm64-bpk9q                            0/1     ContainerCreating          0               5h38m
    kube-system        kube-sriov-cni-ds-arm64-gqks9                            0/1     ContainerCreating          0               5h38m
    kube-system        kube-sriov-cni-ds-arm64-hsx9k                            0/1     ContainerCreating          0               5h38m
    kube-system        kube-sriov-cni-ds-arm64-jrlpb                            0/1     ContainerCreating          0               5h38m
    kube-system        kube-sriov-cni-ds-arm64-krt9n                            0/1     ContainerCreating          0               5h38m
    kube-system        kube-sriov-cni-ds-arm64-m42j4                            0/1     ContainerCreating          0               5h38m
    kube-system        kube-sriov-cni-ds-arm64-nnrw2                            0/1     ContainerCreating          0               5h38m
    kube-system        kube-sriov-cni-ds-arm64-nqshk                            0/1     ContainerCreating          0               5h38m
    kube-system        kube-sriov-cni-ds-arm64-q2stf                            0/1     ContainerCreating          0               5h38m
    kube-system        kube-sriov-cni-ds-arm64-r8b9d                            0/1     ContainerCreating          0               5h38m
    kube-system        kube-sriov-cni-ds-arm64-vn2gj                            0/1     ContainerCreating          0               5h38m
    kube-system        kube-sriov-cni-ds-arm64-vstrn                            0/1     ContainerCreating          0               5h38m
    kube-system        kube-sriov-cni-ds-arm64-wdbcj                            0/1     ContainerCreating          0               5h38m
    kube-system        kube-sriov-cni-ds-arm64-zxgrl                            0/1     ContainerCreating          0               5h38m
    kube-system        metrics-server-7cd5fcb6b7-zbkfj                          0/1     ContainerCreating          0               5
    kubectl describe pod throws the following error for all the pods stuck in the ContainerCreating state
    Warning FailedCreatePodSandBox 55s (x1755 over 6h24m) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "986af2c3af9e173a6f4084fcf73795bccf3b996c98f5a80b9f0a04a554cb8a21": plugin type="multus" name="multus-cni-network" failed (add): [cdi/cdi-apiserver-cdb4566f6-vq2zx/d82b198a-de67-4463-8e76-884e022fdc99:k8s-pod-network]: error adding container to network "k8s-pod-network": plugin type="calico" failed (add): error getting ClusterInformation: connection is unauthorized: Unauthorized
    Journal logs flooded with the following error
    Nov 24 13:00:43 dev3-kv-02 k3s[2648839]: E1124 13:00:43.020907 2648839 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, service account token has expired]"
    s
    • 2
    • 1
  • f

    full-park-34540

    11/24/2022, 3:41 PM
    to clarify We are using k3s + canal as CNI
  • l

    loud-lawyer-79863

    11/24/2022, 4:00 PM
    So I deleted by accident some empty rancher managed namespace on our rancher cluster (so I thought ! but actually super important ns https://github.com/rancher/rancher/issues/33119 ) Trying to restore a backup but it is able to restore everything except what looks like what I deleted ... Anyone ever seen this ?
    ERRO[2022/11/24 15:38:43] Error restoring resource grb-cn99w of type <http://management.cattle.io/v3|management.cattle.io/v3>, Resource=globalrolebindings: restoreResource: err updating resource <http://GlobalRoleBinding.management.cattle.io|GlobalRoleBinding.management.cattle.io> "grb-cn99w" is invalid: metadata.deletionGracePeriodSeconds: Invalid value: 0: field is immutable 
    ERRO[2022/11/24 15:38:45] Error restoring cluster-scoped resources [error restoring grb-lr9qh of type <http://management.cattle.io/v3|management.cattle.io/v3>, Resource=globalrolebindings: restoreResource: err updating resource <http://GlobalRoleBinding.management.cattle.io|GlobalRoleBinding.management.cattle.io> "grb-lr9qh" is invalid: metadata.deletionGracePeriodSeconds: Invalid value: 0: field is immutable error restoring grb-cn99w of type <http://management.cattle.io/v3|management.cattle.io/v3>, Resource=globalrolebindings: restoreResource: err updating resource <http://GlobalRoleBinding.management.cattle.io|GlobalRoleBinding.management.cattle.io> "grb-cn99w" is invalid: metadata.deletionGracePeriodSeconds: Invalid value: 0: field is immutable] 
    ERRO[2022/11/24 15:40:03] Error restoring cluster-scoped resources [error restoring grb-lr9qh of type <http://management.cattle.io/v3|management.cattle.io/v3>, Resource=globalrolebindings: restoreResource: err updating resource <http://GlobalRoleBinding.management.cattle.io|GlobalRoleBinding.management.cattle.io> "grb-lr9qh" is invalid: metadata.deletionGracePeriodSeconds: Invalid value: 0: field is immutable error restoring grb-cn99w of type <http://management.cattle.io/v3|management.cattle.io/v3>, Resource=globalrolebindings: restoreResource: err updating resource <http://GlobalRoleBinding.management.cattle.io|GlobalRoleBinding.management.cattle.io> "grb-cn99w" is invalid: metadata.deletionGracePeriodSeconds: Invalid value: 0: field is immutable] 
    ERRO[2022/11/24 15:40:03] error syncing 'restore-rzrb6': handler restore: error restoring cluster-scoped resources, check logs for exact error, requeuing
  • s

    stocky-article-82001

    11/24/2022, 5:06 PM
    Can you assign a node an internal IP while it is already in a cluster or will I have to remove it and re-add it. We’re moving to use an internal VPN for all node <-> node communications.
  • r

    rich-sundown-87208

    11/24/2022, 6:57 PM
    Ello 🙂 my rancher/rancher GUI has suddenly started telling me that it cannot connect to the cluster - fun fun
  • r

    rich-sundown-87208

    11/24/2022, 6:58 PM
    mostly timeouts in the logs, so I'm thinking SSL issue of some sort?
  • r

    rich-sundown-87208

    11/24/2022, 6:59 PM
    I connected to a shell inside the container and couldn't curl google
  • r

    rich-sundown-87208

    11/24/2022, 7:01 PM
    "Cluster health check failed: Failed to communicate with API server during namespace check: Get "https://10.43.0.1:443/api/v1/namespaces/kube-system?timeout=45s": context deadline exceeded"
  • r

    rich-sundown-87208

    11/24/2022, 7:02 PM
    curl <https://10.43.0.1:443/api/v1/namespaces/kube-system?timeout=45s>
    curl: (60) SSL certificate problem: unable to get local issuer certificate
    More details here: <https://curl.haxx.se/docs/sslcerts.html>
    
    curl failed to verify the legitimacy of the server and therefore could not
    establish a secure connection to it. To learn more about this situation and
    how to fix it, please visit the web page mentioned above.
  • r

    rich-sundown-87208

    11/24/2022, 7:02 PM
    vs
    curl -k <https://10.43.0.1:443/api/v1/namespaces/kube-system?timeout=45s>
    {
      "kind": "Status",
      "apiVersion": "v1",
      "metadata": {
        
      },
      "status": "Failure",
      "message": "Unauthorized",
      "reason": "Unauthorized",
      "code": 401
    }
  • r

    rich-sundown-87208

    11/24/2022, 7:03 PM
    and the response is almost instant, which is in conflict with "context deadline exceeded"
  • r

    rich-sundown-87208

    11/24/2022, 8:37 PM
    okay, so I downloaded the ca pem from here; https://curl.haxx.se/ca/cacert.pem
  • r

    rich-sundown-87208

    11/24/2022, 8:37 PM
    and now i can hit google, but I still can't get to 10.43.0.1
  • e

    elegant-librarian-62192

    11/25/2022, 12:34 AM
    Hi folks, after quitting and relaunching RD, I attempt to start a container that was previously open, and I'm seeing:
    FATA[0000] failed to create shim task: OCI runtime create failed: runc create failed: cannot allocate tty if runc will detach without setting console socket: unknown
    The container was open with a Bash console at the point when I quit, which I suspect may be part of the problem. No problem to recreate, but curious if there's a quick way to recover from this failure mode in future.
    s
    • 2
    • 1
  • e

    elegant-librarian-62192

    11/25/2022, 2:47 AM
    Also noticed that the QEMU process floats between 85%-115% of a core regardless of whether or not there are any containers (other than all the k3s stuff) running. Is this expected behavior?
  • w

    wonderful-airplane-86139

    11/25/2022, 3:47 AM
    Hi, running rancher 2.4 cluster on aws ec2 and from yesterday pods on newly added nodes don't resolve dns. Anyone has an idea?
  • s

    silly-jordan-81965

    11/25/2022, 7:22 AM
    Good Morning - we have a strange behavior regarding Rancher snapshot handling. In one cluster running cilium and vxlan when we take snapshots we receive status on the snapshots and we can see that Rancher reports Status, Name and size of the snapshot. However we have 2 other clusters that is also running Cilium and Bird. These clusters Doesn’t report backup status of taken snapshot or the size of the snapshot but when we do check the control-plane node we can see the snapshot on that node and we can see that the snapshot contain data. On one of the clusters this has worked but stopped working. What can be causing this? Im think we are missing a port opening somewhere - but having checked with our network team and according to them we have all openings that we need. Anyone have an idea where to start to look?
  • a

    acoustic-businessperson-24722

    11/25/2022, 9:03 AM
    Hey, has anyone experience with wrong metrics about used and free ram?
  • s

    silly-jordan-81965

    11/25/2022, 9:15 AM
    Anyone know what the below secret is referring to?
    level=error msg="[K8s] received secret to process that was older than the last secret operated on. (256559510 vs 256559619)"
     level=error msg="error syncing 'fleet-default/bootstrap-template-d4lsj-machine-plan': handler secret-watch: secret received was too old, requeuing"
    b
    • 2
    • 1
  • s

    salmon-portugal-38278

    11/25/2022, 1:02 PM
    Hello, Anyone around have any experience with the new
    rancher-monitoring
    ? Ive been trying to add a
    PrometheusRule
    using Rancher UI following the docs here https://docs.ranchermanager.rancher.io/v2.6/how-to-guides/advanced-user-guides/monitoring-v2[…]uration-guides/advanced-configuration/prometheusrules After creating the object, I tried to see it's really there in my prometheus dashboard. Unfort, it's not there I looked in the
    rulefile
    configmap that watches all the prometheusrules and it's not there too !
    • 1
    • 2
Powered by Linen
Title
s

salmon-portugal-38278

11/25/2022, 1:02 PM
Hello, Anyone around have any experience with the new
rancher-monitoring
? Ive been trying to add a
PrometheusRule
using Rancher UI following the docs here https://docs.ranchermanager.rancher.io/v2.6/how-to-guides/advanced-user-guides/monitoring-v2[…]uration-guides/advanced-configuration/prometheusrules After creating the object, I tried to see it's really there in my prometheus dashboard. Unfort, it's not there I looked in the
rulefile
configmap that watches all the prometheusrules and it's not there too !
I tried to play with some labels and all, but it's not solving my issue
Well adding the label `release: rancher-monitoring`does the trick !
View count: 21