rancher-users #general

Join the conversation Join Slack

general

early-lifeguard-63817

12/01/2022, 2:09 PM

ok so feeling really stupid here.. how do I reset the password of a non-admin user?

dazzling-computer-84464

12/01/2022, 7:32 PM

Has anyone encountered error like this?

2022/12/01 19:28:41 [DEBUG] No active connection for cluster [c-rw259], will wait for about 30 seconds
2022/12/01 19:28:41 [TRACE] dialerFactory: apiEndpoint hostPort for cluster [c-pw867] is [172.20.0.1:443]
2022/12/01 19:28:41 [TRACE] dialerFactory: no tunnel session found for cluster [c-pw867], falling back to nodeDialer

brash-zebra-92886

12/01/2022, 7:33 PM

Any idea when 2.7.0 will be in https://releases.rancher.com/server-charts/stable ?

👀 2

lemon-jelly-91576

12/01/2022, 7:48 PM

I installed Rancher Desktop a few days ago and it was working until this morning when things broke down. when I run

kubectl version

I’m getting this error:

invalid configuration: no configuration has been provided
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: <http://version.Info|version.Info>{Major:"1", Minor:"25", GitVersion:"v1.25.4", GitCommit:"872a965c6c6526caa949f0c6ac028ef7aff3fb78", GitTreeState:"clean", BuildDate:"2022-11-09T13:36:36Z", GoVersion:"go1.19.3", Compiler:"gc", Platform:"darwin/amd64"}
Kustomize Version: v4.5.7
The connection to the server localhost:8080 was refused - did you specify the right host or port?

Something in the configuration seems to have become corrupted somehow. How can I reset the ~/.kube/config or re-establish a new configuration?

✅ 1

👀 1

acceptable-printer-7134

12/01/2022, 8:35 PM

having trouble importing EKS into new rancher version 2.7. As far as i know we have followed procedure. 1. cloud credentials 2. aws

aws-auth

config map entry for the MapUsers where can we check logs regarding importing?

miniature-advantage-78722

12/01/2022, 10:04 PM

When I install sysbox on RKE2 I run into this issue: https://github.com/nestybox/sysbox/issues/567 I can fix it by using the patch for CRI-O that is show on the second to last comment but I'd like to switch kubelet back onto systemd instead. Does anyone know where the kubelet arguments are stored on the host machines (or elsewhere) so that I can manually repair the

--cgroup-driver

flag?

famous-grass-8099

12/01/2022, 11:11 PM

Hello Rancher Team, I’m new to rancher and trying to setup v2.6 over single node with docker. I am having following errors. Appreciate If anyone can help. Here is how my setup is. single node docker image running behind nginx as reverse proxy (it is docker-compose.yml having both containers). That single node is behind AWS classic load balancer. following are the errors being logged by rancher node (docker-compose logs -f).

rancher-server_1  | 2022/12/01 04:54:09 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:49716 (X-Forwarded-For: 44.230.106.196, 172.31.30.132): response 400: websocket: the client is not using the websocket protocol: 'websocket' token not found in 'Upgrade' header
rancher-server_1  | 2022/12/01 04:54:09 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:49716 (X-Forwarded-For: 44.230.106.196, 172.31.30.132): response 400: Error during upgrade for host [c-sxlgx]: websocket: the client is not using the websocket protocol: 'websocket' token not found in 'Upgrade' header

I have another AWS EKS cluster. I am trying to import in above rancher instance. It is showing

waiting

status.

early-lifeguard-63817

12/01/2022, 11:43 PM

what component in rancher syncs the helm repos for Apps section? all my repos are in "In Progress" status for hours and I see no charts

bright-fireman-42144

12/02/2022, 1:32 AM

apologies if this has been asked before. What jetstack chart for cert-manager is compatible with the weird cattle helm install process in rancher 2.6.9?

damp-dinner-23240

12/02/2022, 8:33 AM

Hi Rancher team. It has been a great experience using this app on mac with M1 chip. But I get into some problem when I run docker images (mysql). Can anyone help ?? Below is the docker-compose I’ve used

version: "3"
services:
  db:
    container_name: spring-db
    image: mysql
    platform: linux/amd64
    environment:
      MYSQL_DATABASE: todos
      MYSQL_USER: root
      MYSQL_PASSWORD: root
      MYSQL_ROOT_PASSWORD: root
    volumes:
      - ./db/data:/var/lib/mysql:rw
    ports:
      - "3307:3307"
    restart: always

Below is the error I see

👀 1

kind-waitress-15815

12/02/2022, 9:31 AM

Can anyone recommend a clean method of deploying Rancher on a Harvester cluster for the express purpose of managing Harvester and deploying additional Rancher clusters?

salmon-carpenter-62625

12/02/2022, 10:08 AM

Hello. Is anyone here who know what's the logic behind alertmanager-rancher-monitoring-alertmanager-generated vs alertmanager-rancher-monitoring-alertmanager secrets? I deploy my rancher-monitoring by helm manually where I added own routes and receivers but after installing the chart alertmanager-rancher-monitoring-alertmanager-generated seams to contains my changes for a short period of tiem and then are ovveritend and contains some default values, so the config alertmanager-rancher-monitoring-alertmanager-0 is not the one that I deploy with helm chart.

billions-plastic-92005

12/02/2022, 12:06 PM

Hi Rancher Team! Could you kindly advice how to disable PodSecurityPolicy admission controller in Rancher, if is it possible of course. We are using Rancher 2.6.8 at the moment.

👍 1

creamy-room-58344

12/02/2022, 4:13 PM

Hello there! I'm using Rancher 2.7.0 integrated with vSphere I noticed that the form to create a new cluster in vSphere with rke2/K3s is very different from RKE1 With RKE1, I'm able to use templates to create the VMs On RKE2/K3s I must paste my Cloud Config YAML every time Are there any plans to implement templates on RKE2/K3s? Am I missing something? Thanks in advance

freezing-fireman-44188

12/02/2022, 4:39 PM

here at TMX we have rancher desktop install on our centos machine every time I start it it seems to start 2 copies. the second copy find a port in use and complain. Is that normal with desktop rancher ?

creamy-accountant-88363

12/02/2022, 5:11 PM

Hi, I'm using Rancher 2.6.8 and was wondering how to provision clusters using the

<http://clusters.provisioning.cattle.io/v1|clusters.provisioning.cattle.io/v1>

API using a

kubernetesVersion

that is built/supplied by someone else. Currently this only seems to work with the rke2 or k3s kubernetes versions bundled with Rancher.

gentle-petabyte-40055

12/02/2022, 7:38 PM

Hello. I am having an issue where the Ubuntu machines in a Rancher only cluster were updated we can no longer access rancher. The ingress pods and rancher pods are all running. But port 443 and 80 are not open on the nodes. I have a daily rancher backup on S3 is there a way to fix this or restore it on a new cluster. And if restoring it is the way to go please guide me to the right direction. This was an RKE1 cluster running HA rancher with many clusters on them.

few-carpenter-10741

12/02/2022, 10:10 PM

Hello Everyone, this has never happened to me after working with Rancher for over a year. I did a test cluster with our Rancher server and once I finished the test deleted. The cluster is on updating stage and there is only 1 worker node left, the cluster is on updating stage and it has this message

Waiting for cp-zema-uat1 to finish provisioning

has anyone had this problem before and how can I delete it? thanks in advance

gentle-petabyte-40055

12/03/2022, 4:29 AM

could anyone please assist me I am having a very hard time figuring this out.

gentle-petabyte-40055

12/03/2022, 4:29 AM

gentle-petabyte-40055

12/03/2022, 4:30 AM

gentle-advantage-38637

12/03/2022, 12:35 PM

Hey all, pretty new to rancher, can someone point me to some tutorials about setting up rancher. I mean from the start, I have ubuntu server and docker what's next? What else I need to install?

gentle-advantage-38637

12/03/2022, 9:17 PM

hey all, me again, all good overall, REK1 working prefect but when I switch to RKE2/K3s I'm greeting strange issues. anyone have any idea?

bright-fish-35393

12/05/2022, 1:10 AM

Hi — I have a Rancher/FreeIPA integration issue — we have been running rancher 2.5.8 on RHEL 7 & RKE2 on an isolated network for over a year, and suddenly lost the ability to login to the Rancher GUI using FreeIPA credentials. Error from GUI looks like “unknown error, see your system administrator” Local logins work fine but my users depend on the FreeIPA flavor login. Rebooted all masters recently which did not help. Internal error from API looks like LDAP error 49, but the FreeIPA authorization setup looks fine and appears to work; I know the read-only FreeIPA user’s credentials work, but still no luck logging in to Rancher GUI. Any ideas on what to check? Thanks for any help you can provide..

lively-night-78214

12/05/2022, 7:24 AM

hi, anyone here still using Rancher 1.6?

adorable-photographer-68517

12/05/2022, 10:08 AM

Hi I'm just trigger a upgrade for the APP Rancher-Backup.. And I'm wondering why i can't use an exsisting Volume-Claim? Is there a plausible explanation why?

sparse-potato-80319

12/05/2022, 11:29 AM

Try to run Rancher Desktop on my corporate notebook. Installation world but Kubernetes cannot be started. To me it looks like proxy issues. Where can I set proxy settings for Rancher Desktop?

ancient-air-32350

12/05/2022, 12:12 PM

Does anyone know exactly the “apps” under Apps-> Charts are filtered ? for example there are many more available charts at https://github.com/rancher/charts/tree/release-v2.6.8/charts than listed on my installation. specifically I’m looking for the rancher-vsphere-csi app

ambitious-student-74765

12/05/2022, 1:38 PM

👋 Hi everyone! which version of rancher is supported for elastic kuberneties version V 1.22 currently using v2.5.7

👀 1

thankful-balloon-877

12/05/2022, 4:16 PM

Hi,I am installed Rancher with RKE2 on SLE Micro using the RPM package. I did this two times before, and it always worked great, installing the package,

systemctl start rke2-server

, and waiting for it to come up. This time, the service will not come up - in the kubelet.log file I find several entries of this:

E1205 13:53:15.683661    2726 remote_runtime.go:209] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create containerd task: failed to create shim: OCI runtime create failed: runc create failed: unable to start container process: error during container init: write /proc/self/attr/keycreate: invalid argument: unknown"
E1205 13:53:15.683713    2726 kuberuntime_sandbox.go:70] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create containerd task: failed to create shim: OCI runtime create failed: runc create failed: unable to start container process: error during container init: write /proc/self/attr/keycreate: invalid argument: unknown" pod="kube-system/etcd-rancher-har-nue-01"
E1205 13:53:15.683746    2726 kuberuntime_manager.go:833] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to create containerd task: failed to create shim: OCI runtime create failed: runc create failed: unable to start container process: error during container init: write /proc/self/attr/keycreate: invalid argument: unknown" pod="kube-system/etcd-rancher-har-nue-01"
E1205 13:53:15.683802    2726 pod_workers.go:951] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"etcd-rancher-har-nue-01_kube-system(e18aa5e5b83a5a3c56d78e4054612394)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"etcd-rancher-har-nue-01_kube-system(e18aa5e5b83a5a3c56d78e4054612394)\\\": rpc error: code = Unknown desc = failed to create containerd task: failed to create shim: OCI runtime create failed: runc create failed: unable to start container process: error during container init: write /proc/self/attr/keycreate: invalid argument: unknown\"" pod="kube-system/etcd-rancher-har-nue-01" podUID=e18aa5e5b83a5a3c56d78e4054612394
E1205 13:53:15.723238    2726 kubelet.go:2466] "Error getting node" err="node \"rancher-har-nue-01\" not found"

Am I right in thinking that this is my issue? If yes, any ideas what is happening here and where that "invalid argument: unknown" could come from?

Title

thankful-balloon-877

12/05/2022, 4:16 PM

Hi,I am installed Rancher with RKE2 on SLE Micro using the RPM package. I did this two times before, and it always worked great, installing the package,

systemctl start rke2-server

, and waiting for it to come up. This time, the service will not come up - in the kubelet.log file I find several entries of this:

E1205 13:53:15.683661    2726 remote_runtime.go:209] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create containerd task: failed to create shim: OCI runtime create failed: runc create failed: unable to start container process: error during container init: write /proc/self/attr/keycreate: invalid argument: unknown"
E1205 13:53:15.683713    2726 kuberuntime_sandbox.go:70] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create containerd task: failed to create shim: OCI runtime create failed: runc create failed: unable to start container process: error during container init: write /proc/self/attr/keycreate: invalid argument: unknown" pod="kube-system/etcd-rancher-har-nue-01"
E1205 13:53:15.683746    2726 kuberuntime_manager.go:833] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to create containerd task: failed to create shim: OCI runtime create failed: runc create failed: unable to start container process: error during container init: write /proc/self/attr/keycreate: invalid argument: unknown" pod="kube-system/etcd-rancher-har-nue-01"
E1205 13:53:15.683802    2726 pod_workers.go:951] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"etcd-rancher-har-nue-01_kube-system(e18aa5e5b83a5a3c56d78e4054612394)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"etcd-rancher-har-nue-01_kube-system(e18aa5e5b83a5a3c56d78e4054612394)\\\": rpc error: code = Unknown desc = failed to create containerd task: failed to create shim: OCI runtime create failed: runc create failed: unable to start container process: error during container init: write /proc/self/attr/keycreate: invalid argument: unknown\"" pod="kube-system/etcd-rancher-har-nue-01" podUID=e18aa5e5b83a5a3c56d78e4054612394
E1205 13:53:15.723238    2726 kubelet.go:2466] "Error getting node" err="node \"rancher-har-nue-01\" not found"

Am I right in thinking that this is my issue? If yes, any ideas what is happening here and where that "invalid argument: unknown" could come from?

creamy-pencil-82913

12/05/2022, 5:53 PM

Can you read through the comments at https://github.com/rancher/rke2/issues/851 and see if anything here matches what you’ve done or are experiencing? Sounds very similar.

thankful-balloon-877

12/05/2022, 6:10 PM

Thanks Brandon, I actually found that issue, but figured it does not really match my use case, because I do not have SELinux enforced and neither do I use any non-standard network plugins

creamy-pencil-82913

12/05/2022, 6:26 PM

Are you using your own containerd?

thankful-balloon-877

12/05/2022, 6:26 PM

Nothing of the sorts, it's a stock installation of SLE Micro

This is what I wrote myself together from the previous attempts .. hence I am a bit confused what's different this time 🙂 https://w3.nue.suse.com/~gpfuetzenreuter/init-rke-node.sh.txt

creamy-pencil-82913

12/05/2022, 7:00 PM

you might look at the containerd log file and see if there’s anything else in there that might suggest whats gone wrong

thankful-balloon-877

12/05/2022, 7:16 PM

is it possible I don't even have containerd?

rancher-har-nue-01:~ # ls /var/log/containers/ 
rancher-har-nue-01:~ # rpm -qa|grep container 
container-selinux-2.188.0-150400.1.8.noarch

creamy-pencil-82913

12/05/2022, 7:25 PM

The containerd log, not the container logs

/var/lib/rancher/rke2/agent/containerd/containerd.log

so you do have selinux stuff on here but it’s not in enforcing mode?

what mode is it in?

thankful-balloon-877

12/05/2022, 7:28 PM

oh, sorry, I forgot containerd was shipped together with rke2. containerd.log shows some of this:

time="2022-12-05T19:26:40.803544362Z" level=warning msg="cleanup warnings time=\"2022-12-05T19:26:40Z\" level=info msg=\"starting signal loop\" namespace=k8s.
io pid=17802\ntime=\"2022-12-05T19:26:40Z\" level=warning msg=\"failed to read init pid file\" error=\"open /run/k3s/containerd/io.containerd.runtime.v2.task/
<http://k8s.io/3e36f5b6a0971ee3b62b5597f9c9931d8e58edc45b9a55ecf509272f6bb5a1a2/init.pid|k8s.io/3e36f5b6a0971ee3b62b5597f9c9931d8e58edc45b9a55ecf509272f6bb5a1a2/init.pid>: no such file or directory\"\n" 
time="2022-12-05T19:26:40.803953105Z" level=error msg="copy shim log" error="read /proc/self/fd/21: file already closed" 
time="2022-12-05T19:26:40.818880484Z" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:etcd-rancher-har-nue-01,Uid:e18aa5e5b83a5a3c56d78e4054612394
,Namespace:kube-system,Attempt:0,} failed, error" error="failed to create containerd task: failed to create shim: OCI runtime create failed: runc create faile
d: unable to start container process: error during container init: write /proc/self/attr/keycreate: invalid argument: unknown"

if you want I can upload the full file, but it seems similar to kubelet log?

SELinux shows as permissive, I have not configured anything in regards to it

rancher-har-nue-01:~ # getenforce 
Permissive

creamy-pencil-82913

12/05/2022, 7:29 PM

what versions of SLE Micro and RKE2 are you using?

thankful-balloon-877

12/05/2022, 7:29 PM

rancher-har-nue-01:~ # rpm -qa|grep rke 
rke2-selinux-0.11-1.sle.noarch 
rke2-common-1.23.14~rke2r1-0.x86_64 
rke2-server-1.23.14~rke2r1-0.x86_64 
rancher-har-nue-01:~ # grep PRETTY /etc/os-release 
PRETTY_NAME="SUSE Linux Enterprise Micro 5.3"

creamy-pencil-82913

12/05/2022, 7:32 PM

I think it is related to weird selinux stuff. Either make sure you have all the selinux-related packages installed (latest rke2-selinux) and rke2 started with

selinux: true

or remove the other selinux bits.

https://github.com/containerd/containerd/issues/5864#issuecomment-898625687 suggests that it is related to using selinux contexts that don’t exist, which would make sense if you were missing some selinux bits. Permissive mode doesn’t block anything, but you still need to set up the contexts properly.

thankful-balloon-877

12/05/2022, 7:54 PM

interesting. I now tried it with

selinux: true

added to config.yaml, but that seems to get stuck with the same loop. what I notice is that on one of my existing installations (same setup just slightly older versions)

ls -RZ /var/lib/rancher/|grep container_var_lib
....
system_u:object_r:container_var_lib_t:s0 rke2 
    system_u:object_r:container_var_lib_t:s0 agent 
    system_u:object_r:container_var_lib_t:s0 bin 
    system_u:object_r:container_var_lib_t:s0 server
....

where as on the new one

ls -RZ /var/lib/rancher/|grep container_var_lib
<empty>
rancher-har-nue-01:~ # restorecon -Rvn /var/lib/rancher/|grep container_var_lib
<empty>

the old/working one has these versions

rancher-prv-01:~ # rpm -qa|egrep 'selinux|rke' 
selinux-policy-targeted-20210716-150400.2.3.noarch 
patterns-microos-selinux-5.3.3-150400.1.1.x86_64 
libselinux1-3.1-150400.1.69.x86_64 
selinux-policy-20210716-150400.2.3.noarch 
container-selinux-2.188.0-150400.1.2.noarch 
selinux-tools-3.1-150400.1.69.x86_64 
rke2-selinux-0.9-1.sle.noarch 
rke2-common-1.23.9~rke2r1-0.x86_64 
rke2-server-1.23.9~rke2r1-0.x86_64

creamy-pencil-82913

12/05/2022, 7:57 PM

did you install the rke2-selinux package that I mentioned above?

thankful-balloon-877

12/05/2022, 7:57 PM

yes, it is installed

creamy-pencil-82913

12/05/2022, 7:57 PM

ah but no container-selinux

that should be a dependency for the rke2-selinux package

did you get any errors when installing it?

you appear to be missing a bunch of selinux related stuff, compared to your working node at least

thankful-balloon-877

12/05/2022, 7:58 PM

it is.

rancher-har-nue-01:~ # rpm -q container-selinux 
container-selinux-2.188.0-150400.1.8.noarch 
rancher-har-nue-01:~ # rpm -q rke2-selinux 
rke2-selinux-0.11-1.sle.noarch
rancher-har-nue-01:~ # rpm -q --requires rke2-selinux |grep container 
container-selinux >= 2.164.2-1.1

creamy-pencil-82913

12/05/2022, 7:58 PM

ah ok. I didn’t see it in the package list you posted above

thankful-balloon-877

12/05/2022, 7:58 PM

I think I confused you by using different grep patterns, sorry 😄

creamy-pencil-82913

12/05/2022, 7:58 PM

yeah

I would probably just compare packages between the two, see if there’s anything else you’re missing. It is very odd that you are not getting any container contexts set on the binaries

what contexts DO the rke2 binaries have?

thankful-balloon-877

12/05/2022, 8:01 PM

the package sets seem to be the same.. just the versions are newer, but I guess others are using the same packages too.. on the binary there's another obscurity indeed:

old:
rancher-prv-01:~ # ls -Z /usr/bin/rke2 
system_u:object_r:container_runtime_exec_t:s0 /usr/bin/rke2
new:
rancher-har-nue-01:~ # ls -Z /usr/bin/rke2 
system_u:object_r:bin_t:s0 /usr/bin/rke2

and

restorecon -Rvn /usr/bin/rke2

doesn't return anything 😕

heh, I tried to hack around it by copying the contexts in /etc/selinux from the old to the new one - I forgot, this is SLE Micro, /usr/bin is read-only

rancher-har-nue-01:~ # kubectl get node 
NAME                STATUS  ROLES                      AGE   VERSION 
rancher-har-nue-01  Ready   control-plane,etcd,master  118s  v1.23.14+rke2r1

some combination of copying /etc/selinux from the other setup, installing libselinux and container-selinux devel packages from security:SELinux, ignoring some relabel errors during boot, copying the rke2 binaries to /usr/local/bin and repeated restorecon's on the latter and /var/lib/rancher (it would keep resetting) made it come up now. hm I rather not keep it so macgyvered, wonder if something in the packages changed during versions that made the selinux container policies no longer install correctly themselves

View count: 118