rancher with 2.6.x version

image.png

please advice

Hello Folk's, i try to install k3s on Fedora 38, and i want to specify dedicated --cluster-cidr and --service-cidre, but I get stuck, because the command line parameter have no impact, it always configure the two nets with 10.42 and 10.43. also it ignores the parameters --flannel-backen=none. my complete settings are :

curl -sfL <https://get.k3s.io> | sh -s - server --server --cluster-init  <https://k8s01:6443>  --flannel-backend=none  --cluster-cidr=10.80.0.0/16   --service-cidr=10.77.0.0/16

I would expect that the pods will stuck in pending - until I install the CNI with cilium. but the cluster comes up and have running pods. any ideas? are env-variables are better than --parameter's?

👋 Hi everyone!

Hello folks. Glad to have come across this channel. I'd be happy to get some help resolving this as shown in the snapshot. So many searches and troubleshooting have led to nothing. Has anyone come across this issue? I'll promptly respond to any question for clarification.

Hi All, can some advice why i can't other apps like monitoring, ops gatekeeper, cis , etc under cluster tools.

image.png

same when i go to app => charts, can't see prometheus or monitoring, opa gatekeeper, cis, etc

i can see all other apps, really appreciate your response.

I've recently upgraded from 2.5.17 to 2.6.12. None of my helm charts, built for apps I deploy, are showing up under Repository Charts. I've narrowed the problem down to the version field which is in Major.Minor.Patch-BuildID format; i.e. 3.55.0-12. The apiVersion in my helm chart is set to 2. After reading helm v2 docs, the version format I am using is in fact a valid semantic version 2.0.0 format. If I drop the -BuildID i.e. 3.55.0, the chart shows up in Rancher. Is anyone else experiencing this?

I have found out that 3.55.0+12 works, but 3.55.0-12 does not

I have v2.6.5 and it uses helm 3

I will give it a try

I am not sure if helm. version is even configurable - helm2 vs helm3

v3.55.0 worked

v3.55.0+10 works. The - causes problems

Nice that is the helm version correct?

according to semver, releases with a - in them are prerelease versions.

helm search

and other tools that show chart versions don’t show prerelease versions by default. you need to add the --devel flag to see them.

see https://helm.sh/docs/helm/helm_search_repo/

Copy code

--devel                use development versions (alpha, beta, and release candidate releases), too. Equivalent to version '>0.0.0-0'. If --version is set, this is ignored

for Rancher, I think there’s a checkbox in the UI to show devel releases

I'm trying to recover a broken cluster. I have the cluster's etcd

db

file and some

.snap

files. In the new (replacement cluster) I guess I now need to run the documented

rke etcd snapshot-restore

command? I cannot access the original cluster (it's really broken) but I do have what I think are some key files. So I guess I put a file from the broken cluster onto S3 where the new cluster can find it? I compress the file and this helps (i.e. it appears the file can be found) but I get the error

File [/backup/prior.db] not found in file [/backup/prior.db.zip]

. What's wrong - can I use the original db file? What do I need to put on the S3 bucket?

hi, i was able to find track down the issue with read-only mounts when you enable audit logs, here is cluster configuration: kube-apiserver-arg: - audit-policy-file=/var/lib/rancher/rke2/agent/policy-test.yaml - audit-log-path=/var/log/audit-k8s/policy-log.log kube-apiserver-extra-mount: - /var/log/audit-k8s/:/var/log/audit-k8s/ Sure, you need to create some policy file before on all master nodes. Everything looks fine: -rw------- 1 root root 100582521 May 11 13:43 policy-log-2023-05-11T10-43-26.885.log -rw------- 1 root root 103313216 May 11 13:43 policy-log-2023-05-11T10-43-28.441.log -rw------- 1 root root 79832869 May 11 13:54 policy-log-2023-05-11T10-54-15.567.log -rw------- 1 root root 52660013 May 11 13:54 policy-log.log Here is mounts from pod manifest for kube-api: - hostPath: path: /var/log/audit-k8s/ type: DirectoryOrCreate name: extra-mount-0 - mountPath: /var/log/audit-k8s/ name: extra-mount-0 Everything works until you restart rke2-server, the log file does exist and rke2 generates the following mounts: - hostPath: path: /var/log/audit-k8s/policy-log.log type: File name: file18 - hostPath: path: /var/log/audit-k8s/ type: DirectoryOrCreate name: extra-mount-0 - mountPath: /var/log/audit-k8s/policy-log.log name: file18 readOnly: true - mountPath: /var/log/audit-k8s/ name: extra-mount-0 And of course, kube-api starts to crash: crictl logs cbbfd26e1514d I0511 105757.404782 1 server.go:558] external host was not specified, using 172.24.150.133 I0511 105757.405249 1 server.go:158] Version: v1.24.2+rke2r1 I0511 105757.405290 1 server.go:160] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" I0511 105757.914086 1 shared_informer.go:255] Waiting for caches to sync for node_authorizer E0511 105757.914548 1 run.go:74] "command failed" err="ensureLogFile: open /var/log/audit-k8s/policy-log.log: read-only file system" Removing /var/log/audit-k8s/policy-log.log fixes the issue till the next restart.

Hi All, i have installed RKE2 on a baremetal machine, yesterday i was doing load testing on the machine and server was unresponsive, i had to forefully restart it, not sure why the node is in Not ready state, since this is POC setup, so i have one node. I am using Rook ceph as storage and metallb for load balancer when i describe the node, i get this as the response, can someone please help me

Hi community, I was trying to import the existing K3S cluster v1.26 to Rancher. Thus, have chose "Import existing cluster", but couldn't find a command that would import the K3S: Rancher version: 2.7.3 Deployment: Helm Installed on the K3S cluster version: v1.24 Is it a bug? Thanks in advance,

Anyone know how to avoid piling up cronjobs when it gets an

ImagePullBackOff

error? This is the scenario 1. Cron job runs every minute 2. The pod was set to pull

Always

, unfortunately. Now fixed. 3. A cleanup job deleted the image from the registry 4. Now the pod fails with

ImagePullBackOff

5. Every minute, a new job/pod is kicked off 6. The pods with

ImagePullBackOff

never go away, and overnight I got 600 failed pods, and a new one every minute. I looked at

backOffLimit

but looks like that only applies if the pod gets into a failed state. In this case the status is always

Waiting

I am running Rancher Desktop v1.8.1 on Mac Big Sur. I can't start Rancher due to 'No space left on device' error. When I start Rancher desktop, it doesn't start any containers.

stderr: 'scp: /tmp/rd-nerdctl-maUYHq.nerdctl: No space left on device\n',
  code: 1,
  command: [
    '/Applications/Rancher <http://Desktop.app/Contents/Resources/resources/darwin/lima/bin/limactl|Desktop.app/Contents/Resources/resources/darwin/lima/bin/limactl>',
    'copy',
    '/var/folders/w5/zd3mhbp553l27m7djjd450ph0000gn/T/rd-nerdctl-maUYHq/nerdctl',
    '0:/tmp/rd-nerdctl-maUYHq.nerdctl'
  ]
}

Any idea why?

Filesystem       Size   Used  Avail Capacity iused      ifree %iused  Mounted on
/dev/disk1s1s1  466Gi   14Gi  137Gi    10%  553797 4881899083    0%   /
devfs           190Ki  190Ki    0Bi   100%     658          0  100%   /dev
/dev/disk1s5    466Gi   24Ki  137Gi     1%       0 4882452880    0%   /System/Volumes/VM
/dev/disk1s3    466Gi  734Mi  137Gi     1%    2864 4882450016    0%   /System/Volumes/Preboot
/dev/disk1s6    466Gi  5.4Mi  137Gi     1%      15 4882452865    0%   /System/Volumes/Update
/dev/disk1s2    466Gi  312Gi  137Gi    70% 1030785 4881422095    0%   /System/Volumes/Data
map auto_home     0Bi    0Bi    0Bi   100%       0          0  100%   /System/Volumes/Data/home

Hello, while importing existing cluster to Rancher, I accidentally ran the commands provided by Rancher to another cluster which was already imported on another Is there a way to recover from this? The cluster in which the commands were run was managed by another Rancher installation using older version (2.5.5)

Is there an option or annotation that can be added to a ClusterRepo to allow for --devel to be included in the Repo sync request?

Hi, I installed rancher desktop in windows and pull one of my company image which is almost 6gb. it's starting the container but immediately exiting. Any help will be appreciated.