rancher-users #k3s

creamy-pencil-82913

08/30/2022, 7:17 AM

Outage again :/

cuddly-egg-57762

08/30/2022, 9:34 AM

Hello people, I'm trying to deploy the cilium helm chart via the k3s-integrated helm controller and I want to provide the spec.chartContent instead of the chart name (because I'm working on an airgapped environment). When I generate the yaml and put it under the manfest directory, the k3s service logs reports the following error:

Copy code

k3s[2515485]: time="2022-08-30T09:12:03Z" level=error msg="Failed to process config: failed to process /var/lib/rancher/k3s/server/manifests/cilium.yaml: yaml: line 14: could not find expected ':'"

even if everything looks fine to me:

Copy code

apiVersion: <http://helm.cattle.io/v1|helm.cattle.io/v1>
kind: HelmChart
metadata:
  name: cilium
  namespace: kube-system
spec:
  bootstrap: True
  chartContent: <b64 encoded file>
  targetNamespace: kube-system
  valuesContent: |-
    operator:
      replicas: 2
      image:
        useDigest: false
    tunnel: disabled
    autoDirectNodeRoutes: true
    kubeProxyReplacement: strict
    loadBalancer:
      mode:dsr
    k8sServiceHost: 10.130.42.248
    k8sServicePort: 6443
    nativeRoutingCIDR: 10.0.0.0/16
    image: 
      useDigest: false
      pullPolicy: IfNotPresent

To generate the content of spec.chartContent I do the command "base64 cilium-x.y.z.tgz" and past the result into it. I'm I doing something wrong? Or am I missing something?

dazzling-appointment-98003

08/30/2022, 10:05 AM

Hi @creamy-pencil-82913, currently the default install of k3s through https://get.k3s.io is broken. It's necessary to specify as version. Otherwise the installer won't work. GitHub Issue: https://github.com/k3s-io/k3s/issues/6052 We experienced this issue during a kubernetes workshopt with customers today. Would be great if someone can take a look at the issue and prioritize it.

straight-businessperson-27680

08/30/2022, 6:17 PM

Hetzner-k3s v0.6.2 is out! 🎉 This update makes it possible to configure labels and taints for the nodes, to allow scheduling workloads with particular requirements on specific nodes (for example, apps that require a GPU would be scheduled on nodes with a GPU). https://github.com/vitobotta/hetzner-k3s

👍 1

clever-art-93319

08/31/2022, 7:41 AM

Hi, does anyone encountered the following error? tried number if solutions with no success: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image "rancher/mirrored-pause:3.1": failed to pull image "rancher/mirrored-pause:3.1": failed to pull and unpack image "<http://docker.io/rancher/mirrored-pause:3.1|docker.io/rancher/mirrored-pause:3.1>": failed to resolve reference "<http://docker.io/rancher/mirrored-pause:3.1|docker.io/rancher/mirrored-pause:3.1>": failed to do request: Head "<https://registry-1.docker.io/v2/rancher/mirrored-pause/manifests/3.1>": net/http: TLS handshake timeout

glamorous-flag-56432

08/31/2022, 11:09 AM

Ello, anybody using k3s in a cluster setup? Ie 3 independent instances. We use dedicated ETCD store, however we also launch the k3s within k8s, not bare metal. (We use k3s as a storage for some CRDs and configuration etc). Is there any easy way how to launch all 3 instances with the same set of flags and environment variables? I found only

--cluster-init

etc, but not sure what is required for external ETCD

brainy-electrician-41196

08/31/2022, 7:07 PM

is there a full example of k3s config.yaml? https://rancher.com/docs/k3s/latest/en/installation/install-options/#:~:text=An%20example%20of%20a%20basic%20server%20config%20file%20is%20below%3A

jolly-waitress-71272

08/31/2022, 9:12 PM

What's the right way to become k3s "certified"? There doesn't seem to be defined paths of any kind, but my team is going to start really really using it and I want to get ahead of the curve. I'm looking at https://rancher.com/learn-the-basics but it doesn't seem to be k3s focused at all, with only zero to k3s in seconds being k3s centric.

refined-magician-25478

08/31/2022, 9:50 PM

Does k3s' bundled flannel backend support CNI plugins like multus? We weren't able to quickly figure out how to use multus with the flannel backend but were able to disable it and run flannel in the cluster with multus.

refined-toddler-64572

09/01/2022, 2:43 PM

A few days back upgraded to

1.24.4+k3s1

and I've been monitoring the logs, seeing stuff I haven't noticed before:

refined-toddler-64572

09/01/2022, 2:43 PM

Copy code

k3s02 systemd[1]: Started Lightweight Kubernetes.
k3s02 k3s[567197]: time="2022-09-01T10:27:00-04:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
k3s02 k3s[567197]: time="2022-09-01T10:27:00-04:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:6443/v1-k3s/readyz>: 500 Internal Server Error"
k3s02 k3s[567197]: Flag --cloud-provider has been deprecated, will be removed in 1.24 or later, in favor of removing cloud provider code from Kubelet.
k3s02 k3s[567197]: Flag --containerd has been deprecated, This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.
k3s02 k3s[567197]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.27. Image garbage collector will get sandbox image information from CRI
k3s02 k3s[567197]: I0901 10:27:00.937171  567197 server.go:192] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime

melodic-hamburger-23329

09/02/2022, 12:35 AM

Is there some documentation/manual/etc. regarding certificate and key handling in k3s? There’s quite overwhelming amount of various keys and certs, and I’m just trying to figure out how everything works (also, how cert-manager / vault would fit the picture). Also, I would like to know all possible configuration options regarding key and cert algorithms, cipher suites, etc. E.g., noticed that service account tokens are signed with RS256, but would like to use ES256 instead.

chilly-telephone-51989

09/02/2022, 1:59 AM

I need help setting up a k3s cluster ( at a very short notice. 😞 ) so I ran this

curl -sfL <https://get.k3s.io> | sh -s - server

but i did not get the token anywhere in /etc/rancher/k3s the only file that is there is k3s.yaml where do I get the token?

chilly-telephone-51989

09/02/2022, 2:10 AM

i ran the installation for nodes, using the token. how long does it take to start nodes and join cluster. it seems to be taking forever. stuck at systemd: Starting K3s-agent

chilly-telephone-51989

09/02/2022, 2:14 AM

nodes not joining the cluster at the moment

careful-optician-75900

09/02/2022, 9:45 AM

When we need to use these annotation ? I have some issues with web-sockets. When i check proxy server logs, i got "*Error* getting SSL certificate "default/*-tls": local SSL certificate default/*-tls was not found. Using default certificate" and it keeps updating ingress to classic load balancer and got disconnected every 1mins after upgrading rancher version.

Copy code

annotations:
    <http://field.cattle.io/projectId|field.cattle.io/projectId>: ""

Updating local copy of SSL certificate to classic load balancer every 1 mins. How to troubleshoot these issues ? Nginx-ingress logs: 8 controller.go:177] Configuration changes detected, backend reload required. 8 backend_ssl.go:189] Updating local copy of SSL certificate "cattle-system/tls-rancher-ingress" with missing intermediate CA certs I0830 050732.859819 8 controller.go:195] Backend successfully reloaded.A Any ideas for SSL Uploading every 1 mins ? Many thanks

able-mechanic-45652

09/02/2022, 9:59 AM

after upgrading to 1.24.4 I'm getting "x509: cannot validate certificate for <worker ip> because it doesn't contain any IP SANs" messages when trying to access pod logs or exec individual pods

clever-air-65544

09/02/2022, 5:10 PM

New weekly report is up! https://github.com/k3s-io/k3s/discussions/6096

👍 2

high-controller-26526

09/03/2022, 12:28 PM

still going strong 💪

Copy code

KUBECONFIG=k3s.yaml kubectl get nodes
NAME       STATUS   ROLES                  AGE     VERSION
0-server   Ready    control-plane,master   2y43d   v1.24.4+k3s1
1-agent    Ready    <none>                 2y43d   v1.24.4+k3s1
2-agent    Ready    <none>                 2y43d   v1.24.4+k3s1
3-agent    Ready    <none>                 2y43d   v1.24.4+k3s1

🎉 5

kind-nightfall-56861

09/03/2022, 10:34 PM

Hey small, slightly unrelated question... People have asked me to publish my personal guide on how I have set up my at-home RPi 4 cluster running k3s... And while it's not completed; I'm wondering if it's current set up is workable, haven't really created a public blog before, so looking for pointers. The blog currently only contains RPi and k3s configuration, I'm still wanting to expand it with: • Helm installation • Rancher installation @ Ubuntu • Cert-manager configuration using Let's Encrypt • Deploying "your first application" that I'll provide, using automated TLS-certs • (hopefully) needless to say; but include sources that I have used Anyone willing to give me pointers?

👍 1

prehistoric-diamond-4224

09/04/2022, 9:00 PM

Hi does Traefik v2 work with

Ingress

resources? I am aware that now there is the new

IngressRoute

, but if i were to upgrade now, would v2 be compatible with all the old Iingress resources already present in the cluster?

stale-orange-90901

09/06/2022, 6:27 PM

Howdy all, is it feasible to run a 3 nodes cluster with embedded etcd (masters with scheduling so that workloads also can be run on the control olane) as a small internal (call-home) production edge cluster ?

calm-france-2744

09/07/2022, 12:01 AM

From a worker node, how can I check if the worker successfully connected to the cluster and is in a ready state? The worker nodes don't seem to have kubectl configured, otherwise I could use it to get the node's status. Just checking the systemd service status doesn't seem to tell me if the node successfully connected to the cluster, just if the service is running. Any ideas other than searching the systemd journal for log messages?

numerous-zoo-73399

09/07/2022, 9:37 AM

Hi I have 3 master nodes (

control-plane,etcd,master

) and I am trying to update them. I do not use upgrade plan as I need additional operations, but the flow is the same. I do replace k3s and run kill to the process. it seems that it works on first node but after that 2 others are becoming

NotReady

, i do not see any or may missed the logic regarding the order on doing it in

k3s-upgrade

component. Moreover are there any limitation of k3s version compatabilities? Meaning if I have one of the nodes with version

v1.23.8+k3s1

and 2 others are still

v1.21.4+k3s1

- can it be a limitation? Thanks in advance 🙏 rancher employee

👀 1

😲 1

careful-optician-75900

09/07/2022, 10:02 AM

I0907 100025.937228 8 backend_ssl.go:189] Updating local copy of SSL certificate "cattle-system/tls-rancher-ingress" with missing intermediate CA certs Do you guys have any ideas about intermediate ca certificates for rancher ?

red-musician-8168

09/08/2022, 11:45 PM

Late in the day, but just a survey of whether folks here have experience or interest in running argo-workflows, and possibly argo-cd in their k3s cluster . . .

stale-dinner-99388

09/09/2022, 12:59 PM

Hey is there any way i can pass rancher username at the time of installation. I am using helm chart for install rancher. Command:

Copy code

helm upgrade --install rancher rancher-latest/rancher   --namespace cattle-system   --set hostname=url   --set ingress.tls.source="letsEncrypt" --set bootstrapPassword=abcdxyz  --set letsEncrypt.email="email"   --set letsEncrypt.environment="production"

clever-air-65544

09/09/2022, 6:39 PM

In case anyone wonders what the team was working on this week: https://github.com/k3s-io/k3s/discussions/6116

🙌 1

stocky-sundown-51677

09/10/2022, 9:17 AM

Hey folks, I had a question about adding nodes to a K3S cluster which are on different networks. I have tried searching online but haven't really seen anything discussions around this. What I want to do is dynamically add a node similar to the way we can add gitlab workers on a host (if this makes sense?) Any ideas or directions to the right resources/docs would be really appreciated!

chilly-telephone-51989

09/11/2022, 1:11 AM

trying to setup a k3s cluster or 3 nodes on aws machines. on master k3s installed successfully but on worker node it is stuck at starting agent message. i used external ip of master as url and its toke. this is the log of service and it cant get certificates

Copy code

k3s-agent.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s-agent.service; enabled; vendor preset: enabled)
     Active: activating (start) since Sun 2022-09-11 00:57:46 UTC; 2min 17s ago
       Docs: <https://k3s.io>
    Process: 2842576 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=0/SUCCESS)
    Process: 2842578 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 2842579 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 2842580 (k3s-agent)
      Tasks: 9
     Memory: 15.5M
        CPU: 146ms
     CGroup: /system.slice/k3s-agent.service
             └─2842580 "/usr/local/bin/k3s agent"

Sep 11 00:57:46 ip-172-31-41-97 sh[2842576]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Sep 11 00:57:46 ip-172-31-41-97 sh[2842577]: Failed to get unit file state for nm-cloud-setup.service: No such file or directory
Sep 11 00:57:46 ip-172-31-41-97 k3s[2842580]: time="2022-09-11T00:57:46Z" level=info msg="Starting k3s agent v1.24.4+k3s1 (c3f830e9)"
Sep 11 00:57:46 ip-172-31-41-97 k3s[2842580]: time="2022-09-11T00:57:46Z" level=info msg="Running load balancer k3s-agent-load-balancer 127.0.0.1:6444 -> [3.128.3.142:6443]"
Sep 11 00:58:06 ip-172-31-41-97 k3s[2842580]: time="2022-09-11T00:58:06Z" level=error msg="failed to get CA certs: Get \"<https://127.0.0.1:6444/cacerts>\": context deadline >
Sep 11 00:58:28 ip-172-31-41-97 k3s[2842580]: time="2022-09-11T00:58:28Z" level=error msg="failed to get CA certs: Get \"<https://127.0.0.1:6444/cacerts>\": context deadline >
Sep 11 00:58:50 ip-172-31-41-97 k3s[2842580]: time="2022-09-11T00:58:50Z" level=error msg="failed to get CA certs: Get \"<https://127.0.0.1:6444/cacerts>\": context deadline >
Sep 11 00:59:12 ip-172-31-41-97 k3s[2842580]: time="2022-09-11T00:59:12Z" level=error msg="failed to get CA certs: Get \"<https://127.0.0.1:6444/cacerts>\": context deadline >
Sep 11 00:59:34 ip-172-31-41-97 k3s[2842580]: time="2022-09-11T00:59:34Z" level=error msg="failed to get CA certs: Get \"<https://127.0.0.1:6444/cacerts>\": context deadline >
Sep 11 00:59:56 ip-172-31-41-97 k3s[2842580]: time="2022-09-11T00:59:56Z" level=error msg="failed to get CA certs: Get \"<https://127.0.0.1:6444/cacerts>\": context deadline >
~

i tried using curl but it wont work.