How is the `<http field cattle io publicEndpoints|field catt rancher-users #general

Title

wonderful-rain-13345

03/24/2023, 1:49 AM

How is the

<http://field.cattle.io/publicEndpoints|field.cattle.io/publicEndpoints>

annotation set? i went from single node running old k3os to latest k3s by adding new node and removing old node. Now IP is incorrect and i get a 404. oh IP is set in the load balancer in ingress, but no effect when i update

heh

failed to start cluster controllers

refined-analyst-8898

03/24/2023, 2:19 AM

Using Rancher Desktop on Linux, perchance? https://rancher-users.slack.com/archives/C0200L1N1MM/p1679531325237759?thread_ts=1679527315.426589&cid=C0200L1N1MM

wonderful-rain-13345

03/24/2023, 2:19 AM

no, i'm on rancher running on k3s via helm,

not sure if this matters

=== <http://rkeclusters.rke.cattle.io|rkeclusters.rke.cattle.io> ===
      host: localhost

refined-analyst-8898

03/24/2023, 2:24 AM

Were you requesting the DNS name of the Rancher Server UI when you encountered the 404? Is Rancher Server installed in the same K3S cluster where the node was replaced?

wonderful-rain-13345

03/24/2023, 2:24 AM

yes, it i went from k3s 1.21.x all the way up to 1.25.x version by version

I did it by first joining a new node, 1.22, and then shutting down original node..then upgraded 1 version at a time.

I've since switched the IP back so everything is as it was, except for a new hostname on the node

refined-analyst-8898

03/24/2023, 2:26 AM

Is this an excerpt from the Ingress resource for the Rancher Server UI?

=== <http://rkeclusters.rke.cattle.io|rkeclusters.rke.cattle.io> ===
      host: localhost

wonderful-rain-13345

03/24/2023, 2:27 AM

output from

sudo  kubectl get crd | grep -v ^NAME | awk '{print $1}' | while read CRD; do echo === $CRD ===; sudo kubectl get $CRD -A -o yaml | grep 'localhost'; done

the (now former) new IP 172.16.1.95 is nowhere to be found in the crds. the 172.16.1.167 (old ip, now the current IP):

=== <http://nodes.management.cattle.io|nodes.management.cattle.io> ===
      - address: 172.16.1.167
      <http://flannel.alpha.coreos.com/public-ip|flannel.alpha.coreos.com/public-ip>: 172.16.1.167
      <http://k3s.io/internal-ip|k3s.io/internal-ip>: 172.16.1.167

so only thing that has really changed is the hostname of the k8s node, but i would hope that would not matter.

ahh Error: UPGRADE FAILED: chart requires kubeVersion: < 1.25.0-0 which is incompatible with Kubernetes v1.25.7+k3s1

may have something to do with it

👍 1

hmm downgraded and tried to re-upgrade the chart, but i suspect no changes so idempotent

kube-system                 svclb-traefik-415191bb-rwfqw

is pending and helm traefik installs failed.. hmm

didn't traefik upgrade to v2 or something in iirc 1.21?

refined-analyst-8898

03/24/2023, 2:43 AM

I recall something about k3s moving to Traefik v2.

wonderful-rain-13345

03/24/2023, 2:43 AM

yea

traefik install failed because crds aren't there

+ helm_v3 install traefik-crd <https://10.43.0.1:443/static/charts/traefik-crd-20.3.1+up20.3.0.tgz>                                                                                │
│ Error: INSTALLATION FAILED: cannot re-use a name that is still in use                                                                                                            │
│ Stream closed EOF for kube-system/helm-install-traefik-crd-xnsr7 (helm)

ok, problem was the upgrading of the k3s applied new crds versions for traefik and broke them. the helm install/upgrade failed. Had to force delete (remove finalizers) and delete history manually then

kubectl create -f ./manifests/traefik.yaml

and that fixed it

Dashboard works now

refined-analyst-8898

03/24/2023, 12:01 PM

Nice work figuring that out.

58 Views

#general Join Slack