This message was deleted.

the cloud provider needs to be deployed during cluster provisioning, as the rancher cluster agent won’t be able to run until that’s done. That effectively means that you need to put EVERYTHING necessary to get the out-of-tree cloud provider working properly in a user manifest, so that it is installed during cluster bootstrap.

Ah, interesting. I added the cloud-controller in the addition-manifests section of the cluster config and thought I got all of the tags right. Running

crictl ps -a

on the controlplane shows the aws-cloud-controller-manager with a bunch of failed restarts, last line in the log shows that I must have an old tag lying around somehere

Cloud provider could not be initialized: could not init cloud provider "aws": Found multiple cluster tags with prefix <http://kubernetes.io/cluster/|kubernetes.io/cluster/>

ahh sounds like you’re close!

it looks like that's from trying to describe ec2 instances, so does that mean ANY instance in my VPC with multiple cluster tags will cause that problem? I assumed I'd just need to fix the instances I added to the cluster, but I guess that makes sense it'd look at all of them

I’d defer to the upstream docs on how they expect things to be tagged

ok cool that makes sense, thank you. Aside from that, all of the steps in this section should be all I need? The only other thing I had to do different was the hostname-override because our naming conventions differ form hostname.ec.internal

machineGlobalConfig:
      cni: canal
      disable-kube-proxy: false
      etcd-expose-metrics: false
      kube-apiserver-arg:
        - cloud-provider=external
      kube-controller-manager-arg:
        - cloud-provider=external
      kube-proxy-arg:
        - '--hostname-override="$(hostname -f)"'
      kube-scheduler-arg: []
    machinePools: null
    machineSelectorConfig:
      - config:
          cloud-provider-name: aws
          kubelet-arg:
            - '--hostname-override="$(hostname -f)"'

- '--hostname-override="$(hostname -f)"'

does that… work? I wouldn’t have expected shell expressions to be supported like that.

I have no idea, I tried getting this working in our RKE1 cluster about a year ago, I've been looking through my old tickets today and I noted in one of my tickets that someone in this slack suggested doing that. Fingers crossed haha I wasn't able to find good documentation on how to set those overrides. I see plenty of docs that say to do it for the kubelet on all nodes and the kube-proxy, but couldn't find the exact syntax anywhere. Actually I think it might not be working or there something else I have to set, because the cloud-controller-manager is working now that I cleaned up those three tags, but I'm seeing this in the log

error syncing '<ip_address>our.domain': failed to get provider ID for node <ip_address>.our.domain at cloudprovider: failed to get instance ID from cloud provider: instance not found, requeuing

spec:
  containers:
  - args:
    - --cluster-cidr=10.42.0.0/16
    - --conntrack-max-per-core=0
    - --conntrack-tcp-timeout-close-wait=0s
    - --conntrack-tcp-timeout-established=0s
    - --healthz-bind-address=127.0.0.1
    - --hostname-override="$(hostname -f)"
    - --kubeconfig=/var/lib/rancher/rke2/agent/kubeproxy.kubeconfig
    - --proxy-mode=iptables
    command:
    - kube-proxy

I get this:

root@rke2-server-1:/# cat /var/log/pods/kube-system_kube-proxy-rke2-server-1_e778576655760f9a1185c3599f8f57f0/kube-proxy/0.log
2023-12-19T22:38:09.257758437Z stderr F E1219 22:38:09.257613       1 server.go:1039] "Failed to retrieve node info" err="nodes \"\\\"$(hostname -f)\\\"\" not found"

bummer. Any idea on how to set that dynamically? Or does it need to go in each nodes user data script?

You need to override it to match the DEFAULT naming scheme, not the actual naming scheme in use

Oh I see, I guess I was reading that wrong.

its right there

ok, I guess that's where I'm confused. Then I just need to set it for each node when they join the cluster, I can't put it in the cluster config without some shell commands and parsing of the hostname?

You could drop something like this into the image you’re using as a userdata script to set it up ahead of time:

#!/bin/sh
TOKEN=`curl -s -X PUT "<http://169.254.169.254/latest/api/token>" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`
REGION=`curl -s -H "X-aws-ec2-metadata-token: $TOKEN" <http://169.254.169.254/latest/meta-data/placement/region>`
INSTANCE=`curl -s -H "X-aws-ec2-metadata-token: $TOKEN" <http://169.254.169.254/latest/meta-data/instance-id>`

echo "overriding hostname as ${INSTANCE}.${REGION}.compute.internal"

mkdir -p /etc/rancher/rke2/config.yaml.d
echo <<EOF >/etc/rancher/rke2/config.yaml.d/99-aws-id.yaml
  kubelet-arg+:
    - --hostname-override=${INSTANCE}.${REGION}.compute.internal
  kube-proxy-arg+:
    - --hostname-override=${INSTANCE}.${REGION}.compute.internal
EOF

awesome thank you!

I believe that should work

I'll give it a shot, you've been a huge help and cleared up a lot of stuff, thanks again

lol I wanted

cat <<EOF

not

echo <<EOF

of course but otherewise should work I think

Is there anywhere else I need to set the override? I tweaked it a bit so it's using the ip instead of resource name, but this is what the 99-aws-id.yaml looks like

kubelet-arg+:
    - --hostname-override=<ip_address>.ec2.us-east-1.internal
  kube-proxy-arg+:
    - --hostname-override=<ip_address>.ec2.us-east-1.internal

Now it hasn't even gotten to starting the cloud controller, the rke2.server.service log shows this error

"Waiting for control-plane node <ip_address>.our.domain startup: nodes \"<ip_address>.our.domain\" not found"

So somewhere it's still trying to use the actual hostname of the node, with our custom domain

you’re not overriding the node name, just the hostname

yea but I mean somewhere it must be using the hostname as the nodename still. Do I need to set the node-name in a config file as well? https://docs.rke2.io/install/requirements#prerequisites

ps -ef |grep kubelet

does show the kubelet running

the kubelet should be responsible for creating the node resource, I would check the kubelet logs and try to see why that’s not happening

Ah, yea, the kubelet log on the control plane is full of

"Attempting to register node" node="<IP>.ec2.us-east-1.internal"
"Unable to register node with API server" err="nodes \"<IP>.ec2.us-east-1.internal\" is forbidden: node \"<IP>.our.domain\" is not allowed to modify node \"<IP>.ec2.us-east-1.internal"

you might want to also set

node-name: <whatever>

Cool, I'll add that to the script you gave me. Is that fixable on my nodes currently trying to register and provision or do I need to kill it all and start over? Not big deal only takes a couple minutes now I've done it enough times haha

the servers in particular will be confused if you change that after the fact

ok, I figures as much