Channels
extensions
rancher-extensions
supermicro-sixsq
ozt
os
one-point-x
rio
onlinemeetup
v16-v21-migration
events
onlinetraining
training-1220
training-0110
training-0124
k3s-contributor
submariner
storage
k3os
windows
ci-cd
theranchcast
rfed_ara
nederlands
ukranian
danish
training-0131
masterclass
training-0207
training-0214
terraform-controller
epinio
phillydotnet
swarm
rancher-wrangler
deutsch
russian
chinese
mesos
français
japanese
arm
longhorn-dev
terraform-provider-rke
service-mesh
azure
gcp
academy
random
elemental
espanol
lima
harvester-dev
portugues
kubernetes
kim
hypper
kubewarden
opni
logging
neuvector-security
rancher-setup
developer
office-hours
mexico
cabpr
rke
vsphere
longhorn-storage
k3s
k3d
terraform-provider-rancher2
fleet
amazon
harvester
rke2
s3gw
hobbyfarm
rancher-desktop
general
Title
c
creamy-autumn-87994
05/10/2023, 1:12 AMHas anyone tried to change the template of an RKE2 cluster agent-pool on vSphere? It attempts to upgrade and create a node, but just stops there. Doesn’t register the node or do anything.
I’m able to manually delete a node and it’ll auto-gen a new one using the old template, but it doesn’t affect the update process. Update process is broken.
a
agreeable-oil-87482
05/10/2023, 6:24 AMI've done this before (accidentally!) And it worked ok. On the node it created SSH to it and grab the rancher-system-agent logs.
c
creamy-autumn-87994
05/10/2023, 5:02 PMHm. Yeah it looks to be like an image issue. I have one from 3/27 that works fine and the same one but updated on 5/04 is not working. I just noticed my /etc/passwd looks different..
old
docker:x:1028:100::/home/docker:/bin/shubuntu:x:1028:1029:Ubuntu:/home/ubuntu:/bin/basha
agreeable-oil-87482
05/10/2023, 5:03 PMProbably the default user in the cloud image. If the docker user doesn't exist it hasn't ingested the new cloud config
Has the nodes hostname changed?
c
creamy-autumn-87994
05/10/2023, 5:04 PMnope, never gets to that point
a
agreeable-oil-87482
05/10/2023, 5:04 PMYeah if the node name hasn't changed then it hasn't ingested the rancher generated cloud init user data config
It actually does that before invoking any rancher specific activity
c
creamy-autumn-87994
05/10/2023, 5:06 PMDoes rancher ssh into the node to start the process?
Or sent cloud-init data on provisioning?
a
agreeable-oil-87482
05/10/2023, 5:07 PMNot with rke2. The cloud init user data config writes and executes a script that fires off the agent install. The cloud init config does include the new hostname value though
But the ssh key is used if you SSH into the node using the rancher ui
c
creamy-autumn-87994
05/10/2023, 5:08 PMHm. So I wonder if something is off with the new version of cloud-init
a
agreeable-oil-87482
05/10/2023, 5:08 PMHow did you make your new template?
c
creamy-autumn-87994
05/10/2023, 5:09 PMjenkins job runs a packer job. Packer creates it identically as the last image, the only difference is time and the OS updates.
a
agreeable-oil-87482
05/10/2023, 5:09 PMDid you run cloud init clean as part of the build?
c
creamy-autumn-87994
05/10/2023, 5:09 PMYup
a
agreeable-oil-87482
05/10/2023, 5:10 PMWhich OS is it?
c
creamy-autumn-87994
05/10/2023, 5:10 PMubuntu 20.04
a
agreeable-oil-87482
05/10/2023, 5:11 PMI had to also remove contents of a dir to fully reset it
c
creamy-autumn-87994
05/10/2023, 5:12 PMOh there is one difference. Our images were trying to connect to the metadata URL and take several minutes to boot up, so we added this file
root@ubuntu-server:/etc/cloud/cloud.cfg.d# cat 99_disable_metadata.cfg
datasource_list: [ None ]a
agreeable-oil-87482
05/10/2023, 5:13 PMAh. That'll instruct cloud init not to ingest from the iso rancher mounts
c
creamy-autumn-87994
05/10/2023, 5:13 PMdangit!
Thank you for the extra pair of 👀
a
agreeable-oil-87482
05/10/2023, 5:13 PMYou need to have the nocloud data source added iirc
👍 1
No worries!
c
creamy-autumn-87994
05/10/2023, 5:14 PMI’ll give that a shot, thank you!!
One more question if you have a second! If we are running longhorn with local PVs, does the auto-upgrade mess things up with volume replicas? It’s unfortunate we can’t choose which node to do in order and put the longhorn node in maintenance first.
a
agreeable-oil-87482
05/10/2023, 5:20 PMWhat do you mean longhorn with local pvs?
c
creamy-autumn-87994
05/10/2023, 5:20 PMwith longhorn storing the volume replicas on the worker nodes. Then updating the template will replace the worker nodes with new ones.
a
agreeable-oil-87482
05/10/2023, 5:21 PMAh. Providing you have enough replicas and they're spread across worker nodes. Sure. You can influence how many worker nodes are upgraded in parallel in the cluster options
Id also check out https://longhorn.io/docs/1.4.1/high-availability/auto-balance-replicas/
c
creamy-autumn-87994
05/10/2023, 5:22 PMTypically you would go to the longhorn UI and put the node in maintenance and offload the volume. But with an update to the cluster, updates are automatic and it replaces worker nodes quickly. You don’t get a chance to go to the longhorn UI
a
agreeable-oil-87482
05/10/2023, 5:24 PMInteresting. Can you post that in #longhorn-storage please?
c
creamy-autumn-87994
05/10/2023, 5:24 PMOh, didn’t know about the channel! Thank you!