This message was deleted.

I managed to get Rancher to spin up instances in OpenStack, but the "cacert" fields and the "userdataFile" fields of the driver appear to only honor a single line for entry. Turns out that I do not need the "cacert" field for my environment, but I do need to inject user data in a cloud-init fashion.

Hey Zane I'm also working with rancher on openstack. I'm using the terraform provider tho. It may help convert your scripts/cacert into 1 line. I do this with my private key and things work well

Thanks @creamy-crayon-86622. I did manage to get Rancher to deploy a cluster to OpenStack. I am using the node driver included with Rancher Manager. We are pretty new to OpenStack, so this was a bit of a learning curve. Some updates on deploying resources to OpenStack: cacert - I managed to work around the cacert by checking an ignore cert option in the OpenStack node driver settings in the Rancher UI. Not ideal, but this is a lab environment with Rancher manager and OpenStack behind the same security perimeter. cloud config - After playing with OpenStack and manually deploying an Ubuntu cloud image with some cloud-init text, I did notice that OpenStack converted my cloud-init into a Base-64 encoded single line in the VM definition. Based on that, I converted the cloud-init text that I wanted Rancher to deploy to Base-64 and added it to my Rancher OpenStack deployment configuration. However, it errored out with an invalid cloud config error message. In the end, I chose to go without the cloud config file, as I was only using it to set SSH passwords for troubleshooting. I can use the Rancher UI to open an SSH session, and there weren't any other packages that I needed to add to the cloud image. I wish the node driver had better documentation... terraform - I did use Terraform to deploy a test domain, project, network, router, and VM. This was mostly getting familiar with the provider for the purpose of deploying VMs, but it had less "friction" than using the OpenStack node driver, so I may use Terraform for the initial cluster VM deployment in the future and import the cluster into Rancher. At the end of the day, we coaxed it into working. But the documentation could be better.

Yeah I totally hear ya. I haven't had any luck getting the cloud config working for openstack. The node driver works without the cloud config being set tho. You just have to put in your openstack env vars. I'm using the terraform rancher providers rancher2_machine_config_v2 resource to build out the driver. But I wasn't able to get the outputs of all the IP addresses of all the vms so it didn't seem helpful. It allows me to scale vms tho which is cool. I'm also going the terraform route of building vms and then just adding them to the cluster. Seems to be the easiest and quickest solution so far. And like you mentioned, the docs aren't that great bleh I'm curious tho, are you using openstacks native load balancing solution Octavia? That's the next piece I'm working with now. I'm just using a wildcard DNS record pointing to my k8s workers at the moment which isn't great but it gets me by. I don't want to deploy custom loadbalancers... Yet lol

Not using Octavia (yet). I tend to use ingress controllers (e.g. NGINX or Traefik) deployed inside the clusters and couple them with the MetalLB load balancer. As long as I can get an L2/VLAN IP for MetalLB, then I can expose the apps via ingress controller (front-ended with the MetalLB IP). Plus, using MetalLB keeps things consistent across Rancher-deployed clusters regardless of whether I'm deploying to OpenStack, Harvester, or VMWare as the driver. But that's on-prem. For cloud providers, using the cloud-provided ingress/load balancers makes more sense.

Hi, I started setting up rke2 cluster within OpenStack. Did you manage to get userdataFile to work? I just need to add nfs-common on all my nodes that are deployed via the rancher ui but for some reason the added config never runs.

#cloud-config package_update: true package_upgrade: true packages: ['nfs-common']

Hey Marcel you may have to base64 encode that script. I haven't used user data for k8s tho but openstack instances appear to like that

When I base64 encode it the node provisioning fails with

existing userdata file does not begin with '#!' or '#cloud-config'

Looking at the cloud provider docs (https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/openstack-[…]-controller-manager/using-openstack-cloud-controller-manager.md), while it does not have a reference to a

userDataFile

field, some of the other fields that are referred to as files (e.g.

clouds-file

,

cert-file

, and

key-file

) are described as paths to files and not actually file content. I wonder if the

cacert

and

userDataFile

fields are actually supposed to be file paths/URIs/URLs.

considering the errors that I am getting, when not providing proper content, that doesn't seem to be the case.

Looking at https://cloudinit.readthedocs.io/en/latest/reference/datasources/openstack.html, I decided to replace all carriage returns in my cloud config file with "\n". This gave me a single line configuration that Rancher did not complain about. However, looking at the created instance (VM) on OpenStack, the decoded contents of the User Data seem to indicate that Rancher uses this to inject its scripts. There is no indication that the

userDataFile

contents that I configure (just SSH password settings and quemu guest agent) are being merged.

I have opend a bug report, because that seems to be the case: https://github.com/rancher/rancher/issues/46311 OpenStacks accepts cloud config content like

#!/usr/bin/env -S bash -c "echo Start node >> /home/ubuntu/start.log

therefore it should also work when setting the same in rancher, since it is literally a single line. I want to check what happens if I set Cloud Provider to external within the Cluster Configuration menu, instead of using rke default. Maybe cloud config can be set there with multiline content.

That's a good find! I'm hopeful that your bug report will result in the "userDataFile" field being expanded into a larger text block in the UI that allows users to copy/paste file contents. This would be similar to the UI behavior that we see in both the VMWare and Harvester cloud providers. However, I'm holding my breath on the UI change though, as the UI's for the node drivers and cloud providers generally reflect what is defined by the upstream. Still, what is placed into the "userDataFile" field should, at the very least, be merged with the contents that Rancher uses to run its agent deployment scripts, rather than completely overwritten/ignored by it. And, maybe update the Rancher docs to describe the format that the field expects to see.

Here someone made a good overview for how yaml handles newlines, spaces etc: https://stackoverflow.com/a/21699210 If that is correct I can guess what is going wrong. The single quoted string adds a new line with every space. The cloud-config has spaces which aren't supposed to be new lines, and therefore are not working. With

>-

maybe it is possible to work with

\n_

I did try out to set

CLuster Provider

to External and define

Cloud Provider Config

but I couldn't get it to work. Or do you maybe have another idea?

That's a good find! I have tried replacing my carriage returns with

\n

, but did not try using

|

Hopefully the bug report will get an official response on how this field should be formatted.

Since waiting for a reply/bugfix will probably take a while, I will create my own Cluster Template Helm Chart for OpenStack acording to this repo and comment: https://github.com/rancher/cluster-template-examples https://github.com/rancher/rancher/issues/40226#issuecomment-1399175288 With these I hopefully can get a solution that can be automated so that I don't have to edit the already deployed clusters manually

Hey do either one of you have any configuration examples I can look at? Still using rancher to try deploying my cluster. I'm deploying nodes via the openstack node driver, setting the cluster to the external cloud provider and passing in an additional manifest for the occm. But it appears coredns doesn't spin up and it causes my cattle agent to fail on DNS. If you guys have a run book or a set of steps you use that would be great.

That is more or less what I am currently running in a test environment. I only kept enabled what is required or potentially usefull (like cloud-init). Requirements: - A working rke2 cluster with rancher used as Management Cluster. - OpenStack node Driver enabled - Access to Openstack API via Application Credentials On Management Cluster: - Install the rke2 cluster template:

<https://jugit.fz-juelich.de/iek-10/public/ict-platform/deployment/kubernetes/rke2-rancher-cluster-templates>

- Configure OpenStack Access:

openstack-secrets.yaml

- Fill out the

values.yaml

, especially

x-openstackconfig

- Create a cluster using that file and the template - Login into the created Downstream Cluster (it will not show in rancher as running until you complete the next step) - Create

cloud-config

secret on Downstream Cluster - nano

cloud.conf

-

[Global]
    auth-url="<https://example.com:5000/v3>"
    application-credential-id=""
    application-credential-secret=""
    region=""
    tls-insecure=false

    [LoadBalancer]
    manage-security-groups=false
    floating-network-id=""
    subnet-id=""

-

kubectl create secret -n kube-system generic cloud-config --from-file=./cloud.conf

- Wait until the cluster shows up as running in rancher Important notice:

cluster.config:
  machineGlobalConfig:
    disable_cloud_controller: true
  machineSelectorConfig.kubelet_arg:
    - cloud-provider=external

These two options are crucial, without them OCCM will not work.