This message was deleted.

are you using the cloud controller?

I just install it by

curl -sfL <https://get.rke2.io> | sh -

command in my machine in VirtualBox

Kubernetes (and rke/rke2) itself does not include any implementation of a load balancer controller, so entries you create just stay in pending because nothing does anything with them. Cloud controllers (for AWS, Azure, etc) or specific balancers like MetalLB implement it.

K3s is meant to be run in smaller self-contained environments, does include a stub load-balancer controller, in klipper-lb. RKE2 expects to be deployed to a traditional enterprise/cloud environment where you’ll be using an external cloud provider for LB services, or MetalLB for bare metal, as @full-painter-23916 said.

is there any guilde how to hook-up rke2 to external LB, running on OpenStack?

Deploy the external LB's LB Controller?

sure, there is extended configuation and controller, but doesn't work

is that a limitation of the openstack cloud controller?

I don't think so as I have it under control. Is there any guide how to make that combo working? I just made "gues" based on k8s experience and aws integration on https://docs.rke2.io/advanced/

if it’s not bundled with rke2 then you’d deploy it exactly the same as you would on any other Kubernetes cluster

unfortunately, that doesn't work 😕

What specifically doesn’t work?

I have addedd file: - /etc/rancher/rke2/config.yaml.d/55-openstack.yaml with content: { "cloud-provider-config": "/etc/rancher/rke2/cloud.conf", "cloud-provider-name": "external" } root@cltst-pool1-a9366016-2cq28:~# cat /etc/rancher/rke2/cloud.conf [Global] auth-url="<keystone URL>" domain-name="<domain>" tenant-name="BUILD" username="Admin" password="<password>" tls-insecure = true [LoadBalancer] use-octavia=true lb-provider="octavia" subnet-id="96f8e0b6-e5a4-4b8a-aeea-e6c2f9759a2e" create-monitor=true monitor-delay=60s monitor-timeout=30s monitor-max-retries=5 manage-security-groups=true [Networking] ipv6-support-disabled=true [BlockStorage] bs-version=v2 followed agent restart -> all green Then I made controller deployment in context of cluster -> controller is working with logs: ubuntu@czbrn-sky-jmp-001:~$ kubectl logs -n kube-system pods/cloud-controller-manager-cltst-pool1-a9366016-2cq28 I0610 144418.410692 1 serving.go:354] Generated self-signed cert in-memory I0610 144418.539458 1 controllermanager.go:142] Version: v1.22.1-k3s1 I0610 144418.542287 1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController I0610 144418.542369 1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController I0610 144418.542468 1 configmap_cafile_content.go:201] "Starting controller" name="client-ca:kube systemextension apiserver authentication:client-ca-file" I0610 144418.542527 1 shared_informer.go:240] Waiting for caches to sync for client-ca:kube systemextension apiserver authentication:client-ca-file I0610 144418.542578 1 configmap_cafile_content.go:201] "Starting controller" name="client-ca:kube systemextension apiserver authentication:requestheader-client-ca-file" I0610 144418.542621 1 shared_informer.go:240] Waiting for caches to sync for client-ca:kube systemextension apiserver authentication:requestheader-client-ca-file I0610 144418.542733 1 secure_serving.go:200] Serving securely on 127.0.0.1:10258 I0610 144418.543349 1 tlsconfig.go:240] "Starting DynamicServingCertificateController" I0610 144418.544142 1 leaderelection.go:248] attempting to acquire leader lease kube-system/cloud-controller-manager... I0610 144418.551965 1 leaderelection.go:258] successfully acquired lease kube-system/cloud-controller-manager I0610 144418.552232 1 event.go:291] "Event occurred" object="kube-system/cloud-controller-manager" kind="Lease" apiVersion="coordination.k8s.io/v1" type="Normal" reason="LeaderElection" message="cltst-pool1-a9366016-2cq28_eb15f228-df6e-4734-8575-21c704306e6d became leader" I0610 144418.643526 1 shared_informer.go:247] Caches are synced for client-ca:kube systemextension apiserver authentication:requestheader-client-ca-file I0610 144418.643755 1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController I0610 144418.643872 1 shared_informer.go:247] Caches are synced for client-ca:kube systemextension apiserver authentication:client-ca-file I0610 144418.662187 1 node_controller.go:115] Sending events to api server. I0610 144418.662320 1 controllermanager.go:285] Started "cloud-node" I0610 144418.662372 1 node_controller.go:154] Waiting for informer caches to sync I0610 144418.663674 1 node_lifecycle_controller.go:76] Sending events to api server I0610 144418.663767 1 controllermanager.go:285] Started "cloud-node-lifecycle" E0610 144418.665273 1 core.go:92] Failed to start service controller: the cloud provider does not support external load balancers W0610 144418.665307 1 controllermanager.go:282] Skipping "service" I0610 144418.665357 1 core.go:103] Will not configure cloud provider routes, --configure-cloud-routes: false <----- that bugs me ---> W0610 144418.665374 1 controllermanager.go:282] Skipping "route" I0610 144418.763597 1 node_controller.go:390] Initializing node cltst-pool1-a9366016-2cq28 with cloud provider I0610 144418.777926 1 node_controller.go:454] Successfully initialized node cltst-pool1-a9366016-2cq28 with cloud provider I0610 144418.778260 1 event.go:291] "Event occurred" object="cltst-pool1-a9366016-2cq28" kind="Node" apiVersion="v1" type="Normal" reason="Synced" message="Node synced successfully" I0610 144914.996305 1 node_controller.go:390] Initializing node cltst-pool2-c1951d7e-fnrdv with cloud provider E0610 144915.001133 1 node_controller.go:212] error syncing 'cltst-pool2-c1951d7e-fnrdv': failed to get provider ID for node cltst-pool2-c1951d7e-fnrdv at cloudprovider: failed to get instance ID from cloud provider: address annotations not yet set, requeuing I0610 144915.005955 1 node_controller.go:390] Initializing node cltst-pool2-c1951d7e-fnrdv with cloud provider E0610 144915.006047 1 node_controller.go:212] error syncing 'cltst-pool2-c1951d7e-fnrdv': failed to get provider ID for node cltst-pool2-c1951d7e-fnrdv at cloudprovider: failed to get instance ID from cloud provider: address annotations not yet set, requeuing I0610 144915.016210 1 node_controller.go:390] Initializing node cltst-pool2-c1951d7e-fnrdv with cloud provider E0610 144915.016313 1 node_controller.go:212] error syncing 'cltst-pool2-c1951d7e-fnrdv': failed to get provider ID for node cltst-pool2-c1951d7e-fnrdv at cloudprovider: failed to get instance ID from cloud provider: address annotations not yet set, requeuing I0610 144915.020485 1 node_controller.go:390] Initializing node cltst-pool2-c1951d7e-fnrdv with cloud provider I0610 144915.171694 1 node_controller.go:454] Successfully initialized node cltst-pool2-c1951d7e-fnrdv with cloud provider I0610 144915.174165 1 event.go:291] "Event occurred" object="cltst-pool2-c1951d7e-fnrdv" kind="Node" apiVersion="v1" type="Normal" reason="Synced" message="Node synced successfully" Testing deployment is in pending with no changes in Octavia configuration (LBaaS) jmp-001:~$ kubectl get svc -n default NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 6h41m my-np-service LoadBalancer 10.43.224.222 <pending> 80:30441/TCP 4h15m

cloud-provider-config isn’t used if you use cloud-provider-name=external. You need to actually get your config into the deployment

Correct. I have used statis deployment from: - https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/openstack-cloud-controller-manager/using-openstack-cloud-controller-manager.md Also created secret including cert of Keystone endpoint and cloud.conf for making openstack-cloud-controller-manager-ds.yaml deployment consistent. I use same procedure in opensource k8s for years. It has worked for me, so far. RKE works fine as well, event configuration is slightly different. With RKE2, there is serious lack of information how to put all pieces toghether

RKE2 is just another Kubernetes distro. Our docs cover configuring the packaged components. If you’re deploying something to it yourself, you should be able to do that exactly the same as you would on any other distro.

How can I make it so, using rancher server for deployment. I'm running it, using OpenStack integration, so nodes are being provisioned by Rancher server and cluster config has to performed there. If I change anything manually, it's going to be rewritten back (by Rancher)

what version of Rancher are you using?

the latest 2.6.5 (asuming)

changing the cloud provider after the cluster is already deployed probably won’t work. The nodes will already be labeled with the wrong cloud provider ID

correct, I'm not changing anything on cluster level. All configs has to be performed on Rancher level

right, but I’m saying you will need to provision a new cluster for this

sure

kk

sure, take your time. thank you for help

cloud-provider-config means nothing to rke2 (or Kubernetes in general) if you’re not using one of the in-tree providers, So unless you’re setting cloud-provider-name=aws or cloud-provider-name=gce you might as well not bother.

Hello, I'd like to create OpenStack cluster using RKE2 and run some commands on all nodes before the cluster is formed. I tried to do it by userDataFile like on RKE but for RKE2 it does not work. Is the format of userDataFile different for RKE2? My userDataFile is as follows:

So, I have an update on deployment. First, the recommended procedure works only when a cluster is being built as a single node. When deployed as a two-node cluster (master + worker) and cloud controller set as "external" + add-on manifest, the rancher agent is infinitely stopped at the "connecting agent" step. The only working procedure is when you create a two-node cluster (master + worker) without any cloud controller enabled and let the cluster be reconciled. Then patch YAML configuration end edit cloud controller by "external" and insert add-on manifest into a cluster configuration. Then is cluster reconciled, with updated configuration and "external" cloud controller enabled. BTW, userData input in the OpenStack integration form seems not to be working. During cloud-init, commands are not run in an instance.