https://rancher.com/ logo
#rke2
Title
# rke2
b

billowy-animal-38808

05/31/2022, 2:39 PM
Hello everyone. Does Rke2 allow the creation of LoadBalancer services by default? I have a rke2 Cluster and I created a LoadBalancer service but when listing the services it is in a
pending
state. Can someone explain to me what to do to enable this option?
f

faint-airport-83518

05/31/2022, 2:41 PM
are you using the cloud controller?
you could go look at the logs for the cloud controller pod if so
b

billowy-animal-38808

05/31/2022, 2:42 PM
I just install it by
curl -sfL <https://get.rke2.io> | sh -
command in my machine in VirtualBox
f

full-painter-23916

05/31/2022, 4:15 PM
Kubernetes (and rke/rke2) itself does not include any implementation of a load balancer controller, so entries you create just stay in pending because nothing does anything with them. Cloud controllers (for AWS, Azure, etc) or specific balancers like MetalLB implement it.
c

creamy-pencil-82913

05/31/2022, 6:26 PM
K3s is meant to be run in smaller self-contained environments, does include a stub load-balancer controller, in klipper-lb. RKE2 expects to be deployed to a traditional enterprise/cloud environment where you’ll be using an external cloud provider for LB services, or MetalLB for bare metal, as @full-painter-23916 said.
a

abundant-night-53774

06/10/2022, 5:17 PM
is there any guilde how to hook-up rke2 to external LB, running on OpenStack?
when I use similar config as in RKE or "general K8s" I'm getting following error: I0610 144418.663767 1 controllermanager.go:285] Started "cloud-node-lifecycle" E0610 144418.665273 1 core.go:92] Failed to start service controller: the cloud provider does not support external load balancers W0610 144418.665307 1 controllermanager.go:282] Skipping "service" I0610 144418.665357 1 core.go:103] Will not configure cloud provider routes, --configure-cloud-routes: false
c

creamy-pencil-82913

06/10/2022, 5:19 PM
Deploy the external LB's LB Controller?
If the cloud controller doesn't support load balancers there's probably a separate controller for load balancers.
a

abundant-night-53774

06/10/2022, 5:33 PM
sure, there is extended configuation and controller, but doesn't work
c

creamy-pencil-82913

06/10/2022, 7:00 PM
is that a limitation of the openstack cloud controller?
a

abundant-night-53774

06/10/2022, 8:10 PM
I don't think so as I have it under control. Is there any guide how to make that combo working? I just made "gues" based on k8s experience and aws integration on https://docs.rke2.io/advanced/
c

creamy-pencil-82913

06/10/2022, 8:24 PM
if it’s not bundled with rke2 then you’d deploy it exactly the same as you would on any other Kubernetes cluster
a

abundant-night-53774

06/10/2022, 8:44 PM
unfortunately, that doesn't work 😕
c

creamy-pencil-82913

06/10/2022, 8:55 PM
What specifically doesn’t work?
a

abundant-night-53774

06/10/2022, 9:27 PM
I have addedd file: - /etc/rancher/rke2/config.yaml.d/55-openstack.yaml with content: { "cloud-provider-config": "/etc/rancher/rke2/cloud.conf", "cloud-provider-name": "external" } root@cltst-pool1-a9366016-2cq28:~# cat /etc/rancher/rke2/cloud.conf [Global] auth-url="<keystone URL>" domain-name="<domain>" tenant-name="BUILD" username="Admin" password="<password>" tls-insecure = true [LoadBalancer] use-octavia=true lb-provider="octavia" subnet-id="96f8e0b6-e5a4-4b8a-aeea-e6c2f9759a2e" create-monitor=true monitor-delay=60s monitor-timeout=30s monitor-max-retries=5 manage-security-groups=true [Networking] ipv6-support-disabled=true [BlockStorage] bs-version=v2 followed agent restart -> all green Then I made controller deployment in context of cluster -> controller is working with logs: ubuntu@czbrn-sky-jmp-001:~$ kubectl logs -n kube-system pods/cloud-controller-manager-cltst-pool1-a9366016-2cq28 I0610 144418.410692 1 serving.go:354] Generated self-signed cert in-memory I0610 144418.539458 1 controllermanager.go:142] Version: v1.22.1-k3s1 I0610 144418.542287 1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController I0610 144418.542369 1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController I0610 144418.542468 1 configmap_cafile_content.go:201] "Starting controller" name="client-ca:kube systemextension apiserver authentication:client-ca-file" I0610 144418.542527 1 shared_informer.go:240] Waiting for caches to sync for client-ca:kube systemextension apiserver authentication:client-ca-file I0610 144418.542578 1 configmap_cafile_content.go:201] "Starting controller" name="client-ca:kube systemextension apiserver authentication:requestheader-client-ca-file" I0610 144418.542621 1 shared_informer.go:240] Waiting for caches to sync for client-ca:kube systemextension apiserver authentication:requestheader-client-ca-file I0610 144418.542733 1 secure_serving.go:200] Serving securely on 127.0.0.1:10258 I0610 144418.543349 1 tlsconfig.go:240] "Starting DynamicServingCertificateController" I0610 144418.544142 1 leaderelection.go:248] attempting to acquire leader lease kube-system/cloud-controller-manager... I0610 144418.551965 1 leaderelection.go:258] successfully acquired lease kube-system/cloud-controller-manager I0610 144418.552232 1 event.go:291] "Event occurred" object="kube-system/cloud-controller-manager" kind="Lease" apiVersion="coordination.k8s.io/v1" type="Normal" reason="LeaderElection" message="cltst-pool1-a9366016-2cq28_eb15f228-df6e-4734-8575-21c704306e6d became leader" I0610 144418.643526 1 shared_informer.go:247] Caches are synced for client-ca:kube systemextension apiserver authentication:requestheader-client-ca-file I0610 144418.643755 1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController I0610 144418.643872 1 shared_informer.go:247] Caches are synced for client-ca:kube systemextension apiserver authentication:client-ca-file I0610 144418.662187 1 node_controller.go:115] Sending events to api server. I0610 144418.662320 1 controllermanager.go:285] Started "cloud-node" I0610 144418.662372 1 node_controller.go:154] Waiting for informer caches to sync I0610 144418.663674 1 node_lifecycle_controller.go:76] Sending events to api server I0610 144418.663767 1 controllermanager.go:285] Started "cloud-node-lifecycle" E0610 144418.665273 1 core.go:92] Failed to start service controller: the cloud provider does not support external load balancers W0610 144418.665307 1 controllermanager.go:282] Skipping "service" I0610 144418.665357 1 core.go:103] Will not configure cloud provider routes, --configure-cloud-routes: false <----- that bugs me ---> W0610 144418.665374 1 controllermanager.go:282] Skipping "route" I0610 144418.763597 1 node_controller.go:390] Initializing node cltst-pool1-a9366016-2cq28 with cloud provider I0610 144418.777926 1 node_controller.go:454] Successfully initialized node cltst-pool1-a9366016-2cq28 with cloud provider I0610 144418.778260 1 event.go:291] "Event occurred" object="cltst-pool1-a9366016-2cq28" kind="Node" apiVersion="v1" type="Normal" reason="Synced" message="Node synced successfully" I0610 144914.996305 1 node_controller.go:390] Initializing node cltst-pool2-c1951d7e-fnrdv with cloud provider E0610 144915.001133 1 node_controller.go:212] error syncing 'cltst-pool2-c1951d7e-fnrdv': failed to get provider ID for node cltst-pool2-c1951d7e-fnrdv at cloudprovider: failed to get instance ID from cloud provider: address annotations not yet set, requeuing I0610 144915.005955 1 node_controller.go:390] Initializing node cltst-pool2-c1951d7e-fnrdv with cloud provider E0610 144915.006047 1 node_controller.go:212] error syncing 'cltst-pool2-c1951d7e-fnrdv': failed to get provider ID for node cltst-pool2-c1951d7e-fnrdv at cloudprovider: failed to get instance ID from cloud provider: address annotations not yet set, requeuing I0610 144915.016210 1 node_controller.go:390] Initializing node cltst-pool2-c1951d7e-fnrdv with cloud provider E0610 144915.016313 1 node_controller.go:212] error syncing 'cltst-pool2-c1951d7e-fnrdv': failed to get provider ID for node cltst-pool2-c1951d7e-fnrdv at cloudprovider: failed to get instance ID from cloud provider: address annotations not yet set, requeuing I0610 144915.020485 1 node_controller.go:390] Initializing node cltst-pool2-c1951d7e-fnrdv with cloud provider I0610 144915.171694 1 node_controller.go:454] Successfully initialized node cltst-pool2-c1951d7e-fnrdv with cloud provider I0610 144915.174165 1 event.go:291] "Event occurred" object="cltst-pool2-c1951d7e-fnrdv" kind="Node" apiVersion="v1" type="Normal" reason="Synced" message="Node synced successfully" Testing deployment is in pending with no changes in Octavia configuration (LBaaS) jmp-001:~$ kubectl get svc -n default NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 6h41m my-np-service LoadBalancer 10.43.224.222 <pending> 80:30441/TCP 4h15m
TL;DR openstack looks it's running in cluster, but doesn't operate svc kind LoadBalancer
BTW, it's v1.22.9+rke2r2
c

creamy-pencil-82913

06/10/2022, 9:50 PM
cloud-provider-config isn’t used if you use cloud-provider-name=external. You need to actually get your config into the deployment
how did you deploy it? helm chart?
passing cloud-provider-config into the core Kubernetes components only makes sense if you’re using the in-tree cloud providers
if you’re using out-of-tree providers and deploying them as pods or something then you need to get the config passed into the pods
a

abundant-night-53774

06/10/2022, 10:12 PM
Correct. I have used statis deployment from: - https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/openstack-cloud-controller-manager/using-openstack-cloud-controller-manager.md Also created secret including cert of Keystone endpoint and cloud.conf for making openstack-cloud-controller-manager-ds.yaml deployment consistent. I use same procedure in opensource k8s for years. It has worked for me, so far. RKE works fine as well, event configuration is slightly different. With RKE2, there is serious lack of information how to put all pieces toghether
c

creamy-pencil-82913

06/10/2022, 10:15 PM
RKE2 is just another Kubernetes distro. Our docs cover configuring the packaged components. If you’re deploying something to it yourself, you should be able to do that exactly the same as you would on any other distro.
The logs you show above appear to be for the built-in cloud controller, not the openstack one. Have you tried starting the RKE2 servers with
--disable-cloud-controller
to disable the built-in external cloud controller?
a

abundant-night-53774

06/10/2022, 10:34 PM
How can I make it so, using rancher server for deployment. I'm running it, using OpenStack integration, so nodes are being provisioned by Rancher server and cluster config has to performed there. If I change anything manually, it's going to be rewritten back (by Rancher)
c

creamy-pencil-82913

06/10/2022, 10:35 PM
what version of Rancher are you using?
a

abundant-night-53774

06/10/2022, 10:35 PM
the latest 2.6.5 (asuming)
c

creamy-pencil-82913

06/10/2022, 10:35 PM
changing the cloud provider after the cluster is already deployed probably won’t work. The nodes will already be labeled with the wrong cloud provider ID
a

abundant-night-53774

06/10/2022, 10:37 PM
correct, I'm not changing anything on cluster level. All configs has to be performed on Rancher level
c

creamy-pencil-82913

06/10/2022, 10:37 PM
right, but I’m saying you will need to provision a new cluster for this
a

abundant-night-53774

06/10/2022, 10:38 PM
sure
c

creamy-pencil-82913

06/10/2022, 10:38 PM
kk
let me remind myself what the option is in the UI. I mostly work on RKE2 and K3s, I don’t get into the Rancher web interface too much
a

abundant-night-53774

06/10/2022, 10:39 PM
sure, take your time. thank you for help
Speaking of deploying servers, we've created cluster using Rancher server using OpenStack provision integration. Then I added file: /etc/rancher/rke2/config.yaml.d/55-openstack.yaml with content: { "cloud-provider-config": "/etc/rancher/rke2/cloud.conf", "cloud-provider-name": "external" } + file /etc/rancher/rke2/cloud.conf and perform restar rancher agent Then procedure above.
c

creamy-pencil-82913

06/10/2022, 10:47 PM
cloud-provider-config means nothing to rke2 (or Kubernetes in general) if you’re not using one of the in-tree providers, So unless you’re setting cloud-provider-name=aws or cloud-provider-name=gce you might as well not bother.
do you get the difference between in-tree and out-of-tree cloud providers?
When creating the cluster in the UI, if you set the Cloud Provider to External, and leave the Cloud Provider Config empty, that should get you what you want.
Note that the nodes will remain tainted Unintialized until the cloud provider is active, so you might need to bundle the manifest for that into the cluster under Add-On Config -> Additional Manifest
s

square-car-51889

06/14/2022, 4:42 PM
Hello, I'd like to create OpenStack cluster using RKE2 and run some commands on all nodes before the cluster is formed. I tried to do it by userDataFile like on RKE but for RKE2 it does not work. Is the format of userDataFile different for RKE2? My userDataFile is as follows:
#!/bin/sh touch /tmp/test
a

abundant-night-53774

06/15/2022, 4:22 PM
So, I have an update on deployment. First, the recommended procedure works only when a cluster is being built as a single node. When deployed as a two-node cluster (master + worker) and cloud controller set as "external" + add-on manifest, the rancher agent is infinitely stopped at the "connecting agent" step. The only working procedure is when you create a two-node cluster (master + worker) without any cloud controller enabled and let the cluster be reconciled. Then patch YAML configuration end edit cloud controller by "external" and insert add-on manifest into a cluster configuration. Then is cluster reconciled, with updated configuration and "external" cloud controller enabled. BTW, userData input in the OpenStack integration form seems not to be working. During cloud-init, commands are not run in an instance.
113 Views