https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# general
a
This message was deleted.
h
@bumpy-printer-21267 If you cannot access your rancher ui because the cluster agent is not connected, you might not be able to directly retrieve the master-slave IPs from the rancher itself.
However, I think you can use other methods depending on the underlying infrastructure of your rancher setup.
b
Yeah it's a vsphere RKE, trying to log in directly into masters now to see if there might be old IPs in the log files.
h
I would say you check your dhcp server logs. Depending on how your dhcp server is set up, it might have logged the IP assignments it made. If you have a centralized logging service or system logs from around the time of the change, that could also help you discover the assigned IP addresses.
b
Unfortunately those were trimmed 😞
h
If you have shell access to any nodes in the cluster, you may be able to use 
kubectl
or other k8s commands to retrieve information about the cluster, such as the 
kubectl get nodes -o wide
command. This will show you information about all nodes, including their internal IP addresses.
Or another resource could be the rancher DB. Rancher maintains a database that holds information about all of its managed clusters. If you have access to this database, you might be able to find the IP addresses you are looking for. However, manipulating rancher's internal database directly can be risky and should only be done if you are confident in what you are doing.
b
^^ that's what I was hoping to do. I can access rancher itself just fine.
h
Any luck?
b
Need to dig through packer logs to find node passwords to log into them locally. If I can't do it, I will try rancher DB.
h
Gotcha
b
I can access nodes
But there's no kubectl there lol
h
Where?
How about in the controller node?
b
on the nodes themselves
h
You should be able to set the proper context and see the nodes/pods, etc.
kubectl get nodes -o wide
What can you see with this ^^
b
on rancher?
h
How did you setup rancher?
b
rancher only shows local-node
h
which one?
Are you using rancher-desktop?
b
No, docker installation
h
Cool, so you should see all the nodes if the config is right. Do you know the context?.
Also you can try kubectl get nodes -A
So you can see all the nodes from all namespaces
Probably you are not using the right namespace in your context.
But if you are using docker for the rancher setup i guess you can inspect the docker container
Run 
docker ps
on your Docker host. This will list out all running Docker containers and their IDs. Find the container IDs for your rancher server and agent containers.
docker inspect container_id
replace the container_id with the actual container id
b
there's only one container for docker.
I can bash into it and see all the actual cluster namespaces
but get nodes still only shows local-node 😞
h
How many namespaces can you see in the clusteR?
You have to set the right namespace
b
Copy code
`NAME                                         STATUS   AGE
c-xw54q                                      Active   396d
cattle-fleet-clusters-system                 Active   396d
cattle-fleet-local-system                    Active   396d
cattle-fleet-system                          Active   396d
cattle-global-data                           Active   396d
cattle-global-nt                             Active   396d
cattle-impersonation-system                  Active   396d
cattle-system                                Active   396d
cluster-fleet-default-c-xw54q-5b0d8956f2eb   Active   396d
cluster-fleet-local-local-1a3d67d0a899       Active   396d
default                                      Active   396d
fleet-default                                Active   396d
fleet-local                                  Active   396d
kube-node-lease                              Active   396d
kube-public                                  Active   396d
kube-system                                  Active   396d
local                                        Active   396d
p-6cph6                                      Active   396d
p-82xsz                                      Active   145d
p-9b2xb                                      Active   396d
p-crws4                                      Active   396d
p-jstjs                                      Active   123d
p-lqtvd                                      Active   124d
p-pt9km                                      Active   32d
p-scw4z                                      Active   167d
p-spkh6                                      Active   396d
p-vwzd4                                      Active   396d
p-wsjwc                                      Active   395d
p-wzszj                                      Active   396d
p-xxpjv                                      Active   346d
u-8wmql                                      Active   123d
u-lwd6k                                      Active   123d
u-zxb2j                                      Active   396d
user-qftdg                                   Active   396d
h
Did you try kubectl get nodes -A first so you can validate there's more than the local-node?
^^
b
only shows local-node
h
Umm, well If you're running rancher as a single docker container and your k8s nodes are independent machines, and you don't have direct access to the rancher ui, you can try using another method like: like querying the db within the rancher container.
First access the rancher docker container: docker exec -it container_id bash
b
I do have access to rancher uI, but the problem is it won't let me open that container
h
mysql -u cattle -p cattle: replace the values with your own.
b
s/container/cluster/
h
And then SELECT * FROM host;
Ummm but the vsphere agent is not connected.
b
well yes, because the IPs changed
h
You have a misconfig you need to sort that out first.
What does the logs from the container shows?
docker logs container?
b
Copy code
2023/06/25 17:04:34 [ERROR] error syncing 'c-xw54q': handler cluster-deploy: Get "<https://10.12.2.187:6443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent>": cluster agent disconnected, requeuing
2023/06/25 17:04:39 [INFO] Stopping cluster agent for c-xw54q
2023/06/25 17:04:39 [ERROR] failed to start cluster controllers c-xw54q: context canceled
that's what it loops on
h
Yeah so it cannot find the k8s api server ip
b
would that be one of the masters?
h
Yes, the IP address 10.12.2.187 you provided in the error message should belong to one of the master nodes in the k8s cluster. The master node runs the k8s API server, which is the central management entity of a k8s cluster, and listens on port 6443 by default. This server processes REST operations, validates them, and updates the corresponding objects in etcd.
b
I guess I'll try changing their IPs to that one until it connects
h
Well that's an option, certenly you need to sort that out first.
b
@hundreds-battery-84841 that's a lot for your help, my laptop's about to die, I need to get off the road and plug myself iin
(obviously this happened in the middle of a road trip)
h
No problem man, just let me know.
b
@hundreds-battery-84841 does the docker based rancher have an internal mysql?
h
Well, that was just an example. But I think docker based rancher installation uses SQLite, not MySQL.
b
cool, now I just need to locate it
h
Yeah
If I'm not mistaken it'd be /var/lib/rancher/management-state/rancher.db
sqlite3 /var/lib/rancher/rancher.db
.tables
etc
b
Copy code
a6318174ecdb:/var/lib/rancher # find . -name *.db
./k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/155/fs/usr/lib/sysimage/rpm/Index.db
./k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/155/fs/usr/lib/sysimage/rpm/Packages.db
./k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/156/fs/usr/lib/sysimage/rpm/Index.db
./k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/156/fs/usr/lib/sysimage/rpm/Packages.db
./k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/164/fs/usr/lib/sysimage/rpm/Index.db
./k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/164/fs/usr/lib/sysimage/rpm/Packages.db
./k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/176/fs/usr/lib/sysimage/rpm/Index.db
./k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/176/fs/usr/lib/sysimage/rpm/Packages.db
./k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/159/fs/usr/lib/sysimage/rpm/Index.db
./k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/159/fs/usr/lib/sysimage/rpm/Packages.db
./k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/metadata.db
./k3s/agent/containerd/io.containerd.snapshotter.v1.stargz/snapshotter/metadata.db
./k3s/agent/containerd/io.containerd.metadata.v1.bolt/meta.db
these are the only db files...
didn't they replace it with etcd?
h
Uff yeah i think so
But those files are not related to rancher
They are related to its underelying distro k3s
k3s uses etcd, mysql or postgresql to store its state, including the state of the clusters it manages.
In the case of etcd, the data is not stored in a simple .db file, but rather in a more complex format that's not directly human readable i would say.
In the case of a k3s setup, the rancher state is stored in a different location. When you installed rancher on top of k3s, it used k3s bundled etcd compatible database if i'm not mistaken, named Kine, which by default uses a sqlite database stored in /var/lib/rancher/k3s/server/db/state.db
b
damn
layers upon layers upon layers 🙂
h
hahahaah
Keep in mind that kine is not rancher native. I'ts a component of k3s to replace etcd
b
all right, looks like grepping for CATTLE_ADDRESS in master nodes logs did the trick 😃
And things are back to green. Thanks for your help again.
h
Sure
289 Views
Open in Slack
PreviousNext
Powered by