I have a real old rancher 2.3.2 environment that will be decommissioned this year. I rebuilt the entire rancher cluster, all kubernetes is up but for the elasticsearch workloads the pods all stuck in pending mode. I used a working cluster for comparison purposes and identical to the working but all master nodes and worker nodes for elastic are pending. It does not even try to install, no errors, nothing in logs

kubectl get all --all-namespaces

NAMESPACE NAME READY STATUS RESTARTS AGE ca819-elasticsearch pod/ca819-elastic01-0-679d9c648b-nm26c 0/1 Pending 0 12m ca819-elasticsearch pod/ca819-elastic01-0-756b4c8f7d-xlm6m 0/1 Pending 0 44m ca819-elasticsearch pod/ca819-elastic01-1-68d46db4fd-4szmp 0/1 Pending 0 43m ca819-elasticsearch pod/ca819-elastic01-1-756cc84d9f-vhz5b 0/1 Pending 0 12m ca819-elasticsearch pod/ca819-elastic01-2-5bfbddc7bb-wwlq2 0/1 Pending 0 43m ca819-elasticsearch pod/ca819-elastic01-2-649cc975b5-8xfsp 0/1 Pending 0 12m ca819-elasticsearch pod/ca819-elastic01-3-596f86cdf5-kx8fb 0/1 Pending 0 12m ca819-elasticsearch pod/ca819-elastic01-3-7786c7c5b-mckcb 0/1 Pending 0 43m ca819-elasticsearch pod/ca819-elastic01-master-58445b886c-bvvq2 0/1 Pending 0 13m ca819-elasticsearch pod/ca819-elastic02-0-5c689954c8-skwtp 0/1 Pending 0 42m ca819-elasticsearch pod/ca819-elastic02-0-74849dfc49-s8pxs 0/1 Pending 0 12m ca819-elasticsearch pod/ca819-elastic02-1-5f89594588-ksn2z 0/1 Pending 0 42m ca819-elasticsearch pod/ca819-elastic02-1-6484cc889-trc9w 0/1 Pending 0 12m ca819-elasticsearch pod/ca819-elastic02-2-64d96c7885-ng8pr 0/1 Pending 0 42m ca819-elasticsearch pod/ca819-elastic02-2-6b6c44c49-bj46p 0/1 Pending 0 11m ca819-elasticsearch pod/ca819-elastic02-3-676ddc779d-pjnb6 0/1 Pending 0 11m ca819-elasticsearch pod/ca819-elastic02-3-bb46d7f6-stj72 0/1 Pending 0 41m ca819-elasticsearch pod/ca819-elastic02-master-7bd6d46796-gwr2g 0/1 Pending 0 12m cattle-system pod/cattle-cluster-agent-c88bd9b8c-bjwps 1/1 Running 0 25h cattle-system pod/cattle-node-agent-fncrl 1/1 Running 0 25h cattle-system pod/cattle-node-agent-g72qz 1/1 Running 0 25h cattle-system pod/cattle-node-agent-jmgrr 1/1 Running 0 25h cattle-system pod/cattle-node-agent-qkp4v 1/1 Running 0 25h cattle-system pod/cattle-node-agent-qpp6h 1/1 Running 0 25h cattle-system pod/cattle-node-agent-rt555 1/1 Running 0 25h cattle-system pod/cattle-node-agent-t9c7s 1/1 Running 0 25h cattle-system pod/cattle-node-agent-zbl9r 1/1 Running 0 25h cattle-system pod/kube-api-auth-5zchd 1/1 Running 0 25h cattle-system pod/kube-api-auth-6kqtm 1/1 Running 0 25h cattle-system pod/kube-api-auth-7r5n4 1/1 Running 0 25h cattle-system pod/kube-api-auth-9pb8v 1/1 Running 0 25h cattle-system pod/kube-api-auth-g4w9z 1/1 Running 0 25h cattle-system pod/kube-api-auth-nvznt 1/1 Running 0 25h cattle-system pod/kube-api-auth-p8k5v 1/1 Running 0 25h cattle-system pod/kube-api-auth-xnrkt 1/1 Running 0 25h ingress-nginx pod/default-http-backend-5bcc9fd598-dllpn 1/1 Running 0 25h ingress-nginx pod/nginx-ingress-controller-2qqqg 1/1 Running 0 25h ingress-nginx pod/nginx-ingress-controller-47c4f 1/1 Running 0 25h ingress-nginx pod/nginx-ingress-controller-97qrl 1/1 Running 0 25h ingress-nginx pod/nginx-ingress-controller-dx2rl 1/1 Running 0 25h ingress-nginx pod/nginx-ingress-controller-n84nj 1/1 Running 0 25h ingress-nginx pod/nginx-ingress-controller-pps7j 1/1 Running 0 25h ingress-nginx pod/nginx-ingress-controller-qjsrh 1/1 Running 0 25h ingress-nginx pod/nginx-ingress-controller-v2l8r 1/1 Running 0 25h kube-system pod/canal-2mcnc 2/2 Running 1 25h kube-system pod/canal-85497 2/2 Running 0 25h kube-system pod/canal-86zl8 2/2 Running 0 25h kube-system pod/canal-g6jb6 2/2 Running 0 25h kube-system pod/canal-k2jcw 2/2 Running 1 25h kube-system pod/canal-nmrck 2/2 Running 0 25h kube-system pod/canal-vm2kg 2/2 Running 0 25h kube-system pod/canal-vthm6 2/2 Running 1 25h kube-system pod/coredns-799dffd9c4-4lnqb 1/1 Running 4 25h kube-system pod/coredns-799dffd9c4-6vbss 1/1 Running 11 25h kube-system pod/coredns-799dffd9c4-mxplc 1/1 Running 7 25h kube-system pod/coredns-autoscaler-84766fbb4-779nx 1/1 Running 0 25h kube-system pod/metrics-server-5846c94685-d7kxq 1/1 Running 0 25h kube-system pod/rke-coredns-addon-deploy-job-nr6hs 0/1 Completed 0 25h kube-system pod/rke-ingress-controller-deploy-job-7sj7f 0/1 Completed 0 25h kube-system pod/rke-metrics-addon-deploy-job-bnphk 0/1 Completed 0 25h kube-system pod/rke-network-plugin-deploy-job-h5vlp 0/1 Completed 0 25h no idea why

anything in events of the namespace ?

kubectl describe pod -n ca819-elasticsearch ca819-elastic01-master-58445b886c-bvvq2

Name: ca819-elastic01-master-58445b886c-bvvq2 Namespace: ca819-elasticsearch Priority: 0 Node: <none> Labels: pod-template-hash=58445b886c workload.user.cattle.io/workloadselector=deployment-ca819-elasticsearch-ca819-elastic01-master Annotations: cattle.io/timestamp: 2025-04-24T233213Z field.cattle.io/ports: [[{"containerPort":9200,"dnsName":"ca819-elastic01-master-nodeport","kind":"NodePort","name":"9200tcp01","protocol":"TCP","sourcePort":0},... workload.cattle.io/state: {"Y2E4MTktY2FwdHVyZTAx":"c-xb6ln:m-68d0ab2cc0b7"} Status: Pending IP: IPs: <none> Controlled By: ReplicaSet/ca819-elastic01-master-58445b886c Containers: ca819-elastic01-master: Image: docker.elastic.co/elasticsearch/elasticsearch:6.8.21 Ports: 9200/TCP, 9300/TCP Host Ports: 0/TCP, 0/TCP Liveness: tcp-socket :9200 delay=60s timeout=2s period=2s #success=1 #failure=3 Readiness: tcp-socket :9200 delay=60s timeout=2s period=2s #success=2 #failure=3 Environment: ES_JAVA_OPTS: -Xms4g -Xmx4g cluster.name: ca819-pcap discovery.zen.minimum_master_nodes: 2 discovery.zen.ping.unicast.hosts: ca819-elastic01-master,ca819-elastic01-0,ca819-elastic01-1,ca819-elastic01-2,ca819-elastic01-3,ca819-elastic02-master,ca819-elastic02-0,ca819-elastic02-1,ca819-elastic02-2,ca819-elastic02-3 node.data: false node.master: true node.name: ca819-elastic01-master xpack.graph.enabled: true xpack.ml.enabled: false xpack.monitoring.collection.enabled: true xpack.monitoring.enabled: true xpack.security.enabled: false xpack.watcher.enabled: false Mounts: /usr/share/elasticsearch/data from data (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-87xzr (ro) Volumes: data: Type: HostPath (bare host directory volume) Path: /var/lib/elasticsearch4/ca819-elastic01.scrub.scrub.com-ca819-elastic01-4 HostPathType: default-token-87xzr: Type: Secret (a volume populated by a Secret) SecretName: default-token-87xzr Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: <none>

pretty much just me redeploying

kubectl get events -n ca819-elasticsearch

LAST SEEN TYPE REASON OBJECT MESSAGE 18m Normal SuccessfulDelete replicaset/ca819-elastic01-0-585c448d47 Deleted pod: ca819-elastic01-0-585c448d47-jvhmb 18m Normal SuccessfulCreate replicaset/ca819-elastic01-0-679d9c648b Created pod: ca819-elastic01-0-679d9c648b-nm26c 50m Normal SuccessfulCreate replicaset/ca819-elastic01-0-756b4c8f7d Created pod: ca819-elastic01-0-756b4c8f7d-xlm6m 50m Normal SuccessfulDelete replicaset/ca819-elastic01-0-9f6979488 Deleted pod: ca819-elastic01-0-9f6979488-qrrr2 50m Normal ScalingReplicaSet deployment/ca819-elastic01-0 Scaled down replica set ca819-elastic01-0-9f6979488 to 0 50m Normal ScalingReplicaSet deployment/ca819-elastic01-0 Scaled up replica set ca819-elastic01-0-756b4c8f7d to 1 18m Normal ScalingReplicaSet deployment/ca819-elastic01-0 Scaled down replica set ca819-elastic01-0-585c448d47 to 0 18m Normal ScalingReplicaSet deployment/ca819-elastic01-0 Scaled up replica set ca819-elastic01-0-679d9c648b to 1 18m Normal SuccessfulDelete replicaset/ca819-elastic01-1-54c56cf8ff Deleted pod: ca819-elastic01-1-54c56cf8ff-28w5l 50m Normal SuccessfulDelete replicaset/ca819-elastic01-1-568f6bb6c8 Deleted pod: ca819-elastic01-1-568f6bb6c8-2sqrq 50m Normal SuccessfulCreate replicaset/ca819-elastic01-1-68d46db4fd Created pod: ca819-elastic01-1-68d46db4fd-4szmp 18m Normal SuccessfulCreate replicaset/ca819-elastic01-1-756cc84d9f Created pod: ca819-elastic01-1-756cc84d9f-vhz5b 50m Normal ScalingReplicaSet deployment/ca819-elastic01-1 Scaled down replica set ca819-elastic01-1-568f6bb6c8 to 0 50m Normal ScalingReplicaSet deployment/ca819-elastic01-1 Scaled up replica set ca819-elastic01-1-68d46db4fd to 1 18m Normal ScalingReplicaSet deployment/ca819-elastic01-1 Scaled down replica set ca819-elastic01-1-54c56cf8ff to 0 18m Normal ScalingReplicaSet deployment/ca819-elastic01-1 Scaled up replica set ca819-elastic01-1-756cc84d9f to 1

Are you deploying elastic with helm chart? Is it a bitnami chart or the elastic operator? Do you have any nodeselectors/affinities set in your deployment yamls that are not on the node?

try to deploy something simpler like this https://kubernetes.io/docs/tasks/run-application/run-stateless-application-deployment/#creating-and-exploring-an-nginx-deployment if this works then review the requirements for ES...

all i have is the docker image

Yeah, for sure yaml and even when you get them up, you need svc yamls and have the master talk to the ingest nodes.

dumped the yaml's between working and failing, typo < schedulerName: default-schedule ---

schedulerName: default-scheduler

I’d also recommend take on the learning curve of deploying this with helm. Either with the elastic operator or bitnami chart if there is one. It will be easier to maintain in the future.

ty

Nice! Glad you figure it out. Just in case, some reading material

ty