This message was deleted.
# generala
adamant-kite-43734
09/12/2022, 2:22 PMThis message was deleted.
🙏 1
b
brash-planet-10109
09/12/2022, 2:27 PMOMG
Is it?
Are you sure, it is because of modifications
e
eager-refrigerator-66976
09/13/2022, 8:03 AMI am pretty sure that any modification to the cluster configuration applied via terraform cause cluster deletion occasionally…
eager-refrigerator-66976
09/13/2022, 8:04 AMI’ve lost couple times prod clusters already just by applying manifests change…
eager-refrigerator-66976
09/13/2022, 8:05 AMI was doing some more testing, and this doesn’t happen if I modify
Clustereager-refrigerator-66976
09/13/2022, 8:06 AMalso I’ve tried to reproduce this applying cluster modification via rancher API directly, so this also was very stable for 2k+ iterations
eager-refrigerator-66976
09/13/2022, 8:07 AMso far this happens for me only If I change my cluster configuration with terraform provider
b
brash-planet-10109
09/13/2022, 8:08 AMThat's really scary
I will use UI then for provisioning and managing state
e
eager-refrigerator-66976
09/13/2022, 8:09 AMwell, this doesn’t scale if you have to manage many cluster in gitOps maner
đź‘Ť 1
eager-refrigerator-66976
09/13/2022, 8:11 AMbut yeh this is a disaster, first time it happened to me I was applying some label change in production cluster and it just destroyed the whole cluster in seconds
b
brash-planet-10109
09/13/2022, 8:12 AMYeah correct
Please share your analysis after resolution
b
best-microphone-20624
09/15/2022, 12:05 AMHave you considered using a k8s gitops cd tool to apply cluster custom resource updates from git as opposed to using terraform? Unfortunately the rancher experience for gitops provisioning and managing custom clusters is not great. We try to use our IaaS tool as little as possible for this and use our k8s gitops tool as much as possible for managing the clusters and their addons.
e
eager-refrigerator-66976
09/15/2022, 7:55 AM@best-microphone-20624 yeh, this is something I am currently considering to use instead as using terraform is just dangerous
the thing is actually this might be not terraform provider issue, all it does is calling norman (rancher apis) APIs so this might be API bug
b
best-microphone-20624
09/15/2022, 12:07 PMI would expect fleet-based cluster mgmt at scale to be better tested than terraform-based cluster mgmt. It may be that terraform uses the same underlying apis but does so in a less reliable manner.
4 Views