Has anyone faced an issue where even setting the taints, tolerations, and node selector, the longhorn workloads still go to the wrong nodes? For instance, we have 2 nodes dedicated to database related workloads, including longhorn, and they are being tainted and labelled correctly. I have made the needed changes on Longhorn-ui, Longhorn manager, and the driver deployer, but still the CSI components for example are on the wrong nodes (non tainted). Not really sure what is going on, I have been looking into this for the past 2 days. I have been trying to get a hold of someone from longhorn team but had no luck so far. I saw some issues opened about that, but all of them were resolved with the taints and tolerations setting.

Did you also set the

System Managed Components Node Selector

? https://longhorn.io/docs/1.2.2/advanced-resources/deploy/node-selector/#setting-up-node-selector-after-longhorn-has-been-installed

Yes I did, but still not working. I am on the latest version (1.3.2). Our deployment is using

kubectl

, So I have been following the docs, the pages that seemed most relevant are this https://longhorn.io/docs/1.3.2/advanced-resources/deploy/taint-toleration/ and this https://longhorn.io/docs/1.3.2/advanced-resources/deploy/customizing-default-settings/#using-the-longhorn-deployment-yaml-file. I have setup on the ConfigMap the Node selector for system components, but still not able to get it working. I was initially thinking it was a memory/cpu allocation issue, but it is not.

You mean the related components (like attacher/provisioner/resizer rather than the plugin) do not apply the tolerations you set? To add the tolerations for this kind of System Managed Components (components that are dynamically deployed by longhorn manager after the installation), I would recommend to modify the setting mentioned in the doc rather than directly editing the ConfigMap

longhorn-default-setting

You are correct about the components giving me trouble. By the setting mentioned, do you mean the System Managed Components Node Selector setting on the Longhorn UI?