This message was deleted Rancher Users #rke2

Join Slack

This message was deleted.

# rke2

adamant-kite-43734

11/24/2025, 10:19 AM

This message was deleted.

narrow-guitar-87575

11/24/2025, 12:01 PM

ok, not related to multus itself but to the rke machinery somehow - it is happening also with 4.2.2 multus version.

creamy-pencil-82913

11/24/2025, 8:56 PM

I’ve not seen this before. Note that RKE2 itself doesn’t do anything to directly manage the multus credentials. It is 100% managed by things in the multus helm chart.

narrow-guitar-87575

11/24/2025, 8:57 PM

For me it is reproducible. 1.33.5 is ok, 1.33.6 fails after 1 hour of multus running as token is not refreshed. Reverting 1.33.6 back to 1.33.5 solved the issue (with some restarts of multus).

narrow-guitar-87575

11/24/2025, 8:58 PM

ok, so I could try to just revert multus and try a bit more restarts in 1.33.6. will let you know

creamy-pencil-82913

11/24/2025, 8:58 PM

the apiserver refreshes SA tokens hourly on all Kubernetes versions. There would not be any difference in that behavior between RKE2 releases. Could be something broken in upstream Multus.

narrow-guitar-87575

11/24/2025, 8:59 PM

ok, will re-test and let you know. If I will find something multus related, will report to multus

creamy-pencil-82913

11/24/2025, 9:01 PM

have you customized the rke2-multus helmchartconfig in some way?

narrow-guitar-87575

11/24/2025, 9:03 PM

this is the only config I use:

Copy code

apiVersion: <http://helm.cattle.io/v1|helm.cattle.io/v1>
kind: HelmChartConfig
metadata:
  name: rke2-multus
  namespace: kube-system
spec:
  valuesContent: |-
    rke2-whereabouts:
      enabled: true
      resources:
        limits:
            cpu: 1
            memory: 300Mi
        requests:
            cpu: 1m
            memory: 50Mi

creamy-pencil-82913

11/24/2025, 9:03 PM

eager-iron-71848

11/25/2025, 3:16 PM

I tried to reproduce the issue on a fresh rke2 v1.33.6 install but I couldn’t. I can create pods using multus even after 1h. Do you see the issue only on an upgrade from 1.33.5 to 1.33.6?

narrow-guitar-87575

11/25/2025, 3:17 PM

yes, all cases I see they were upgrades from 1.33.5 to 133.6

eager-iron-71848

11/25/2025, 3:18 PM

Ok I’ll try to test that

narrow-guitar-87575

11/25/2025, 3:18 PM

thank you

narrow-guitar-87575

11/26/2025, 9:12 PM

this is probably some change in API machinery. multus from 1.33.5 rke2 version has token:

Copy code

{
  "aud": [
    "<https://kubernetes.default.svc.cluster.local>",
    "rke2"
  ],
  "exp": 1795726175,
  "iat": 1764190175,
  "iss": "<https://kubernetes.default.svc.cluster.local>",
  "jti": "x",
  "<http://kubernetes.io|kubernetes.io>": {
    "namespace": "kube-system",
    "node": {
      "name": "<http://kub-a5.priv.cerit-sc.cz|kub-a5.priv.cerit-sc.cz>",
      "uid": "5e66478b-b931-4d06-86f7-eb9d7e781ca1"
    },
    "pod": {
      "name": "rke2-multus-4gs84",
      "uid": "7ab64287-80cc-4735-9049-3ebb8d28e917"
    },
    "serviceaccount": {
      "name": "multus",
      "uid": "302045cb-9d93-4baa-b049-03e30a9338b8"
    },
    "warnafter": 1764193782
  },
  "nbf": 1764190175,
  "sub": "system:serviceaccount:kube-system:multus"
}

while rke2 1.33.6 issues token:

Copy code

{
  "aud": [
    "<https://kubernetes.default.svc.cluster.local>",
    "rke2"
  ],
  "exp": 1764194795,
  "iat": 1764191188,
  "iss": "<https://kubernetes.default.svc.cluster.local>",
  "jti": "x",
  "<http://kubernetes.io|kubernetes.io>": {
    "namespace": "kube-system",
    "node": {
      "name": "<http://app001.ostr.nrp1.du.cesnet.cz|app001.ostr.nrp1.du.cesnet.cz>",
      "uid": "0cef76c3-2679-40d1-bd07-2a05ced1cec6"
    },
    "pod": {
      "name": "rke2-multus-pb78x",
      "uid": "6c9a00d2-075f-438c-8a37-2f8556258314"
    },
    "serviceaccount": {
      "name": "multus",
      "uid": "03a9e926-29b6-4890-a800-a2c0125b210d"
    }
  },
  "nbf": 1764191188,
  "sub": "system:serviceaccount:kube-system:multus"
}

see that exp field in the 1.33.5 version is set to 1y ahead, while exp field in 1.33.6 is set to +1h. both versions mount the token using:

Copy code

- name: kube-api-access-lm6pk
      projected:
        defaultMode: 420
        sources:
          - serviceAccountToken:
              expirationSeconds: 3607
              path: token

so it looks like the expiry did not work before, but now it does. or does anybody understand things here?

creamy-pencil-82913

11/26/2025, 9:20 PM

This is how it always should have worked. Were you overriding the service account token expiration period in your apiserver config? https://kubernetes.io/docs/reference/access-authn-authz/service-accounts-admin/ > A

serviceAccountToken

source, that contains a token that the kubelet acquires from kube-apiserver. The kubelet fetches time-bound tokens using the TokenRequest API. A token served for a TokenRequest expires either when the pod is deleted or after a defined lifespan (by default, that is 1 hour). The kubelet also refreshes that token before the token expires. The token is bound to the specific Pod and has the kube-apiserver as its audience.

creamy-pencil-82913

11/26/2025, 9:23 PM

that would be the

--service-account-max-token-expiration

apiserver flag I believe

narrow-guitar-87575

11/26/2025, 9:24 PM

I guess it would be broken all the time then. It seems to work this way: multus pod gets api token, it creates kubeconfig from it to the host location /etc/cni/net.d/multus.d/multus.kubeconfig. pod's token is refreshed after 1 hour in both cases, however the multus pod never updates the host multus.kubeconfig again in both cases.

creamy-pencil-82913

11/26/2025, 9:25 PM

what is your primary CNI?

narrow-guitar-87575

11/26/2025, 9:25 PM

no, I do not use that flag.

Copy code

kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss-cerit.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --advertise-address=10.16.62.14 --allow-privileged=true --anonymous-auth=false --api-audiences=<https://kubernetes.default.svc.cluster.local>,rke2 --audit-log-format=json --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/log/kube-audit/audit.log --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --encryption-provider-config-automatic-reload=true --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=<https://127.0.0.1:2379> --feature-gates=UserNamespacesPodSecurityStandards=true,InPlacePodVerticalScalingExclusiveCPUs=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=<https://kubernetes.default.svc.cluster.local> --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16,2001:718:801:42cb:8:3::/112 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key

narrow-guitar-87575

11/26/2025, 9:25 PM

primary cni is calico (but same happens with primary cilium)

creamy-pencil-82913

11/26/2025, 9:37 PM

I just spun up a new server and I am seeing that the token is valid for 1 year

narrow-guitar-87575

11/26/2025, 9:37 PM

despite the limitation in the manifest?

creamy-pencil-82913

11/26/2025, 9:38 PM

this is the decoded token from /etc/cni/net.d/multus.d/multus.kubeconfig

Copy code

root@rke2-server-001:/# kubectl get node -o wide
NAME                          STATUS   ROLES                       AGE     VERSION          INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION    CONTAINER-RUNTIME
<http://rke2-server-001.example.com|rke2-server-001.example.com>   Ready    control-plane,etcd,master   9m38s   v1.33.6+rke2r1   172.17.0.4    <none>        Ubuntu 22.04.5 LTS   6.14.0-1012-aws   <containerd://2.1.5-k3s1>

narrow-guitar-87575

11/26/2025, 9:39 PM

and you should see the exactly the same token in

/var/lib/kubelet/pods/[multuspodid]/volumes/kubernetes.io~projected/kube-api-access-*/token

is that so?

creamy-pencil-82913

11/26/2025, 9:46 PM

yes.

Copy code

root@rke2-server-001:/# for ID in $(kubectl get pod -n kube-system -l app=rke2-multus -o jsonpath={.items[].metadata.uid}); do cat /var/lib/kubelet/pods/$ID/volumes/kubernetes.io~projected/kube-api-access-*/token; echo; done

creamy-pencil-82913

11/26/2025, 9:47 PM

Copy code

root@rke2-server-001:/# for ID in $(kubectl get pod -n kube-system -l app=rke2-multus -o jsonpath={.items[].metadata.uid}); do cat /var/lib/kubelet/pods/$ID/volumes/kubernetes.io~projected/kube-api-access-*/token; echo; done
eyJhbGciOiJSUzI1NiIsImtpZCI6InJSY0NXYUFjQXBMb21mX3AwclY4Mk13Q0w0aVF0NjhCTkpxMjVJYUhLMVEifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiLCJya2UyIl0sImV4cCI6MTc5NTcyODU2MSwiaWF0IjoxNzY0MTkyNTYxLCJpc3MiOiJodHRwczovL2t1YmVybmV0ZXMuZGVmYXVsdC5zdmMuY2x1c3Rlci5sb2NhbCIsImp0aSI6ImZlODU4MTQ4LTY2OTMtNGQwZS1iMjFhLTJjYmExY2ZhNGYwNCIsImt1YmVybmV0ZXMuaW8iOnsibmFtZXNwYWNlIjoia3ViZS1zeXN0ZW0iLCJub2RlIjp7Im5hbWUiOiJya2UyLXNlcnZlci0wMDEuZXhhbXBsZS5jb20iLCJ1aWQiOiJlZmY3NTI2Yy02MGIzLTQ0OGYtYjVjMC0xNjQwMTE3MGYwMWIifSwicG9kIjp7Im5hbWUiOiJya2UyLW11bHR1cy05bjVxdyIsInVpZCI6ImM3ZjdlNTBmLWRkMDEtNGNkZC1hYjVlLTBlNzBlYWI1MTMwNCJ9LCJzZXJ2aWNlYWNjb3VudCI6eyJuYW1lIjoibXVsdHVzIiwidWlkIjoiYTI2OGQwMjMtNDBlNy00OTZlLWFhZDQtNTYzZmVjYmEwMjdiIn0sIndhcm5hZnRlciI6MTc2NDE5NjE2OH0sIm5iZiI6MTc2NDE5MjU2MSwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOm11bHR1cyJ9.PgiJziYUx2B23OoyIHDv8irXd8JRtBtSZHSA6IRzv8bPZsf-0o9jCIUWSYaRx2cLyy1A_cAbGyIS-ThAdO62ZIvklnQV3xkQZ3pBVmLO5sLD8QbbvQfEcxgpCWXjSzgDpXgFayJlLVMl06ZYzGCB-G6eC8FcopVFhwdawWZdSF9x80w42AROnFmdDtlaBEviKFR08T4p-F2wNY9xx9tkd47XEt5rLqyxkJ9PI7KjqiIcUaffKFcHkWPW4w9kekMVXvUTEYVdWFKaouTbtZm19jp7tqwM7cqB-FDTml4zQ-wqluwvv2f3rrSWWMRkiXpwnQad9cRwjDlWuAbUBXXlFQ
root@rke2-server-001:/# grep token /etc/cni/net.d/multus.d/multus.kubeconfig
    token: "eyJhbGciOiJSUzI1NiIsImtpZCI6InJSY0NXYUFjQXBMb21mX3AwclY4Mk13Q0w0aVF0NjhCTkpxMjVJYUhLMVEifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiLCJya2UyIl0sImV4cCI6MTc5NTcyODU2MSwiaWF0IjoxNzY0MTkyNTYxLCJpc3MiOiJodHRwczovL2t1YmVybmV0ZXMuZGVmYXVsdC5zdmMuY2x1c3Rlci5sb2NhbCIsImp0aSI6ImZlODU4MTQ4LTY2OTMtNGQwZS1iMjFhLTJjYmExY2ZhNGYwNCIsImt1YmVybmV0ZXMuaW8iOnsibmFtZXNwYWNlIjoia3ViZS1zeXN0ZW0iLCJub2RlIjp7Im5hbWUiOiJya2UyLXNlcnZlci0wMDEuZXhhbXBsZS5jb20iLCJ1aWQiOiJlZmY3NTI2Yy02MGIzLTQ0OGYtYjVjMC0xNjQwMTE3MGYwMWIifSwicG9kIjp7Im5hbWUiOiJya2UyLW11bHR1cy05bjVxdyIsInVpZCI6ImM3ZjdlNTBmLWRkMDEtNGNkZC1hYjVlLTBlNzBlYWI1MTMwNCJ9LCJzZXJ2aWNlYWNjb3VudCI6eyJuYW1lIjoibXVsdHVzIiwidWlkIjoiYTI2OGQwMjMtNDBlNy00OTZlLWFhZDQtNTYzZmVjYmEwMjdiIn0sIndhcm5hZnRlciI6MTc2NDE5NjE2OH0sIm5iZiI6MTc2NDE5MjU2MSwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOm11bHR1cyJ9.PgiJziYUx2B23OoyIHDv8irXd8JRtBtSZHSA6IRzv8bPZsf-0o9jCIUWSYaRx2cLyy1A_cAbGyIS-ThAdO62ZIvklnQV3xkQZ3pBVmLO5sLD8QbbvQfEcxgpCWXjSzgDpXgFayJlLVMl06ZYzGCB-G6eC8FcopVFhwdawWZdSF9x80w42AROnFmdDtlaBEviKFR08T4p-F2wNY9xx9tkd47XEt5rLqyxkJ9PI7KjqiIcUaffKFcHkWPW4w9kekMVXvUTEYVdWFKaouTbtZm19jp7tqwM7cqB-FDTml4zQ-wqluwvv2f3rrSWWMRkiXpwnQad9cRwjDlWuAbUBXXlFQ"
root@rke2-server-001:/#rke2

creamy-pencil-82913

11/26/2025, 9:47 PM

same token

narrow-guitar-87575

11/26/2025, 9:47 PM

so the api somehow issues 1y token even if request was for 3607s

narrow-guitar-87575

11/26/2025, 9:47 PM

but not always 😉

creamy-pencil-82913

11/26/2025, 9:48 PM

well, and only for you

creamy-pencil-82913

11/26/2025, 9:48 PM

I do not see this on the same version of RKE2

narrow-guitar-87575

11/26/2025, 9:49 PM

I guess that intention here was that the multus pod rotates the host token, but for some reason, it does not. Will check the sources if it was meant to do so.

narrow-guitar-87575

11/26/2025, 9:49 PM

could it be upgrade related? I started the cluster with 1.33.5 and upgraded to 1.33.6

narrow-guitar-87575

11/26/2025, 9:50 PM

and yes, only for me it does what it is supposed to do? 😉

narrow-guitar-87575

11/26/2025, 9:50 PM

i.e., limit the token expiration to 1h+7s

creamy-pencil-82913

11/26/2025, 9:52 PM

well even if it does expire after 1 hour… literally all the multus pod does is sit there refreshing the kubeconfig every second

creamy-pencil-82913

11/26/2025, 9:52 PM

https://github.com/k8snetworkplumbingwg/multus-cni/blob/master/cmd/thin_entrypoint/main.go#L627-L639

creamy-pencil-82913

11/26/2025, 9:52 PM

so, maybe check multus pod logs to see what it says?

narrow-guitar-87575

11/26/2025, 9:54 PM

Copy code

kubeconfig is created in /host/etc/cni/net.d/multus.d/multus.kubeconfig
kubeconfig file is created.
master capabilities is get from conflist
multus config file is created.

and nothing more

creamy-pencil-82913

11/26/2025, 9:55 PM

what are the args to your multus pod? It should say

Entering watch loop…

creamy-pencil-82913

11/26/2025, 9:55 PM

Copy code

containers:
    - args:
      - --multus-conf-file=auto
      - --cni-conf-dir=/host/etc/cni/net.d
      - --cni-bin-dir=/host/opt/cni/bin
      - --multus-autoconfig-dir=/host/etc/cni/net.d
      - --multus-kubeconfig-file-host=/etc/cni/net.d/multus.d/multus.kubeconfig
      command:
      - /thin_entrypoint
      env:
      - name: KUBERNETES_NODE_NAME
        valueFrom:
          fieldRef:
            apiVersion: v1
            fieldPath: spec.nodeName
      image: rancher/hardened-multus-cni:v4.2.3-build20251031

creamy-pencil-82913

11/26/2025, 9:56 PM

oh hmm, at some point they gated the watching on

--cleanup-config-on-exit

flag

narrow-guitar-87575

11/26/2025, 9:56 PM

Copy code

- args:
        - '--multus-conf-file=auto'
        - '--cni-conf-dir=/host/etc/cni/net.d'
        - '--cni-bin-dir=/host/opt/cni/bin'
        - '--multus-autoconfig-dir=/host/etc/cni/net.d'
        - >-
          --multus-kubeconfig-file-host=/etc/cni/net.d/multus.d/multus.kubeconfig
      command:
        - /thin_entrypoint

creamy-pencil-82913

11/26/2025, 9:57 PM

I still don’t know why you’re seeing a different expiration for the token though

creamy-pencil-82913

11/26/2025, 9:59 PM

you might try this:

Copy code

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-multus
  namespace: kube-system
spec:
  valuesContent: |-
    config:
      cni_conf:
        cleanupConfigOnExit: true
    rke2-whereabouts:
      enabled: true
      resources:
        limits:
            cpu: 1
            memory: 300Mi
        requests:
            cpu: 1m
            memory: 50Mi

creamy-pencil-82913

11/26/2025, 10:00 PM

The chart values suggest that option is true by default but that is not the case

narrow-guitar-87575

11/26/2025, 10:01 PM

yes, now I have:

Copy code

kubeconfig is created in /host/etc/cni/net.d/multus.d/multus.kubeconfig
kubeconfig file is created.
master capabilities is get from conflist
multus config file is created.
Entering watch loop...

creamy-pencil-82913

11/26/2025, 10:01 PM

see if it works past an hour now

narrow-guitar-87575

11/26/2025, 10:02 PM

yes, will let you know. thank you

creamy-pencil-82913

11/26/2025, 10:02 PM

is your token still only valid for 1 hour?

narrow-guitar-87575

11/26/2025, 10:03 PM

yes, still only 1h

creamy-pencil-82913

11/26/2025, 10:06 PM

if that fixes it, would you mind opening an issue in the rancher/rke2 repo?

narrow-guitar-87575

11/26/2025, 10:07 PM

it seems that I looked at the wrong kube-api server as you ask about the flag:

--service-account-extend-token-expiration=false

it is actually set to this value

narrow-guitar-87575

11/26/2025, 10:07 PM

but I do not explicitly set this flag

creamy-pencil-82913

11/26/2025, 10:09 PM

Ahh that is from CIS profile

creamy-pencil-82913

11/26/2025, 10:09 PM

you did not mention you were running with hardened CIS profile

narrow-guitar-87575

11/26/2025, 10:09 PM

yes, sorry, I do use CIS profile

creamy-pencil-82913

11/26/2025, 10:09 PM

that would do it

narrow-guitar-87575

11/26/2025, 10:10 PM

but in both cases, 1.33.5 and 1.33.6

creamy-pencil-82913

11/26/2025, 10:10 PM

yeah it was just added for compat with latest CIS benchmark

creamy-pencil-82913

11/26/2025, 10:10 PM

I’ll open an issue

narrow-guitar-87575

11/26/2025, 10:11 PM

thank you for help

creamy-pencil-82913

11/26/2025, 10:25 PM

https://github.com/rancher/rke2/issues/9313

narrow-guitar-87575

11/26/2025, 11:17 PM

so yes, it seems that multus config workaround is ok

creamy-pencil-82913

11/26/2025, 11:20 PM

great!

70 Views

Open in Slack

Previous Next