Hi! I'm looking for information how Rancher stores information about changes done to deployments via UI (e.g. modifications of resources limits) We have an app installed under Rancher 2.4 and someone did some manual changes a while ago without leaving a trace what exactly was changed. Is there any way to browse the Rancher's CRDs and find what changes were done via Rancher UI? I don't see anything in AppRevision objects, nor in App objects.

Gotta say it -> 2.4 is extremely EOL and unsupportable I would upgrade ASAP. To answer your question - no, likely you wont be able to see a historical view of the actions on the cluster as rancher does not store that (rancher is not a version control system). If you want a historical view of the actions being done on your cluster - you need to enable a feature called 'audit logging' in your kubernetes distro: https://rancher.com/docs/rke/latest/en/config-options/audit-log/ Audit logs store a "who and what" trace of all of the requests to the kubernetes API server. Each cluster will need this enabled in order to see the requests to all the cluster. Rancher itself talks to the kubernetes API of each cluster so all actions on downstream clusters from the rancher UI will appear in the audit log as well.

Yes, I know it's EOL and upgrade to 2.6 is in the works on our side Regarding the audit - I can live without the information "who and what" at this moment, but I need to at least find "what" has been changed. The problem we face right now is that developers change Helm chart for application, run the deploy and after deploy they find that a change is not applied just because there was a manual change via UI. We found this only because developers were updating resources limits for one of the services and once they asked DevOps engineer to help them understand why pods are still saturating CPU. I found that even though Helm chart said 800m, deployment was set to 200m and it was done manually a while ago by another engineer. So what I'm looking for now is to check whole application and verify if there are other paramters, which were modified manually, so we can ensure that futuer Helm chart updates will get applied as expected

Rancher Stores its configuration in CRDs on the local cluster. Things like 'users', 'clusters' and 'global roles' are CRDs stored in etcd. You can use

kubectl get crds

on the to see all of these custom resources. As alluded to earlier Rancher is not 'git' so it doesn't 'version' resources or keep a historical log of 'who did what'. git is actually a good use for this. Put the app chart inside of a gitrepo - version it and you can see who changed what. Use an Admission controller (like kubewarden or OPA) and RBAC to prevent manual changes to resources such as pods and deployments in the cluster and use something like fleet to continuously deploy revisions of the App into the cluster. Devs should never talk to kubernetes directly nor should they have more than read-only access to the cluster. but if they must - enable audit logging on the rancher cluster to determine what kubernetes resources where modified when.

@bulky-sunset-52084 thank you for your answer I fully agree on your view regarding direct access to Kubernetes cluster by developers and we usually don't provide such access; sadly in this case it's an app which grew from PoC to production-ready and there are some changes done via Rancher UI by DevOps guy, while he was tinkering with PoC and simply Project Team didn't notice it until they've run into troubles... what I was looking for in my initial question was "which CRD is holding changes done via Rancher UI" as I have trouble finding it - I believe I checked all resources in the namespace reserved for project in Rancher's etcd (I mean the

p-xyzuw

like name for this particular app) and I still didn't find how Rancher stores it