https://rancher.com/ logo
Powered by
Title
g

gorgeous-cat-50570

01/11/2023, 10:07 AM
Hello, recently I encountered a problem in rancher v2.7.0. I was using the GUI to delete a project but in resulted in this error: 
Waiting on project-precan-alert-controller_c-vfm92
Inspecting container logs : 
[ERROR] error syncing 'c-vfm92/creator-cluster-owner': handler mgmt-auth-crtb-controller: couldn't create role cluster-owner: <http://roles.rbac.authorization.k8s.io|roles.rbac.authorization.k8s.io> "cluster-owner" is forbidden: unable to create new content in namespace p-jwcwp because it is being terminated, requeuing
[ERROR] error syncing 'c-vfm92/u-mxcjeyd7a6-admin': handler mgmt-auth-crtb-controller: couldn't create role cluster-owner: <http://roles.rbac.authorization.k8s.io|roles.rbac.authorization.k8s.io> "cluster-owner" is forbidden: unable to create new content in namespace p-jwcwp because it is being terminated, requeuing
[ERROR] error syncing 'c-vfm92/c-vfm92-fleet-default-owner': handler mgmt-auth-crtb-controller: couldn't create role cluster-owner: <http://roles.rbac.authorization.k8s.io|roles.rbac.authorization.k8s.io> "cluster-owner" is forbidden: unable to create new content in namespace p-jwcwp because it is being terminated, requeuing
l

limited-pizza-33551

01/11/2023, 10:39 AM
Hello bennii, good day! • Could you provide more information about your environment? • Was the user cluster-owner created by you? If yes, did you notice an error during creation? Are you able to log off and log on to the Rancher GUI as cluster-owner?
g

gorgeous-cat-50570

01/11/2023, 10:42 AM
NAME="Ubuntu"
VERSION_ID="22.04"
Rancher container image = v2.7.0
I can login / logout as cluster-owner . I can create other projects / delete them and so on
l

limited-pizza-33551

01/11/2023, 10:51 AM
So if I understand correctly, there is only a single project that you're facing this issue?
g

gorgeous-cat-50570

01/11/2023, 10:52 AM
yes exactly
And it needs to be deleted before doing anything (I need to use same project name)
l

limited-pizza-33551

01/11/2023, 10:54 AM
Can you execute kubectl get ns and provide the output here, please? I reckon that one of the rancher-created namespaces is stuck in Terminating from the above message but I'd just like to be sure
g

gorgeous-cat-50570

01/11/2023, 10:55 AM
the namespace is deleted (I have no ns in terminating state) and that what is suspicious 🤨
l

limited-pizza-33551

01/11/2023, 10:56 AM
Hmm, that certainly is suspicious. p-jwcwp is no longer there, you say? Did you retry the deletion of the project after you noticed that the namespace isn't there?
g

gorgeous-cat-50570

01/11/2023, 10:57 AM
yep I tried many times but same error , I used also the api to provision the project / ns and its config but no luck I get 
Status code was 403 and not [201]: HTTP Error 403: Forbidden
Also from the log I see that it tries to create the ns 
[mgmt-auth-crtb-controller] Creating role cluster-owner in namespace p-jwcwp
l

limited-pizza-33551

01/11/2023, 11:01 AM
So, the challenge here is that I think there is a finalizer that needs to be deleted. You will need to find it and delete it, otherwise I reckon it'll go on ad infinitum
g

gorgeous-cat-50570

01/11/2023, 11:03 AM
on which part
because I tried to look for something like that but I do not know from where to start hh
l

limited-pizza-33551

01/11/2023, 11:04 AM
kubectl get namespace <YOUR_NAMESPACE> -o json > <YOUR_NAMESPACE>.json
-- Can you try executing this command and see if anything shows up in the json file? Replace YOUR_NAMESPACE with the namespace name, please.
g

gorgeous-cat-50570

01/11/2023, 11:08 AM
I have no namespace in terminating state.
l

limited-pizza-33551

01/11/2023, 11:09 AM
bennii, I understand that you have nothing in the terminating state. But when you execute the above command, you'll be able to get the finalizer associated with the resource spec
Which is why I am asking you to execute it. I believe you when you say that you don't have any in the terminating state. But to get the finalizers, we need to check if anything shows up in this json file.
✔️ 1
g

gorgeous-cat-50570

01/11/2023, 11:11 AM
I know but how can I execute an api request of something not existing ?
error from server (NotFound): namespaces "xxxx" not found
l

limited-pizza-33551

01/11/2023, 11:13 AM
Okay, thank you. So the resource + finalizer problem is ruled out.
g

gorgeous-cat-50570

01/11/2023, 11:13 AM
I also restarted rancher container but same issue
one more thing , I have not created a namespace called 
p-jwcwp
but in rancher logs I get the error I posted above
l

limited-pizza-33551

01/11/2023, 11:23 AM
kubectl get projects.management.cattle.io --all-namespaces
Can you execute this command, please?
And see if there are any dead projects lingering?
g

gorgeous-cat-50570

01/11/2023, 11:25 AM
error: the server doesn't have a resource type "projects"
l

limited-pizza-33551

01/11/2023, 11:29 AM
Is this namespace - c-vfm92 still alive?
Sorry *resource
g

gorgeous-cat-50570

01/11/2023, 11:31 AM
I guess those weird naming are coming from rancher container , because I have no namespaces named as c-xxx or t-xxx or whatever
l

limited-pizza-33551

01/11/2023, 11:34 AM
Was the project created using terraform?
g

gorgeous-cat-50570

01/11/2023, 11:35 AM
nope ansible
l

limited-pizza-33551

01/11/2023, 11:37 AM
In the YAML file, there will be a finalizer added I'm pretty sure for 
<http://lifecycle.cattle.io/create.project-precan-alert-controller_|lifecycle.cattle.io/create.project-precan-alert-controller_>
✔️ 1
g

gorgeous-cat-50570

01/11/2023, 11:37 AM
{
"id": "c-vfm92/p-jwcwp",
"type": "management.cattle.io.project",
"links": {
"remove": "blocked",
"self": "…/v1/management.cattle.io.projects/c-vfm92/p-jwcwp",
"update": "blocked",
"view": "…/apis/management.cattle.io/v3/namespaces/c-vfm92/projects/p-jwcwp"
},
"apiVersion": "<http://management.cattle.io/v3|management.cattle.io/v3>",
"kind": "Project",
"metadata": {
"annotations": {
"<http://authz.management.cattle.io/creator-role-bindings|authz.management.cattle.io/creator-role-bindings>": "{\"created\":[\"project-owner\"],\"required\":[\"project-owner\"]}",
"<http://field.cattle.io/creatorId|field.cattle.io/creatorId>": "user-55flh",
"<http://lifecycle.cattle.io/create.mgmt-project-rbac-remove|lifecycle.cattle.io/create.mgmt-project-rbac-remove>": "true",
"<http://lifecycle.cattle.io/create.project-namespace-auth_c-vfm92|lifecycle.cattle.io/create.project-namespace-auth_c-vfm92>": "true",
"<http://lifecycle.cattle.io/create.project-precan-alert-controller_c-vfm92|lifecycle.cattle.io/create.project-precan-alert-controller_c-vfm92>": "true"
},
"creationTimestamp": "2023-01-10T15:06:46Z",
"deletionGracePeriodSeconds": 0,
"deletionTimestamp": "2023-01-11T08:55:43Z",
"fields": [ 2 items
"p-jwcwp",
"20h"
],
"finalizers": [
"<http://clusterscoped.controller.cattle.io/project-precan-alert-controller_c-vfm92|clusterscoped.controller.cattle.io/project-precan-alert-controller_c-vfm92>"
],
this is a part of the api file for that project
the problem I can’t edit the yaml file and send the request back
l

limited-pizza-33551

01/11/2023, 11:46 AM
Not a recommended approach, but you can use kubectl replace --raw
g

gorgeous-cat-50570

01/11/2023, 11:47 AM
after a few investigation I went to see the ns in the local cluster that rancher container uses and it turns out that there is a namespace in terminating state
l

limited-pizza-33551

01/11/2023, 11:48 AM
Yeah - so you have to use kubectl edit namespace <your_namespace>
Remove the associated finalizer
Then delete the namespace.
g

gorgeous-cat-50570

01/11/2023, 11:49 AM
finalizers: {}
should be a dict right ?
l

limited-pizza-33551

01/11/2023, 11:50 AM
"finalizers":[]
You can refer to this issue - https://github.com/rancher/rancher/issues/14715. There are many ways to do it easily.
Lmk how it goes
g

gorgeous-cat-50570

01/11/2023, 11:55 AM
there is no finalizers in metadata for that ns
l

limited-pizza-33551

01/11/2023, 11:57 AM
Okay - can you execute kubectl get namespace <YOUR_NAMESPACE> -o json > <YOUR_NAMESPACE>.json?
Whatever your namespace name is
👍 1
g

gorgeous-cat-50570

01/11/2023, 11:58 AM
done
only finalizer in spec not in metadata.
l

limited-pizza-33551

01/11/2023, 11:59 AM
Remove the kubernetes from finalizers array which is under spec
Then execute this - 
kubectl replace --raw "/api/v1/namespaces/<YOUR_NAMESPACE>/finalize" -f ./<YOUR_NAMESPACE>.json
g

gorgeous-cat-50570

01/11/2023, 12:00 PM
done
l

limited-pizza-33551

01/11/2023, 12:00 PM
kubectl get namespace
Your namespace should be gone
g

gorgeous-cat-50570

01/11/2023, 12:01 PM
SomeResourcesRemain","message":"Some resources are remaining: <http://projectalertgroups.management.cattle.io|projectalertgroups.management.cattle.io> has 1 resource instances, <http://projectalertrules.management.cattle.io|projectalertrules.management.cattle.io> has 2 resource instances"},{"type":"NamespaceFinalizersRemaining","status":"True","lastTransitionTime":"2023-01-11T08:55:48Z","reason":"SomeFinalizersRemain","message":"Some content in the namespace has finalizers remaining: <http://clusterscoped.controller.cattle.io/pod-target-alert-watcher_c-vfm92|clusterscoped.controller.cattle.io/pod-target-alert-watcher_c-vfm92> in 2 resource instances, <http://clusterscoped.controller.cattle.io/project-alert-group-lifecycle_c-vfm92|clusterscoped.controller.cattle.io/project-alert-group-lifecycle_c-vfm92> in 1 resource instances"
not really it is binded to some stuff that should be resolved first 🤦‍♂️
l

limited-pizza-33551

01/11/2023, 12:02 PM
Oh okay, it still has stuff hanging around. That's really surprising
Can you open another terminal and run kubectl proxy?
In that terminal, could you run curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json https://localhost:8001/api/v1/namespaces/YOUR_NAMESPACE/finalize
g

gorgeous-cat-50570

01/11/2023, 12:11 PM
I can’t open a proxy . and some resources are hanging , I can see them using GUI . force deletion not working
l

limited-pizza-33551

01/11/2023, 12:13 PM
If that doesn't work, we will need to check if any of the apiservices are unavailable by executing and not serving its resources: • kubectl get apiservice|grep False Once that's done, check the resources associated with the apiservice in your namespace that still exist • kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get -n $your-ns-to-delete Then use kubectl delete • kubectl delete APIService <resource_name>
g

gorgeous-cat-50570

01/11/2023, 12:16 PM
all of them are True :melting_face:
l

limited-pizza-33551

01/11/2023, 12:17 PM
In the local cluster, as well?
g

gorgeous-cat-50570

01/11/2023, 12:17 PM
yep
l

limited-pizza-33551

01/11/2023, 12:18 PM
What are the resources that you can see on the GUI then?
Can you execute kubectl delete APIService <name_of_resource>
Is this the command that hangs?
g

gorgeous-cat-50570

01/11/2023, 12:21 PM
• list the resource: 
kubectl get ProjectAlertRules -n p-jwcwp
• delete: 
kubectl delete ProjectAlertRules less-than-half-workload-available --force --grace-period=0 -n p-jwcwp
l

limited-pizza-33551

01/11/2023, 12:23 PM
• kubectl get apiservice | grep False -n p-jwcwp
Can you execute this?
g

gorgeous-cat-50570

01/11/2023, 12:24 PM
nothing
l

limited-pizza-33551

01/11/2023, 12:26 PM
kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get -n p-jwcwp
g

gorgeous-cat-50570

01/11/2023, 12:26 PM
No resources found in p-jwcwp namespace
l

limited-pizza-33551

01/11/2023, 12:27 PM
Can you execute kubectl get namespace?
g

gorgeous-cat-50570

01/11/2023, 12:29 PM
i have a list
l

limited-pizza-33551

01/11/2023, 12:31 PM
of?
g

gorgeous-cat-50570

01/11/2023, 12:34 PM
all namespaces and the ns in terminating state still persists
but I guess I found teh solution
one moment
I patched the hanging resources using kubectl and the ns is gone
now if I want to delete the project that is hanging I get this : 
[ERROR] Error during subscribe websocket: close sent
l

limited-pizza-33551

01/11/2023, 12:54 PM
Could be because your CPU/Ram is overloaded
Could you logout and login and retry?
If that doesn't work, please try restarting the node.
g

gorgeous-cat-50570

01/11/2023, 1:06 PM
I have no time now . but if you are working with rancher community there are some other issue • minimize terminal on rancher GUI and reopen it agin -> loosing session • installing an application does not show related helm errors like previous version 2.5.15 • Gui is slow • Extension not showing installed once
19 Views
#generalJoin Slack
Powered by