Hello recently I encountered a problem in rancher v2 7 0 I w rancher-users #general

Title

gorgeous-cat-50570

01/11/2023, 10:07 AM

Hello, recently I encountered a problem in rancher v2.7.0. I was using the GUI to delete a project but in resulted in this error:

Waiting on project-precan-alert-controller_c-vfm92

Inspecting container logs :

Copy code

[ERROR] error syncing 'c-vfm92/creator-cluster-owner': handler mgmt-auth-crtb-controller: couldn't create role cluster-owner: <http://roles.rbac.authorization.k8s.io|roles.rbac.authorization.k8s.io> "cluster-owner" is forbidden: unable to create new content in namespace p-jwcwp because it is being terminated, requeuing
[ERROR] error syncing 'c-vfm92/u-mxcjeyd7a6-admin': handler mgmt-auth-crtb-controller: couldn't create role cluster-owner: <http://roles.rbac.authorization.k8s.io|roles.rbac.authorization.k8s.io> "cluster-owner" is forbidden: unable to create new content in namespace p-jwcwp because it is being terminated, requeuing
[ERROR] error syncing 'c-vfm92/c-vfm92-fleet-default-owner': handler mgmt-auth-crtb-controller: couldn't create role cluster-owner: <http://roles.rbac.authorization.k8s.io|roles.rbac.authorization.k8s.io> "cluster-owner" is forbidden: unable to create new content in namespace p-jwcwp because it is being terminated, requeuing

limited-pizza-33551

01/11/2023, 10:39 AM

Hello bennii, good day! • Could you provide more information about your environment? • Was the user cluster-owner created by you? If yes, did you notice an error during creation? Are you able to log off and log on to the Rancher GUI as cluster-owner?

gorgeous-cat-50570

01/11/2023, 10:42 AM

Copy code

NAME="Ubuntu"
VERSION_ID="22.04"
Rancher container image = v2.7.0

I can login / logout as cluster-owner . I can create other projects / delete them and so on

limited-pizza-33551

01/11/2023, 10:51 AM

So if I understand correctly, there is only a single project that you're facing this issue?

gorgeous-cat-50570

01/11/2023, 10:52 AM

yes exactly

And it needs to be deleted before doing anything (I need to use same project name)

limited-pizza-33551

01/11/2023, 10:54 AM

Can you execute kubectl get ns and provide the output here, please? I reckon that one of the rancher-created namespaces is stuck in Terminating from the above message but I'd just like to be sure

gorgeous-cat-50570

01/11/2023, 10:55 AM

the namespace is deleted (I have no ns in terminating state) and that what is suspicious 🤨

limited-pizza-33551

01/11/2023, 10:56 AM

Hmm, that certainly is suspicious. p-jwcwp is no longer there, you say? Did you retry the deletion of the project after you noticed that the namespace isn't there?

gorgeous-cat-50570

01/11/2023, 10:57 AM

yep I tried many times but same error , I used also the api to provision the project / ns and its config but no luck I get

Status code was 403 and not [201]: HTTP Error 403: Forbidden

Also from the log I see that it tries to create the ns

[mgmt-auth-crtb-controller] Creating role cluster-owner in namespace p-jwcwp

limited-pizza-33551

01/11/2023, 11:01 AM

So, the challenge here is that I think there is a finalizer that needs to be deleted. You will need to find it and delete it, otherwise I reckon it'll go on ad infinitum

gorgeous-cat-50570

01/11/2023, 11:03 AM

on which part

because I tried to look for something like that but I do not know from where to start hh

limited-pizza-33551

01/11/2023, 11:04 AM

Copy code

kubectl get namespace <YOUR_NAMESPACE> -o json > <YOUR_NAMESPACE>.json

-- Can you try executing this command and see if anything shows up in the json file? Replace YOUR_NAMESPACE with the namespace name, please.

gorgeous-cat-50570

01/11/2023, 11:08 AM

I have no namespace in terminating state.

limited-pizza-33551

01/11/2023, 11:09 AM

bennii, I understand that you have nothing in the terminating state. But when you execute the above command, you'll be able to get the finalizer associated with the resource spec

Which is why I am asking you to execute it. I believe you when you say that you don't have any in the terminating state. But to get the finalizers, we need to check if anything shows up in this json file.

✔️ 1

gorgeous-cat-50570

01/11/2023, 11:11 AM

I know but how can I execute an api request of something not existing ?

error from server (NotFound): namespaces "xxxx" not found

limited-pizza-33551

01/11/2023, 11:13 AM

Okay, thank you. So the resource + finalizer problem is ruled out.

gorgeous-cat-50570

01/11/2023, 11:13 AM

I also restarted rancher container but same issue

one more thing , I have not created a namespace called

p-jwcwp

but in rancher logs I get the error I posted above

limited-pizza-33551

01/11/2023, 11:23 AM

kubectl get projects.management.cattle.io --all-namespaces

Can you execute this command, please?

And see if there are any dead projects lingering?

gorgeous-cat-50570

01/11/2023, 11:25 AM

error: the server doesn't have a resource type "projects"

limited-pizza-33551

01/11/2023, 11:29 AM

Is this namespace - c-vfm92 still alive?

Sorry *resource

gorgeous-cat-50570

01/11/2023, 11:31 AM

I guess those weird naming are coming from rancher container , because I have no namespaces named as c-xxx or t-xxx or whatever

limited-pizza-33551

01/11/2023, 11:34 AM

Was the project created using terraform?

gorgeous-cat-50570

01/11/2023, 11:35 AM

nope ansible

limited-pizza-33551

01/11/2023, 11:37 AM

In the YAML file, there will be a finalizer added I'm pretty sure for

Copy code

<http://lifecycle.cattle.io/create.project-precan-alert-controller_|lifecycle.cattle.io/create.project-precan-alert-controller_>

✔️ 1

gorgeous-cat-50570

01/11/2023, 11:37 AM

Copy code

{
"id": "c-vfm92/p-jwcwp",
"type": "management.cattle.io.project",
"links": {
"remove": "blocked",
"self": "…/v1/management.cattle.io.projects/c-vfm92/p-jwcwp",
"update": "blocked",
"view": "…/apis/management.cattle.io/v3/namespaces/c-vfm92/projects/p-jwcwp"
},
"apiVersion": "<http://management.cattle.io/v3|management.cattle.io/v3>",
"kind": "Project",
"metadata": {
"annotations": {
"<http://authz.management.cattle.io/creator-role-bindings|authz.management.cattle.io/creator-role-bindings>": "{\"created\":[\"project-owner\"],\"required\":[\"project-owner\"]}",
"<http://field.cattle.io/creatorId|field.cattle.io/creatorId>": "user-55flh",
"<http://lifecycle.cattle.io/create.mgmt-project-rbac-remove|lifecycle.cattle.io/create.mgmt-project-rbac-remove>": "true",
"<http://lifecycle.cattle.io/create.project-namespace-auth_c-vfm92|lifecycle.cattle.io/create.project-namespace-auth_c-vfm92>": "true",
"<http://lifecycle.cattle.io/create.project-precan-alert-controller_c-vfm92|lifecycle.cattle.io/create.project-precan-alert-controller_c-vfm92>": "true"
},
"creationTimestamp": "2023-01-10T15:06:46Z",
"deletionGracePeriodSeconds": 0,
"deletionTimestamp": "2023-01-11T08:55:43Z",
"fields": [ 2 items
"p-jwcwp",
"20h"
],
"finalizers": [
"<http://clusterscoped.controller.cattle.io/project-precan-alert-controller_c-vfm92|clusterscoped.controller.cattle.io/project-precan-alert-controller_c-vfm92>"
],

this is a part of the api file for that project

the problem I can’t edit the yaml file and send the request back

limited-pizza-33551

01/11/2023, 11:46 AM

Not a recommended approach, but you can use kubectl replace --raw

gorgeous-cat-50570

01/11/2023, 11:47 AM

after a few investigation I went to see the ns in the local cluster that rancher container uses and it turns out that there is a namespace in terminating state

limited-pizza-33551

01/11/2023, 11:48 AM

Yeah - so you have to use kubectl edit namespace <your_namespace>

Remove the associated finalizer

Then delete the namespace.

gorgeous-cat-50570

01/11/2023, 11:49 AM

finalizers: {}

should be a dict right ?

limited-pizza-33551

01/11/2023, 11:50 AM

Copy code

"finalizers":[]

You can refer to this issue - https://github.com/rancher/rancher/issues/14715. There are many ways to do it easily.

Lmk how it goes

gorgeous-cat-50570

01/11/2023, 11:55 AM

there is no finalizers in metadata for that ns

limited-pizza-33551

01/11/2023, 11:57 AM

Okay - can you execute kubectl get namespace <YOUR_NAMESPACE> -o json > <YOUR_NAMESPACE>.json?

Whatever your namespace name is

👍 1

gorgeous-cat-50570

01/11/2023, 11:58 AM

done

only finalizer in spec not in metadata.

limited-pizza-33551

01/11/2023, 11:59 AM

Remove the kubernetes from finalizers array which is under spec

Then execute this -

Copy code

kubectl replace --raw "/api/v1/namespaces/<YOUR_NAMESPACE>/finalize" -f ./<YOUR_NAMESPACE>.json

gorgeous-cat-50570

01/11/2023, 12:00 PM

done

limited-pizza-33551

01/11/2023, 12:00 PM

kubectl get namespace

Your namespace should be gone

gorgeous-cat-50570

01/11/2023, 12:01 PM

SomeResourcesRemain","message":"Some resources are remaining: <http://projectalertgroups.management.cattle.io|projectalertgroups.management.cattle.io> has 1 resource instances, <http://projectalertrules.management.cattle.io|projectalertrules.management.cattle.io> has 2 resource instances"},{"type":"NamespaceFinalizersRemaining","status":"True","lastTransitionTime":"2023-01-11T08:55:48Z","reason":"SomeFinalizersRemain","message":"Some content in the namespace has finalizers remaining: <http://clusterscoped.controller.cattle.io/pod-target-alert-watcher_c-vfm92|clusterscoped.controller.cattle.io/pod-target-alert-watcher_c-vfm92> in 2 resource instances, <http://clusterscoped.controller.cattle.io/project-alert-group-lifecycle_c-vfm92|clusterscoped.controller.cattle.io/project-alert-group-lifecycle_c-vfm92> in 1 resource instances"

not really it is binded to some stuff that should be resolved first 🤦‍♂️

limited-pizza-33551

01/11/2023, 12:02 PM

Oh okay, it still has stuff hanging around. That's really surprising

Can you open another terminal and run kubectl proxy?

In that terminal, could you run curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json https://localhost:8001/api/v1/namespaces/YOUR_NAMESPACE/finalize

gorgeous-cat-50570

01/11/2023, 12:11 PM

I can’t open a proxy . and some resources are hanging , I can see them using GUI . force deletion not working

limited-pizza-33551

01/11/2023, 12:13 PM

If that doesn't work, we will need to check if any of the apiservices are unavailable by executing and not serving its resources: • kubectl get apiservice|grep False Once that's done, check the resources associated with the apiservice in your namespace that still exist • kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get -n $your-ns-to-delete Then use kubectl delete • kubectl delete APIService <resource_name>

gorgeous-cat-50570

01/11/2023, 12:16 PM

all of them are True 🫠

limited-pizza-33551

01/11/2023, 12:17 PM

In the local cluster, as well?

gorgeous-cat-50570

01/11/2023, 12:17 PM

yep

limited-pizza-33551

01/11/2023, 12:18 PM

What are the resources that you can see on the GUI then?

Can you execute kubectl delete APIService <name_of_resource>

Is this the command that hangs?

gorgeous-cat-50570

01/11/2023, 12:21 PM

• list the resource:

kubectl get ProjectAlertRules -n p-jwcwp

• delete:

kubectl delete ProjectAlertRules less-than-half-workload-available --force --grace-period=0 -n p-jwcwp

limited-pizza-33551

01/11/2023, 12:23 PM

• kubectl get apiservice | grep False -n p-jwcwp

Can you execute this?

gorgeous-cat-50570

01/11/2023, 12:24 PM

nothing

limited-pizza-33551

01/11/2023, 12:26 PM

kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get -n p-jwcwp

gorgeous-cat-50570

01/11/2023, 12:26 PM

No resources found in p-jwcwp namespace

limited-pizza-33551

01/11/2023, 12:27 PM

Can you execute kubectl get namespace?

gorgeous-cat-50570

01/11/2023, 12:29 PM

i have a list

limited-pizza-33551

01/11/2023, 12:31 PM

of?

gorgeous-cat-50570

01/11/2023, 12:34 PM

all namespaces and the ns in terminating state still persists

but I guess I found teh solution

one moment

I patched the hanging resources using kubectl and the ns is gone

now if I want to delete the project that is hanging I get this :

[ERROR] Error during subscribe websocket: close sent

limited-pizza-33551

01/11/2023, 12:54 PM

Could be because your CPU/Ram is overloaded

Could you logout and login and retry?

If that doesn't work, please try restarting the node.

gorgeous-cat-50570

01/11/2023, 1:06 PM

I have no time now . but if you are working with rancher community there are some other issue • minimize terminal on rancher GUI and reopen it agin -> loosing session • installing an application does not show related helm errors like previous version 2.5.15 • Gui is slow • Extension not showing installed once

172 Views