https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# general
a
This message was deleted.
l
Hello bennii, good day! • Could you provide more information about your environment? • Was the user cluster-owner created by you? If yes, did you notice an error during creation? Are you able to log off and log on to the Rancher GUI as cluster-owner?
g
Copy code
NAME="Ubuntu"
VERSION_ID="22.04"
Rancher container image = v2.7.0
I can login / logout as cluster-owner . I can create other projects / delete them and so on
l
So if I understand correctly, there is only a single project that you're facing this issue?
g
yes exactly
And it needs to be deleted before doing anything (I need to use same project name)
l
Can you execute kubectl get ns and provide the output here, please? I reckon that one of the rancher-created namespaces is stuck in Terminating from the above message but I'd just like to be sure
g
the namespace is deleted (I have no ns in terminating state) and that what is suspicious 🤨
l
Hmm, that certainly is suspicious. p-jwcwp is no longer there, you say? Did you retry the deletion of the project after you noticed that the namespace isn't there?
g
yep I tried many times but same error , I used also the api to provision the project / ns and its config but no luck I get 
Status code was 403 and not [201]: HTTP Error 403: Forbidden
Also from the log I see that it tries to create the ns 
[mgmt-auth-crtb-controller] Creating role cluster-owner in namespace p-jwcwp
l
So, the challenge here is that I think there is a finalizer that needs to be deleted. You will need to find it and delete it, otherwise I reckon it'll go on ad infinitum
g
on which part
because I tried to look for something like that but I do not know from where to start hh
l
Copy code
kubectl get namespace <YOUR_NAMESPACE> -o json > <YOUR_NAMESPACE>.json
-- Can you try executing this command and see if anything shows up in the json file? Replace YOUR_NAMESPACE with the namespace name, please.
g
I have no namespace in terminating state.
l
bennii, I understand that you have nothing in the terminating state. But when you execute the above command, you'll be able to get the finalizer associated with the resource spec
Which is why I am asking you to execute it. I believe you when you say that you don't have any in the terminating state. But to get the finalizers, we need to check if anything shows up in this json file.
✔️ 1
g
I know but how can I execute an api request of something not existing ?
error from server (NotFound): namespaces "xxxx" not found
l
Okay, thank you. So the resource + finalizer problem is ruled out.
g
I also restarted rancher container but same issue
one more thing , I have not created a namespace called 
p-jwcwp
but in rancher logs I get the error I posted above
l
kubectl get projects.management.cattle.io --all-namespaces
Can you execute this command, please?
And see if there are any dead projects lingering?
g
error: the server doesn't have a resource type "projects"
l
Is this namespace - c-vfm92 still alive?
Sorry *resource
g
I guess those weird naming are coming from rancher container , because I have no namespaces named as c-xxx or t-xxx or whatever
l
Was the project created using terraform?
g
nope ansible
l
In the YAML file, there will be a finalizer added I'm pretty sure for
Copy code
<http://lifecycle.cattle.io/create.project-precan-alert-controller_|lifecycle.cattle.io/create.project-precan-alert-controller_>
✔️ 1
g
Copy code
{
"id": "c-vfm92/p-jwcwp",
"type": "management.cattle.io.project",
"links": {
"remove": "blocked",
"self": "…/v1/management.cattle.io.projects/c-vfm92/p-jwcwp",
"update": "blocked",
"view": "…/apis/management.cattle.io/v3/namespaces/c-vfm92/projects/p-jwcwp"
},
"apiVersion": "<http://management.cattle.io/v3|management.cattle.io/v3>",
"kind": "Project",
"metadata": {
"annotations": {
"<http://authz.management.cattle.io/creator-role-bindings|authz.management.cattle.io/creator-role-bindings>": "{\"created\":[\"project-owner\"],\"required\":[\"project-owner\"]}",
"<http://field.cattle.io/creatorId|field.cattle.io/creatorId>": "user-55flh",
"<http://lifecycle.cattle.io/create.mgmt-project-rbac-remove|lifecycle.cattle.io/create.mgmt-project-rbac-remove>": "true",
"<http://lifecycle.cattle.io/create.project-namespace-auth_c-vfm92|lifecycle.cattle.io/create.project-namespace-auth_c-vfm92>": "true",
"<http://lifecycle.cattle.io/create.project-precan-alert-controller_c-vfm92|lifecycle.cattle.io/create.project-precan-alert-controller_c-vfm92>": "true"
},
"creationTimestamp": "2023-01-10T15:06:46Z",
"deletionGracePeriodSeconds": 0,
"deletionTimestamp": "2023-01-11T08:55:43Z",
"fields": [ 2 items
"p-jwcwp",
"20h"
],
"finalizers": [
"<http://clusterscoped.controller.cattle.io/project-precan-alert-controller_c-vfm92|clusterscoped.controller.cattle.io/project-precan-alert-controller_c-vfm92>"
],
this is a part of the api file for that project
the problem I can’t edit the yaml file and send the request back
l
Not a recommended approach, but you can use kubectl replace --raw
g
after a few investigation I went to see the ns in the local cluster that rancher container uses and it turns out that there is a namespace in terminating state
l
Yeah - so you have to use kubectl edit namespace <your_namespace>
Remove the associated finalizer
Then delete the namespace.
g
finalizers: {}
should be a dict right ?
l
Copy code
"finalizers":[]
You can refer to this issue - https://github.com/rancher/rancher/issues/14715. There are many ways to do it easily.
Lmk how it goes
g
there is no finalizers in metadata for that ns
l
Okay - can you execute kubectl get namespace <YOUR_NAMESPACE> -o json > <YOUR_NAMESPACE>.json?
Whatever your namespace name is
👍 1
g
done
only finalizer in spec not in metadata.
l
Remove the kubernetes from finalizers array which is under spec
Then execute this -
Copy code
kubectl replace --raw "/api/v1/namespaces/<YOUR_NAMESPACE>/finalize" -f ./<YOUR_NAMESPACE>.json
g
done
l
kubectl get namespace
Your namespace should be gone
g
SomeResourcesRemain","message":"Some resources are remaining: <http://projectalertgroups.management.cattle.io|projectalertgroups.management.cattle.io> has 1 resource instances, <http://projectalertrules.management.cattle.io|projectalertrules.management.cattle.io> has 2 resource instances"},{"type":"NamespaceFinalizersRemaining","status":"True","lastTransitionTime":"2023-01-11T08:55:48Z","reason":"SomeFinalizersRemain","message":"Some content in the namespace has finalizers remaining: <http://clusterscoped.controller.cattle.io/pod-target-alert-watcher_c-vfm92|clusterscoped.controller.cattle.io/pod-target-alert-watcher_c-vfm92> in 2 resource instances, <http://clusterscoped.controller.cattle.io/project-alert-group-lifecycle_c-vfm92|clusterscoped.controller.cattle.io/project-alert-group-lifecycle_c-vfm92> in 1 resource instances"
not really it is binded to some stuff that should be resolved first 🤦‍♂️
l
Oh okay, it still has stuff hanging around. That's really surprising
Can you open another terminal and run kubectl proxy?
In that terminal, could you run curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json https://localhost:8001/api/v1/namespaces/YOUR_NAMESPACE/finalize
g
I can’t open a proxy . and some resources are hanging , I can see them using GUI . force deletion not working
l
If that doesn't work, we will need to check if any of the apiservices are unavailable by executing and not serving its resources: • kubectl get apiservice|grep False Once that's done, check the resources associated with the apiservice in your namespace that still exist • kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get -n $your-ns-to-delete Then use kubectl delete • kubectl delete APIService <resource_name>
g
all of them are True 🫠
l
In the local cluster, as well?
g
yep
l
What are the resources that you can see on the GUI then?
g
these on local cluster of the namespace that is in terminating state. I am trying to force the deletion of those resources but still nothing the command just hang
l
Can you execute kubectl delete APIService <name_of_resource>
Is this the command that hangs?
g
• list the resource: 
kubectl get ProjectAlertRules -n p-jwcwp
• delete: 
kubectl delete ProjectAlertRules less-than-half-workload-available --force --grace-period=0 -n p-jwcwp
l
• kubectl get apiservice | grep False -n p-jwcwp
Can you execute this?
g
nothing
l
kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get -n p-jwcwp
g
No resources found in p-jwcwp namespace
l
Can you execute kubectl get namespace?
g
i have a list
l
of?
g
all namespaces and the ns in terminating state still persists
but I guess I found teh solution
one moment
I patched the hanging resources using kubectl and the ns is gone
now if I want to delete the project that is hanging I get this : 
[ERROR] Error during subscribe websocket: close sent
l
Could be because your CPU/Ram is overloaded
Could you logout and login and retry?
If that doesn't work, please try restarting the node.
g
I have no time now . but if you are working with rancher community there are some other issue • minimize terminal on rancher GUI and reopen it agin -> loosing session • installing an application does not show related helm errors like previous version 2.5.15 • Gui is slow • Extension not showing installed once
460 Views
Open in Slack
PreviousNext
Powered by