gorgeous-cat-50570
01/11/2023, 10:07 AMWaiting on project-precan-alert-controller_c-vfm92
Inspecting container logs :
[ERROR] error syncing 'c-vfm92/creator-cluster-owner': handler mgmt-auth-crtb-controller: couldn't create role cluster-owner: <http://roles.rbac.authorization.k8s.io|roles.rbac.authorization.k8s.io> "cluster-owner" is forbidden: unable to create new content in namespace p-jwcwp because it is being terminated, requeuing
[ERROR] error syncing 'c-vfm92/u-mxcjeyd7a6-admin': handler mgmt-auth-crtb-controller: couldn't create role cluster-owner: <http://roles.rbac.authorization.k8s.io|roles.rbac.authorization.k8s.io> "cluster-owner" is forbidden: unable to create new content in namespace p-jwcwp because it is being terminated, requeuing
[ERROR] error syncing 'c-vfm92/c-vfm92-fleet-default-owner': handler mgmt-auth-crtb-controller: couldn't create role cluster-owner: <http://roles.rbac.authorization.k8s.io|roles.rbac.authorization.k8s.io> "cluster-owner" is forbidden: unable to create new content in namespace p-jwcwp because it is being terminated, requeuing
limited-pizza-33551
01/11/2023, 10:39 AMgorgeous-cat-50570
01/11/2023, 10:42 AMNAME="Ubuntu"
VERSION_ID="22.04"
Rancher container image = v2.7.0
I can login / logout as cluster-owner .
I can create other projects / delete them and so onlimited-pizza-33551
01/11/2023, 10:51 AMgorgeous-cat-50570
01/11/2023, 10:52 AMlimited-pizza-33551
01/11/2023, 10:54 AMgorgeous-cat-50570
01/11/2023, 10:55 AMlimited-pizza-33551
01/11/2023, 10:56 AMgorgeous-cat-50570
01/11/2023, 10:57 AMStatus code was 403 and not [201]: HTTP Error 403: Forbidden
[mgmt-auth-crtb-controller] Creating role cluster-owner in namespace p-jwcwp
limited-pizza-33551
01/11/2023, 11:01 AMgorgeous-cat-50570
01/11/2023, 11:03 AMlimited-pizza-33551
01/11/2023, 11:04 AMkubectl get namespace <YOUR_NAMESPACE> -o json > <YOUR_NAMESPACE>.json
-- Can you try executing this command and see if anything shows up in the json file? Replace YOUR_NAMESPACE with the namespace name, please.gorgeous-cat-50570
01/11/2023, 11:08 AMlimited-pizza-33551
01/11/2023, 11:09 AMgorgeous-cat-50570
01/11/2023, 11:11 AMerror from server (NotFound): namespaces "xxxx" not found
limited-pizza-33551
01/11/2023, 11:13 AMgorgeous-cat-50570
01/11/2023, 11:13 AMp-jwcwp
but in rancher logs I get the error I posted abovelimited-pizza-33551
01/11/2023, 11:23 AMgorgeous-cat-50570
01/11/2023, 11:25 AMerror: the server doesn't have a resource type "projects"
limited-pizza-33551
01/11/2023, 11:29 AMgorgeous-cat-50570
01/11/2023, 11:31 AMlimited-pizza-33551
01/11/2023, 11:34 AMgorgeous-cat-50570
01/11/2023, 11:35 AMlimited-pizza-33551
01/11/2023, 11:37 AM<http://lifecycle.cattle.io/create.project-precan-alert-controller_|lifecycle.cattle.io/create.project-precan-alert-controller_>
gorgeous-cat-50570
01/11/2023, 11:37 AM{
"id": "c-vfm92/p-jwcwp",
"type": "management.cattle.io.project",
"links": {
"remove": "blocked",
"self": "…/v1/management.cattle.io.projects/c-vfm92/p-jwcwp",
"update": "blocked",
"view": "…/apis/management.cattle.io/v3/namespaces/c-vfm92/projects/p-jwcwp"
},
"apiVersion": "<http://management.cattle.io/v3|management.cattle.io/v3>",
"kind": "Project",
"metadata": {
"annotations": {
"<http://authz.management.cattle.io/creator-role-bindings|authz.management.cattle.io/creator-role-bindings>": "{\"created\":[\"project-owner\"],\"required\":[\"project-owner\"]}",
"<http://field.cattle.io/creatorId|field.cattle.io/creatorId>": "user-55flh",
"<http://lifecycle.cattle.io/create.mgmt-project-rbac-remove|lifecycle.cattle.io/create.mgmt-project-rbac-remove>": "true",
"<http://lifecycle.cattle.io/create.project-namespace-auth_c-vfm92|lifecycle.cattle.io/create.project-namespace-auth_c-vfm92>": "true",
"<http://lifecycle.cattle.io/create.project-precan-alert-controller_c-vfm92|lifecycle.cattle.io/create.project-precan-alert-controller_c-vfm92>": "true"
},
"creationTimestamp": "2023-01-10T15:06:46Z",
"deletionGracePeriodSeconds": 0,
"deletionTimestamp": "2023-01-11T08:55:43Z",
"fields": [ 2 items
"p-jwcwp",
"20h"
],
"finalizers": [
"<http://clusterscoped.controller.cattle.io/project-precan-alert-controller_c-vfm92|clusterscoped.controller.cattle.io/project-precan-alert-controller_c-vfm92>"
],
this is a part of the api file for that projectlimited-pizza-33551
01/11/2023, 11:46 AMgorgeous-cat-50570
01/11/2023, 11:47 AMlimited-pizza-33551
01/11/2023, 11:48 AMgorgeous-cat-50570
01/11/2023, 11:49 AMfinalizers: {}
should be a dict right ?limited-pizza-33551
01/11/2023, 11:50 AM"finalizers":[]
gorgeous-cat-50570
01/11/2023, 11:55 AMlimited-pizza-33551
01/11/2023, 11:57 AMgorgeous-cat-50570
01/11/2023, 11:58 AMlimited-pizza-33551
01/11/2023, 11:59 AMkubectl replace --raw "/api/v1/namespaces/<YOUR_NAMESPACE>/finalize" -f ./<YOUR_NAMESPACE>.json
gorgeous-cat-50570
01/11/2023, 12:00 PMlimited-pizza-33551
01/11/2023, 12:00 PMgorgeous-cat-50570
01/11/2023, 12:01 PMSomeResourcesRemain","message":"Some resources are remaining: <http://projectalertgroups.management.cattle.io|projectalertgroups.management.cattle.io> has 1 resource instances, <http://projectalertrules.management.cattle.io|projectalertrules.management.cattle.io> has 2 resource instances"},{"type":"NamespaceFinalizersRemaining","status":"True","lastTransitionTime":"2023-01-11T08:55:48Z","reason":"SomeFinalizersRemain","message":"Some content in the namespace has finalizers remaining: <http://clusterscoped.controller.cattle.io/pod-target-alert-watcher_c-vfm92|clusterscoped.controller.cattle.io/pod-target-alert-watcher_c-vfm92> in 2 resource instances, <http://clusterscoped.controller.cattle.io/project-alert-group-lifecycle_c-vfm92|clusterscoped.controller.cattle.io/project-alert-group-lifecycle_c-vfm92> in 1 resource instances"
limited-pizza-33551
01/11/2023, 12:02 PMgorgeous-cat-50570
01/11/2023, 12:11 PMlimited-pizza-33551
01/11/2023, 12:13 PMgorgeous-cat-50570
01/11/2023, 12:16 PMlimited-pizza-33551
01/11/2023, 12:17 PMgorgeous-cat-50570
01/11/2023, 12:17 PMlimited-pizza-33551
01/11/2023, 12:18 PMgorgeous-cat-50570
01/11/2023, 12:21 PMkubectl get ProjectAlertRules -n p-jwcwp
• delete: kubectl delete ProjectAlertRules less-than-half-workload-available --force --grace-period=0 -n p-jwcwp
limited-pizza-33551
01/11/2023, 12:23 PMgorgeous-cat-50570
01/11/2023, 12:24 PMlimited-pizza-33551
01/11/2023, 12:26 PMgorgeous-cat-50570
01/11/2023, 12:26 PMNo resources found in p-jwcwp namespace
limited-pizza-33551
01/11/2023, 12:27 PMgorgeous-cat-50570
01/11/2023, 12:29 PMlimited-pizza-33551
01/11/2023, 12:31 PMgorgeous-cat-50570
01/11/2023, 12:34 PM[ERROR] Error during subscribe websocket: close sent
limited-pizza-33551
01/11/2023, 12:54 PMgorgeous-cat-50570
01/11/2023, 1:06 PM