breezy-autumn-81048
03/16/2023, 11:44 AMTrace[1068908304]: [30.003276269s] [30.003276269s] END
E0314 15:02:02.236947 1 reflector.go:140] <http://k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169|k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169>: Failed to watch *v1.Secret: failed to list *v1.Secret: Get "<https://10.43.0.1:443/api/v1/namespaces/cert-manager/secrets?fieldSelector=metadata.name%3Dcert-manager-webhook-ca&resourceVersion=360915>": dial tcp 10.43.0.1:443: i/o timeout
W0314 15:03:28.953687 1 reflector.go:424] <http://k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169|k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169>: failed to list *v1.Secret: Get "<https://10.43.0.1:443/api/v1/namespaces/cert-manager/secrets?fieldSelector=metadata.name%3Dcert-manager-webhook-ca&resourceVersion=360915>": dial tcp 10.43.0.1:443: i/o timeout
I0314 15:03:28.953816 1 trace.go:219] Trace[516939538]: "Reflector ListAndWatch" name:<http://k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169|k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169> (14-Mar-2023 15:02:58.949) (total time: 30004ms):
Trace[516939538]: ---"Objects listed" error:Get "<https://10.43.0.1:443/api/v1/namespaces/cert-manager/secrets?fieldSelector=metadata.name%3Dcert-manager-webhook-ca&resourceVersion=360915>": dial tcp 10.43.0.1:443: i/o timeout 30004ms (15:03:28.953)
Trace[516939538]: [30.004226263s] [30.004226263s] END
E0314 15:03:28.953837 1 reflector.go:140] <http://k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169|k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169>: Failed to watch *v1.Secret: failed to list *v1.Secret: Get "<https://10.43.0.1:443/api/v1/namespaces/cert-manager/secrets?fieldSelector=metadata.name%3Dcert-manager-webhook-ca&resourceVersion=360915>": dial tcp 10.43.0.1:443: i/o timeout
W0314 15:04:44.919380 1 reflector.go:424] <http://k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169|k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169>: failed to list *v1.Secret: Get "<https://10.43.0.1:443/api/v1/namespaces/cert-manager/secrets?fieldSelector=metadata.name%3Dcert-manager-webhook-ca&resourceVersion=360915>": dial tcp 10.43.0.1:443: i/o timeout
I0314 15:04:44.919458 1 trace.go:219] Trace[430405071]: "Reflector ListAndWatch" name:<http://k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169|k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169> (14-Mar-2023 15:04:14.918) (total time: 30000ms):
Trace[430405071]: ---"Objects listed" error:Get "<https://10.43.0.1:443/api/v1/namespaces/cert-manager/secrets?fieldSelector=metadata.name%3Dcert-manager-webhook-ca&resourceVersion=360915>": dial tcp 10.43.0.1:443: i/o timeout 30000ms (15:04:44.919)
Trace[430405071]: [30.000964846s] [30.000964846s] END
E0314 15:04:44.919472 1 reflector.go:140] <http://k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169|k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169>: Failed to watch *v1.Secret: failed to list *v1.Secret: Get "<https://10.43.0.1:443/api/v1/namespaces/cert-manager/secrets?fieldSelector=metadata.name%3Dcert-manager-webhook-ca&resourceVersion=360915>": dial tcp 10.43.0.1:443: i/o timeout
Can someone explain what's wrong? It feels that I can't fully install a helm chart because of this issue. ( I noticed this issue when was trying to install a helm chart of actions-runner-controller, and the error I got: Error: Internal error occurred: failed calling webhook "<http://webhook.cert-manager.io|webhook.cert-manager.io>": failed to call webhook: Post "<https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s>": context deadline exceeded )
As well, here are some logs from the pod of actions-runner-controller:
Warning FailedMount 17m kubelet Unable to attach or mount volumes: unmounted volumes=[cert], unattached volumes=[kube-api-access-v48zj secret tmp cert]: timed out waiting for the condition
Warning FailedMount 8m32s kubelet Unable to attach or mount volumes: unmounted volumes=[cert], unattached volumes=[tmp cert kube-api-access-v48zj secret]: timed out waiting for the condition
Warning FailedMount 6m18s (x5 over 19m) kubelet Unable to attach or mount volumes: unmounted volumes=[cert], unattached volumes=[secret tmp cert kube-api-access-v48zj]: timed out waiting for the condition
Warning FailedMount 103s (x2 over 4m1s) kubelet Unable to attach or mount volumes: unmounted volumes=[cert], unattached volumes=[cert kube-api-access-v48zj secret tmp]: timed out waiting for the condition
Warning FailedMount 86s (x18 over 21m) kubelet MountVolume.SetUp failed for volume "cert" : secret "actions-runner-controller-serving-cert" not found
Thanks in advance,kubectl .kube\config.yaml apply -f test-resources.yaml
namespace/cert-manager-test unchanged
Error from server (InternalError): error when creating "test-resources.yaml": Internal error occurred: failed calling webhook "<http://webhook.cert-manager.io|webhook.cert-manager.io>": failed to call webhook: Post "<https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s>": context deadline exceeded
Error from server (InternalError): error when creating "test-resources.yaml": Internal error occurred: failed calling webhook "<http://webhook.cert-manager.io|webhook.cert-manager.io>": failed to call webhook: Post "<https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s>": context deadline exceeded
So, it's not an issue with actions-runner-controller, but with the cert-manager or K3S deployed using Rancher.
Could someone recommend how to fix this issue?E0315 19:51:36.923104 1 cluster.go:161] cert-manager "msg"="Failed to get API Group-Resources" "error"="Get \"<https://10.43.0.1:443/api?timeout=32s>\": dial tcp 10.43.0.1:443: i/o timeout"
Error: error creating manager: Get "<https://10.43.0.1:443/api?timeout=32s>": dial tcp 10.43.0.1:443: i/o timeout
Usage:
ca-injector [flags]
Flags:
--add_dir_header If true, adds the file directory to the header of the log messages
--alsologtostderr log to standard error as well as files (no effect when -logtostderr=true)
--enable-profiling Enable profiling for cainjector
--feature-gates mapStringBool A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
AllAlpha=true|false (ALPHA - default=false)
AllBeta=true|false (BETA - default=false)
-h, --help help for ca-injector
--kubeconfig string Paths to a kubeconfig. Only required if out-of-cluster.
--leader-elect If true, cainjector will perform leader election between instances to ensure no more than one instance of cainjector operates at a time (default true)
--leader-election-lease-duration duration The duration that non-leader candidates will wait after observing a leadership renewal until attempting to acquire leadership of a led but unrenewed leader slot. This is effectively the maximum duration that a leader can be stopped before it is replaced by another candidate. This is only applicable if leader election is enabled. (default 1m0s)
--leader-election-namespace string Namespace used to perform leader election. Only used if leader election is enabled (default "kube-system")
--leader-election-renew-deadline duration The interval between attempts by the acting master to renew a leadership slot before it stops leading. This must be less than or equal to the lease duration. This is only applicable if leader election is enabled. (default 40s)
--leader-election-retry-period duration The duration the clients should wait between attempting acquisition and renewal of a leadership. This is only applicable if leader election is enabled. (default 15s)
--log-flush-frequency duration Maximum number of seconds between log flushes (default 5s)
--log_backtrace_at traceLocation when logging hits line file:N, emit a stack trace (default :0)
--log_dir string If non-empty, write log files in this directory (no effect when -logtostderr=true)
--log_file string If non-empty, use this log file (no effect when -logtostderr=true)
--log_file_max_size uint Defines the maximum size a log file can grow to (no effect when -logtostderr=true). Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
--logtostderr log to standard error instead of files (default true)
--namespace string If set, this limits the scope of cainjector to a single namespace. If set, cainjector will not update resources with certificates outside of the configured namespace.
--one_output If true, only write logs to their native severity level (vs also writing to each lower severity level; no effect when -logtostderr=true)
--profiler-address string Address of the Go profiler (pprof) if enabled. This should never be exposed on a public interface. (default "localhost:6060")
--skip_headers If true, avoid header prefixes in the log messages
--skip_log_headers If true, avoid headers when opening log files (no effect when -logtostderr=true)
--stderrthreshold severity logs at or above this threshold go to stderr when writing to files and stderr (no effect when -logtostderr=true or -alsologtostderr=false) (default 2)
-v, --v Level number for the log level verbosity (default 0)
--vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging
error creating manager: Get "<https://10.43.0.1:443/api?timeout=32s>": dial tcp 10.43.0.1:443: i/o timeout
rough-farmer-49135
03/16/2023, 1:04 PMbreezy-autumn-81048
03/16/2023, 1:38 PM* Added cert-manager-webhook.cert-manager.svc:10250:127.0.0.1 to DNS cache
* Trying 10.154.146.178:443...
* Connected to <http://my.host.net|my.host.net> (10.154.146.178) port 443 (#0)
* schannel: disabled automatic use of client certificate
* ALPN: offers http/1.1
* ALPN: server accepted http/1.1
* Server auth using Basic with user 'kubeconfig-user-9qc1111111'
> POST /k8s/clusters/c-m-qjr7kbg9/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy/validate HTTP/1.1
> Host: <http://my.host.net|my.host.net>
> Authorization: Basic gaewfawefwefwefdsfeturyurrtrte4534th4572r23fwef
> User-Agent: curl/7.83.1
> Accept: */*
> Content-Type: application/json
> Content-Length: 517
>* Mark bundle as not supporting multiuse
< HTTP/1.1 503 Service Unavailable
< Audit-Id: c5b1d7a6-c4e5-444f-a067-372e1e06f524
< Cache-Control: no-cache, no-store, must-revalidate
< Cache-Control: no-cache, private
< Content-Length: 276
< Content-Type: application/json
< Date: Thu, 16 Mar 2023 13:04:03 GMT
< X-Api-Cattle-Auth: true
< X-Content-Type-Options: nosniff
<{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "error trying to reach service: proxy error from 127.0.0.1:6443 while dialing 10.42.7.12:53, code 503: 503 Service Unavailable",
"reason": "ServiceUnavailable",
"code": 503
}*
10.42.7.12 - codedns pod
So, it means that something is wrong with network?rough-farmer-49135
03/16/2023, 1:43 PMkubectl get pods -A -o wide
will tell you what node a given pod is running on.breezy-autumn-81048
03/16/2023, 1:43 PMI0316 13:37:46.929570 1 trace.go:205] Trace[942292090]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167 (16-Mar-2023 13:37:16.928) (total time: 30000ms):
Trace[942292090]: ---"Objects listed" error:Get "<https://10.43.0.1:443/api/v1/namespaces?resourceVersion=6721552>": dial tcp 10.43.0.1:443: i/o timeout 30000ms (13:37:46.929)
Trace[942292090]: [30.000559454s] [30.000559454s] END
E0316 13:37:46.929584 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: Failed to watch *v1.Namespace: failed to list *v1.Namespace: Get "<https://10.43.0.1:443/api/v1/namespaces?resourceVersion=6721552>": dial tcp 10.43.0.1:443: i/o timeout
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
W0316 13:38:21.128007 1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: failed to list *v1.EndpointSlice: Get "<https://10.43.0.1:443/apis/discovery.k8s.io/v1/endpointslices?resourceVersion=6723908>": dial tcp 10.43.0.1:443: i/o timeout
I0316 13:38:21.128083 1 trace.go:205] Trace[1677112065]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167 (16-Mar-2023 13:37:51.127) (total time: 30000ms):
Trace[1677112065]: ---"Objects listed" error:Get "<https://10.43.0.1:443/apis/discovery.k8s.io/v1/endpointslices?resourceVersion=6723908>": dial tcp 10.43.0.1:443: i/o timeout 30000ms (13:38:21.127)
Trace[1677112065]: [30.000941767s] [30.000941767s] END
E0316 13:38:21.128102 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Get "<https://10.43.0.1:443/apis/discovery.k8s.io/v1/endpointslices?resourceVersion=6723908>": dial tcp 10.43.0.1:443: i/o timeout
W0316 13:38:30.196262 1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: failed to list *v1.Service: Get "<https://10.43.0.1:443/api/v1/services?resourceVersion=6723076>": dial tcp 10.43.0.1:443: i/o timeout
I0316 13:38:30.196330 1 trace.go:205] Trace[518327474]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167 (16-Mar-2023 13:38:00.195) (total time: 30000ms):
Trace[518327474]: ---"Objects listed" error:Get "<https://10.43.0.1:443/api/v1/services?resourceVersion=6723076>": dial tcp 10.43.0.1:443: i/o timeout 30000ms (13:38:30.196)
Trace[518327474]: [30.000532191s] [30.000532191s] END
E0316 13:38:30.196344 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: Failed to watch *v1.Service: failed to list *v1.Service: Get "<https://10.43.0.1:443/api/v1/services?resourceVersion=6723076>": dial tcp 10.43.0.1:443: i/o timeout
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
W0316 13:39:03.327155 1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: failed to list *v1.Namespace: Get "<https://10.43.0.1:443/api/v1/namespaces?resourceVersion=6721552>": dial tcp 10.43.0.1:443: i/o timeout
I0316 13:39:03.327238 1 trace.go:205] Trace[472849857]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167 (16-Mar-2023 13:38:33.326) (total time: 30000ms):
Trace[472849857]: ---"Objects listed" error:Get "<https://10.43.0.1:443/api/v1/namespaces?resourceVersion=6721552>": dial tcp 10.43.0.1:443: i/o timeout 30000ms (13:39:03.327)
Trace[472849857]: [30.000579133s] [30.000579133s] END
E0316 13:39:03.327258 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: Failed to watch *v1.Namespace: failed to list *v1.Namespace: Get "<https://10.43.0.1:443/api/v1/namespaces?resourceVersion=6721552>": dial tcp 10.43.0.1:443: i/o timeout
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
W0316 13:39:25.231933 1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: failed to list *v1.EndpointSlice: Get "<https://10.43.0.1:443/apis/discovery.k8s.io/v1/endpointslices?resourceVersion=6723908>": dial tcp 10.43.0.1:443: i/o timeout
I0316 13:39:25.232044 1 trace.go:205] Trace[677905320]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167 (16-Mar-2023 13:38:55.230) (total time: 30001ms):
Trace[677905320]: ---"Objects listed" error:Get "<https://10.43.0.1:443/apis/discovery.k8s.io/v1/endpointslices?resourceVersion=6723908>": dial tcp 10.43.0.1:443: i/o timeout 30000ms (13:39:25.231)
Trace[677905320]: [30.001014748s] [30.001014748s] END
E0316 13:39:25.232059 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Get "<https://10.43.0.1:443/apis/discovery.k8s.io/v1/endpointslices?resourceVersion=6723908>": dial tcp 10.43.0.1:443: i/o timeout
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
W0316 13:39:59.853741 1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: failed to list *v1.Service: Get "<https://10.43.0.1:443/api/v1/services?resourceVersion=6723076>": dial tcp 10.43.0.1:443: i/o timeout
I0316 13:39:59.853855 1 trace.go:205] Trace[1629891480]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167 (16-Mar-2023 13:39:29.852) (total time: 30000ms):
Trace[1629891480]: ---"Objects listed" error:Get "<https://10.43.0.1:443/api/v1/services?resourceVersion=6723076>": dial tcp 10.43.0.1:443: i/o timeout 30000ms (13:39:59.853)
Trace[1629891480]: [30.000976129s] [30.000976129s] END
E0316 13:39:59.853872 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: Failed to watch *v1.Service: failed to list *v1.Service: Get "<https://10.43.0.1:443/api/v1/services?resourceVersion=6723076>": dial tcp 10.43.0.1:443: i/o timeout
W0316 13:40:17.731262 1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: failed to list *v1.Namespace: Get "<https://10.43.0.1:443/api/v1/namespaces?resourceVersion=6721552>": dial tcp 10.43.0.1:443: i/o timeout
I0316 13:40:17.731324 1 trace.go:205] Trace[1242097369]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167 (16-Mar-2023 13:39:47.730) (total time: 30001ms):
Trace[1242097369]: ---"Objects listed" error:Get "<https://10.43.0.1:443/api/v1/namespaces?resourceVersion=6721552>": dial tcp 10.43.0.1:443: i/o timeout 30001ms (13:40:17.731)
Trace[1242097369]: [30.001195763s] [30.001195763s] END
E0316 13:40:17.731341 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: Failed to watch *v1.Namespace: failed to list *v1.Namespace: Get "<https://10.43.0.1:443/api/v1/namespaces?resourceVersion=6721552>": dial tcp 10.43.0.1:443: i/o timeout
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
W0316 13:40:34.564570 1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: failed to list *v1.EndpointSlice: Get "<https://10.43.0.1:443/apis/discovery.k8s.io/v1/endpointslices?resourceVersion=6723908>": dial tcp 10.43.0.1:443: i/o timeout
I0316 13:40:34.564659 1 trace.go:205] Trace[1746491914]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167 (16-Mar-2023 13:40:04.563) (total time: 30001ms):
Trace[1746491914]: ---"Objects listed" error:Get "<https://10.43.0.1:443/apis/discovery.k8s.io/v1/endpointslices?resourceVersion=6723908>": dial tcp 10.43.0.1:443: i/o timeout 30001ms (13:40:34.564)
Trace[1746491914]: [30.001397329s] [30.001397329s] END
E0316 13:40:34.564671 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Get "<https://10.43.0.1:443/apis/discovery.k8s.io/v1/endpointslices?resourceVersion=6723908>": dial tcp 10.43.0.1:443: i/o timeout
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
W0316 13:41:04.424294 1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: failed to list *v1.Service: Get "<https://10.43.0.1:443/api/v1/services?resourceVersion=6723076>": dial tcp 10.43.0.1:443: i/o timeout
I0316 13:41:04.424391 1 trace.go:205] Trace[953581850]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167 (16-Mar-2023 13:40:34.423) (total time: 30000ms):
Trace[953581850]: ---"Objects listed" error:Get "<https://10.43.0.1:443/api/v1/services?resourceVersion=6723076>": dial tcp 10.43.0.1:443: i/o timeout 30000ms (13:41:04.424)
Trace[953581850]: [30.000987325s] [30.000987325s] END
E0316 13:41:04.424405 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: Failed to watch *v1.Service: failed to list *v1.Service: Get "<https://10.43.0.1:443/api/v1/services?resourceVersion=6723076>": dial tcp 10.43.0.1:443: i/o timeout
rough-farmer-49135
03/16/2023, 1:49 PMbreezy-autumn-81048
03/16/2023, 1:52 PMrough-farmer-49135
03/16/2023, 1:53 PMjournalctl -u k3s-server
(I presume, I've done more RKE2, so I might be off there).breezy-autumn-81048
03/16/2023, 1:56 PM