able-wall-25846
11/29/2022, 5:22 PMroot@knode03:~# crictl pull <http://registry01.sys.nc4.iad0.nsscloud.net:5000/ns-ubuntu-20-04-qemu:latest|registry01.sys.nc4.iad0.nsscloud.net:5000/ns-ubuntu-20-04-qemu:latest>
FATA[0000] pulling image: rpc error: code = Unknown desc = failed to pull and unpack image "<http://registry01.sys.nc4.iad0.nsscloud.net:5000/ns-ubuntu-20-04-qemu:latest|registry01.sys.nc4.iad0.nsscloud.net:5000/ns-ubuntu-20-04-qemu:latest>": failed to resolve reference "<http://registry01.sys.nc4.iad0.nsscloud.net:5000/ns-ubuntu-20-04-qemu:latest|registry01.sys.nc4.iad0.nsscloud.net:5000/ns-ubuntu-20-04-qemu:latest>": failed to do request: Head "<https://registry01.sys.nc4.iad0.nsscloud.net:5000/v2/ns-ubuntu-20-04-qemu/manifests/latest>": http: server gave HTTP response to HTTPS client
able-wall-25846
11/29/2022, 5:23 PMable-wall-25846
11/29/2022, 5:25 PM/var/lib/rancher/rke2/agent/etc/containerd/config.toml
able-wall-25846
11/29/2022, 5:25 PMable-wall-25846
11/29/2022, 5:25 PM[plugins.cri.registry.mirrors."<http://registry01.sys.nc4.iad0.nsscloud.net|registry01.sys.nc4.iad0.nsscloud.net>"]
endpoint = ["<http://registry01.sys.nc4.iad0.nsscloud.net:5000>"]
able-wall-25846
11/29/2022, 5:25 PMflaky-shampoo-86024
11/29/2022, 6:33 PMgentle-petabyte-40055
11/30/2022, 5:22 AMbest-address-42882
11/30/2022, 5:34 AMsalmon-chef-55025
11/30/2022, 9:18 AM"unable to decode an event from the watch stream: backed up reader") has prevented the request from succeeding
[main] 2022/11/23 14:38:18 Starting Tiller v2.16.5-rancher1 (tls=false)
[main] 2022/11/23 14:38:18 GRPC listening on :61000
[main] 2022/11/23 14:38:18 Probes listening on :50491
[main] 2022/11/23 14:38:18 Storage driver is ConfigMap
[main] 2022/11/23 14:38:18 Max history per release is 10
[tiller] 2022/11/23 14:38:19 getting history for release ne-campaign
[storage] 2022/11/23 14:38:19 getting release history for "ne-campaign"
[storage/driver] 2022/11/23 14:38:19 query: failed to query with labels: Get <https://localhost:443/k8s/clusters/c-xzmr2/api/v1/namespaces/ne-campaign/configmaps?labelSelector=NAME%3Dne-campaign%2COWNER%3DTILLER>: dial tcp 127.0.0.1:443: connect: cannot assign requested address
[tiller] 2022/11/23 14:38:19 preparing update for ne-campaign
[storage] 2022/11/23 14:38:19 getting deployed releases from "ne-campaign" history
[storage/driver] 2022/11/23 14:38:19 query: failed to query with labels: Get <https://localhost:443/k8s/clusters/c-xzmr2/api/v1/namespaces/ne-campaign/configmaps?labelSelector=NAME%3Dne-campaign%2COWNER%3DTILLER%2CSTATUS%3DDEPLOYED>: dial tcp 127.0.0.1:443: connect: cannot assign requested address
[tiller] 2022/11/23 14:38:19 failed to prepare update: Get <https://localhost:443/k8s/clusters/c-xzmr2/api/v1/namespaces/ne-campaign/configmaps?labelSelector=NAME%3Dne-campaign%2COWNER%3DTILLER%2CSTATUS%3DDEPLOYED>: dial tcp 127.0.0.1:443: connect: cannot assign requested address
[tiller] 2022/11/23 14:38:19 performing force update for ne-campaign
[storage] 2022/11/23 14:38:19 getting last revision of "ne-campaign"
[storage] 2022/11/23 14:38:19 getting release history for "ne-campaign"
[storage/driver] 2022/11/23 14:38:19 query: failed to query with labels: Get <https://localhost:443/k8s/clusters/c-xzmr2/api/v1/namespaces/ne-campaign/configmaps?labelSelector=NAME%3Dne-campaign%2COWNER%3DTILLER>: dial tcp 127.0.0.1:443: connect: cannot assign requested address
[main] 2022/11/23 14:38:19 Starting Tiller v2.16.5-rancher1 (tls=false)
[main] 2022/11/23 14:38:19 GRPC listening on :50491
[main] 2022/11/23 14:38:19 Probes listening on :50491
[main] 2022/11/23 14:38:19 Storage driver is ConfigMap
[main] 2022/11/23 14:38:19 Max history per release is 10
[main] 2022/11/23 14:38:19 Probes server died: listen tcp :50491: bind: address already in use
[main] 2022/11/23 14:48:20 Starting Tiller v2.16.5-rancher1 (tls=false)
[main] 2022/11/23 14:48:20 GRPC listening on :50491
[main] 2022/11/23 14:48:20 Probes listening on :61000
[main] 2022/11/23 14:48:20 Storage driver is ConfigMap
[main] 2022/11/23 14:48:20 Max history per release is 10
[tiller] 2022/11/23 14:48:21 getting history for release ne-campaign
[storage] 2022/11/23 14:48:21 getting release history for "ne-campaign"
[storage/driver] 2022/11/23 14:48:21 query: failed to query with labels: Get <https://localhost:443/k8s/clusters/c-xzmr2/api/v1/namespaces/ne-campaign/configmaps?labelSelector=NAME%3Dne-campaign%2COWNER%3DTILLER>: dial tcp 127.0.0.1:443: connect: cannot assign requested address
[tiller] 2022/11/23 14:48:21 preparing update for ne-campaign
[storage] 2022/11/23 14:48:21 getting deployed releases from "ne-campaign" history
[storage/driver] 2022/11/23 14:48:21 query: failed to query with labels: Get <https://localhost:443/k8s/clusters/c-xzmr2/api/v1/namespaces/ne-campaign/configmaps?labelSelector=NAME%3Dne-campaign%2COWNER%3DTILLER%2CSTATUS%3DDEPLOYED>: dial tcp 127.0.0.1:443: connect: cannot assign requested address
[tiller] 2022/11/23 14:48:21 failed to prepare update: Get <https://localhost:443/k8s/clusters/c-xzmr2/api/v1/namespaces/ne-campaign/configmaps?labelSelector=NAME%3Dne-campaign%2COWNER%3DTILLER%2CSTATUS%3DDEPLOYED>: dial tcp 127.0.0.1:443: connect: cannot assign requested address
[tiller] 2022/11/23 14:48:21 performing force update for ne-campaign
[storage] 2022/11/23 14:48:21 getting last revision of "ne-campaign"
[storage] 2022/11/23 14:48:21 getting release history for "ne-campaign"
[storage/driver] 2022/11/23 14:48:21 query: failed to query with labels: Get <https://localhost:443/k8s/clusters/c-xzmr2/api/v1/namespaces/ne-campaign/configmaps?labelSelector=NAME%3Dne-campaign%2COWNER%3DTILLER>: dial tcp 127.0.0.1:443: connect: cannot assign requested address
[main] 2022/11/23 14:48:22 Starting Tiller v2.16.5-rancher1 (tls=false)
[main] 2022/11/23 14:48:22 GRPC listening on :50491
[main] 2022/11/23 14:48:22 Probes listening on :61000
[main] 2022/11/23 14:48:22 Storage driver is ConfigMap
[main] 2022/11/23 14:48:22 Max history per release is 10
[tiller] 2022/11/23 14:48:23 getting history for release ne-campaign
[storage] 2022/11/23 14:48:23 getting release history for "ne-campaign"
[storage/driver] 2022/11/23 14:48:23 query: failed to query with labels: Get <https://localhost:443/k8s/clusters/c-xzmr2/api/v1/namespaces/ne-campaign/configmaps?labelSelector=NAME%3Dne-campaign%2COWNER%3DTILLER>: dial tcp 127.0.0.1:443: connect: cannot assign requested address
[tiller] 2022/11/23 14:48:23 preparing update for ne-campaign
[storage] 2022/11/23 14:48:23 getting deployed releases from "ne-campaign" history
[storage/driver] 2022/11/23 14:48:23 query: failed to query with labels: Get <https://localhost:443/k8s/clusters/c-xzmr2/api/v1/namespaces/ne-campaign/configmaps?labelSelector=NAME%3Dne-campaign%2COWNER%3DTILLER%2CSTATUS%3DDEPLOYED>: dial tcp 127.0.0.1:443: connect: cannot assign requested address
[tiller] 2022/11/23 14:48:23 failed to prepare update: Get <https://localhost:443/k8s/clusters/c-xzmr2/api/v1/namespaces/ne-campaign/configmaps?labelSelector=NAME%3Dne-campaign%2COWNER%3DTILLER%2CSTATUS%3DDEPLOYED>: dial tcp 127.0.0.1:443: connect: cannot assign requested address
[tiller] 2022/11/23 14:48:23 performing force update for ne-campaign
[storage] 2022/11/23 14:48:23 getting last revision of "ne-campaign"
[storage] 2022/11/23 14:48:23 getting release history for "ne-campaign"
[storage/driver] 2022/11/23 14:48:23 query: failed to query with labels: Get <https://localhost:443/k8s/clusters/c-xzmr2/api/v1/namespaces/ne-campaign/configmaps?labelSelector=NAME%3Dne-campaign%2COWNER%3DTILLER>: dial tcp 127.0.0.1:443: connect: cannot assign requested address
We have tried restarting the master nodes (& docker service ) on downstream Kubernetes nodes .. but same has not helped.
We have tried deploying latest code on other micro services hosted on the same cluster but that is also failing.
This error seems to have worsened now .. PFB the recent logs where its getting blocked at even earlier stage.
2022/11/28 14:40:29 [ERROR] Error certs=InvalidBodyContent 422: failed to find cert that matched private key parsing cert template-repository-prod-tls. Will not display correctly in UI
2022/11/28 14:43:12 [INFO] Installing chart using helm version: rancher-helm
[main] 2022/11/28 14:43:12 Starting Tiller v2.16.5-rancher1 (tls=false)
[main] 2022/11/28 14:43:12 GRPC listening on :61000
[main] 2022/11/28 14:43:12 Probes listening on :61000
[main] 2022/11/28 14:43:12 Storage driver is ConfigMap
[main] 2022/11/28 14:43:12 Max history per release is 10
[main] 2022/11/28 14:43:12 Probes server died: listen tcp :61000: bind: address already in use
UPGRADE FAILED
Error: context deadline exceeded
2022/11/28 14:53:12 [ERROR] AppController p-746c7/ecomm-support [helm-controller] failed with : failed to install app ecomm-support. Error: UPGRADE FAILED: context deadline exceeded
billowy-vegetable-58311
11/30/2022, 10:21 AMbrew install --cask --verbose --force docker
Error shows as ... "Error: It seems there is already a Binary at '/usr/local/share/zsh/site-functions/_docker'.
Following is the terminal logs which I am sharing
vishwanathangadi@vishwanaths-MacBook-Pro ~ % brew install --cask --verbose --force docker
Running `brew update --auto-update`...
==> Auto-updated Homebrew!
Updated 1 tap (homebrew/core).
You have 18 outdated formulae installed.
You can upgrade them with brew upgrade
or list them with brew outdated.
==> Downloading <https://desktop.docker.com/mac/main/amd64/91661/Docker.dmg>
Already downloaded: /Users/vishwanathangadi/Library/Caches/Homebrew/downloads/1ffb31f522268752a888fbee3b48dcfa70666365c75c92b6d179e0dd22e82e89--Docker.dmg
==> Verifying checksum for cask 'docker'
==> Installing Cask docker
hdiutil attach -plist -nobrowse -readonly -mountrandom /private/tmp/d20221130-1928-ongdwu /Users/vishwanathangadi/Library/Caches/Homebrew/downloads/1ffb31f522268752a888fbee3b48dcfa70666365c75c92b6d179e0dd22e82e89--Docker.dmg
mkbom -s -i /private/tmp/20221130-1928-zb8c19.list -- /private/tmp/20221130-1928-1z0cblx.bom
ditto --bom /private/tmp/20221130-1928-1z0cblx.bom -- /private/tmp/d20221130-1928-ongdwu/dmg.khdXlw /private/tmp/d20221130-1928-1j5lsiq
diskutil info -plist /private/tmp/d20221130-1928-ongdwu/dmg.khdXlw
diskutil eject /private/tmp/d20221130-1928-ongdwu/dmg.khdXlw
cp -pR /private/tmp/d20221130-1928-1j5lsiq/Docker.app/. /usr/local/Caskroom/docker/4.14.1,91661/Docker.app
chmod -Rf +w /private/tmp/d20221130-1928-1j5lsiq
==> Moving App '<http://Docker.app|Docker.app>' to '/Applications/Docker.app'
==> Linking Binary 'docker-compose.bash-completion' to '/usr/local/etc/bash_completion.d/docker-compose'
==> Unlinking Binary '/usr/local/etc/bash_completion.d/docker-compose'
==> Backing App '<http://Docker.app|Docker.app>' up to '/usr/local/Caskroom/docker/4.14.1,91661/Docker.app'
==> Removing App '/Applications/Docker.app'
==> Purging files for version 4.14.1,91661 of Cask docker
Error: It seems there is already a Binary at '/usr/local/share/zsh/site-functions/_docker'.
future-alarm-92026
11/30/2022, 2:05 PMbitter-noon-90925
11/30/2022, 2:28 PMfierce-nest-35400
11/30/2022, 2:58 PMfierce-nest-35400
11/30/2022, 2:59 PMfierce-nest-35400
11/30/2022, 2:59 PMorange-gigabyte-8918
11/30/2022, 3:08 PMlittle-father-38603
11/30/2022, 3:45 PM/stats/summary
endpoint takes a really long time to respondlittle-father-38603
11/30/2022, 4:42 PMdocker stats
hangs, I suppose kubelet calls that. If anyone knows why this could happen, please let me know :)early-lunch-37616
11/30/2022, 5:15 PMbreezy-ram-80329
12/01/2022, 4:18 AMfamous-grass-8099
12/01/2022, 5:03 AMrancher-server_1 | 2022/12/01 04:54:09 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:49716 (X-Forwarded-For: 44.230.106.196, 172.31.30.132): response 400: websocket: the client is not using the websocket protocol: 'websocket' token not found in 'Upgrade' header
rancher-server_1 | 2022/12/01 04:54:09 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:49716 (X-Forwarded-For: 44.230.106.196, 172.31.30.132): response 400: Error during upgrade for host [c-sxlgx]: websocket: the client is not using the websocket protocol: 'websocket' token not found in 'Upgrade' header
I have another AWS EKS cluster. I am trying to import in above rancher instance. It is showing waiting
status.early-lifeguard-63817
12/01/2022, 8:50 AMshy-tent-66642
12/01/2022, 10:18 AMkind-analyst-38351
12/01/2022, 10:51 AMgifted-shampoo-48019
12/01/2022, 12:16 PMcluster-fb7393592079
. is there a way for me to change that name?acceptable-printer-7134
12/01/2022, 12:24 PMhelpful-student-22864
12/01/2022, 1:16 PMearly-lifeguard-63817
12/01/2022, 2:09 PMdazzling-computer-84464
12/01/2022, 7:32 PM2022/12/01 19:28:41 [DEBUG] No active connection for cluster [c-rw259], will wait for about 30 seconds
2022/12/01 19:28:41 [TRACE] dialerFactory: apiEndpoint hostPort for cluster [c-pw867] is [172.20.0.1:443]
2022/12/01 19:28:41 [TRACE] dialerFactory: no tunnel session found for cluster [c-pw867], falling back to nodeDialer