https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# k3s
a
This message was deleted.
c
Check the containerd logs to see what exactly it's doing. It looks like you're getting an error page from the proxy.
You might also mention what specific version you're trying to upgrade from and to?
b
Thanks Brad! I will check the containerd-logs! We are running e2e-tests for our controllers with k3s, so we use multiple versions of k3s - since we want to test our controllers against multiple upstream K8s versions. This was working fine with the next to latest versions of k3s 1.27.x, 1.28.x and 1.29.x. With the latest versions of k3s all our pipeline jobs are breaking.
All working with v1.27.9-k3s1, v1.28.5-k3s1 and v1.29.0-k3s1. All failing with v1.27.10-k3s1, v1.28.6-k3s1 and v1.29.1-k3s1.
Yes, I agree there seems to be an issue with our proxy:
Copy code
time="2024-01-31T15:55:04.317832003Z" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:local-path-provisioner-84db5d44d9-rr6fl,Uid:9bf3b704-a305-44fe-b5ae-dfa256cba4cb,Namespace:kube-system,Attempt:0,}"
time="2024-01-31T15:55:04.325182440Z" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:coredns-6799fbcd5-pt8rn,Uid:665cfbcd-0273-4a2e-9b07-1f468eafeb5d,Namespace:kube-system,Attempt:0,}"
time="2024-01-31T15:55:04.325212560Z" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:metrics-server-67c658944b-grcm7,Uid:f3946703-34c1-49b3-a0d3-5767edf36761,Namespace:kube-system,Attempt:0,}"
time="2024-01-31T15:55:04.433939815Z" level=warning msg="reference for unknown type: text/html" digest="sha256:fbbb2793fdc6d093453b774a929b489b8b951267822b33de47ddd6907ad30816" mediatype=text/html size=3029
time="2024-01-31T15:55:04.505056974Z" level=warning msg="reference for unknown type: text/html" digest="sha256:fbbb2793fdc6d093453b774a929b489b8b951267822b33de47ddd6907ad30816" mediatype=text/html size=3029
time="2024-01-31T15:55:04.509513703Z" level=info msg="ImageCreate event name:\"<http://docker.io/rancher/mirrored-pause:3.6\|docker.io/rancher/mirrored-pause:3.6\>"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}  labels:{key:\"io.cri-containerd.pinned\"  value:\"pinned\"}"
time="2024-01-31T15:55:04.509570905Z" level=error msg="Failed to handle event name:\"<http://docker.io/rancher/mirrored-pause:3.6\|docker.io/rancher/mirrored-pause:3.6\>"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}  labels:{key:\"io.cri-containerd.pinned\"  value:\"pinned\"} for <http://docker.io/rancher/mirrored-pause:3.6|docker.io/rancher/mirrored-pause:3.6>" error="update image store for \"<http://docker.io/rancher/mirrored-pause:3.6\|docker.io/rancher/mirrored-pause:3.6\>": get image info from containerd: get image diffIDs: unexpected media type text/html for sha256:fbbb2793fdc6d093453b774a929b489b8b951267822b33de47ddd6907ad30816: not found"
time="2024-01-31T15:55:04.517885593Z" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:local-path-provisioner-84db5d44d9-rr6fl,Uid:9bf3b704-a305-44fe-b5ae-dfa256cba4cb,Namespace:kube-system,Attempt:0,} failed, error" error="rpc error: code = NotFound desc = failed to get sandbox image \"rancher/mirrored-pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"<http://docker.io/rancher/mirrored-pause:3.6\|docker.io/rancher/mirrored-pause:3.6\>": failed to unpack image on snapshotter overlayfs: unexpected media type text/html for sha256:fbbb2793fdc6d093453b774a929b489b8b951267822b33de47ddd6907ad30816: not found"
time="2024-01-31T15:55:04.517963424Z" level=info msg="stop pulling image <http://docker.io/rancher/mirrored-pause:3.6|docker.io/rancher/mirrored-pause:3.6>: active requests=0, bytes read=6638"
time="2024-01-31T15:55:04.522351085Z" level=warning msg="reference for unknown type: text/html" digest="sha256:fbbb2793fdc6d093453b774a929b489b8b951267822b33de47ddd6907ad30816" mediatype=text/html size=3029
time="2024-01-31T15:55:04.527354782Z" level=info msg="Events for \"<http://docker.io/rancher/mirrored-pause:3.6\|docker.io/rancher/mirrored-pause:3.6\>" is in backoff, enqueue event name:\"<http://docker.io/rancher/mirrored-pause:3.6\|docker.io/rancher/mirrored-pause:3.6\>"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}  labels:{key:\"io.cri-containerd.pinned\"  value:\"pinned\"}"
time="2024-01-31T15:55:04.534853118Z" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:metrics-server-67c658944b-grcm7,Uid:f3946703-34c1-49b3-a0d3-5767edf36761,Namespace:kube-system,Attempt:0,} failed, error" error="rpc error: code = NotFound desc = failed to get sandbox image \"rancher/mirrored-pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"<http://docker.io/rancher/mirrored-pause:3.6\|docker.io/rancher/mirrored-pause:3.6\>": failed to unpack image on snapshotter overlayfs: unexpected media type text/html for sha256:fbbb2793fdc6d093453b774a929b489b8b951267822b33de47ddd6907ad30816: not found"
time="2024-01-31T15:55:04.534881644Z" level=info msg="stop pulling image <http://docker.io/rancher/mirrored-pause:3.6|docker.io/rancher/mirrored-pause:3.6>: active requests=0, bytes read=3319"
time="2024-01-31T15:55:04.543668484Z" level=info msg="Events for \"<http://docker.io/rancher/mirrored-pause:3.6\|docker.io/rancher/mirrored-pause:3.6\>" is in backoff, enqueue event name:\"<http://docker.io/rancher/mirrored-pause:3.6\|docker.io/rancher/mirrored-pause:3.6\>"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}  labels:{key:\"io.cri-containerd.pinned\"  value:\"pinned\"}"
time="2024-01-31T15:55:04.547424276Z" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:coredns-6799fbcd5-pt8rn,Uid:665cfbcd-0273-4a2e-9b07-1f468eafeb5d,Namespace:kube-system,Attempt:0,} failed, error" error="rpc error: code = NotFound desc = failed to get sandbox image \"rancher/mirrored-pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"<http://docker.io/rancher/mirrored-pause:3.6\|docker.io/rancher/mirrored-pause:3.6\>": failed to unpack image on snapshotter overlayfs: unexpected media type text/html for sha256:fbbb2793fdc6d093453b774a929b489b8b951267822b33de47ddd6907ad30816: not found"
time="2024-01-31T15:55:04.547485498Z" level=info msg="stop pulling image <http://docker.io/rancher/mirrored-pause:3.6|docker.io/rancher/mirrored-pause:3.6>: active requests=0, bytes read=3319"
time="2024-01-31T15:55:05.335644838Z" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:cert-manager-5d6675966f-gvjfg,Uid:500da9a5-9757-4804-80d1-31cdcea702df,Namespace:cert-manager,Attempt:0,}"
time="2024-01-31T15:55:05.336501871Z" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:cert-manager-cainjector-d7f8b5464-hlh5r,Uid:d123a5b9-f9be-4278-b19a-483d35d9e396,Namespace:cert-manager,Attempt:0,}"
time="2024-01-31T15:55:05.341365924Z" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:cert-manager-webhook-84ffb579c6-zr855,Uid:1d81b02a-f272-4eba-8e04-5dced6f2138f,Namespace:cert-manager,Attempt:0,}"
time="2024-01-31T15:55:05.467322009Z" level=warning msg="reference for unknown type: text/html" digest="sha256:fbbb2793fdc6d093453b774a929b489b8b951267822b33de47ddd6907ad30816" mediatype=text/html size=3029
time="2024-01-31T15:55:05.477530706Z" level=warning msg="reference for unknown type: text/html" digest="sha256:fbbb2793fdc6d093453b774a929b489b8b951267822b33de47ddd6907ad30816" mediatype=text/html size=3029
time="2024-01-31T15:55:05.481385028Z" level=info msg="Events for \"<http://docker.io/rancher/mirrored-pause:3.6\|docker.io/rancher/mirrored-pause:3.6\>" is in backoff, enqueue event name:\"<http://docker.io/rancher/mirrored-pause:3.6\|docker.io/rancher/mirrored-pause:3.6\>"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}  labels:{key:\"io.cri-containerd.pinned\"  value:\"pinned\"}"
time="2024-01-31T15:55:05.488937532Z" level=warning msg="reference for unknown type: text/html" digest="sha256:fbbb2793fdc6d093453b774a929b489b8b951267822b33de47ddd6907ad30816" mediatype=text/html size=3029
time="2024-01-31T15:55:05.492732382Z" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:cert-manager-cainjector-d7f8b5464-hlh5r,Uid:d123a5b9-f9be-4278-b19a-483d35d9e396,Namespace:cert-manager,Attempt:0,} failed, error" error="rpc error: code = NotFound desc = failed to get sandbox image \"rancher/mirrored-pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"<http://docker.io/rancher/mirrored-pause:3.6\|docker.io/rancher/mirrored-pause:3.6\>": failed to unpack image on snapshotter overlayfs: unexpected media type text/html for sha256:fbbb2793fdc6d093453b774a929b489b8b951267822b33de47ddd6907ad30816: not found"
time="2024-01-31T15:55:05.492794492Z" level=info msg="stop pulling image <http://docker.io/rancher/mirrored-pause:3.6|docker.io/rancher/mirrored-pause:3.6>: active requests=0, bytes read=3319"
time="2024-01-31T15:55:05.496763773Z" level=info msg="Events for \"<http://docker.io/rancher/mirrored-pause:3.6\|docker.io/rancher/mirrored-pause:3.6\>" is in backoff, enqueue event name:\"<http://docker.io/rancher/mirrored-pause:3.6\|docker.io/rancher/mirrored-pause:3.6\>"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}  labels:{key:\"io.cri-containerd.pinned\"  value:\"pinned\"}"
time="2024-01-31T15:55:05.509045943Z" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:cert-manager-webhook-84ffb579c6-zr855,Uid:1d81b02a-f272-4eba-8e04-5dced6f2138f,Namespace:cert-manager,Attempt:0,} failed, error" error="rpc error: code = NotFound desc = failed to get sandbox image \"rancher/mirrored-pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"<http://docker.io/rancher/mirrored-pause:3.6\|docker.io/rancher/mirrored-pause:3.6\>": failed to unpack image on snapshotter overlayfs: unexpected media type text/html for sha256:fbbb2793fdc6d093453b774a929b489b8b951267822b33de47ddd6907ad30816: not found"
time="2024-01-31T15:55:05.509115086Z" level=info msg="stop pulling image <http://docker.io/rancher/mirrored-pause:3.6|docker.io/rancher/mirrored-pause:3.6>: active requests=0, bytes read=3319"
time="2024-01-31T15:55:05.519976731Z" level=info msg="Events for \"<http://docker.io/rancher/mirrored-pause:3.6\|docker.io/rancher/mirrored-pause:3.6\>" is in backoff, enqueue event name:\"<http://docker.io/rancher/mirrored-pause:3.6\|docker.io/rancher/mirrored-pause:3.6\>"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}  labels:{key:\"io.cri-containerd.pinned\"  value:\"pinned\"}"
time="2024-01-31T15:55:05.523602764Z" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:cert-manager-5d6675966f-gvjfg,Uid:500da9a5-9757-4804-80d1-31cdcea702df,Namespace:cert-manager,Attempt:0,} failed, error" error="rpc error: code = NotFound desc = failed to get sandbox image \"rancher/mirrored-pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"<http://docker.io/rancher/mirrored-pause:3.6\|docker.io/rancher/mirrored-pause:3.6\>": failed to unpack image on snapshotter overlayfs: unexpected media type text/html for sha256:fbbb2793fdc6d093453b774a929b489b8b951267822b33de47ddd6907ad30816: not found"
time="2024-01-31T15:55:05.523633357Z" level=info msg="stop pulling image <http://docker.io/rancher/mirrored-pause:3.6|docker.io/rancher/mirrored-pause:3.6>: active requests=0, bytes read=3319"
time="2024-01-31T15:55:06.025219998Z" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:helm-install-traefik-crd-6n6dw,Uid:f061afa1-b900-4568-a0ee-f50e0bc7404a,Namespace:kube-system,Attempt:0,}"
time="2024-01-31T15:55:06.026901977Z" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:helm-install-traefik-s97d5,Uid:b5ab08f0-d095-40f9-b110-54e1d20b8c5a,Namespace:kube-system,Attempt:0,}"
time="2024-01-31T15:55:06.098308065Z" level=warning msg="reference for unknown type: text/html" digest="sha256:fbbb2793fdc6d093453b774a929b489b8b951267822b33de47ddd6907ad30816" mediatype=text/html size=3029
time="2024-01-31T15:55:06.111147092Z" level=info msg="Events for \"<http://docker.io/rancher/mirrored-pause:3.6\|docker.io/rancher/mirrored-pause:3.6\>" is in backoff, enqueue event name:\"<http://docker.io/rancher/mirrored-pause:3.6\|docker.io/rancher/mirrored-pause:3.6\>"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}  labels:{key:\"io.cri-containerd.pinned\"  value:\"pinned\"}"
time="2024-01-31T15:55:06.114867988Z" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:helm-install-traefik-crd-6n6dw,Uid:f061afa1-b900-4568-a0ee-f50e0bc7404a,Namespace:kube-system,Attempt:0,} failed, error" error="rpc error: code = NotFound desc = failed to get sandbox image \"rancher/mirrored-pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"<http://docker.io/rancher/mirrored-pause:3.6\|docker.io/rancher/mirrored-pause:3.6\>": failed to unpack image on snapshotter overlayfs: unexpected media type text/html for sha256:fbbb2793fdc6d093453b774a929b489b8b951267822b33de47ddd6907ad30816: not found"
time="2024-01-31T15:55:06.114953383Z" level=info msg="stop pulling image <http://docker.io/rancher/mirrored-pause:3.6|docker.io/rancher/mirrored-pause:3.6>: active requests=0, bytes read=3319"
But as this is working fine with previous versions of k3s it would be nice to know what new is expected.
c
Are you using the releases that came out yesterday?
Also, that looks like kubelet log, not containerd log.
b
Are you using the releases that came out yesterday?
Yes, those are the releases that are failing for us.
c
Can you look at containerd log, and also confirm whether or not you are using a custom containerd config.toml template?
b
Not using any custom containerd config.toml AFAIK. We are just configuring registry mirrors.
c
ok. can you share the contents of 
/var/lib/rancher/k3s/agent/etc/containerd/config.toml
and 
/var/lib/rancher/k3s/agent/containerd/containerd.log
?
b
By investigating the logs of our proxy, we figured out that the mirror config supplied to k3s had to change to work with the latest release. After appending 
/v2
to our proxy hostname it now works. So
Copy code
<http://docker.io|docker.io>:
    endpoint:
      - <proxy-host>
had to be changed to
Copy code
<http://docker.io|docker.io>:
    endpoint:
      - <proxy-host>/v2
Copy code
/ # cat /var/lib/rancher/k3s/agent/etc/containerd/config.toml
# File generated by k3s. DO NOT EDIT. Use config.toml.tmpl instead.
version = 2

[plugins."io.containerd.internal.v1.opt"]
  path = "/var/lib/rancher/k3s/agent/containerd"
[plugins."io.containerd.grpc.v1.cri"]
  stream_server_address = "127.0.0.1"
  stream_server_port = "10010"
  enable_selinux = false
  enable_unprivileged_ports = true
  enable_unprivileged_icmp = true
  sandbox_image = "rancher/mirrored-pause:3.6"

[plugins."io.containerd.grpc.v1.cri".containerd]
  snapshotter = "overlayfs"
  disable_snapshot_annotations = true



[plugins."io.containerd.grpc.v1.cri".cni]
  bin_dir = "/bin"
  conf_dir = "/var/lib/rancher/k3s/agent/etc/cni/net.d"


[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  SystemdCgroup = false

[plugins."io.containerd.grpc.v1.cri".registry]
  config_path = "/var/lib/rancher/k3s/agent/etc/containerd/certs.d"






/ #
c
hmm. That shouldn’t be necessary, if the endpoint address doesn’t have a path component, /v2 is assumed by containerd. Did you perhaps have 
<https://proxy-host/>
instead of 
<https://proxy-host>
?
b
No, the mirrors are configured with plain hostnames.
c
hmm that is an unexpected change in behavior on the containerd side.
wait
you said the endpoint is just 
proxy-host
without a scheme or anything?
It was always supposed to be a URI
b
We use k3d config to configure this. Here is the redacted contents:
Copy code
---
apiVersion: <http://k3d.io/v1alpha5|k3d.io/v1alpha5>
kind: Simple
metadata:
  name: local
image: artifactory.<company-domain>/dockerhub-docker-remote/rancher/k3s:v1.29.0-k3s1
registries:
  config: |
    mirrors:
      <http://docker.io|docker.io>:
        endpoint:
          - dockerhub-docker-remote.hub.<company-domain>
      <http://gcr.io|gcr.io>:
        endpoint:
          - gcr-docker-remote.hub.<company-domain>
      <http://ghcr.io|ghcr.io>:
        endpoint:
          - ghcr-docker-remote.hub.<company-domain>
      <http://mcr.microsoft.com|mcr.microsoft.com>:
        endpoint:
          - docker-mcr-microsoft-com-remote.hub.<company-domain>
      <http://registry.k8s.io|registry.k8s.io>:
        endpoint:
          - k8s-docker-remote.hub.<company-domain>
      <http://k8s.gcr.io|k8s.gcr.io>:
        endpoint:
          - k8s-docker-remote.hub.<company-domain>
      <http://quay.io|quay.io>:
        endpoint:
          - quay-docker-remote.hub.<company-domain>
c
yeah, that is not the expected format for the endpoint. it should be a URI, not a bare hostname.
b
According to https://docs.k3s.io/installation/private-registry#registries-configuration-file we are missing a few things. But this was working up till the releases yesterday.
Thanks for your help!
c
We bumped the containerd version in the latest release, I suspect something changed with how it handles bare hostnames in the endpoint field
👍 1
b
Prefixing the mirror hostnames with 
https://
also works. So it's either that or append 
/v2
. Strange! 🙂
c
yep. I suspect it’s specifically the handling of it not being a URI that changed.
Can take a look at addressing that for the next release
🙌 1
262 Views
Open in Slack
PreviousNext
Powered by