❓ -> Hi all , I have recently upgraded from rke2

1.31.4+rke2r1

to

1.32.3+rke2r1

and started facing this issue in

ctr

command -

/var/lib/rancher/rke2/bin/ctr --address /run/k3s/containerd/containerd.sock -n k8s.io images pull SOMEFQDN:PORT/SOMEIMAGE:VERSION --plain-http=true
 WARN[0000] DEPRECATION: The `configs` property of `[plugins."io.containerd.grpc.v1.cri".registry]` is deprecated since containerd v1.5 and will be removed in containerd v2.1.Use `config_path` instead. 
 ctr: rpc error: code = Unknown desc = failed to resolve image: failed to do request: Head "<https://SOMEFQDN>:PORT/v2/SOMEIMAGE/manifests/VERSION": http: server gave HTTP response to HTTPS client

, We have a registry running on the VM at that FQDN and port . It worked if we swap SOMEFQDN with

localhost

.So it confirms the image is present . Has there being any change of

--plain-http

flag (I can see it in help section of ctr command) or its precedence got changed in recent releases and why it was not honoured ? When are we expecting

1.32.4+rke2r1

? I am guessing some difference of logic in minor versions as previously using

1.31.4

.

Based on the warnings it looks like you're using a custom containerd config template that needs to be upgraded to modern syntax.

Thanks @creamy-pencil-82913 for the reply , how to deal the error btw -

rpc error: code = Unknown desc = failed to resolve image: failed to do request: Head "<https://FQDN>:PORT_2/v2/SOMEIMAGE/manifests/VERSION": http: server gave HTTP response to HTTPS client

My registries.yaml looks like -

mirrors:
  <http://docker.io|docker.io>:
    endpoint:
    - <https://localhost>:PORT_1
    - <http://FQDN>:PORT_2
    - <https://SUB_DOMAIN.FQDN>
    rewrite:
      ^library/(.*): $1
  <http://index.docker.io|index.docker.io>:
    endpoint:
    - <https://localhost>:PORT_1
    - <http://FQDN>:PORT_2
    - <https://SUB_DOMAIN.FQDN>
  localhost:PORT_1:
    endpoint:
    - <https://localhost>:PORT_1
    - <http://FQDN>:PORT_2
    - <https://SUB_DOMAIN.FQDN>
  FQDN:PORT_2:
    endpoint:
    - <https://localhost>:PORT_1
    - <http://FQDN>:PORT_2
    - <https://SUB_DOMAIN.FQDN>
configs:
  localhost:PORT_1:
    auth:
      username: XXX
      password: YYY
    tls:
      insecure_skip_verify: true
  SUBDOMAIN.FQDN:
    auth:
      username: XXX
      password: YYY
    tls:
      insecure_skip_verify: true
  FQDN:PORT_2:
    auth: {}
    tls:
      insecure_skip_verify: true

I dont see any mistake in this as it was working fine in older version of RKE2 . Would be grateful for any help .

No, your containerd config template. Not registries.yaml

# File generated by rke2. DO NOT EDIT. Use config.toml.tmpl instead.
version = 3
root = "/var/lib/rancher/rke2/agent/containerd"
state = "/run/k3s/containerd"

[grpc]
  address = "/run/k3s/containerd/containerd.sock"

[plugins.'io.containerd.internal.v1.opt']
  path = "/var/lib/rancher/rke2/agent/containerd"

[plugins.'io.containerd.grpc.v1.cri']
  stream_server_address = "127.0.0.1"
  stream_server_port = "10010"

[plugins.'io.containerd.cri.v1.runtime']
  enable_selinux = false
  enable_unprivileged_ports = true
  enable_unprivileged_icmp = true
  device_ownership_from_security_context = false

[plugins.'io.containerd.cri.v1.images']
  snapshotter = "overlayfs"
  disable_snapshot_annotations = true

[plugins.'io.containerd.cri.v1.images'.pinned_images]
  sandbox = "index.docker.io/rancher/mirrored-pause:3.6"

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc.options]
  SystemdCgroup = true

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runhcs-wcow-process]
  runtime_type = "io.containerd.runhcs.v1"

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.'crun']
  runtime_type = "io.containerd.runc.v2"

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.'crun'.options]
  BinaryName = "/usr/bin/crun"
  SystemdCgroup = true

[plugins.'io.containerd.cri.v1.images'.registry]
  config_path = "/var/lib/rancher/rke2/agent/etc/containerd/certs.d"

[plugins.'io.containerd.cri.v1.images'.registry.configs.'localhost:30071'.auth]
  username = "XXX
  password = "YYY"

[plugins.'io.containerd.cri.v1.images'.registry.configs.'SUB_DOMAIN.FQDN'.auth]
  username = "XXX"
  password = "YYY"

[plugins.'io.containerd.cri.v1.images'.registry.configs.'FQDN:PORT_2'.auth]

This is my

config.toml

file . We have

/var/lib/rancher/rke2/agent/etc/containerd/certs.d/FQDN/host.toml

as -

# File generated by rke2. DO NOT EDIT.
server = "<https://FQDN>:PORT_2/v2"
capabilities = ["pull", "resolve", "push"]
skip_verify = true

[host]

[host."<https://localhost:30071/v2>"]
  capabilities = ["pull", "resolve"]
  skip_verify = true

[host."<http://FQDN>:PORT_2/v2"]
  capabilities = ["pull", "resolve"]
  skip_verify = true

[host."<https://SUB_DOMAIN.FQDN/v2>"]
  capabilities = ["pull", "resolve"]
  skip_verify = true