https://rancher.com/ logo
Title
p

prehistoric-judge-25958

10/03/2022, 5:59 PM
Hi, i have setup a private registry (harbor) inside my k3s cluster. I have pushed images succesfully to this registry and now i want to pull images from the registry. When i put this yaml config below in
/etc/rancher/k3s/registries.yaml
my node become in a NotReady state after "systemcl restart k3s"
mirrors:
  harbor.k8s.lan:
    endpoint:
      - "<https://harbor.k8s.lan:443>"
configs:
  "harbor.k8s.lan:443":
    tls:
      cert_file: /etc/rancher/k3s/certs/cert.pem
      key_file:  /etc/rancher/k3s/certs/cert-key.pem
      ca_file:   /etc/rancher/k3s/certs/k8s-lan.crt
      insecure_skip_verify: "true"
I am using self-signed certificates for my k8s.lan domain and put them in the directory /etc/rancher/k3s/certs/ describe node k3s-master-01 output:
Normal   Starting                 23m                kubelet          Starting kubelet.
  Warning  InvalidDiskCapacity      23m                kubelet          invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  23m                kubelet          Updated Node Allocatable limit across pods
  Normal   NodeHasSufficientMemory  23m (x2 over 23m)  kubelet          Node k3s-master-01 status is now: NodeHasSufficientMemory
  Normal   NodeHasNoDiskPressure    23m (x2 over 23m)  kubelet          Node k3s-master-01 status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID     23m (x2 over 23m)  kubelet          Node k3s-master-01 status is now: NodeHasSufficientPID
  Normal   NodeReady                23m                kubelet          Node k3s-master-01 status is now: NodeReady
  Normal   NodeNotReady             20m (x3 over 63m)  node-controller  Node k3s-master-01 status is now: NodeNotReady
1
c

creamy-pencil-82913

10/03/2022, 6:20 PM
are you sure you have the correct content in all those files? why are you passing CA and client certificates if you’ve turned off TLS verification?
It sounds like k3s is failing to start on that node, I’d check the logs to figure out why
p

prehistoric-judge-25958

10/03/2022, 6:23 PM
I thought
insecure_skip_verify: "true"
was needed to avoid checking the certificates with a known authority and so should be true when using self-signed certificates and own CA. Stupid question but where to find logs files of a node?
c

creamy-pencil-82913

10/03/2022, 6:24 PM
there are two ways to handle certs not issued by a public CA. You can either turn off TLS verification (less secure), or pass in the CA Certificate that signed the certificate so that it becomes trusted (more secure). You’re doing both.
where are the logs? In journald, ususally.
p

prehistoric-judge-25958

10/03/2022, 6:37 PM
Hm removed the cert part out of the yaml
mirrors:
  harbor.k8s.lan:
    endpoint:
      - "<https://harbor.k8s.lan:443>"
configs:
  "harbor.k8s.lan":
    tls:
      insecure_skip_verify: "true"
but still
Yeah! got i working. Two things was going on. I had
insecure_skip_verify: "true"
set as a string value instead of a boolean. ->
insecure_skip_verify: true
and like you said @creamy-pencil-82913 I removed the cert part likt this below and now it is working. thanks for the help!
mirrors:
  harbor.k8s.lan:
    endpoint:
      - "<https://harbor.k8s.lan:443>"
configs:
  "harbor.k8s.lan:443":
    tls:
      insecure_skip_verify: true