https://rancher.com/ logo
Title
k

kind-librarian-59515

02/15/2023, 2:40 PM
Hi all, with @flat-finland-50817 we tried to install Harvester on a bare metal server using the .ISO. We did a manual installation without the use of the configuration file. After reboot, we have an "imagePullBackOff" on this image: `*`rancher/harvester-cluster-repo:v1.1.1`*`. We've read this thread: https://rancher-users.slack.com/archives/C01GKHKAG0K/p1668802622497439. But we do not know if this is the same problem. We also search for the image on the SUSE registry:
<http://registry.suse.com|registry.suse.com>
; but we did not find it. * Below are more details:
# cat /etc/os-release

NAME="SLE Micro"
VERSION="5.2"
VERSION_ID="5.2"
PRETTY_NAME="Harvester v1.1.1"
ID="sle-micro-rancher"
ID_LIKE="suse"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sle-micro-rancher:5.2"
VARIANT="Harvester"
VARIANT_ID="Harvester-20221201"
GRUB_ENTRY_NAME="Harvester v1.1.1"
# cat /etc/harvester-release.yaml

harvester: v1.1.1
harvesterChart: 1.1.1
os: Harvester v1.1.1
kubernetes: v1.24.7+rke2r1
rancher: v2.6.9
monitoringChart: 100.1.0+up19.0.3
loggingChart: 100.1.3+up3.17.7
kubevirt: 0.54.0-150400.3.7.1
minUpgradableVersion: 'v1.0.3'
rancherDependencies:
  fleet:
    chart: 100.1.0+up0.4.0
    app: 0.4.0
  fleet-crd:
    chart: 100.1.0+up0.4.0
    app: 0.4.0
  rancher-webhook:
    chart: 1.0.6+up0.2.7
    app: 0.2.7
# kubectl get nodes -o wide

NAME           STATUS   ROLES                       AGE     VERSION          INTERNAL-IP   EXTERNAL-IP   OS-IMAGE           KERNEL-VERSION                 CONTAINER-RUNTIME
harvester123   Ready    control-plane,etcd,master   3h27m   v1.24.7+rke2r1   10.11.12.13   <none>        Harvester v1.1.1   5.3.18-150300.59.101-default   <containerd://1.6.8-k3s1>
# kubectl get pods -A

NAMESPACE                   NAME                                                    READY   STATUS         RESTARTS        AGE
cattle-fleet-local-system   fleet-agent-86dc84bdb7-67q4z                            1/1     Running        0               3h25m
cattle-fleet-system         fleet-controller-6695647998-wsr7r                       1/1     Running        0               3h27m
cattle-fleet-system         gitjob-5c5979d844-grk9p                                 1/1     Running        0               3h27m
cattle-system               harvester-cluster-repo-5d95ff67d-9pf7j                  0/1     ErrImagePull   0               6m11s
cattle-system               rancher-7d87758d74-bjv44                                1/1     Running        0               3h30m
cattle-system               rancher-webhook-84c7967dd5-kjnjk                        1/1     Running        0               3h26m
cattle-system               system-upgrade-controller-79fc9c84b7-v8g4f              1/1     Running        0               3h25m
kube-system                 cloud-controller-manager-harvester113                   1/1     Running        0               3h30m
kube-system                 etcd-harvester113                                       1/1     Running        0               3h29m
kube-system                 helm-install-rke2-canal-fptbs                           0/1     Completed      0               3h30m
kube-system                 helm-install-rke2-coredns-cs7wb                         0/1     Completed      0               3h30m
kube-system                 helm-install-rke2-ingress-nginx-9wssc                   0/1     Completed      0               3h26m
kube-system                 helm-install-rke2-metrics-server-btvx4                  0/1     Completed      0               3h30m
kube-system                 helm-install-rke2-multus-5ftgw                          0/1     Completed      0               3h30m
kube-system                 kube-apiserver-harvester113                             1/1     Running        0               3h30m
kube-system                 kube-controller-manager-harvester113                    1/1     Running        1 (3h24m ago)   3h30m
kube-system                 kube-proxy-harvester113                                 1/1     Running        0               3h30m
kube-system                 kube-scheduler-harvester113                             1/1     Running        1 (3h24m ago)   3h30m
kube-system                 rke2-canal-k6zqw                                        2/2     Running        0               3h29m
kube-system                 rke2-coredns-rke2-coredns-58fd75f64b-7p6vq              1/1     Running        0               3h29m
kube-system                 rke2-coredns-rke2-coredns-autoscaler-768bfc5985-vqr57   1/1     Running        0               3h29m
kube-system                 rke2-ingress-nginx-controller-9rsbn                     1/1     Running        0               3h25m
kube-system                 rke2-metrics-server-67697454f8-tmqkz                    1/1     Running        0               3h29m
kube-system                 rke2-multus-ds-p4czm                                    1/1     Running        0               3h30m
```
kubectl describe pod -n cattle-system harvester-cluster-repo-5d95ff67d-9pf7j
Events:
  Type     Reason          Age                    From               Message
  ----     ------          ----                   ----               -------
  Normal   Scheduled       7m12s                  default-scheduler  Successfully assigned cattle-system/harvester-cluster-repo-5d95ff67d-9pf7j to harvester113
  Normal   AddedInterface  7m11s                  multus             Add eth0 [10.52.0.30/32] from k8s-pod-network
  Normal   Pulling         5m40s (x4 over 7m11s)  kubelet            Pulling image "rancher/harvester-cluster-repo:v1.1.1"
  Warning  Failed          5m40s (x4 over 7m10s)  kubelet            Failed to pull image "rancher/harvester-cluster-repo:v1.1.1": rpc error: code = Unknown desc = failed to pull and unpack image "<http://docker.io/rancher/harvester-cluster-repo:v1.1.1|docker.io/rancher/harvester-cluster-repo:v1.1.1>": failed to resolve reference "<http://docker.io/rancher/harvester-cluster-repo:v1.1.1|docker.io/rancher/harvester-cluster-repo:v1.1.1>": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
  Warning  Failed          5m40s (x4 over 7m10s)  kubelet            Error: ErrImagePull
  Warning  Failed          5m28s (x6 over 7m10s)  kubelet            Error: ImagePullBackOff
  Normal   BackOff         2m9s (x20 over 7m10s)  kubelet            Back-off pulling image "rancher/harvester-cluster-repo:v1.1.1"
Using `crane`:
harvester-cluster-repo
does not exist on
<http://docker.io|docker.io>
crane ls <http://docker.io/rancher/harvester-cluster-repo|docker.io/rancher/harvester-cluster-repo>

Error: reading tags for <http://docker.io/rancher/harvester-cluster-repo|docker.io/rancher/harvester-cluster-repo>: GET <https://index.docker.io/v2/rancher/harvester-cluster-repo/tags/list?n=1000>: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:rancher/harvester-cluster-repo Type:repository]]
👍 1
g

great-bear-19718

02/15/2023, 10:53 PM
this image is an image created at runtime and only contains internal artefacts and is packaged within the iso
during the install process, we copy images and extract them for rke2
i have seen in some scenarios when using iso, the extract can fail silently, likely because image extraction took over 30 mins, in which case you can see node boot and complain about the missing image
best option would be to retry iso install or try a PXE based install, which downloads iso locally onto node, and runs into less issues during the image extract
i hope this helps
f

flat-finland-50817

02/16/2023, 8:32 AM
Thanks for the explanations, we will retry the install. May I ask why this image is not available publicly in a registry ? What is the benefit of only packaging it in the iso and not give access to it by registry ?
We were able to finish the install by extracting the
harvester-cluster-repo
image manually from the iso and importing it with docker on the machine we installed Harvester on. @great-bear-19718 if you can provide an explanation about why the image is not in a public registry I'd be glad.
g

great-bear-19718

02/20/2023, 9:27 AM
its only use is to hold individual helm charts for some of the core components already published as separate releases
f

flat-finland-50817

02/20/2023, 9:34 AM
But why not uploading it to a public registry (like docker.io or registry.suse.com) ? As it could prevent installation failures ? Is it vendor proprietary or something like that ?
g

great-bear-19718

02/20/2023, 9:37 AM
one it will not work in air-gapped env, and issue is to figure out why extraction is failing silently
k

kind-librarian-59515

02/20/2023, 10:02 AM
issue is to figure out why extraction is failing silently
You are correct 🙂: we can create an issue out of this Slack conversation if you want.
one it will not work in air-gapped env
The question is not wether you should embed or not the container images in the harvester ISO: of course you should embed all the needed container images in the Harvester ISO file. The question is to allow people to audit and review the container images that are embedded in the ISO. This is exactly what is done on the k3s github: there are files called "k3s-airgap-images-amd64.tar*", which anyone can review and audit and then use in their own k3s airgap installation 😊. The image "*harvester-cluster-repo*" is nowhere to be found, exept in the harvester ISO. We do not know where and how the ISO is fabricated. We do not know where and how the container image is fabricated. Does this mean it is a rancher vendor/proprietary container image, and not an open source container image? Are there other container images that follows the same behaviour (for RKE2)? What is the licensing behind Harvester?
g

great-bear-19718

02/20/2023, 10:03 AM
if you have docker on x86 just run
make build-iso
k

kind-librarian-59515

02/20/2023, 10:09 AM
[edit link] Thank you, this helps 🙂. Can you confirm that this image
rancher/harvester-cluster-repo:v1.1.1
corresponds to this Containerfile? • https://github.com/harvester/harvester-installer/blob/c75e7922ac789c8bd725b894395219806da0e689/package/harvester-repo/Dockerfilehttps://github.com/harvester/harvester-installer/commit/9d237eb9caa816b69b75c7d55d60fc0f6683a246
Thanks to your help, I also discover the rancher tool "wharfie", which seems to have the same sort of features than "`crane export`" or "`podman/docker save`" or "`skopeo copy`". https://github.com/rancher/wharfie