Hi all, with @flat-finland-50817 we tried to install Harvester on a bare metal server using the .ISO. We did a manual installation without the use of the configuration file. After reboot, we have an "imagePullBackOff" on this image: `*`rancher/harvester-cluster-repo:v1.1.1`*`. We've read this thread: https://rancher-users.slack.com/archives/C01GKHKAG0K/p1668802622497439. But we do not know if this is the same problem. We also search for the image on the SUSE registry:

<http://registry.suse.com|registry.suse.com>

; but we did not find it. * Below are more details:

# cat /etc/os-release

NAME="SLE Micro"
VERSION="5.2"
VERSION_ID="5.2"
PRETTY_NAME="Harvester v1.1.1"
ID="sle-micro-rancher"
ID_LIKE="suse"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sle-micro-rancher:5.2"
VARIANT="Harvester"
VARIANT_ID="Harvester-20221201"
GRUB_ENTRY_NAME="Harvester v1.1.1"

# cat /etc/harvester-release.yaml

harvester: v1.1.1
harvesterChart: 1.1.1
os: Harvester v1.1.1
kubernetes: v1.24.7+rke2r1
rancher: v2.6.9
monitoringChart: 100.1.0+up19.0.3
loggingChart: 100.1.3+up3.17.7
kubevirt: 0.54.0-150400.3.7.1
minUpgradableVersion: 'v1.0.3'
rancherDependencies:
  fleet:
    chart: 100.1.0+up0.4.0
    app: 0.4.0
  fleet-crd:
    chart: 100.1.0+up0.4.0
    app: 0.4.0
  rancher-webhook:
    chart: 1.0.6+up0.2.7
    app: 0.2.7

# kubectl get nodes -o wide

NAME           STATUS   ROLES                       AGE     VERSION          INTERNAL-IP   EXTERNAL-IP   OS-IMAGE           KERNEL-VERSION                 CONTAINER-RUNTIME
harvester123   Ready    control-plane,etcd,master   3h27m   v1.24.7+rke2r1   10.11.12.13   <none>        Harvester v1.1.1   5.3.18-150300.59.101-default   <containerd://1.6.8-k3s1>

# kubectl get pods -A

NAMESPACE                   NAME                                                    READY   STATUS         RESTARTS        AGE
cattle-fleet-local-system   fleet-agent-86dc84bdb7-67q4z                            1/1     Running        0               3h25m
cattle-fleet-system         fleet-controller-6695647998-wsr7r                       1/1     Running        0               3h27m
cattle-fleet-system         gitjob-5c5979d844-grk9p                                 1/1     Running        0               3h27m
cattle-system               harvester-cluster-repo-5d95ff67d-9pf7j                  0/1     ErrImagePull   0               6m11s
cattle-system               rancher-7d87758d74-bjv44                                1/1     Running        0               3h30m
cattle-system               rancher-webhook-84c7967dd5-kjnjk                        1/1     Running        0               3h26m
cattle-system               system-upgrade-controller-79fc9c84b7-v8g4f              1/1     Running        0               3h25m
kube-system                 cloud-controller-manager-harvester113                   1/1     Running        0               3h30m
kube-system                 etcd-harvester113                                       1/1     Running        0               3h29m
kube-system                 helm-install-rke2-canal-fptbs                           0/1     Completed      0               3h30m
kube-system                 helm-install-rke2-coredns-cs7wb                         0/1     Completed      0               3h30m
kube-system                 helm-install-rke2-ingress-nginx-9wssc                   0/1     Completed      0               3h26m
kube-system                 helm-install-rke2-metrics-server-btvx4                  0/1     Completed      0               3h30m
kube-system                 helm-install-rke2-multus-5ftgw                          0/1     Completed      0               3h30m
kube-system                 kube-apiserver-harvester113                             1/1     Running        0               3h30m
kube-system                 kube-controller-manager-harvester113                    1/1     Running        1 (3h24m ago)   3h30m
kube-system                 kube-proxy-harvester113                                 1/1     Running        0               3h30m
kube-system                 kube-scheduler-harvester113                             1/1     Running        1 (3h24m ago)   3h30m
kube-system                 rke2-canal-k6zqw                                        2/2     Running        0               3h29m
kube-system                 rke2-coredns-rke2-coredns-58fd75f64b-7p6vq              1/1     Running        0               3h29m
kube-system                 rke2-coredns-rke2-coredns-autoscaler-768bfc5985-vqr57   1/1     Running        0               3h29m
kube-system                 rke2-ingress-nginx-controller-9rsbn                     1/1     Running        0               3h25m
kube-system                 rke2-metrics-server-67697454f8-tmqkz                    1/1     Running        0               3h29m
kube-system                 rke2-multus-ds-p4czm                                    1/1     Running        0               3h30m
```

kubectl describe pod -n cattle-system harvester-cluster-repo-5d95ff67d-9pf7j

Events:
  Type     Reason          Age                    From               Message
  ----     ------          ----                   ----               -------
  Normal   Scheduled       7m12s                  default-scheduler  Successfully assigned cattle-system/harvester-cluster-repo-5d95ff67d-9pf7j to harvester113
  Normal   AddedInterface  7m11s                  multus             Add eth0 [10.52.0.30/32] from k8s-pod-network
  Normal   Pulling         5m40s (x4 over 7m11s)  kubelet            Pulling image "rancher/harvester-cluster-repo:v1.1.1"
  Warning  Failed          5m40s (x4 over 7m10s)  kubelet            Failed to pull image "rancher/harvester-cluster-repo:v1.1.1": rpc error: code = Unknown desc = failed to pull and unpack image "<http://docker.io/rancher/harvester-cluster-repo:v1.1.1|docker.io/rancher/harvester-cluster-repo:v1.1.1>": failed to resolve reference "<http://docker.io/rancher/harvester-cluster-repo:v1.1.1|docker.io/rancher/harvester-cluster-repo:v1.1.1>": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
  Warning  Failed          5m40s (x4 over 7m10s)  kubelet            Error: ErrImagePull
  Warning  Failed          5m28s (x6 over 7m10s)  kubelet            Error: ImagePullBackOff
  Normal   BackOff         2m9s (x20 over 7m10s)  kubelet            Back-off pulling image "rancher/harvester-cluster-repo:v1.1.1"

Using `crane`:

harvester-cluster-repo

does not exist on

<http://docker.io|docker.io>

crane ls <http://docker.io/rancher/harvester-cluster-repo|docker.io/rancher/harvester-cluster-repo>

Error: reading tags for <http://docker.io/rancher/harvester-cluster-repo|docker.io/rancher/harvester-cluster-repo>: GET <https://index.docker.io/v2/rancher/harvester-cluster-repo/tags/list?n=1000>: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:rancher/harvester-cluster-repo Type:repository]]

this image is an image created at runtime and only contains internal artefacts and is packaged within the iso

Thanks for the explanations, we will retry the install. May I ask why this image is not available publicly in a registry ? What is the benefit of only packaging it in the iso and not give access to it by registry ?

its only use is to hold individual helm charts for some of the core components already published as separate releases

But why not uploading it to a public registry (like docker.io or registry.suse.com) ? As it could prevent installation failures ? Is it vendor proprietary or something like that ?

one it will not work in air-gapped env, and issue is to figure out why extraction is failing silently

issue is to figure out why extraction is failing silently

You are correct 🙂: we can create an issue out of this Slack conversation if you want.

one it will not work in air-gapped env

The question is not wether you should embed or not the container images in the harvester ISO: of course you should embed all the needed container images in the Harvester ISO file. The question is to allow people to audit and review the container images that are embedded in the ISO. This is exactly what is done on the k3s github: there are files called "k3s-airgap-images-amd64.tar*", which anyone can review and audit and then use in their own k3s airgap installation 😊. The image "*harvester-cluster-repo*" is nowhere to be found, exept in the harvester ISO. We do not know where and how the ISO is fabricated. We do not know where and how the container image is fabricated. Does this mean it is a rancher vendor/proprietary container image, and not an open source container image? Are there other container images that follows the same behaviour (for RKE2)? What is the licensing behind Harvester?

[edit link] Thank you, this helps 🙂. Can you confirm that this image

rancher/harvester-cluster-repo:v1.1.1

corresponds to this Containerfile? • https://github.com/harvester/harvester-installer/blob/c75e7922ac789c8bd725b894395219806da0e689/package/harvester-repo/Dockerfile • https://github.com/harvester/harvester-installer/commit/9d237eb9caa816b69b75c7d55d60fc0f6683a246