https://rancher.com/ logo
Powered by
Title
a

astonishing-engine-86351

04/12/2023, 3:13 PM
Looks like the keypair annotation is there, but the ImportKeyPair isn't created 😞
This code in digitalocean_interface is not doing anything 
if _, err = controllerutil.CreateOrUpdate(ctx, r.Client, keyPair, func() error {
		keyPair.Spec.PublicKey = pubKey
		keyPair.Spec.Secret = credSecret

		if err := controllerutil.SetControllerReference(vm, keyPair, r.Scheme); err != nil {
			r.Log.Error(err, "unable to set ownerReference for DO keypair")
			return err
		}
		return nil
	}); err != nil {
		return status, err
	}
If I check kubectl I get no importkeypairs
@worried-fountain-60974 is this at all related to the status changes you had to fix on another object?
They importkeypair is generated and the create or update call made, but nothing happens
w

worried-fountain-60974

04/12/2023, 3:58 PM
I suspect you are right.
a

astonishing-engine-86351

04/12/2023, 3:59 PM
So it's creating the keypair. But it's now failing at the liveness check
w

worried-fountain-60974

04/12/2023, 3:59 PM
I have that code up, let me add the status update bits and cut an RC for you
a

astonishing-engine-86351

04/12/2023, 3:59 PM
vm.Spec.Keypair is empty
w

worried-fountain-60974

04/12/2023, 4:00 PM
hrm that shouldn't be part of it
a

astonishing-engine-86351

04/12/2023, 4:00 PM
I'm running it all in goland right now since I can't make heads or tails of it
That's gotten past this earlier stuff, which is probably what you're needing the rc for, but I have this other issue now 😞
w

worried-fountain-60974

04/12/2023, 4:01 PM
lemme see if I can repro so I can help most effectively.
a

astonishing-engine-86351

04/12/2023, 4:01 PM
But I can't find where vm.Spec.KeyPair is getting written
But the secret exists scheduled-c29efa32-ad2ce76d-secret Opaque 2 5m20s
w

worried-fountain-60974

04/12/2023, 4:06 PM
What version of hf-shim-operator are you using?
a

astonishing-engine-86351

04/12/2023, 4:06 PM
Let me pull latest
w

worried-fountain-60974

04/12/2023, 4:06 PM
hold up
a

astonishing-engine-86351

04/12/2023, 4:07 PM
I'm using the code I cloned, but haven't pulled in a while
w

worried-fountain-60974

04/12/2023, 4:07 PM
I just tagged 
v0.2.0
it should be releasing now
I think that will help
a

astonishing-engine-86351

04/12/2023, 4:08 PM
ok
What's my best path for deploying droplet-operator. Doing this right now export IMG=hobbyfarm/droplet-operator:v0.1.3 make all make install make deploy
w

worried-fountain-60974

04/12/2023, 4:10 PM
For the meantime, that is probably best. i haven't gotten to getting the chart releasing working on that repo yet
a

astonishing-engine-86351

04/12/2023, 4:10 PM
let me know when hf-shim is released 🙂
w

worried-fountain-60974

04/12/2023, 4:43 PM
Done
a

astonishing-engine-86351

04/12/2023, 4:45 PM
This is all i'm getting from droplet-operator manager E0412 16:44:48.176504 1 leaderelection.go:325] error retrieving resource lock hobbyfarm/1036face.cattle.io: leases.coordination.k8s.io "1036face.cattle.io" is forbidden: User "system😒erviceaccount:hobbyfarm:default" cannot get │ │ manager E0412 16:44:51.474561 1 leaderelection.go:325] error retrieving resource lock hobbyfarm/1036face.cattle.io: leases.coordination.k8s.io "1036face.cattle.io" is forbidden: User "system😒erviceaccount:hobbyfarm:default" cannot get │ │ manager E0412 16:44:55.794358 1 leaderelection.go:325] error retrieving resource lock hobbyfarm/1036face.cattle.io: leases.coordination.k8s.io "1036face.cattle.io" is forbidden: User "system😒erviceaccount:hobbyfarm:default" cannot get │ │ manager E0412 16:44:59.717700 1 leaderelection.go:325] error retrieving resource lock hobbyfarm/1036face.cattle.io: leases.coordination.k8s.io "1036face.cattle.io" is forbidden: User "system😒erviceaccount:hobbyfarm:default" cannot get │ │ manager E0412 16:45:01.999148 1 leaderelection.go:325] error retrieving resource lock hobbyfarm/1036face.cattle.io: leases.coordination.k8s.io "1036face.cattle.io" is forbidden: User "system😒erviceaccount:hobbyfarm:default" cannot get │ │
I'm going to try running it manuallly
Shim is stuck here 
<http://gs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|gs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>                                                                                                                                                          │
│     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266                                                                                                                                                  │
│ <http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>                                                                                                                                                                │
│     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227                                                                                                                                                  │
│ 1.6813179594285004e+09    ERROR    controller.virtualmachine    Reconciler error    {"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "scheduled-bfd7aa8e-0cb0050a", "namespace": "hobbyfarm", "error": "unable t │
│ <http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>                                                                                                                                                          │
│     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266                                                                                                                                                  │
│ <http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>                                                                                                                                                                │
│     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227                                                                                                                                                  │
│
Guessing it's because droplet operator is not doing its job
@worried-fountain-60974 Looks like no annotations are being created, or they're being lost at createSecret
jason in ~/src/github.com/jasonvanbrackel/introduction-to-kubernetes on main ● ● 🦄 kubectl get virtualmachine -n hobbyfarm -o yaml
apiVersion: v1
items:
- apiVersion: <http://hobbyfarm.io/v1|hobbyfarm.io/v1>
  kind: VirtualMachine
  metadata:
    creationTimestamp: "2023-04-12T16:49:47Z"
    finalizers:
    - <http://finalizer.hobbyfarm.io/vmset|finalizer.hobbyfarm.io/vmset>
    generation: 2
    labels:
      bound: "false"
      dynamic: "false"
      environment: env-digitalocean-external
      <http://hobbyfarm.io/scheduledevent|hobbyfarm.io/scheduledevent>: se-se-h7jwpyinkh
      ready: "false"
      restrictedbind: "true"
      restrictedbindvalue: se-se-h7jwpyinkh
      template: vmt-digitalocean-small
      vmset: se-se-se-h7jwpyinkh-vms-scheduled-bfd7aa8e
    name: scheduled-bfd7aa8e-cd78b87d
    namespace: hobbyfarm
    ownerReferences:
    - apiVersion: <http://hobbyfarm.io/v1|hobbyfarm.io/v1>
      kind: VirtualMachineSet
      name: se-se-se-h7jwpyinkh-vms-scheduled-bfd7aa8e
      uid: e190b445-7ee1-44df-a914-16df920972b7
    resourceVersion: "2101"
    uid: cb7d26b4-9c96-4da0-b017-17203b340f59
  spec:
    id: scheduled-bfd7aa8e-cd78b87d
    provision: false
    ssh_username: ""
    user: ""
    vm_claim_id: ""
    vm_set_id: se-se-se-h7jwpyinkh-vms-scheduled-bfd7aa8e
    vm_template_id: vmt-digitalocean-small
  status:
    allocated: false
    environment_id: env-digitalocean-external
    hostname: ""
    private_ip: ""
    public_ip: ""
    status: SecretCreated
    tainted: false
    ws_endpoint: <http://shell.127.0.0.1.nip.io|shell.127.0.0.1.nip.io>
- apiVersion: <http://hobbyfarm.io/v1|hobbyfarm.io/v1>
  kind: VirtualMachine
  metadata:
    creationTimestamp: "2023-04-12T17:15:51Z"
    finalizers:
    - <http://finalizer.hobbyfarm.io/vmset|finalizer.hobbyfarm.io/vmset>
    generation: 2
    labels:
      bound: "false"
      dynamic: "false"
      environment: env-digitalocean-external
      <http://hobbyfarm.io/scheduledevent|hobbyfarm.io/scheduledevent>: se-se-h7jwpyinkh
      ready: "false"
      restrictedbind: "true"
      restrictedbindvalue: se-se-h7jwpyinkh
      template: vmt-digitalocean-small
      vmset: se-se-se-h7jwpyinkh-vms-scheduled-bfd7aa8e
    name: scheduled-bfd7aa8e-75b5b2d8
    namespace: hobbyfarm
    ownerReferences:
    - apiVersion: <http://hobbyfarm.io/v1|hobbyfarm.io/v1>
      kind: VirtualMachineSet
      name: se-se-se-h7jwpyinkh-vms-scheduled-bfd7aa8e
      uid: e190b445-7ee1-44df-a914-16df920972b7
    resourceVersion: "3971"
    uid: 177cc4c7-2d1f-47de-9e23-9ca9853f87bd
  spec:
    id: scheduled-bfd7aa8e-75b5b2d8
    provision: false
    ssh_username: ""
    user: ""
    vm_claim_id: ""
    vm_set_id: se-se-se-h7jwpyinkh-vms-scheduled-bfd7aa8e
    vm_template_id: vmt-digitalocean-small
  status:
    allocated: false
    environment_id: env-digitalocean-external
    hostname: ""
    private_ip: ""
    public_ip: ""
    status: SecretCreated
    tainted: false
    ws_endpoint: <http://shell.127.0.0.1.nip.io|shell.127.0.0.1.nip.io>
kind: List
metadata:
  resourceVersion: ""
Getting this from the shim 
│ 1.681320840714841e+09    ERROR    controller.virtualmachine    Reconciler error    {"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "dynamic-pediu5imwf-7e0db18a", "namespace": "hobbyfarm", "error": "Operation │
│  cannot be fulfilled on <http://virtualmachines.hobbyfarm.io|virtualmachines.hobbyfarm.io> \"dynamic-pediu5imwf-7e0db18a\": the object has been modified; please apply your changes to the latest version and try again"}                                                               │
│ <http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>                                                                                                                                                          │
│     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266                                                                                                                                                  │
│ <http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>                                                                                                                                                                │
│     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
maybe that update order is off or something?
w

worried-fountain-60974

04/12/2023, 5:35 PM
I get that sometimes but it goes away. I can look again
a

astonishing-engine-86351

04/12/2023, 5:37 PM
Stopped the scenario and restarted. I'm getting virutalmachine objects with no annotations on them
image.png
That error did end up going away, the version one. So the annotation issue is where I'm stuck at the moment.
w

worried-fountain-60974

04/12/2023, 5:59 PM
Oh fsck I think I know what it is
a

astonishing-engine-86351

04/12/2023, 6:08 PM
I'm all 👂s 🙂. Been long enough since I wrote controller logic, that I'm not seeing it.
w

worried-fountain-60974

04/12/2023, 6:09 PM
https://github.com/hobbyfarm/hf-shim-operator/pull/18
That explains it
a

astonishing-engine-86351

04/12/2023, 6:10 PM
Oh that's nasty
w

worried-fountain-60974

04/12/2023, 6:10 PM
Yeah it's just stupid update logic
if we weren't using a status field as a status subresource we wouldn't have this problem because you can update the whole resource in one go
a

astonishing-engine-86351

04/12/2023, 6:10 PM
That's a bad side-effect, if the intent is just update status
w

worried-fountain-60974

04/12/2023, 6:10 PM
but I would rather do things the right way
yeah. it's fixable with the DeepCopy()
since we don't care about the returned value from a status update just to persist the annotations on the next line
a

astonishing-engine-86351

04/12/2023, 6:11 PM
But if you update the resource the status subresource is unaffected?
w

worried-fountain-60974

04/12/2023, 6:11 PM
Correct.
In a properly defined CRD with Status as a "subresource", updates of the main resource will not affect the status subresource (and vice versa).
Hence you need two calls to fully update the resource. And those calls have to be managed carefully because of the overwrite of the obj reference in the code as you see in that PR
a

astonishing-engine-86351

04/12/2023, 6:13 PM
Yep, I see it now. I had completely forgotten about that behavior
Let me know when the next version is available. I'll give it a test.
w

worried-fountain-60974

04/12/2023, 6:14 PM
once ci finishes i'll cut a v0.2.1
a

astonishing-engine-86351

04/12/2023, 6:14 PM
👍
w

worried-fountain-60974

04/12/2023, 6:20 PM
v0.2.1 building now, give it 10 for docker buildx to choke
then it'll be out
a

astonishing-engine-86351

04/12/2023, 6:23 PM
I appreciate the help. Thank you
If this works. I'll set a class up and give it a test. Then I'll start sending some docs PRs. I also have a couple small changes to make to the droplet operator based on digital ocean api changes. Remove a deprecated field and add a couple new ones
w

worried-fountain-60974

04/12/2023, 6:24 PM
TIA for all that. Contributions like that are sorely needed
Help bandaid some of these thousand cuts we have 😄
a

astonishing-engine-86351

04/12/2023, 6:25 PM
Life of an OSS project 🙂
Name:         dynamic-itkjiikr7x-81612630
Namespace:    hobbyfarm
Labels:       bound=true
              dynamic=true
              environment=env-digitalocean-external
              <http://hobbyfarm.io/provisioner=external|hobbyfarm.io/provisioner=external>
              <http://hobbyfarm.io/scheduledevent=se-se-nkdip3e5ji|hobbyfarm.io/scheduledevent=se-se-nkdip3e5ji>
              <http://hobbyfarm.io/vmtemplate=vmt-digitalocean-small|hobbyfarm.io/vmtemplate=vmt-digitalocean-small>
              ready=false
              restrictedbind=true
              restrictedbindvalue=se-se-nkdip3e5ji
              template=vmt-digitalocean-small
              vmc=dynamic-itkjiikr7x
Annotations:  <none>
@worried-fountain-60974 Still blank
Secret is there
w

worried-fountain-60974

04/12/2023, 7:06 PM
Is this the same object you were working with before?
a

astonishing-engine-86351

04/12/2023, 7:07 PM
New install
Yes virutal machine
w

worried-fountain-60974

04/12/2023, 7:08 PM
hmmm
Can you share with me the manifest of the ImportKeyPair and VirtualMachine
a

astonishing-engine-86351

04/12/2023, 7:32 PM
The Import Key Pair is never created. Here's the VM
jason in ~/src/github.com/jasonvanbrackel/introduction-to-kubernetes on main ● ● 🦄 kubectl get virtualmachine -n hobbyfarm dynamic-itkjiikr7x-81612630 -o yaml
apiVersion: <http://hobbyfarm.io/v1|hobbyfarm.io/v1>
kind: VirtualMachine
metadata:
  creationTimestamp: "2023-04-12T19:03:01Z"
  generation: 1
  labels:
    bound: "true"
    dynamic: "true"
    environment: env-digitalocean-external
    <http://hobbyfarm.io/provisioner|hobbyfarm.io/provisioner>: external
    <http://hobbyfarm.io/scheduledevent|hobbyfarm.io/scheduledevent>: se-se-nkdip3e5ji
    <http://hobbyfarm.io/vmtemplate|hobbyfarm.io/vmtemplate>: vmt-digitalocean-small
    ready: "false"
    restrictedbind: "true"
    restrictedbindvalue: se-se-nkdip3e5ji
    template: vmt-digitalocean-small
    vmc: dynamic-itkjiikr7x
  name: dynamic-itkjiikr7x-81612630
  namespace: hobbyfarm
  ownerReferences:
  - apiVersion: <http://hobbyfarm.io/v1|hobbyfarm.io/v1>
    kind: VirtualMachineClaim
    name: dynamic-itkjiikr7x
    uid: 1288ed1c-679f-4060-9cc2-360104a9967d
  resourceVersion: "1214"
  uid: 9e63b6f5-e4bc-47c0-a726-0e6b90b129c6
spec:
  id: dynamic-itkjiikr7x-81612630
  protocol: ssh
  provision: false
  secret_name: ""
  ssh_username: ""
  user: admin
  vm_claim_id: dynamic-itkjiikr7x
  vm_set_id: ""
  vm_template_id: vmt-digitalocean-small
status:
  allocated: true
  environment_id: env-digitalocean-external
  hostname: ""
  private_ip: ""
  public_ip: ""
  status: SecretCreated
  tainted: false
  ws_endpoint: <http://shell.127.0.0.1.nip.io|shell.127.0.0.1.nip.io>
jason in ~/src/github.com/jasonvanbrackel/introduction-to-kubernetes on main ● ● 🦄 kubectl get secrets -n hobbyfarm
NAME                                     TYPE                                  DATA   AGE
hobbyfarm-webhook-secret                 Opaque                                3      31m
hobbyfarm-gargantua-token-5wgrg          <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token>   3      31m
default-token-vzsds                      <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token>   3      31m
sh.helm.release.v1.hobbyfarm.v1          <http://helm.sh/release.v1|helm.sh/release.v1>                    1      31m
hf-shim-operator-token-2gxgs             <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token>   3      31m
sh.helm.release.v1.hf-shim-operator.v1   <http://helm.sh/release.v1|helm.sh/release.v1>                    1      31m
ec2-operator-token-tgx4j                 <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token>   3      30m
sh.helm.release.v1.ec2-operator.v1       <http://helm.sh/release.v1|helm.sh/release.v1>                    1      30m
digitalocean-secret                      Opaque                                1      30m
dynamic-itkjiikr7x-81612630-secret       Opaque                                2      29m
Droplet operator is also not happy, but I have no idea if that's a problem 
kube-rbac-proxy I0412 19:01:38.527632       1 main.go:186] Valid token audiences:                                                                                                                                                                 │
│ kube-rbac-proxy I0412 19:01:38.530212       1 main.go:232] Generating self signed cert as no cert is provided                                                                                                                                     │
│ kube-rbac-proxy I0412 19:01:41.478365       1 main.go:281] Starting TCP socket on 0.0.0.0:8443                                                                                                                                                    │
│ kube-rbac-proxy I0412 19:01:41.482841       1 main.go:288] Listening securely on 0.0.0.0:8443                                                                                                                                                     │
│ manager I0412 19:01:55.504714       1 request.go:655] Throttling request took 1.101529s, request: GET:<https://10.43.0.1:443/apis/autoscaling/v2beta2?timeout=32s>                                                                                  │
│ manager 2023-04-12T19:02:01.506Z    INFO    controller-runtime.metrics    metrics server is starting to listen    {"addr": "127.0.0.1:8080"}                                                                                                      │
│ manager 2023-04-12T19:02:01.905Z    INFO    setup    starting manager                                                                                                                                                                             │
│ manager I0412 19:02:02.006167       1 leaderelection.go:243] attempting to acquire leader lease hobbyfarm/1036face.cattle.io...                                                                                                                   │
│ manager 2023-04-12T19:02:02.121Z    INFO    controller-runtime.manager    starting metrics server    {"path": "/metrics"}                                                                                                                         │
│ manager E0412 19:02:04.008325       1 leaderelection.go:329] error initially creating leader election record: <http://leases.coordination.k8s.io|leases.coordination.k8s.io> is forbidden: User "system:serviceaccount:hobbyfarm:default" cannot create resource "leases" in API grou │
│ p "<http://coordination.k8s.io|coordination.k8s.io>" in the namespace "hobbyfarm"                                                                                                                                                                                              │
│ manager E0412 19:02:07.608492       1 leaderelection.go:325] error retrieving resource lock hobbyfarm/1036face.cattle.io: <http://leases.coordination.k8s.io|leases.coordination.k8s.io> "<http://1036face.cattle.io|1036face.cattle.io>" is forbidden: User "system:serviceaccount:hobbyfarm:default" cannot get │
│  resource "leases" in API group "<http://coordination.k8s.io|coordination.k8s.io>" in the namespace "hobbyfarm"                                                                                                                                                                │
│ manager E0412 19:02:11.910931       1 leaderelection.go:325] error retrieving resource lock hobbyfarm/1036face.cattle.io: <http://leases.coordination.k8s.io|leases.coordination.k8s.io> "<http://1036face.cattle.io|1036face.cattle.io>" is forbidden: User "system:serviceaccount:hobbyfarm:default" cannot get │
│  resource "leases" in API group "<http://coordination.k8s.io|coordination.k8s.io>" in the namespace "hobbyfarm"                                                                                                                                                                │
│ manager E0412 19:02:15.523777       1 leaderelection.go:325] error retrieving resource lock hobbyfarm/1036face.cattle.io: <http://leases.coordination.k8s.io|leases.coordination.k8s.io> "<http://1036face.cattle.io|1036face.cattle.io>" is forbidden: User "system:serviceaccount:hobbyfarm:default" cannot get │
│  resource "leases" in API group "<http://coordination.k8s.io|coordination.k8s.io>" in the namespace "hobbyfarm"                                                                                                                                                                │
│ manager E0412 19:02:18.612734       1 leaderelection.go:325] error retrieving resource lock hobbyfarm/1036face.cattle.io: <http://leases.coordination.k8s.io|leases.coordination.k8s.io> "<http://1036face.cattle.io|1036face.cattle.io>" is forbidden: User "system:serviceaccount:hobbyfarm:default" cannot get │
│  resource "leases" in API group "<http://coordination.k8s.io|coordination.k8s.io>" in the namespace "hobbyfarm"                                                                                                                                                                │
│ manager E0412 19:02:21.907572       1 leaderelection.go:325] error retrieving resource lock hobbyfarm/1036face.cattle.io: <http://leases.coordination.k8s.io|leases.coordination.k8s.io> "<http://1036face.cattle.io|1036face.cattle.io>" is forbidden: User "system:serviceaccount:hobbyfarm:default" cannot get │
│  resource "leases" in API group "<http://coordination.k8s.io|coordination.k8s.io>" in the namespace "hobbyfarm"                                                                                                                                                                │
│ manager E0412 19:02:25.609285       1 leaderelection.go:325] error retrieving resource lock hobbyfarm/1036face.cattle.io: <http://leases.coordination.k8s.io|leases.coordination.k8s.io> "<http://1036face.cattle.io|1036face.cattle.io>" is forbidden: User "system:serviceaccount:hobbyfarm:default" cannot get │
│  resource "leases" in API group "<http://coordination.k8s.io|coordination.k8s.io>" in the namespace "hobbyfarm"                                                                                                                                                                │
│ manager E0412 19:02:27.814251       1 leaderelection.go:325] error retrieving resource lock hobbyfarm/1036face.cattle.io: <http://leases.coordination.k8s.io|leases.coordination.k8s.io> "<http://1036face.cattle.io|1036face.cattle.io>" is forbidden: User "system:serviceaccount:hobbyfarm:default" cannot get │
│  resource "leases" in API group "<http://coordination.k8s.io|coordination.k8s.io>" in the namespace "hobbyfarm"                                                                                                                                                                │
│ manager E0412 19:02:30.223264       1 leaderelection.go:325] error retrieving resource lock hobbyfarm/1036face.cattle.io: <http://leases.coordination.k8s.io|leases.coordination.k8s.io> "<http://1036face.cattle.io|1036face.cattle.io>" is forbidden: User "system:serviceaccount:hobbyfarm:default" cannot get │
│  resource "leases" in API group "<http://coordination.k8s.io|coordination.k8s.io>" in the namespace "hobbyfarm"                                                                                                                                                                │
│ manager E0412 19:02:32.489466       1 leaderelection.go:325] error retrieving resource lock hobbyfarm/1036face.cattle.io: <http://leases.coordination.k8s.io|leases.coordination.k8s.io> "<http://1036face.cattle.io|1036face.cattle.io>" is forbidden: User "system:serviceaccount:hobbyfarm:default" cannot get │
│  resource "leases" in API group "<http://coordination.k8s.io|coordination.k8s.io>" in the namespace "hobbyfarm"                                                                                                                                                                │
│ manager E0412 19:02:35.223782       1 leaderelection.go:325] error retrieving resource lock hobbyfarm/1036face.cattle.io: <http://leases.coordination.k8s.io|leases.coordination.k8s.io> "<http://1036face.cattle.io|1036face.cattle.io>" is forbidden: User "system:serviceaccount:hobbyfarm:default" cannot get │
│  resource "leases" in API group "<http://coordination.k8s.io|coordination.k8s.io>" in the namespace "hobbyfarm"                                                                                                                                                                │
│ manager E0412 19:02:38.493384       1 leaderelection.go:325] error retrieving resource lock hobbyfarm/1036face.cattle.io: <http://leases.coordination.k8s.io|leases.coordination.k8s.io> "<http://1036face.cattle.io|1036face.cattle.io>" is forbidden: User "system:serviceaccount:hobbyfarm:default" cannot get │
│  resource "leases" in API group "<http://coordination.k8s.io|coordination.k8s.io>" in the namespace "hobbyfarm"
w

worried-fountain-60974

04/12/2023, 7:36 PM
The ImportKeyPair is never created
That's because of the above issue. It's a perms thing
It needs to obtain a lease.
Fully expect this is a documentation issue
a

astonishing-engine-86351

04/12/2023, 7:37 PM
Does that explain the lack of annotations?
Because the former is in the shim the latter from droplet operator
w

worried-fountain-60974

04/12/2023, 7:38 PM
Yeah because the annotation for a keypair name can never be created if the ImportKeyPair never actually comes into existence.
At least that's my theory
I didn't run into this because I have been testing with an admin kubeconfig 🤦‍♂️
a

astonishing-engine-86351

04/12/2023, 7:39 PM
Am I missing a clusterrole or something?
w

worried-fountain-60974

04/12/2023, 7:39 PM
Yes but it's not your fault.
Create a Role with * on coordination.k8s.io/leases in the hobbyfarm namespace
then rolebinding, with subject of serviceaccount default in hobbyfarm namespace
a

astonishing-engine-86351

04/12/2023, 7:48 PM
Is that what this is for?
image.png
w

worried-fountain-60974

04/12/2023, 7:48 PM
that's what it used to be for, I think before leases existed? idk
This was using an older version of KubeBuilder and has been stepped up
a

astonishing-engine-86351

04/12/2023, 7:49 PM
This is being executed by the make install / make deploy so I'll add it here and see if that works
Ok that cleared that error
Going to try to create a vm
@worried-fountain-60974 Shim is still misbehaving.
apiVersion: <http://hobbyfarm.io/v1|hobbyfarm.io/v1>
kind: VirtualMachine
metadata:
  creationTimestamp: "2023-04-12T19:54:38Z"
  generation: 1
  labels:
    bound: "true"
    dynamic: "true"
    environment: env-digitalocean-external
    <http://hobbyfarm.io/provisioner|hobbyfarm.io/provisioner>: external
    <http://hobbyfarm.io/scheduledevent|hobbyfarm.io/scheduledevent>: se-se-2vyaok4txz
    <http://hobbyfarm.io/vmtemplate|hobbyfarm.io/vmtemplate>: vmt-digitalocean-small
    ready: "false"
    restrictedbind: "true"
    restrictedbindvalue: se-se-2vyaok4txz
    template: vmt-digitalocean-small
    vmc: dynamic-2wss7xwiql
  name: dynamic-2wss7xwiql-1a9b63a6
  namespace: hobbyfarm
  ownerReferences:
  - apiVersion: <http://hobbyfarm.io/v1|hobbyfarm.io/v1>
    kind: VirtualMachineClaim
    name: dynamic-2wss7xwiql
    uid: 9100dc64-2199-4c8c-9708-34307eebcac3
  resourceVersion: "1267"
  uid: 044a28b7-6639-472b-af3e-d2418b678958
spec:
  id: dynamic-2wss7xwiql-1a9b63a6
  protocol: ssh
  provision: false
  secret_name: ""
  ssh_username: ""
  user: admin
  vm_claim_id: dynamic-2wss7xwiql
  vm_set_id: ""
  vm_template_id: vmt-digitalocean-small
status:
  allocated: true
  environment_id: env-digitalocean-external
  hostname: ""
  private_ip: ""
  public_ip: ""
  status: SecretCreated
  tainted: false
  ws_endpoint: <http://shell.127.0.0.1.nip.io|shell.127.0.0.1.nip.io>
w

worried-fountain-60974

04/12/2023, 7:57 PM
oh ffs
any errors?
a

astonishing-engine-86351

04/12/2023, 7:59 PM
jason in ~/src/github.com/jasonvanbrackel/introduction-to-kubernetes on main ● ● 🦄 kubectl -n hobbyfarm logs hf-shim-operator-b94f64597-fhpn8
I0412 19:52:47.858847       1 request.go:665] Waited for 1.04282575s due to client-side throttling, not priority and fairness, request: GET:<https://10.43.0.1:443/apis/discovery.k8s.io/v1?timeout=32s>
1.6813291686296883e+09	INFO	controller-runtime.metrics	Metrics server is starting to listen	{"addr": ":8080"}
1.6813291686418562e+09	INFO	setup	starting manager
1.6813291686505077e+09	INFO	Starting server	{"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
1.6813291686545818e+09	INFO	controller.virtualmachine	Starting EventSource	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "source": "kind source: *v1.VirtualMachine"}
1.6813291686550047e+09	INFO	controller.virtualmachine	Starting EventSource	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "source": "kind source: *v1alpha1.Instance"}
1.6813291686550746e+09	INFO	controller.virtualmachine	Starting EventSource	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "source": "kind source: *v1alpha1.ImportKeyPair"}
1.6813291686551135e+09	INFO	controller.virtualmachine	Starting EventSource	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "source": "kind source: *v1alpha1.Instance"}
1.6813291686551623e+09	INFO	controller.virtualmachine	Starting EventSource	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "source": "kind source: *v1alpha1.ImportKeyPair"}
1.6813291686552024e+09	INFO	controller.virtualmachine	Starting EventSource	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "source": "kind source: *v1.Secret"}
1.6813291686553493e+09	INFO	controller.virtualmachine	Starting Controller	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine"}
1.6813291725122058e+09	ERROR	controller-runtime.source	if kind is a CRD, it should be installed before calling Start	{"kind": "<http://ImportKeyPair.ec2.cattle.io|ImportKeyPair.ec2.cattle.io>", "error": "no matches for kind \"ImportKeyPair\" in version \"<http://ec2.cattle.io/v1alpha1\|ec2.cattle.io/v1alpha1\>""}
<http://sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1.1|sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1.1>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/source/source.go:137
<http://k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext|k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext>
	/go/pkg/mod/k8s.io/apimachinery@v0.23.0/pkg/util/wait/wait.go:233
<http://k8s.io/apimachinery/pkg/util/wait.poll|k8s.io/apimachinery/pkg/util/wait.poll>
	/go/pkg/mod/k8s.io/apimachinery@v0.23.0/pkg/util/wait/wait.go:580
<http://k8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext|k8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext>
	/go/pkg/mod/k8s.io/apimachinery@v0.23.0/pkg/util/wait/wait.go:545
<http://sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1|sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/source/source.go:131
1.681329176518322e+09	ERROR	controller-runtime.source	if kind is a CRD, it should be installed before calling Start	{"kind": "<http://Instance.droplet.cattle.io|Instance.droplet.cattle.io>", "error": "no matches for kind \"Instance\" in version \"<http://droplet.cattle.io/v1alpha1\|droplet.cattle.io/v1alpha1\>""}
<http://sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1.1|sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1.1>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/source/source.go:137
<http://k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext|k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext>
	/go/pkg/mod/k8s.io/apimachinery@v0.23.0/pkg/util/wait/wait.go:233
<http://k8s.io/apimachinery/pkg/util/wait.poll|k8s.io/apimachinery/pkg/util/wait.poll>
	/go/pkg/mod/k8s.io/apimachinery@v0.23.0/pkg/util/wait/wait.go:580
<http://k8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext|k8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext>
	/go/pkg/mod/k8s.io/apimachinery@v0.23.0/pkg/util/wait/wait.go:545
<http://sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1|sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/source/source.go:131
I0412 19:52:57.913400       1 request.go:665] Waited for 1.295655376s due to client-side throttling, not priority and fairness, request: GET:<https://10.43.0.1:443/apis/flowcontrol.apiserver.k8s.io/v1beta2?timeout=32s>
1.6813291805199664e+09	ERROR	controller-runtime.source	if kind is a CRD, it should be installed before calling Start	{"kind": "<http://ImportKeyPair.droplet.cattle.io|ImportKeyPair.droplet.cattle.io>", "error": "no matches for kind \"ImportKeyPair\" in version \"<http://droplet.cattle.io/v1alpha1\|droplet.cattle.io/v1alpha1\>""}
<http://sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1.1|sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1.1>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/source/source.go:137
<http://k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext|k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext>
	/go/pkg/mod/k8s.io/apimachinery@v0.23.0/pkg/util/wait/wait.go:233
<http://k8s.io/apimachinery/pkg/util/wait.poll|k8s.io/apimachinery/pkg/util/wait.poll>
	/go/pkg/mod/k8s.io/apimachinery@v0.23.0/pkg/util/wait/wait.go:580
<http://k8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext|k8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext>
	/go/pkg/mod/k8s.io/apimachinery@v0.23.0/pkg/util/wait/wait.go:545
<http://sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1|sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/source/source.go:131
1.6813291906229315e+09	INFO	controller.virtualmachine	Starting workers	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "worker count": 20}
1.6813292781361418e+09	ERROR	controller.virtualmachine	Reconciler error	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "dynamic-2wss7xwiql-1a9b63a6", "namespace": "hobbyfarm", "error": "Operation cannot be fulfilled on <http://virtualmachines.hobbyfarm.io|virtualmachines.hobbyfarm.io> \"dynamic-2wss7xwiql-1a9b63a6\": the object has been modified; please apply your changes to the latest version and try again"}
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
time="2023-04-12T19:54:38Z" level=info msg="creating new keypair"
1.6813292799591446e+09	ERROR	controller.virtualmachine	Reconciler error	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "dynamic-2wss7xwiql-1a9b63a6", "namespace": "hobbyfarm", "error": "Secret \"dynamic-2wss7xwiql-1a9b63a6-secret\" not found"}
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
1.6813292799725318e+09	ERROR	controller.virtualmachine	Reconciler error	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "dynamic-2wss7xwiql-1a9b63a6", "namespace": "hobbyfarm", "error": "Operation cannot be fulfilled on <http://virtualmachines.hobbyfarm.io|virtualmachines.hobbyfarm.io> \"dynamic-2wss7xwiql-1a9b63a6\": the object has been modified; please apply your changes to the latest version and try again"}
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
1.681329279973107e+09	ERROR	controller.virtualmachine	Reconciler error	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "dynamic-2wss7xwiql-1a9b63a6", "namespace": "hobbyfarm", "error": "unable to find label pubKey on VM"}
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
1.681329283138751e+09	ERROR	controller.virtualmachine	Reconciler error	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "dynamic-2wss7xwiql-1a9b63a6", "namespace": "hobbyfarm", "error": "unable to find label pubKey on VM"}
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
1.681329293146719e+09	ERROR	controller.virtualmachine	Reconciler error	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "dynamic-2wss7xwiql-1a9b63a6", "namespace": "hobbyfarm", "error": "unable to find label pubKey on VM"}
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
1.6813293031481442e+09	ERROR	controller.virtualmachine	Reconciler error	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "dynamic-2wss7xwiql-1a9b63a6", "namespace": "hobbyfarm", "error": "unable to find label pubKey on VM"}
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
1.6813293131524856e+09	ERROR	controller.virtualmachine	Reconciler error	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "dynamic-2wss7xwiql-1a9b63a6", "namespace": "hobbyfarm", "error": "unable to find label pubKey on VM"}
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
1.6813293231571817e+09	ERROR	controller.virtualmachine	Reconciler error	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "dynamic-2wss7xwiql-1a9b63a6", "namespace": "hobbyfarm", "error": "unable to find label pubKey on VM"}
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
1.6813293331615367e+09	ERROR	controller.virtualmachine	Reconciler error	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "dynamic-2wss7xwiql-1a9b63a6", "namespace": "hobbyfarm", "error": "unable to find label pubKey on VM"}
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>
That's the shim
Nothing from the droplet operator
manager I0412 19:53:23.756117       1 request.go:655] Throttling request took 1.199809s, request: GET:<https://10.43.0.1:443/apis/apiextensions.k8s.io/v1?timeout=32s>
kube-rbac-proxy I0412 19:53:09.163178       1 main.go:186] Valid token audiences:
kube-rbac-proxy I0412 19:53:09.165665       1 main.go:232] Generating self signed cert as no cert is provided
kube-rbac-proxy I0412 19:53:12.616029       1 main.go:281] Starting TCP socket on 0.0.0.0:8443
kube-rbac-proxy I0412 19:53:12.620980       1 main.go:288] Listening securely on 0.0.0.0:8443
manager 2023-04-12T19:53:28.955Z    INFO    controller-runtime.metrics    metrics server is starting to listen    {"addr": "127.0.0.1:8080"}
manager 2023-04-12T19:53:29.157Z    INFO    setup    starting manager
manager I0412 19:53:29.261324       1 leaderelection.go:243] attempting to acquire leader lease hobbyfarm/1036face.cattle.io...
manager 2023-04-12T19:53:29.653Z    INFO    controller-runtime.manager    starting metrics server    {"path": "/metrics"}
manager I0412 19:53:30.958003       1 leaderelection.go:253] successfully acquired lease hobbyfarm/1036face.cattle.io
manager 2023-04-12T19:53:30.456Z    DEBUG    controller-runtime.manager.events    Normal    {"object": {"kind":"ConfigMap","namespace":"hobbyfarm","name":"<http://1036face.cattle.io|1036face.cattle.io>","uid":"14863b17-84a1-4d58-8853-43ea048b49bb","apiVersion":"v1","resour
manager 2023-04-12T19:53:31.552Z    DEBUG    controller-runtime.manager.events    Normal    {"object": {"kind":"Lease","namespace":"hobbyfarm","name":"<http://1036face.cattle.io|1036face.cattle.io>","uid":"cdaa3afd-e9b5-499f-bf73-76f9fb9db664","apiVersion":"coordination.k8
manager 2023-04-12T19:53:31.655Z    INFO    controller-runtime.manager.controller.instance    Starting EventSource    {"reconciler group": "<http://droplet.cattle.io|droplet.cattle.io>", "reconciler kind": "Instance", "source": "kind source: /, Kind="}
manager 2023-04-12T19:53:31.654Z    INFO    controller-runtime.manager.controller.importkeypair    Starting EventSource    {"reconciler group": "<http://droplet.cattle.io|droplet.cattle.io>", "reconciler kind": "ImportKeyPair", "source": "kind source: /, Kind="}
manager 2023-04-12T19:53:36.955Z    INFO    controller-runtime.manager.controller.instance    Starting Controller    {"reconciler group": "<http://droplet.cattle.io|droplet.cattle.io>", "reconciler kind": "Instance"}
manager 2023-04-12T19:53:37.153Z    INFO    controller-runtime.manager.controller.instance    Starting workers    {"reconciler group": "<http://droplet.cattle.io|droplet.cattle.io>", "reconciler kind": "Instance", "worker count": 1}
manager 2023-04-12T19:53:37.362Z    INFO    controller-runtime.manager.controller.importkeypair    Starting Controller    {"reconciler group": "<http://droplet.cattle.io|droplet.cattle.io>", "reconciler kind": "ImportKeyPair"}
manager 2023-04-12T19:53:37.554Z    INFO    controller-runtime.manager.controller.importkeypair    Starting workers    {"reconciler group": "<http://droplet.cattle.io|droplet.cattle.io>", "reconciler kind": "ImportKeyPair", "worker count": 1}
That's the droplet operator
VM with no annotations, No ImportKey
w

worried-fountain-60974

04/12/2023, 8:00 PM
The CRD errors are weird. What is the output of 
kubectl get CRD | grep droplet
?
a

astonishing-engine-86351

04/12/2023, 8:00 PM
kubectl get customresourcedefinitions.apiextensions.k8s.io | grep droplet importkeypairs.droplet.cattle.io 2023-04-12T19:53:02Z instances.droplet.cattle.io 2023-04-12T19:53:02Z
w

worried-fountain-60974

04/12/2023, 8:01 PM
hrrm
a

astonishing-engine-86351

04/12/2023, 8:02 PM
Maybe I need to install the operators before the shim
w

worried-fountain-60974

04/12/2023, 8:02 PM
The CRDs at least need to be installed first.
a

astonishing-engine-86351

04/12/2023, 8:09 PM
No more CRD errors, still no annotations
kubectl -n hobbyfarm logs hf-shim-operator-b94f64597-pqmg8
1.6813299987394779e+09	INFO	controller-runtime.metrics	Metrics server is starting to listen	{"addr": ":8080"}
1.6813299987493336e+09	INFO	setup	starting manager
1.6813299987561092e+09	INFO	Starting server	{"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
1.6813299987610338e+09	INFO	controller.virtualmachine	Starting EventSource	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "source": "kind source: *v1.VirtualMachine"}
1.681329998761367e+09	INFO	controller.virtualmachine	Starting EventSource	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "source": "kind source: *v1alpha1.Instance"}
1.6813299987614744e+09	INFO	controller.virtualmachine	Starting EventSource	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "source": "kind source: *v1alpha1.ImportKeyPair"}
1.6813299987615116e+09	INFO	controller.virtualmachine	Starting EventSource	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "source": "kind source: *v1alpha1.Instance"}
1.6813299987615592e+09	INFO	controller.virtualmachine	Starting EventSource	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "source": "kind source: *v1alpha1.ImportKeyPair"}
1.6813299987615962e+09	INFO	controller.virtualmachine	Starting EventSource	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "source": "kind source: *v1.Secret"}
1.6813299987619522e+09	INFO	controller.virtualmachine	Starting Controller	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine"}
1.6813299988830366e+09	INFO	controller.virtualmachine	Starting workers	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "worker count": 20}
1.6813300778200343e+09	ERROR	controller.virtualmachine	Reconciler error	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "dynamic-wpk7e3mlop-0cd8059e", "namespace": "hobbyfarm", "error": "Operation cannot be fulfilled on <http://virtualmachines.hobbyfarm.io|virtualmachines.hobbyfarm.io> \"dynamic-wpk7e3mlop-0cd8059e\": the object has been modified; please apply your changes to the latest version and try again"}
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
time="2023-04-12T20:07:57Z" level=info msg="creating new keypair"
1.6813300787538097e+09	ERROR	controller.virtualmachine	Reconciler error	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "dynamic-wpk7e3mlop-0cd8059e", "namespace": "hobbyfarm", "error": "Secret \"dynamic-wpk7e3mlop-0cd8059e-secret\" not found"}
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
1.6813300787671258e+09	ERROR	controller.virtualmachine	Reconciler error	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "dynamic-wpk7e3mlop-0cd8059e", "namespace": "hobbyfarm", "error": "Operation cannot be fulfilled on <http://virtualmachines.hobbyfarm.io|virtualmachines.hobbyfarm.io> \"dynamic-wpk7e3mlop-0cd8059e\": the object has been modified; please apply your changes to the latest version and try again"}
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
1.6813300787676568e+09	ERROR	controller.virtualmachine	Reconciler error	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "dynamic-wpk7e3mlop-0cd8059e", "namespace": "hobbyfarm", "error": "unable to find label pubKey on VM"}
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
1.681330082825227e+09	ERROR	controller.virtualmachine	Reconciler error	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "dynamic-wpk7e3mlop-0cd8059e", "namespace": "hobbyfarm", "error": "unable to find label pubKey on VM"}
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
1.681330092827374e+09	ERROR	controller.virtualmachine	Reconciler error	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "dynamic-wpk7e3mlop-0cd8059e", "namespace": "hobbyfarm", "error": "unable to find label pubKey on VM"}
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
1.6813301028293977e+09	ERROR	controller.virtualmachine	Reconciler error	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "dynamic-wpk7e3mlop-0cd8059e", "namespace": "hobbyfarm", "error": "unable to find label pubKey on VM"}
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
1.6813301128345685e+09	ERROR	controller.virtualmachine	Reconciler error	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "dynamic-wpk7e3mlop-0cd8059e", "namespace": "hobbyfarm", "error": "unable to find label pubKey on VM"}
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
1.6813301228384094e+09	ERROR	controller.virtualmachine	Reconciler error	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "dynamic-wpk7e3mlop-0cd8059e", "namespace": "hobbyfarm", "error": "unable to find label pubKey on VM"}
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
kubectl -n hobbyfarm get virtualmachine dynamic-wpk7e3mlop-0cd8059e -o yaml
apiVersion: <http://hobbyfarm.io/v1|hobbyfarm.io/v1>
kind: VirtualMachine
metadata:
  creationTimestamp: "2023-04-12T20:07:57Z"
  generation: 1
  labels:
    bound: "true"
    dynamic: "true"
    environment: env-digitalocean-external
    <http://hobbyfarm.io/provisioner|hobbyfarm.io/provisioner>: external
    <http://hobbyfarm.io/scheduledevent|hobbyfarm.io/scheduledevent>: se-se-osmxawhj7w
    <http://hobbyfarm.io/vmtemplate|hobbyfarm.io/vmtemplate>: vmt-digitalocean-small
    ready: "false"
    restrictedbind: "true"
    restrictedbindvalue: se-se-osmxawhj7w
    template: vmt-digitalocean-small
    vmc: dynamic-wpk7e3mlop
  name: dynamic-wpk7e3mlop-0cd8059e
  namespace: hobbyfarm
  ownerReferences:
  - apiVersion: <http://hobbyfarm.io/v1|hobbyfarm.io/v1>
    kind: VirtualMachineClaim
    name: dynamic-wpk7e3mlop
    uid: d0dff310-06a1-407b-8bbb-52671b3350d7
  resourceVersion: "1251"
  uid: 111e451d-4deb-444c-842c-eade13599505
spec:
  id: dynamic-wpk7e3mlop-0cd8059e
  protocol: ssh
  provision: false
  secret_name: ""
  ssh_username: ""
  user: admin
  vm_claim_id: dynamic-wpk7e3mlop
  vm_set_id: ""
  vm_template_id: vmt-digitalocean-small
status:
  allocated: true
  environment_id: env-digitalocean-external
  hostname: ""
  private_ip: ""
  public_ip: ""
  status: SecretCreated
  tainted: false
  ws_endpoint: <http://shell.127.0.0.1.nip.io|shell.127.0.0.1.nip.io>
It's creating the secret
kubectl get secrets -n hobbyfarm dynamic-wpk7e3mlop-0cd8059e-secret -o yaml
apiVersion: v1
data:
  private_key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb3dJQkFBS0NBUUVBeTlZbXlFd0JULzNjMmwzYTNoWTJVNlNyaDhPSEpoejBpZG4rY2tlMTY5RUMvZzJLClBHcWZ5Vmg3UmNhaElBMWZzNEJodVFGRWRFZlovTW1jSEtVN25vUGIwa213VHZBOTUxOWdneFlDZFpOYUZ5aFoKZEJOSXlQOUhtWHl0SmRLcy91emxVWm5vOFQyTTlkdXZacWFLeTQ1WDJXK0ljMlpmZGJRSkkvTkh5bHpSem1CLwp5dzBndWRBeWhMQitkaXBMdFZqQkQwdGlUTVcybmRYNDdFRG9QRUtNemxYL1hzdDE2RmdycThydFE0SDRsamhjClE4Y0hvUExzd1FYMHRRbGtnNmdRc0xEb3VKendrK0k1M1RDMWRaRmtOVkVLMUIrb0dyenN6eG1MMm5KY1BaVVUKQlB5S2pWU2tZRUhVN3FYYU5Lcnl3R0UwVHBvRGc0M3BGR2tETVFJREFRQUJBb0lCQVFDeDBidzdkN0tkYWNWRQo2aU9KSXE5NkFHbHpJbXZBSC9wa1Y3dlZUaUd4K3NBMWlQSlRDbUVpY1FINU9QSk8xOXZvNTlKWVpuYWxTaGtzCm9XU0N5TkpkMG9IdHpVTDFXamRtcEptNVFYWWJ3OVFQNTY5Qng2RHpWMS9LMGl6cWxnTTgwTVNzanl2YlorcVUKZ0hWZmcySUplYi9CZTc3NWpSRXh5VUZBRDNkTlhHTGRUc1F1TTBsNGxuRTNpcG9jTXd6NTMxK1JETG0zMGZaNAo5dko4SnU3VVlUYitCVGorcmdmdEhyVjJzTGdrUWFqR2Zkay92NjVUdU9ZU2pUSkhoVi9GMDlKOTd5RjQ2QXU1CkVua1RrbFRRZ0Y4WmdScXFPL3g4a0d3dEN6MzJVMWF0Y21NZys0Qno1NnRTK3JRTFJxYUlQS0p5QXlCNDZoQysKK3ZhWnRMSVJBb0dCQU15T3l0UXFXSXNwMUx6S3dHY3FjaGZtMFBFcGNzdWlzR2M4U3VTeTlWWTFNeWxQWkdZWApLTGZOWXh2OFQwUUdQTHQ0Yi90cW9zZjZhajBIOERRY2pSS0dDcURiVHhORmFPaGU4eVpkYzgxWXRWYUQwY0grCm1ESm9SRmdJam5NSW5EeXlHNU1FV3QxOVZTZ1k3UGNVczFqZGJaMjJLTTBJaDRiR3ZHVVlQSDhqQW9HQkFQOFkKN1BxdWxyWGJOVkpoR2N4S1dtWGREZjV0cWE5OGhQbGNKVS9hWmFjNEhpWm54Q0NoOXU0T2RpdE1PcXIwVkZDbwpDdmpadnhRcVphNDRqMVBiQnFxK0d0am9vUXM4ZGZvZnN4NXU4Szl1R1JQZFBCQjBJc25ibTdOMzNRKy84QWtuCkdPMExLMVVmYjhXcVhwaTVkbnQ0azA5c0JrY1dkQ3BxSUJESGhPT2JBb0dBRG5SY29zZG5RRll4WVRIV2pMTzAKSWpvK00zeGhUMEJzeEs2L0FCQWhRL2s0WjBmajZVN2pvckZSdW8yR3NHRGNjOGJiNnVFckpZUnRjMmlYTWFkKwoxSHRDVHdHUVFlVmF0MmpXTjdUTTFRcmFyWVZkRWg5Q2MvVm9OdXhMM0pjVUt6bk90d0hNeGUvd1Q0Z0xYcDVJCjJoT3poanh6LzZFRnFxRnFlSjVRMEE4Q2dZQVRyQnBxWXBoZ3dkemswM1FuMnZ3SnF2aHZUSW12Yk16Y1pnekMKa2JjUkpEUy9CWFdNaHJFQ1ByOUNLNGpDc2tEbVQyODVMYUdjcm9CVTVTdTRiY1dmMlRVODBRUEtXQWxKRnJTNwpmaGExa0tvQ2JDYzdwWmtoTS94WDVNQjVmSFhJWHVsL0xrTlROeTlwY1BBR214YldBc0JwZVkwcVlrTG5HblFLCmx6aGFUUUtCZ0NTRTlBaVR2Q3VzZlpuL0EwOEtmNWJ1NXVDUjZhSkNpdEJqem9uYXBEZkpaTWxDN0VheThDTU4Ka1BTd1FXZkpGUVFDUENzT2tkVmhvUlltWHRydlFnSUROc3ZUWHdrWjMzVlIzdkZYcWpKVkdhMW5lQ0dZUXBQWQoxVEdlbFVkcFROWmR4OWlqTC9JdGpwRWYwbkllWVY4azVqVCsxeGtTZHpqUTJqVS9keHFYCi0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg==
  public_key: c3NoLXJzYSBBQUFBQjNOemFDMXljMkVBQUFBREFRQUJBQUFCQVFETDFpYklUQUZQL2R6YVhkcmVGalpUcEt1SHc0Y21IUFNKMmY1eVI3WHIwUUwrRFlvOGFwL0pXSHRGeHFFZ0RWK3pnR0c1QVVSMFI5bjh5WndjcFR1ZWc5dlNTYkJPOEQzblgyQ0RGZ0oxazFvWEtGbDBFMGpJLzBlWmZLMGwwcXorN09WUm1lanhQWXoxMjY5bXBvckxqbGZaYjRoelpsOTF0QWtqODBmS1hOSE9ZSC9MRFNDNTBES0VzSDUyS2t1MVdNRVBTMkpNeGJhZDFmanNRT2c4UW96T1ZmOWV5M1hvV0N1cnl1MURnZmlXT0Z4RHh3ZWc4dXpCQmZTMUNXU0RxQkN3c09pNG5QQ1Q0am5kTUxWMWtXUTFVUXJVSDZnYXZPelBHWXZhY2x3OWxSUUUvSXFOVktSZ1FkVHVwZG8wcXZMQVlUUk9tZ09EamVrVWFRTXgK
kind: Secret
metadata:
  creationTimestamp: "2023-04-12T20:07:58Z"
  name: dynamic-wpk7e3mlop-0cd8059e-secret
  namespace: hobbyfarm
  ownerReferences:
  - apiVersion: <http://hobbyfarm.io/v1|hobbyfarm.io/v1>
    blockOwnerDeletion: true
    controller: true
    kind: VirtualMachine
    name: dynamic-wpk7e3mlop-0cd8059e
    uid: 111e451d-4deb-444c-842c-eade13599505
  resourceVersion: "1250"
  uid: 664705a6-624f-4d84-8682-4ec685a73342
type: Opaque
But not an ImportKeyPair 
jason in ~/src/github.com/hobbyfarm/hf-shim-operator on master 🦄 kubectl get -n hobbyfarm <http://importkeypairs.droplet.cattle.io|importkeypairs.droplet.cattle.io>
No resources found in hobbyfarm namespace.
w

worried-fountain-60974

04/12/2023, 8:11 PM
any drop[let operator logs?
droplet operator is what creates the ImportKeyPair
a

astonishing-engine-86351

04/12/2023, 8:12 PM
kubectl -n hobbyfarm logs droplet-operator-controller-manager-655b5df87c-5zfbm
Defaulted container "kube-rbac-proxy" out of: kube-rbac-proxy, manager
I0412 20:05:56.990889       1 main.go:186] Valid token audiences:
I0412 20:05:56.993337       1 main.go:232] Generating self signed cert as no cert is provided
I0412 20:05:58.256049       1 main.go:281] Starting TCP socket on 0.0.0.0:8443
I0412 20:05:58.260104       1 main.go:288] Listening securely on 0.0.0.0:8443
w

worried-fountain-60974

04/12/2023, 8:12 PM
-c manager
a

astonishing-engine-86351

04/12/2023, 8:12 PM
manager 2023-04-12T20:06:15.854Z    INFO    controller-runtime.metrics    metrics server is starting to listen    {"addr": "127.0.0.1:8080"}
manager 2023-04-12T20:06:16.052Z    INFO    setup    starting manager
manager I0412 20:06:16.169304       1 leaderelection.go:243] attempting to acquire leader lease hobbyfarm/1036face.cattle.io...
kube-rbac-proxy I0412 20:05:56.990889       1 main.go:186] Valid token audiences:
kube-rbac-proxy I0412 20:05:56.993337       1 main.go:232] Generating self signed cert as no cert is provided
kube-rbac-proxy I0412 20:05:58.256049       1 main.go:281] Starting TCP socket on 0.0.0.0:8443
kube-rbac-proxy I0412 20:05:58.260104       1 main.go:288] Listening securely on 0.0.0.0:8443
manager 2023-04-12T20:06:16.251Z    INFO    controller-runtime.manager    starting metrics server    {"path": "/metrics"}
manager I0412 20:06:21.158797       1 leaderelection.go:253] successfully acquired lease hobbyfarm/1036face.cattle.io
manager 2023-04-12T20:06:21.352Z    DEBUG    controller-runtime.manager.events    Normal    {"object": {"kind":"ConfigMap","namespace":"hobbyfarm","name":"<http://1036face.cattle.io|1036face.cattle.io>","uid":"fe0f1d2e-adef-4549-8908-2897452315c0","apiVersion":"v1","resour
manager 2023-04-12T20:06:22.063Z    DEBUG    controller-runtime.manager.events    Normal    {"object": {"kind":"Lease","namespace":"hobbyfarm","name":"<http://1036face.cattle.io|1036face.cattle.io>","uid":"06e5b002-a6b1-4b91-8109-6a02f0946c85","apiVersion":"coordination.k8
manager 2023-04-12T20:06:22.256Z    INFO    controller-runtime.manager.controller.instance    Starting EventSource    {"reconciler group": "<http://droplet.cattle.io|droplet.cattle.io>", "reconciler kind": "Instance", "source": "kind source: /, Kind="}
manager 2023-04-12T20:06:22.851Z    INFO    controller-runtime.manager.controller.importkeypair    Starting EventSource    {"reconciler group": "<http://droplet.cattle.io|droplet.cattle.io>", "reconciler kind": "ImportKeyPair", "source": "kind source: /, Kind="}
manager 2023-04-12T20:06:24.655Z    INFO    controller-runtime.manager.controller.importkeypair    Starting Controller    {"reconciler group": "<http://droplet.cattle.io|droplet.cattle.io>", "reconciler kind": "ImportKeyPair"}
manager 2023-04-12T20:06:25.154Z    INFO    controller-runtime.manager.controller.instance    Starting Controller    {"reconciler group": "<http://droplet.cattle.io|droplet.cattle.io>", "reconciler kind": "Instance"}
manager 2023-04-12T20:06:25.253Z    INFO    controller-runtime.manager.controller.importkeypair    Starting workers    {"reconciler group": "<http://droplet.cattle.io|droplet.cattle.io>", "reconciler kind": "ImportKeyPair", "worker count": 1}
manager 2023-04-12T20:06:25.253Z    INFO    controller-runtime.manager.controller.instance    Starting workers    {"reconciler group": "<http://droplet.cattle.io|droplet.cattle.io>", "reconciler kind": "Instance", "worker count": 1}
Error is being thrown in the VirtualMachineController func (r *VirtualMachineReconciler) createImportKeyPair(ctx context.Context, vm *hfv1.VirtualMachine) (status *hfv1.VirtualMachineStatus, err error) { status = vm.Status.DeepCopy() b64PubKey, ok := vm.Annotations["pubKey"] if !ok { return status, fmt.Errorf("unable to find label pubKey on VM") } pubKeyByte, err := b64.StdEncoding.DecodeString(b64PubKey) if err != nil { return status, err }
So it's doing createSecret, but can't do createImportKeyPair
w

worried-fountain-60974

04/12/2023, 8:16 PM
I was wrong its the shim that creates the keypair.
a

astonishing-engine-86351

04/12/2023, 8:28 PM
All the other calls use CreateOrUpdate. Why does VirtualMachine controller only do update?
Nevermind, for the reconciler they all use update for things for that controller's object
CreateOrUpdate is for the Secret
I'm back
w

worried-fountain-60974

04/12/2023, 10:00 PM
I opened a pr in shim repo for the fix for this.
I didn’t tag it yet and I’m out of pocket for the Next few hrs
a

astonishing-engine-86351

04/12/2023, 10:15 PM
I'll give a go local and report back, and test once you have the official out
👍 1
It provisions now but fails the liveness check
// DO liveness check
func (r *VirtualMachineReconciler) doLivenessCheck(ctx context.Context, vm *hfv1.VirtualMachine,
	instance *dropletv1alpha1.Instance) (ready bool, err error) {
	keySecret := &v1.Secret{}
	var username, address string
	err = r.Get(ctx, types.NamespacedName{Name: vm.Spec.KeyPair, Namespace: vm.Namespace}, keySecret)
	if err != nil {
		return ready, err
	}
	if len(vm.Spec.SshUsername) != 0 {
		username = vm.Spec.SshUsername
	} else {
		username = "root"
	}
Can't find the secret
1.681342582050377e+09   ERROR   controller.virtualmachine       Reconciler error        {"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "dynamic-6gk5wu6d23-5f149a48", "namespace": "hobbyfarm", "error": "Secret \"\" not found"}
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>
        /Users/jason/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>
        /Users/jason/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
vm.Spec.KeyPair is blank
I can't find anywhere in code this gets written to
Should be here in createSecret 
vm.Spec.KeyPair = secretName
	status.Status = secretCreated
	vm.Annotations["secret"] = "created"
	vm.Annotations["secretName"] = secretName
Stupid thing is the annotation is there.
So I added this to the liveness check #hack
func (r *VirtualMachineReconciler) doLivenessCheck(ctx context.Context, vm *hfv1.VirtualMachine,
	instance *dropletv1alpha1.Instance) (ready bool, err error) {
	keySecret := &v1.Secret{}
	var username, address string
	var secretName string
	secretName = vm.Spec.KeyPair
	if secretName == "" {
		secretName = vm.Annotations["secretName"]
	}
rc.Remote(command) is failing

func PerformLivenessCheck(address string, userName string, privateKey string, command string) (ready bool, err error) {

	rc, err := ssh.NewRemoteConnection(address, userName, privateKey)
	if err != nil {
		return ready, err
	}
	_, err = rc.Remote(command)
	if err != nil {
		return ready, err
	}
	ready = true
	return ready, nil
}
1.681343447487985e+09 ERROR controller.virtualmachine Reconciler error {"reconciler group": "hobbyfarm.io", "reconciler kind": "VirtualMachine", "name": "dynamic-m7ksjrccr3-03af39e7", "namespace": "hobbyfarm", "error": "ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem /Users/jason/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2 /Users/jason/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
I can connect to it manually 
ssh root@192.241.143.122 -i ./key.key
The authenticity of host '192.241.143.122 (192.241.143.122)' can't be established.
ED25519 key fingerprint is SHA256:hcabWgPyhRQyqZL3l3CnOWw5yy6M4i59bJKKWAd7Tqc.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.241.143.122' (ED25519) to the list of known hosts.
Welcome to Ubuntu 22.10 (GNU/Linux 5.19.0-23-generic x86_64)

 * Documentation:  <https://help.ubuntu.com>
 * Management:     <https://landscape.canonical.com>
 * Support:        <https://ubuntu.com/advantage>

  System information as of Thu Apr 13 00:14:54 UTC 2023

  System load:  0.04541015625      Users logged in:       0
  Usage of /:   1.0% of 154.96GB   IPv4 address for eth0: 192.241.143.122
  Memory usage: 2%                 IPv4 address for eth0: 10.10.0.5
  Swap usage:   0%                 IPv4 address for eth1: 10.116.0.2
  Processes:    122

0 updates can be applied immediately.



The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.
I pulled the key from a breakpoint in code
I'm stuck here. If I bypass the code I can "start" the scenario but the screen doesn't work.
w

worried-fountain-60974

04/13/2023, 7:10 PM
Okay let's recap. What did you change in the hf-shim code?
a

astonishing-engine-86351

04/14/2023, 1:42 PM
I modified digitalocean_interface.go > doLivenessCheck starts at 191 
var secretName string
	secretName = vm.Spec.KeyPair
	if secretName == "" {
		secretName = vm.Annotations["secretName"]
	}
	err = r.Get(ctx, types.NamespacedName{Name: secretName, Namespace: vm.Namespace}, keySecret)
for some reason vm.Spec.KeyPair is not getting written 😞
It is written in the code, but doesn't end up in etcd
This a dirty hack, but I wanted to see if that would get me step further down the path
8 Views
#hobbyfarmJoin Slack
Powered by