https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# hobbyfarm
a
This message was deleted.
a
This code in digitalocean_interface is not doing anything
Copy code
if _, err = controllerutil.CreateOrUpdate(ctx, r.Client, keyPair, func() error {
		keyPair.Spec.PublicKey = pubKey
		keyPair.Spec.Secret = credSecret

		if err := controllerutil.SetControllerReference(vm, keyPair, r.Scheme); err != nil {
			r.Log.Error(err, "unable to set ownerReference for DO keypair")
			return err
		}
		return nil
	}); err != nil {
		return status, err
	}
If I check kubectl I get no importkeypairs
@worried-fountain-60974 is this at all related to the status changes you had to fix on another object?
They importkeypair is generated and the create or update call made, but nothing happens
w
I suspect you are right.
a
So it's creating the keypair. But it's now failing at the liveness check
w
I have that code up, let me add the status update bits and cut an RC for you
a
vm.Spec.Keypair is empty
w
hrm that shouldn't be part of it
a
I'm running it all in goland right now since I can't make heads or tails of it
That's gotten past this earlier stuff, which is probably what you're needing the rc for, but I have this other issue now 😞
w
lemme see if I can repro so I can help most effectively.
a
But I can't find where vm.Spec.KeyPair is getting written
But the secret exists scheduled-c29efa32-ad2ce76d-secret Opaque 2 5m20s
w
What version of hf-shim-operator are you using?
a
Let me pull latest
w
hold up
a
I'm using the code I cloned, but haven't pulled in a while
w
I just tagged 
v0.2.0
it should be releasing now
I think that will help
a
ok
What's my best path for deploying droplet-operator. Doing this right now export IMG=hobbyfarm/droplet-operator:v0.1.3 make all make install make deploy
w
For the meantime, that is probably best. i haven't gotten to getting the chart releasing working on that repo yet
a
let me know when hf-shim is released 🙂
w
Done
a
This is all i'm getting from droplet-operator manager E0412 164448.176504 1 leaderelection.go:325] error retrieving resource lock hobbyfarm/1036face.cattle.io: leases.coordination.k8s.io "1036face.cattle.io" is forbidden: User "systemserviceaccounthobbyfarm:default" cannot get │ │ manager E0412 164451.474561 1 leaderelection.go:325] error retrieving resource lock hobbyfarm/1036face.cattle.io: leases.coordination.k8s.io "1036face.cattle.io" is forbidden: User "systemserviceaccounthobbyfarm:default" cannot get │ │ manager E0412 164455.794358 1 leaderelection.go:325] error retrieving resource lock hobbyfarm/1036face.cattle.io: leases.coordination.k8s.io "1036face.cattle.io" is forbidden: User "systemserviceaccounthobbyfarm:default" cannot get │ │ manager E0412 164459.717700 1 leaderelection.go:325] error retrieving resource lock hobbyfarm/1036face.cattle.io: leases.coordination.k8s.io "1036face.cattle.io" is forbidden: User "systemserviceaccounthobbyfarm:default" cannot get │ │ manager E0412 164501.999148 1 leaderelection.go:325] error retrieving resource lock hobbyfarm/1036face.cattle.io: leases.coordination.k8s.io "1036face.cattle.io" is forbidden: User "systemserviceaccounthobbyfarm:default" cannot get │ │
I'm going to try running it manuallly
Shim is stuck here
Copy code
<http://gs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|gs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>                                                                                                                                                          │
│     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266                                                                                                                                                  │
│ <http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>                                                                                                                                                                │
│     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227                                                                                                                                                  │
│ 1.6813179594285004e+09    ERROR    controller.virtualmachine    Reconciler error    {"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "scheduled-bfd7aa8e-0cb0050a", "namespace": "hobbyfarm", "error": "unable t │
│ <http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>                                                                                                                                                          │
│     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266                                                                                                                                                  │
│ <http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>                                                                                                                                                                │
│     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227                                                                                                                                                  │
│
Guessing it's because droplet operator is not doing its job
@worried-fountain-60974 Looks like no annotations are being created, or they're being lost at createSecret
Copy code
jason in ~/src/github.com/jasonvanbrackel/introduction-to-kubernetes on main ● ● 🦄 kubectl get virtualmachine -n hobbyfarm -o yaml
apiVersion: v1
items:
- apiVersion: <http://hobbyfarm.io/v1|hobbyfarm.io/v1>
  kind: VirtualMachine
  metadata:
    creationTimestamp: "2023-04-12T16:49:47Z"
    finalizers:
    - <http://finalizer.hobbyfarm.io/vmset|finalizer.hobbyfarm.io/vmset>
    generation: 2
    labels:
      bound: "false"
      dynamic: "false"
      environment: env-digitalocean-external
      <http://hobbyfarm.io/scheduledevent|hobbyfarm.io/scheduledevent>: se-se-h7jwpyinkh
      ready: "false"
      restrictedbind: "true"
      restrictedbindvalue: se-se-h7jwpyinkh
      template: vmt-digitalocean-small
      vmset: se-se-se-h7jwpyinkh-vms-scheduled-bfd7aa8e
    name: scheduled-bfd7aa8e-cd78b87d
    namespace: hobbyfarm
    ownerReferences:
    - apiVersion: <http://hobbyfarm.io/v1|hobbyfarm.io/v1>
      kind: VirtualMachineSet
      name: se-se-se-h7jwpyinkh-vms-scheduled-bfd7aa8e
      uid: e190b445-7ee1-44df-a914-16df920972b7
    resourceVersion: "2101"
    uid: cb7d26b4-9c96-4da0-b017-17203b340f59
  spec:
    id: scheduled-bfd7aa8e-cd78b87d
    provision: false
    ssh_username: ""
    user: ""
    vm_claim_id: ""
    vm_set_id: se-se-se-h7jwpyinkh-vms-scheduled-bfd7aa8e
    vm_template_id: vmt-digitalocean-small
  status:
    allocated: false
    environment_id: env-digitalocean-external
    hostname: ""
    private_ip: ""
    public_ip: ""
    status: SecretCreated
    tainted: false
    ws_endpoint: <http://shell.127.0.0.1.nip.io|shell.127.0.0.1.nip.io>
- apiVersion: <http://hobbyfarm.io/v1|hobbyfarm.io/v1>
  kind: VirtualMachine
  metadata:
    creationTimestamp: "2023-04-12T17:15:51Z"
    finalizers:
    - <http://finalizer.hobbyfarm.io/vmset|finalizer.hobbyfarm.io/vmset>
    generation: 2
    labels:
      bound: "false"
      dynamic: "false"
      environment: env-digitalocean-external
      <http://hobbyfarm.io/scheduledevent|hobbyfarm.io/scheduledevent>: se-se-h7jwpyinkh
      ready: "false"
      restrictedbind: "true"
      restrictedbindvalue: se-se-h7jwpyinkh
      template: vmt-digitalocean-small
      vmset: se-se-se-h7jwpyinkh-vms-scheduled-bfd7aa8e
    name: scheduled-bfd7aa8e-75b5b2d8
    namespace: hobbyfarm
    ownerReferences:
    - apiVersion: <http://hobbyfarm.io/v1|hobbyfarm.io/v1>
      kind: VirtualMachineSet
      name: se-se-se-h7jwpyinkh-vms-scheduled-bfd7aa8e
      uid: e190b445-7ee1-44df-a914-16df920972b7
    resourceVersion: "3971"
    uid: 177cc4c7-2d1f-47de-9e23-9ca9853f87bd
  spec:
    id: scheduled-bfd7aa8e-75b5b2d8
    provision: false
    ssh_username: ""
    user: ""
    vm_claim_id: ""
    vm_set_id: se-se-se-h7jwpyinkh-vms-scheduled-bfd7aa8e
    vm_template_id: vmt-digitalocean-small
  status:
    allocated: false
    environment_id: env-digitalocean-external
    hostname: ""
    private_ip: ""
    public_ip: ""
    status: SecretCreated
    tainted: false
    ws_endpoint: <http://shell.127.0.0.1.nip.io|shell.127.0.0.1.nip.io>
kind: List
metadata:
  resourceVersion: ""
Getting this from the shim
Copy code
│ 1.681320840714841e+09    ERROR    controller.virtualmachine    Reconciler error    {"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "dynamic-pediu5imwf-7e0db18a", "namespace": "hobbyfarm", "error": "Operation │
│  cannot be fulfilled on <http://virtualmachines.hobbyfarm.io|virtualmachines.hobbyfarm.io> \"dynamic-pediu5imwf-7e0db18a\": the object has been modified; please apply your changes to the latest version and try again"}                                                               │
│ <http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>                                                                                                                                                          │
│     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266                                                                                                                                                  │
│ <http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>                                                                                                                                                                │
│     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
maybe that update order is off or something?
w
I get that sometimes but it goes away. I can look again
a
Stopped the scenario and restarted. I'm getting virutalmachine objects with no annotations on them
That error did end up going away, the version one. So the annotation issue is where I'm stuck at the moment.
w
Oh fsck I think I know what it is
a
I'm all 👂s 🙂. Been long enough since I wrote controller logic, that I'm not seeing it.
w
That explains it
a
Oh that's nasty
w
Yeah it's just stupid update logic
if we weren't using a status field as a status subresource we wouldn't have this problem because you can update the whole resource in one go
a
That's a bad side-effect, if the intent is just update status
w
but I would rather do things the right way
yeah. it's fixable with the DeepCopy()
since we don't care about the returned value from a status update just to persist the annotations on the next line
a
But if you update the resource the status subresource is unaffected?
w
Correct.
In a properly defined CRD with Status as a "subresource", updates of the main resource will not affect the status subresource (and vice versa).
Hence you need two calls to fully update the resource. And those calls have to be managed carefully because of the overwrite of the obj reference in the code as you see in that PR
a
Yep, I see it now. I had completely forgotten about that behavior
Let me know when the next version is available. I'll give it a test.
w
once ci finishes i'll cut a v0.2.1
a
👍
w
v0.2.1 building now, give it 10 for docker buildx to choke
then it'll be out
a
I appreciate the help. Thank you
If this works. I'll set a class up and give it a test. Then I'll start sending some docs PRs. I also have a couple small changes to make to the droplet operator based on digital ocean api changes. Remove a deprecated field and add a couple new ones
w
TIA for all that. Contributions like that are sorely needed
Help bandaid some of these thousand cuts we have 😄
a
Life of an OSS project 🙂
Copy code
Name:         dynamic-itkjiikr7x-81612630
Namespace:    hobbyfarm
Labels:       bound=true
              dynamic=true
              environment=env-digitalocean-external
              <http://hobbyfarm.io/provisioner=external|hobbyfarm.io/provisioner=external>
              <http://hobbyfarm.io/scheduledevent=se-se-nkdip3e5ji|hobbyfarm.io/scheduledevent=se-se-nkdip3e5ji>
              <http://hobbyfarm.io/vmtemplate=vmt-digitalocean-small|hobbyfarm.io/vmtemplate=vmt-digitalocean-small>
              ready=false
              restrictedbind=true
              restrictedbindvalue=se-se-nkdip3e5ji
              template=vmt-digitalocean-small
              vmc=dynamic-itkjiikr7x
Annotations:  <none>
@worried-fountain-60974 Still blank
Secret is there
w
Is this the same object you were working with before?
a
New install
Yes virutal machine
w
hmmm
Can you share with me the manifest of the ImportKeyPair and VirtualMachine
a
The Import Key Pair is never created. Here's the VM
Copy code
jason in ~/src/github.com/jasonvanbrackel/introduction-to-kubernetes on main ● ● 🦄 kubectl get virtualmachine -n hobbyfarm dynamic-itkjiikr7x-81612630 -o yaml
apiVersion: <http://hobbyfarm.io/v1|hobbyfarm.io/v1>
kind: VirtualMachine
metadata:
  creationTimestamp: "2023-04-12T19:03:01Z"
  generation: 1
  labels:
    bound: "true"
    dynamic: "true"
    environment: env-digitalocean-external
    <http://hobbyfarm.io/provisioner|hobbyfarm.io/provisioner>: external
    <http://hobbyfarm.io/scheduledevent|hobbyfarm.io/scheduledevent>: se-se-nkdip3e5ji
    <http://hobbyfarm.io/vmtemplate|hobbyfarm.io/vmtemplate>: vmt-digitalocean-small
    ready: "false"
    restrictedbind: "true"
    restrictedbindvalue: se-se-nkdip3e5ji
    template: vmt-digitalocean-small
    vmc: dynamic-itkjiikr7x
  name: dynamic-itkjiikr7x-81612630
  namespace: hobbyfarm
  ownerReferences:
  - apiVersion: <http://hobbyfarm.io/v1|hobbyfarm.io/v1>
    kind: VirtualMachineClaim
    name: dynamic-itkjiikr7x
    uid: 1288ed1c-679f-4060-9cc2-360104a9967d
  resourceVersion: "1214"
  uid: 9e63b6f5-e4bc-47c0-a726-0e6b90b129c6
spec:
  id: dynamic-itkjiikr7x-81612630
  protocol: ssh
  provision: false
  secret_name: ""
  ssh_username: ""
  user: admin
  vm_claim_id: dynamic-itkjiikr7x
  vm_set_id: ""
  vm_template_id: vmt-digitalocean-small
status:
  allocated: true
  environment_id: env-digitalocean-external
  hostname: ""
  private_ip: ""
  public_ip: ""
  status: SecretCreated
  tainted: false
  ws_endpoint: <http://shell.127.0.0.1.nip.io|shell.127.0.0.1.nip.io>
Copy code
jason in ~/src/github.com/jasonvanbrackel/introduction-to-kubernetes on main ● ● 🦄 kubectl get secrets -n hobbyfarm
NAME                                     TYPE                                  DATA   AGE
hobbyfarm-webhook-secret                 Opaque                                3      31m
hobbyfarm-gargantua-token-5wgrg          <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token>   3      31m
default-token-vzsds                      <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token>   3      31m
sh.helm.release.v1.hobbyfarm.v1          <http://helm.sh/release.v1|helm.sh/release.v1>                    1      31m
hf-shim-operator-token-2gxgs             <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token>   3      31m
sh.helm.release.v1.hf-shim-operator.v1   <http://helm.sh/release.v1|helm.sh/release.v1>                    1      31m
ec2-operator-token-tgx4j                 <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token>   3      30m
sh.helm.release.v1.ec2-operator.v1       <http://helm.sh/release.v1|helm.sh/release.v1>                    1      30m
digitalocean-secret                      Opaque                                1      30m
dynamic-itkjiikr7x-81612630-secret       Opaque                                2      29m
Droplet operator is also not happy, but I have no idea if that's a problem
Copy code
kube-rbac-proxy I0412 19:01:38.527632       1 main.go:186] Valid token audiences:                                                                                                                                                                 │
│ kube-rbac-proxy I0412 19:01:38.530212       1 main.go:232] Generating self signed cert as no cert is provided                                                                                                                                     │
│ kube-rbac-proxy I0412 19:01:41.478365       1 main.go:281] Starting TCP socket on 0.0.0.0:8443                                                                                                                                                    │
│ kube-rbac-proxy I0412 19:01:41.482841       1 main.go:288] Listening securely on 0.0.0.0:8443                                                                                                                                                     │
│ manager I0412 19:01:55.504714       1 request.go:655] Throttling request took 1.101529s, request: GET:<https://10.43.0.1:443/apis/autoscaling/v2beta2?timeout=32s>                                                                                  │
│ manager 2023-04-12T19:02:01.506Z    INFO    controller-runtime.metrics    metrics server is starting to listen    {"addr": "127.0.0.1:8080"}                                                                                                      │
│ manager 2023-04-12T19:02:01.905Z    INFO    setup    starting manager                                                                                                                                                                             │
│ manager I0412 19:02:02.006167       1 leaderelection.go:243] attempting to acquire leader lease hobbyfarm/1036face.cattle.io...                                                                                                                   │
│ manager 2023-04-12T19:02:02.121Z    INFO    controller-runtime.manager    starting metrics server    {"path": "/metrics"}                                                                                                                         │
│ manager E0412 19:02:04.008325       1 leaderelection.go:329] error initially creating leader election record: <http://leases.coordination.k8s.io|leases.coordination.k8s.io> is forbidden: User "system:serviceaccount:hobbyfarm:default" cannot create resource "leases" in API grou │
│ p "<http://coordination.k8s.io|coordination.k8s.io>" in the namespace "hobbyfarm"                                                                                                                                                                                              │
│ manager E0412 19:02:07.608492       1 leaderelection.go:325] error retrieving resource lock hobbyfarm/1036face.cattle.io: <http://leases.coordination.k8s.io|leases.coordination.k8s.io> "<http://1036face.cattle.io|1036face.cattle.io>" is forbidden: User "system:serviceaccount:hobbyfarm:default" cannot get │
│  resource "leases" in API group "<http://coordination.k8s.io|coordination.k8s.io>" in the namespace "hobbyfarm"                                                                                                                                                                │
│ manager E0412 19:02:11.910931       1 leaderelection.go:325] error retrieving resource lock hobbyfarm/1036face.cattle.io: <http://leases.coordination.k8s.io|leases.coordination.k8s.io> "<http://1036face.cattle.io|1036face.cattle.io>" is forbidden: User "system:serviceaccount:hobbyfarm:default" cannot get │
│  resource "leases" in API group "<http://coordination.k8s.io|coordination.k8s.io>" in the namespace "hobbyfarm"                                                                                                                                                                │
│ manager E0412 19:02:15.523777       1 leaderelection.go:325] error retrieving resource lock hobbyfarm/1036face.cattle.io: <http://leases.coordination.k8s.io|leases.coordination.k8s.io> "<http://1036face.cattle.io|1036face.cattle.io>" is forbidden: User "system:serviceaccount:hobbyfarm:default" cannot get │
│  resource "leases" in API group "<http://coordination.k8s.io|coordination.k8s.io>" in the namespace "hobbyfarm"                                                                                                                                                                │
│ manager E0412 19:02:18.612734       1 leaderelection.go:325] error retrieving resource lock hobbyfarm/1036face.cattle.io: <http://leases.coordination.k8s.io|leases.coordination.k8s.io> "<http://1036face.cattle.io|1036face.cattle.io>" is forbidden: User "system:serviceaccount:hobbyfarm:default" cannot get │
│  resource "leases" in API group "<http://coordination.k8s.io|coordination.k8s.io>" in the namespace "hobbyfarm"                                                                                                                                                                │
│ manager E0412 19:02:21.907572       1 leaderelection.go:325] error retrieving resource lock hobbyfarm/1036face.cattle.io: <http://leases.coordination.k8s.io|leases.coordination.k8s.io> "<http://1036face.cattle.io|1036face.cattle.io>" is forbidden: User "system:serviceaccount:hobbyfarm:default" cannot get │
│  resource "leases" in API group "<http://coordination.k8s.io|coordination.k8s.io>" in the namespace "hobbyfarm"                                                                                                                                                                │
│ manager E0412 19:02:25.609285       1 leaderelection.go:325] error retrieving resource lock hobbyfarm/1036face.cattle.io: <http://leases.coordination.k8s.io|leases.coordination.k8s.io> "<http://1036face.cattle.io|1036face.cattle.io>" is forbidden: User "system:serviceaccount:hobbyfarm:default" cannot get │
│  resource "leases" in API group "<http://coordination.k8s.io|coordination.k8s.io>" in the namespace "hobbyfarm"                                                                                                                                                                │
│ manager E0412 19:02:27.814251       1 leaderelection.go:325] error retrieving resource lock hobbyfarm/1036face.cattle.io: <http://leases.coordination.k8s.io|leases.coordination.k8s.io> "<http://1036face.cattle.io|1036face.cattle.io>" is forbidden: User "system:serviceaccount:hobbyfarm:default" cannot get │
│  resource "leases" in API group "<http://coordination.k8s.io|coordination.k8s.io>" in the namespace "hobbyfarm"                                                                                                                                                                │
│ manager E0412 19:02:30.223264       1 leaderelection.go:325] error retrieving resource lock hobbyfarm/1036face.cattle.io: <http://leases.coordination.k8s.io|leases.coordination.k8s.io> "<http://1036face.cattle.io|1036face.cattle.io>" is forbidden: User "system:serviceaccount:hobbyfarm:default" cannot get │
│  resource "leases" in API group "<http://coordination.k8s.io|coordination.k8s.io>" in the namespace "hobbyfarm"                                                                                                                                                                │
│ manager E0412 19:02:32.489466       1 leaderelection.go:325] error retrieving resource lock hobbyfarm/1036face.cattle.io: <http://leases.coordination.k8s.io|leases.coordination.k8s.io> "<http://1036face.cattle.io|1036face.cattle.io>" is forbidden: User "system:serviceaccount:hobbyfarm:default" cannot get │
│  resource "leases" in API group "<http://coordination.k8s.io|coordination.k8s.io>" in the namespace "hobbyfarm"                                                                                                                                                                │
│ manager E0412 19:02:35.223782       1 leaderelection.go:325] error retrieving resource lock hobbyfarm/1036face.cattle.io: <http://leases.coordination.k8s.io|leases.coordination.k8s.io> "<http://1036face.cattle.io|1036face.cattle.io>" is forbidden: User "system:serviceaccount:hobbyfarm:default" cannot get │
│  resource "leases" in API group "<http://coordination.k8s.io|coordination.k8s.io>" in the namespace "hobbyfarm"                                                                                                                                                                │
│ manager E0412 19:02:38.493384       1 leaderelection.go:325] error retrieving resource lock hobbyfarm/1036face.cattle.io: <http://leases.coordination.k8s.io|leases.coordination.k8s.io> "<http://1036face.cattle.io|1036face.cattle.io>" is forbidden: User "system:serviceaccount:hobbyfarm:default" cannot get │
│  resource "leases" in API group "<http://coordination.k8s.io|coordination.k8s.io>" in the namespace "hobbyfarm"
w
The ImportKeyPair is never created
That's because of the above issue. It's a perms thing
It needs to obtain a lease.
Fully expect this is a documentation issue
a
Does that explain the lack of annotations?
Because the former is in the shim the latter from droplet operator
w
Yeah because the annotation for a keypair name can never be created if the ImportKeyPair never actually comes into existence.
At least that's my theory
I didn't run into this because I have been testing with an admin kubeconfig 🤦‍♂️
a
Am I missing a clusterrole or something?
w
Yes but it's not your fault.
Create a Role with * on coordination.k8s.io/leases in the hobbyfarm namespace
then rolebinding, with subject of serviceaccount default in hobbyfarm namespace
a
Is that what this is for?
w
that's what it used to be for, I think before leases existed? idk
This was using an older version of KubeBuilder and has been stepped up
a
This is being executed by the make install / make deploy so I'll add it here and see if that works
Ok that cleared that error
Going to try to create a vm
@worried-fountain-60974 Shim is still misbehaving.
Copy code
apiVersion: <http://hobbyfarm.io/v1|hobbyfarm.io/v1>
kind: VirtualMachine
metadata:
  creationTimestamp: "2023-04-12T19:54:38Z"
  generation: 1
  labels:
    bound: "true"
    dynamic: "true"
    environment: env-digitalocean-external
    <http://hobbyfarm.io/provisioner|hobbyfarm.io/provisioner>: external
    <http://hobbyfarm.io/scheduledevent|hobbyfarm.io/scheduledevent>: se-se-2vyaok4txz
    <http://hobbyfarm.io/vmtemplate|hobbyfarm.io/vmtemplate>: vmt-digitalocean-small
    ready: "false"
    restrictedbind: "true"
    restrictedbindvalue: se-se-2vyaok4txz
    template: vmt-digitalocean-small
    vmc: dynamic-2wss7xwiql
  name: dynamic-2wss7xwiql-1a9b63a6
  namespace: hobbyfarm
  ownerReferences:
  - apiVersion: <http://hobbyfarm.io/v1|hobbyfarm.io/v1>
    kind: VirtualMachineClaim
    name: dynamic-2wss7xwiql
    uid: 9100dc64-2199-4c8c-9708-34307eebcac3
  resourceVersion: "1267"
  uid: 044a28b7-6639-472b-af3e-d2418b678958
spec:
  id: dynamic-2wss7xwiql-1a9b63a6
  protocol: ssh
  provision: false
  secret_name: ""
  ssh_username: ""
  user: admin
  vm_claim_id: dynamic-2wss7xwiql
  vm_set_id: ""
  vm_template_id: vmt-digitalocean-small
status:
  allocated: true
  environment_id: env-digitalocean-external
  hostname: ""
  private_ip: ""
  public_ip: ""
  status: SecretCreated
  tainted: false
  ws_endpoint: <http://shell.127.0.0.1.nip.io|shell.127.0.0.1.nip.io>
w
oh ffs
any errors?
a
Copy code
jason in ~/src/github.com/jasonvanbrackel/introduction-to-kubernetes on main ● ● 🦄 kubectl -n hobbyfarm logs hf-shim-operator-b94f64597-fhpn8
I0412 19:52:47.858847       1 request.go:665] Waited for 1.04282575s due to client-side throttling, not priority and fairness, request: GET:<https://10.43.0.1:443/apis/discovery.k8s.io/v1?timeout=32s>
1.6813291686296883e+09	INFO	controller-runtime.metrics	Metrics server is starting to listen	{"addr": ":8080"}
1.6813291686418562e+09	INFO	setup	starting manager
1.6813291686505077e+09	INFO	Starting server	{"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
1.6813291686545818e+09	INFO	controller.virtualmachine	Starting EventSource	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "source": "kind source: *v1.VirtualMachine"}
1.6813291686550047e+09	INFO	controller.virtualmachine	Starting EventSource	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "source": "kind source: *v1alpha1.Instance"}
1.6813291686550746e+09	INFO	controller.virtualmachine	Starting EventSource	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "source": "kind source: *v1alpha1.ImportKeyPair"}
1.6813291686551135e+09	INFO	controller.virtualmachine	Starting EventSource	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "source": "kind source: *v1alpha1.Instance"}
1.6813291686551623e+09	INFO	controller.virtualmachine	Starting EventSource	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "source": "kind source: *v1alpha1.ImportKeyPair"}
1.6813291686552024e+09	INFO	controller.virtualmachine	Starting EventSource	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "source": "kind source: *v1.Secret"}
1.6813291686553493e+09	INFO	controller.virtualmachine	Starting Controller	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine"}
1.6813291725122058e+09	ERROR	controller-runtime.source	if kind is a CRD, it should be installed before calling Start	{"kind": "<http://ImportKeyPair.ec2.cattle.io|ImportKeyPair.ec2.cattle.io>", "error": "no matches for kind \"ImportKeyPair\" in version \"<http://ec2.cattle.io/v1alpha1\|ec2.cattle.io/v1alpha1\>""}
<http://sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1.1|sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1.1>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/source/source.go:137
<http://k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext|k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext>
	/go/pkg/mod/k8s.io/apimachinery@v0.23.0/pkg/util/wait/wait.go:233
<http://k8s.io/apimachinery/pkg/util/wait.poll|k8s.io/apimachinery/pkg/util/wait.poll>
	/go/pkg/mod/k8s.io/apimachinery@v0.23.0/pkg/util/wait/wait.go:580
<http://k8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext|k8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext>
	/go/pkg/mod/k8s.io/apimachinery@v0.23.0/pkg/util/wait/wait.go:545
<http://sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1|sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/source/source.go:131
1.681329176518322e+09	ERROR	controller-runtime.source	if kind is a CRD, it should be installed before calling Start	{"kind": "<http://Instance.droplet.cattle.io|Instance.droplet.cattle.io>", "error": "no matches for kind \"Instance\" in version \"<http://droplet.cattle.io/v1alpha1\|droplet.cattle.io/v1alpha1\>""}
<http://sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1.1|sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1.1>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/source/source.go:137
<http://k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext|k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext>
	/go/pkg/mod/k8s.io/apimachinery@v0.23.0/pkg/util/wait/wait.go:233
<http://k8s.io/apimachinery/pkg/util/wait.poll|k8s.io/apimachinery/pkg/util/wait.poll>
	/go/pkg/mod/k8s.io/apimachinery@v0.23.0/pkg/util/wait/wait.go:580
<http://k8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext|k8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext>
	/go/pkg/mod/k8s.io/apimachinery@v0.23.0/pkg/util/wait/wait.go:545
<http://sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1|sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/source/source.go:131
I0412 19:52:57.913400       1 request.go:665] Waited for 1.295655376s due to client-side throttling, not priority and fairness, request: GET:<https://10.43.0.1:443/apis/flowcontrol.apiserver.k8s.io/v1beta2?timeout=32s>
1.6813291805199664e+09	ERROR	controller-runtime.source	if kind is a CRD, it should be installed before calling Start	{"kind": "<http://ImportKeyPair.droplet.cattle.io|ImportKeyPair.droplet.cattle.io>", "error": "no matches for kind \"ImportKeyPair\" in version \"<http://droplet.cattle.io/v1alpha1\|droplet.cattle.io/v1alpha1\>""}
<http://sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1.1|sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1.1>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/source/source.go:137
<http://k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext|k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext>
	/go/pkg/mod/k8s.io/apimachinery@v0.23.0/pkg/util/wait/wait.go:233
<http://k8s.io/apimachinery/pkg/util/wait.poll|k8s.io/apimachinery/pkg/util/wait.poll>
	/go/pkg/mod/k8s.io/apimachinery@v0.23.0/pkg/util/wait/wait.go:580
<http://k8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext|k8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext>
	/go/pkg/mod/k8s.io/apimachinery@v0.23.0/pkg/util/wait/wait.go:545
<http://sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1|sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/source/source.go:131
1.6813291906229315e+09	INFO	controller.virtualmachine	Starting workers	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "worker count": 20}
1.6813292781361418e+09	ERROR	controller.virtualmachine	Reconciler error	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "dynamic-2wss7xwiql-1a9b63a6", "namespace": "hobbyfarm", "error": "Operation cannot be fulfilled on <http://virtualmachines.hobbyfarm.io|virtualmachines.hobbyfarm.io> \"dynamic-2wss7xwiql-1a9b63a6\": the object has been modified; please apply your changes to the latest version and try again"}
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
time="2023-04-12T19:54:38Z" level=info msg="creating new keypair"
1.6813292799591446e+09	ERROR	controller.virtualmachine	Reconciler error	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "dynamic-2wss7xwiql-1a9b63a6", "namespace": "hobbyfarm", "error": "Secret \"dynamic-2wss7xwiql-1a9b63a6-secret\" not found"}
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
1.6813292799725318e+09	ERROR	controller.virtualmachine	Reconciler error	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "dynamic-2wss7xwiql-1a9b63a6", "namespace": "hobbyfarm", "error": "Operation cannot be fulfilled on <http://virtualmachines.hobbyfarm.io|virtualmachines.hobbyfarm.io> \"dynamic-2wss7xwiql-1a9b63a6\": the object has been modified; please apply your changes to the latest version and try again"}
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
1.681329279973107e+09	ERROR	controller.virtualmachine	Reconciler error	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "dynamic-2wss7xwiql-1a9b63a6", "namespace": "hobbyfarm", "error": "unable to find label pubKey on VM"}
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
1.681329283138751e+09	ERROR	controller.virtualmachine	Reconciler error	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "dynamic-2wss7xwiql-1a9b63a6", "namespace": "hobbyfarm", "error": "unable to find label pubKey on VM"}
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
1.681329293146719e+09	ERROR	controller.virtualmachine	Reconciler error	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "dynamic-2wss7xwiql-1a9b63a6", "namespace": "hobbyfarm", "error": "unable to find label pubKey on VM"}
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
1.6813293031481442e+09	ERROR	controller.virtualmachine	Reconciler error	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "dynamic-2wss7xwiql-1a9b63a6", "namespace": "hobbyfarm", "error": "unable to find label pubKey on VM"}
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
1.6813293131524856e+09	ERROR	controller.virtualmachine	Reconciler error	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "dynamic-2wss7xwiql-1a9b63a6", "namespace": "hobbyfarm", "error": "unable to find label pubKey on VM"}
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
1.6813293231571817e+09	ERROR	controller.virtualmachine	Reconciler error	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "dynamic-2wss7xwiql-1a9b63a6", "namespace": "hobbyfarm", "error": "unable to find label pubKey on VM"}
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
1.6813293331615367e+09	ERROR	controller.virtualmachine	Reconciler error	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "dynamic-2wss7xwiql-1a9b63a6", "namespace": "hobbyfarm", "error": "unable to find label pubKey on VM"}
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>
That's the shim
Nothing from the droplet operator
Copy code
manager I0412 19:53:23.756117       1 request.go:655] Throttling request took 1.199809s, request: GET:<https://10.43.0.1:443/apis/apiextensions.k8s.io/v1?timeout=32s>
kube-rbac-proxy I0412 19:53:09.163178       1 main.go:186] Valid token audiences:
kube-rbac-proxy I0412 19:53:09.165665       1 main.go:232] Generating self signed cert as no cert is provided
kube-rbac-proxy I0412 19:53:12.616029       1 main.go:281] Starting TCP socket on 0.0.0.0:8443
kube-rbac-proxy I0412 19:53:12.620980       1 main.go:288] Listening securely on 0.0.0.0:8443
manager 2023-04-12T19:53:28.955Z    INFO    controller-runtime.metrics    metrics server is starting to listen    {"addr": "127.0.0.1:8080"}
manager 2023-04-12T19:53:29.157Z    INFO    setup    starting manager
manager I0412 19:53:29.261324       1 leaderelection.go:243] attempting to acquire leader lease hobbyfarm/1036face.cattle.io...
manager 2023-04-12T19:53:29.653Z    INFO    controller-runtime.manager    starting metrics server    {"path": "/metrics"}
manager I0412 19:53:30.958003       1 leaderelection.go:253] successfully acquired lease hobbyfarm/1036face.cattle.io
manager 2023-04-12T19:53:30.456Z    DEBUG    controller-runtime.manager.events    Normal    {"object": {"kind":"ConfigMap","namespace":"hobbyfarm","name":"<http://1036face.cattle.io|1036face.cattle.io>","uid":"14863b17-84a1-4d58-8853-43ea048b49bb","apiVersion":"v1","resour
manager 2023-04-12T19:53:31.552Z    DEBUG    controller-runtime.manager.events    Normal    {"object": {"kind":"Lease","namespace":"hobbyfarm","name":"<http://1036face.cattle.io|1036face.cattle.io>","uid":"cdaa3afd-e9b5-499f-bf73-76f9fb9db664","apiVersion":"coordination.k8
manager 2023-04-12T19:53:31.655Z    INFO    controller-runtime.manager.controller.instance    Starting EventSource    {"reconciler group": "<http://droplet.cattle.io|droplet.cattle.io>", "reconciler kind": "Instance", "source": "kind source: /, Kind="}
manager 2023-04-12T19:53:31.654Z    INFO    controller-runtime.manager.controller.importkeypair    Starting EventSource    {"reconciler group": "<http://droplet.cattle.io|droplet.cattle.io>", "reconciler kind": "ImportKeyPair", "source": "kind source: /, Kind="}
manager 2023-04-12T19:53:36.955Z    INFO    controller-runtime.manager.controller.instance    Starting Controller    {"reconciler group": "<http://droplet.cattle.io|droplet.cattle.io>", "reconciler kind": "Instance"}
manager 2023-04-12T19:53:37.153Z    INFO    controller-runtime.manager.controller.instance    Starting workers    {"reconciler group": "<http://droplet.cattle.io|droplet.cattle.io>", "reconciler kind": "Instance", "worker count": 1}
manager 2023-04-12T19:53:37.362Z    INFO    controller-runtime.manager.controller.importkeypair    Starting Controller    {"reconciler group": "<http://droplet.cattle.io|droplet.cattle.io>", "reconciler kind": "ImportKeyPair"}
manager 2023-04-12T19:53:37.554Z    INFO    controller-runtime.manager.controller.importkeypair    Starting workers    {"reconciler group": "<http://droplet.cattle.io|droplet.cattle.io>", "reconciler kind": "ImportKeyPair", "worker count": 1}
That's the droplet operator
VM with no annotations, No ImportKey
w
The CRD errors are weird. What is the output of 
kubectl get CRD | grep droplet
?
a
w
hrrm
a
Maybe I need to install the operators before the shim
w
The CRDs at least need to be installed first.
a
No more CRD errors, still no annotations
Copy code
kubectl -n hobbyfarm logs hf-shim-operator-b94f64597-pqmg8
1.6813299987394779e+09	INFO	controller-runtime.metrics	Metrics server is starting to listen	{"addr": ":8080"}
1.6813299987493336e+09	INFO	setup	starting manager
1.6813299987561092e+09	INFO	Starting server	{"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
1.6813299987610338e+09	INFO	controller.virtualmachine	Starting EventSource	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "source": "kind source: *v1.VirtualMachine"}
1.681329998761367e+09	INFO	controller.virtualmachine	Starting EventSource	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "source": "kind source: *v1alpha1.Instance"}
1.6813299987614744e+09	INFO	controller.virtualmachine	Starting EventSource	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "source": "kind source: *v1alpha1.ImportKeyPair"}
1.6813299987615116e+09	INFO	controller.virtualmachine	Starting EventSource	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "source": "kind source: *v1alpha1.Instance"}
1.6813299987615592e+09	INFO	controller.virtualmachine	Starting EventSource	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "source": "kind source: *v1alpha1.ImportKeyPair"}
1.6813299987615962e+09	INFO	controller.virtualmachine	Starting EventSource	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "source": "kind source: *v1.Secret"}
1.6813299987619522e+09	INFO	controller.virtualmachine	Starting Controller	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine"}
1.6813299988830366e+09	INFO	controller.virtualmachine	Starting workers	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "worker count": 20}
1.6813300778200343e+09	ERROR	controller.virtualmachine	Reconciler error	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "dynamic-wpk7e3mlop-0cd8059e", "namespace": "hobbyfarm", "error": "Operation cannot be fulfilled on <http://virtualmachines.hobbyfarm.io|virtualmachines.hobbyfarm.io> \"dynamic-wpk7e3mlop-0cd8059e\": the object has been modified; please apply your changes to the latest version and try again"}
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
time="2023-04-12T20:07:57Z" level=info msg="creating new keypair"
1.6813300787538097e+09	ERROR	controller.virtualmachine	Reconciler error	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "dynamic-wpk7e3mlop-0cd8059e", "namespace": "hobbyfarm", "error": "Secret \"dynamic-wpk7e3mlop-0cd8059e-secret\" not found"}
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
1.6813300787671258e+09	ERROR	controller.virtualmachine	Reconciler error	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "dynamic-wpk7e3mlop-0cd8059e", "namespace": "hobbyfarm", "error": "Operation cannot be fulfilled on <http://virtualmachines.hobbyfarm.io|virtualmachines.hobbyfarm.io> \"dynamic-wpk7e3mlop-0cd8059e\": the object has been modified; please apply your changes to the latest version and try again"}
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
1.6813300787676568e+09	ERROR	controller.virtualmachine	Reconciler error	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "dynamic-wpk7e3mlop-0cd8059e", "namespace": "hobbyfarm", "error": "unable to find label pubKey on VM"}
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
1.681330082825227e+09	ERROR	controller.virtualmachine	Reconciler error	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "dynamic-wpk7e3mlop-0cd8059e", "namespace": "hobbyfarm", "error": "unable to find label pubKey on VM"}
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
1.681330092827374e+09	ERROR	controller.virtualmachine	Reconciler error	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "dynamic-wpk7e3mlop-0cd8059e", "namespace": "hobbyfarm", "error": "unable to find label pubKey on VM"}
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
1.6813301028293977e+09	ERROR	controller.virtualmachine	Reconciler error	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "dynamic-wpk7e3mlop-0cd8059e", "namespace": "hobbyfarm", "error": "unable to find label pubKey on VM"}
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
1.6813301128345685e+09	ERROR	controller.virtualmachine	Reconciler error	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "dynamic-wpk7e3mlop-0cd8059e", "namespace": "hobbyfarm", "error": "unable to find label pubKey on VM"}
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
1.6813301228384094e+09	ERROR	controller.virtualmachine	Reconciler error	{"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "dynamic-wpk7e3mlop-0cd8059e", "namespace": "hobbyfarm", "error": "unable to find label pubKey on VM"}
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
Copy code
kubectl -n hobbyfarm get virtualmachine dynamic-wpk7e3mlop-0cd8059e -o yaml
apiVersion: <http://hobbyfarm.io/v1|hobbyfarm.io/v1>
kind: VirtualMachine
metadata:
  creationTimestamp: "2023-04-12T20:07:57Z"
  generation: 1
  labels:
    bound: "true"
    dynamic: "true"
    environment: env-digitalocean-external
    <http://hobbyfarm.io/provisioner|hobbyfarm.io/provisioner>: external
    <http://hobbyfarm.io/scheduledevent|hobbyfarm.io/scheduledevent>: se-se-osmxawhj7w
    <http://hobbyfarm.io/vmtemplate|hobbyfarm.io/vmtemplate>: vmt-digitalocean-small
    ready: "false"
    restrictedbind: "true"
    restrictedbindvalue: se-se-osmxawhj7w
    template: vmt-digitalocean-small
    vmc: dynamic-wpk7e3mlop
  name: dynamic-wpk7e3mlop-0cd8059e
  namespace: hobbyfarm
  ownerReferences:
  - apiVersion: <http://hobbyfarm.io/v1|hobbyfarm.io/v1>
    kind: VirtualMachineClaim
    name: dynamic-wpk7e3mlop
    uid: d0dff310-06a1-407b-8bbb-52671b3350d7
  resourceVersion: "1251"
  uid: 111e451d-4deb-444c-842c-eade13599505
spec:
  id: dynamic-wpk7e3mlop-0cd8059e
  protocol: ssh
  provision: false
  secret_name: ""
  ssh_username: ""
  user: admin
  vm_claim_id: dynamic-wpk7e3mlop
  vm_set_id: ""
  vm_template_id: vmt-digitalocean-small
status:
  allocated: true
  environment_id: env-digitalocean-external
  hostname: ""
  private_ip: ""
  public_ip: ""
  status: SecretCreated
  tainted: false
  ws_endpoint: <http://shell.127.0.0.1.nip.io|shell.127.0.0.1.nip.io>
It's creating the secret
Copy code
kubectl get secrets -n hobbyfarm dynamic-wpk7e3mlop-0cd8059e-secret -o yaml
apiVersion: v1
data:
  private_key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb3dJQkFBS0NBUUVBeTlZbXlFd0JULzNjMmwzYTNoWTJVNlNyaDhPSEpoejBpZG4rY2tlMTY5RUMvZzJLClBHcWZ5Vmg3UmNhaElBMWZzNEJodVFGRWRFZlovTW1jSEtVN25vUGIwa213VHZBOTUxOWdneFlDZFpOYUZ5aFoKZEJOSXlQOUhtWHl0SmRLcy91emxVWm5vOFQyTTlkdXZacWFLeTQ1WDJXK0ljMlpmZGJRSkkvTkh5bHpSem1CLwp5dzBndWRBeWhMQitkaXBMdFZqQkQwdGlUTVcybmRYNDdFRG9QRUtNemxYL1hzdDE2RmdycThydFE0SDRsamhjClE4Y0hvUExzd1FYMHRRbGtnNmdRc0xEb3VKendrK0k1M1RDMWRaRmtOVkVLMUIrb0dyenN6eG1MMm5KY1BaVVUKQlB5S2pWU2tZRUhVN3FYYU5Lcnl3R0UwVHBvRGc0M3BGR2tETVFJREFRQUJBb0lCQVFDeDBidzdkN0tkYWNWRQo2aU9KSXE5NkFHbHpJbXZBSC9wa1Y3dlZUaUd4K3NBMWlQSlRDbUVpY1FINU9QSk8xOXZvNTlKWVpuYWxTaGtzCm9XU0N5TkpkMG9IdHpVTDFXamRtcEptNVFYWWJ3OVFQNTY5Qng2RHpWMS9LMGl6cWxnTTgwTVNzanl2YlorcVUKZ0hWZmcySUplYi9CZTc3NWpSRXh5VUZBRDNkTlhHTGRUc1F1TTBsNGxuRTNpcG9jTXd6NTMxK1JETG0zMGZaNAo5dko4SnU3VVlUYitCVGorcmdmdEhyVjJzTGdrUWFqR2Zkay92NjVUdU9ZU2pUSkhoVi9GMDlKOTd5RjQ2QXU1CkVua1RrbFRRZ0Y4WmdScXFPL3g4a0d3dEN6MzJVMWF0Y21NZys0Qno1NnRTK3JRTFJxYUlQS0p5QXlCNDZoQysKK3ZhWnRMSVJBb0dCQU15T3l0UXFXSXNwMUx6S3dHY3FjaGZtMFBFcGNzdWlzR2M4U3VTeTlWWTFNeWxQWkdZWApLTGZOWXh2OFQwUUdQTHQ0Yi90cW9zZjZhajBIOERRY2pSS0dDcURiVHhORmFPaGU4eVpkYzgxWXRWYUQwY0grCm1ESm9SRmdJam5NSW5EeXlHNU1FV3QxOVZTZ1k3UGNVczFqZGJaMjJLTTBJaDRiR3ZHVVlQSDhqQW9HQkFQOFkKN1BxdWxyWGJOVkpoR2N4S1dtWGREZjV0cWE5OGhQbGNKVS9hWmFjNEhpWm54Q0NoOXU0T2RpdE1PcXIwVkZDbwpDdmpadnhRcVphNDRqMVBiQnFxK0d0am9vUXM4ZGZvZnN4NXU4Szl1R1JQZFBCQjBJc25ibTdOMzNRKy84QWtuCkdPMExLMVVmYjhXcVhwaTVkbnQ0azA5c0JrY1dkQ3BxSUJESGhPT2JBb0dBRG5SY29zZG5RRll4WVRIV2pMTzAKSWpvK00zeGhUMEJzeEs2L0FCQWhRL2s0WjBmajZVN2pvckZSdW8yR3NHRGNjOGJiNnVFckpZUnRjMmlYTWFkKwoxSHRDVHdHUVFlVmF0MmpXTjdUTTFRcmFyWVZkRWg5Q2MvVm9OdXhMM0pjVUt6bk90d0hNeGUvd1Q0Z0xYcDVJCjJoT3poanh6LzZFRnFxRnFlSjVRMEE4Q2dZQVRyQnBxWXBoZ3dkemswM1FuMnZ3SnF2aHZUSW12Yk16Y1pnekMKa2JjUkpEUy9CWFdNaHJFQ1ByOUNLNGpDc2tEbVQyODVMYUdjcm9CVTVTdTRiY1dmMlRVODBRUEtXQWxKRnJTNwpmaGExa0tvQ2JDYzdwWmtoTS94WDVNQjVmSFhJWHVsL0xrTlROeTlwY1BBR214YldBc0JwZVkwcVlrTG5HblFLCmx6aGFUUUtCZ0NTRTlBaVR2Q3VzZlpuL0EwOEtmNWJ1NXVDUjZhSkNpdEJqem9uYXBEZkpaTWxDN0VheThDTU4Ka1BTd1FXZkpGUVFDUENzT2tkVmhvUlltWHRydlFnSUROc3ZUWHdrWjMzVlIzdkZYcWpKVkdhMW5lQ0dZUXBQWQoxVEdlbFVkcFROWmR4OWlqTC9JdGpwRWYwbkllWVY4azVqVCsxeGtTZHpqUTJqVS9keHFYCi0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg==
  public_key: c3NoLXJzYSBBQUFBQjNOemFDMXljMkVBQUFBREFRQUJBQUFCQVFETDFpYklUQUZQL2R6YVhkcmVGalpUcEt1SHc0Y21IUFNKMmY1eVI3WHIwUUwrRFlvOGFwL0pXSHRGeHFFZ0RWK3pnR0c1QVVSMFI5bjh5WndjcFR1ZWc5dlNTYkJPOEQzblgyQ0RGZ0oxazFvWEtGbDBFMGpJLzBlWmZLMGwwcXorN09WUm1lanhQWXoxMjY5bXBvckxqbGZaYjRoelpsOTF0QWtqODBmS1hOSE9ZSC9MRFNDNTBES0VzSDUyS2t1MVdNRVBTMkpNeGJhZDFmanNRT2c4UW96T1ZmOWV5M1hvV0N1cnl1MURnZmlXT0Z4RHh3ZWc4dXpCQmZTMUNXU0RxQkN3c09pNG5QQ1Q0am5kTUxWMWtXUTFVUXJVSDZnYXZPelBHWXZhY2x3OWxSUUUvSXFOVktSZ1FkVHVwZG8wcXZMQVlUUk9tZ09EamVrVWFRTXgK
kind: Secret
metadata:
  creationTimestamp: "2023-04-12T20:07:58Z"
  name: dynamic-wpk7e3mlop-0cd8059e-secret
  namespace: hobbyfarm
  ownerReferences:
  - apiVersion: <http://hobbyfarm.io/v1|hobbyfarm.io/v1>
    blockOwnerDeletion: true
    controller: true
    kind: VirtualMachine
    name: dynamic-wpk7e3mlop-0cd8059e
    uid: 111e451d-4deb-444c-842c-eade13599505
  resourceVersion: "1250"
  uid: 664705a6-624f-4d84-8682-4ec685a73342
type: Opaque
But not an ImportKeyPair
Copy code
jason in ~/src/github.com/hobbyfarm/hf-shim-operator on master 🦄 kubectl get -n hobbyfarm <http://importkeypairs.droplet.cattle.io|importkeypairs.droplet.cattle.io>
No resources found in hobbyfarm namespace.
w
any drop[let operator logs?
droplet operator is what creates the ImportKeyPair
a
Copy code
kubectl -n hobbyfarm logs droplet-operator-controller-manager-655b5df87c-5zfbm
Defaulted container "kube-rbac-proxy" out of: kube-rbac-proxy, manager
I0412 20:05:56.990889       1 main.go:186] Valid token audiences:
I0412 20:05:56.993337       1 main.go:232] Generating self signed cert as no cert is provided
I0412 20:05:58.256049       1 main.go:281] Starting TCP socket on 0.0.0.0:8443
I0412 20:05:58.260104       1 main.go:288] Listening securely on 0.0.0.0:8443
w
-c manager
a
Copy code
manager 2023-04-12T20:06:15.854Z    INFO    controller-runtime.metrics    metrics server is starting to listen    {"addr": "127.0.0.1:8080"}
manager 2023-04-12T20:06:16.052Z    INFO    setup    starting manager
manager I0412 20:06:16.169304       1 leaderelection.go:243] attempting to acquire leader lease hobbyfarm/1036face.cattle.io...
kube-rbac-proxy I0412 20:05:56.990889       1 main.go:186] Valid token audiences:
kube-rbac-proxy I0412 20:05:56.993337       1 main.go:232] Generating self signed cert as no cert is provided
kube-rbac-proxy I0412 20:05:58.256049       1 main.go:281] Starting TCP socket on 0.0.0.0:8443
kube-rbac-proxy I0412 20:05:58.260104       1 main.go:288] Listening securely on 0.0.0.0:8443
manager 2023-04-12T20:06:16.251Z    INFO    controller-runtime.manager    starting metrics server    {"path": "/metrics"}
manager I0412 20:06:21.158797       1 leaderelection.go:253] successfully acquired lease hobbyfarm/1036face.cattle.io
manager 2023-04-12T20:06:21.352Z    DEBUG    controller-runtime.manager.events    Normal    {"object": {"kind":"ConfigMap","namespace":"hobbyfarm","name":"<http://1036face.cattle.io|1036face.cattle.io>","uid":"fe0f1d2e-adef-4549-8908-2897452315c0","apiVersion":"v1","resour
manager 2023-04-12T20:06:22.063Z    DEBUG    controller-runtime.manager.events    Normal    {"object": {"kind":"Lease","namespace":"hobbyfarm","name":"<http://1036face.cattle.io|1036face.cattle.io>","uid":"06e5b002-a6b1-4b91-8109-6a02f0946c85","apiVersion":"coordination.k8
manager 2023-04-12T20:06:22.256Z    INFO    controller-runtime.manager.controller.instance    Starting EventSource    {"reconciler group": "<http://droplet.cattle.io|droplet.cattle.io>", "reconciler kind": "Instance", "source": "kind source: /, Kind="}
manager 2023-04-12T20:06:22.851Z    INFO    controller-runtime.manager.controller.importkeypair    Starting EventSource    {"reconciler group": "<http://droplet.cattle.io|droplet.cattle.io>", "reconciler kind": "ImportKeyPair", "source": "kind source: /, Kind="}
manager 2023-04-12T20:06:24.655Z    INFO    controller-runtime.manager.controller.importkeypair    Starting Controller    {"reconciler group": "<http://droplet.cattle.io|droplet.cattle.io>", "reconciler kind": "ImportKeyPair"}
manager 2023-04-12T20:06:25.154Z    INFO    controller-runtime.manager.controller.instance    Starting Controller    {"reconciler group": "<http://droplet.cattle.io|droplet.cattle.io>", "reconciler kind": "Instance"}
manager 2023-04-12T20:06:25.253Z    INFO    controller-runtime.manager.controller.importkeypair    Starting workers    {"reconciler group": "<http://droplet.cattle.io|droplet.cattle.io>", "reconciler kind": "ImportKeyPair", "worker count": 1}
manager 2023-04-12T20:06:25.253Z    INFO    controller-runtime.manager.controller.instance    Starting workers    {"reconciler group": "<http://droplet.cattle.io|droplet.cattle.io>", "reconciler kind": "Instance", "worker count": 1}
Error is being thrown in the VirtualMachineController func (r *VirtualMachineReconciler) createImportKeyPair(ctx context.Context, vm *hfv1.VirtualMachine) (status *hfv1.VirtualMachineStatus, err error) { status = vm.Status.DeepCopy() b64PubKey, ok := vm.Annotations["pubKey"] if !ok { return status, fmt.Errorf("unable to find label pubKey on VM") } pubKeyByte, err := b64.StdEncoding.DecodeString(b64PubKey) if err != nil { return status, err }
So it's doing createSecret, but can't do createImportKeyPair
w
I was wrong its the shim that creates the keypair.
a
All the other calls use CreateOrUpdate. Why does VirtualMachine controller only do update?
Nevermind, for the reconciler they all use update for things for that controller's object
CreateOrUpdate is for the Secret
I'm back
w
I opened a pr in shim repo for the fix for this.
I didn’t tag it yet and I’m out of pocket for the Next few hrs
a
I'll give a go local and report back, and test once you have the official out
👍 1
It provisions now but fails the liveness check
Copy code
// DO liveness check
func (r *VirtualMachineReconciler) doLivenessCheck(ctx context.Context, vm *hfv1.VirtualMachine,
	instance *dropletv1alpha1.Instance) (ready bool, err error) {
	keySecret := &v1.Secret{}
	var username, address string
	err = r.Get(ctx, types.NamespacedName{Name: vm.Spec.KeyPair, Namespace: vm.Namespace}, keySecret)
	if err != nil {
		return ready, err
	}
	if len(vm.Spec.SshUsername) != 0 {
		username = vm.Spec.SshUsername
	} else {
		username = "root"
	}
Can't find the secret
Copy code
1.681342582050377e+09   ERROR   controller.virtualmachine       Reconciler error        {"reconciler group": "<http://hobbyfarm.io|hobbyfarm.io>", "reconciler kind": "VirtualMachine", "name": "dynamic-6gk5wu6d23-5f149a48", "namespace": "hobbyfarm", "error": "Secret \"\" not found"}
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>
        /Users/jason/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>
        /Users/jason/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
vm.Spec.KeyPair is blank
I can't find anywhere in code this gets written to
Should be here in createSecret
Copy code
vm.Spec.KeyPair = secretName
	status.Status = secretCreated
	vm.Annotations["secret"] = "created"
	vm.Annotations["secretName"] = secretName
Stupid thing is the annotation is there.
So I added this to the liveness check #hack
Copy code
func (r *VirtualMachineReconciler) doLivenessCheck(ctx context.Context, vm *hfv1.VirtualMachine,
	instance *dropletv1alpha1.Instance) (ready bool, err error) {
	keySecret := &v1.Secret{}
	var username, address string
	var secretName string
	secretName = vm.Spec.KeyPair
	if secretName == "" {
		secretName = vm.Annotations["secretName"]
	}
Copy code
rc.Remote(command) is failing

func PerformLivenessCheck(address string, userName string, privateKey string, command string) (ready bool, err error) {

	rc, err := ssh.NewRemoteConnection(address, userName, privateKey)
	if err != nil {
		return ready, err
	}
	_, err = rc.Remote(command)
	if err != nil {
		return ready, err
	}
	ready = true
	return ready, nil
}
1.681343447487985e+09 ERROR controller.virtualmachine Reconciler error {"reconciler group": "hobbyfarm.io", "reconciler kind": "VirtualMachine", "name": "dynamic-m7ksjrccr3-03af39e7", "namespace": "hobbyfarm", "error": "ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem /Users/jason/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2 /Users/jason/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
I can connect to it manually
Copy code
ssh root@192.241.143.122 -i ./key.key
The authenticity of host '192.241.143.122 (192.241.143.122)' can't be established.
ED25519 key fingerprint is SHA256:hcabWgPyhRQyqZL3l3CnOWw5yy6M4i59bJKKWAd7Tqc.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.241.143.122' (ED25519) to the list of known hosts.
Welcome to Ubuntu 22.10 (GNU/Linux 5.19.0-23-generic x86_64)

 * Documentation:  <https://help.ubuntu.com>
 * Management:     <https://landscape.canonical.com>
 * Support:        <https://ubuntu.com/advantage>

  System information as of Thu Apr 13 00:14:54 UTC 2023

  System load:  0.04541015625      Users logged in:       0
  Usage of /:   1.0% of 154.96GB   IPv4 address for eth0: 192.241.143.122
  Memory usage: 2%                 IPv4 address for eth0: 10.10.0.5
  Swap usage:   0%                 IPv4 address for eth1: 10.116.0.2
  Processes:    122

0 updates can be applied immediately.



The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.
I pulled the key from a breakpoint in code
I'm stuck here. If I bypass the code I can "start" the scenario but the screen doesn't work.
w
Okay let's recap. What did you change in the hf-shim code?
a
I modified digitalocean_interface.go > doLivenessCheck starts at 191
Copy code
var secretName string
	secretName = vm.Spec.KeyPair
	if secretName == "" {
		secretName = vm.Annotations["secretName"]
	}
	err = r.Get(ctx, types.NamespacedName{Name: secretName, Namespace: vm.Namespace}, keySecret)
for some reason vm.Spec.KeyPair is not getting written 😞
It is written in the code, but doesn't end up in etcd
This a dirty hack, but I wanted to see if that would get me step further down the path
22 Views
Open in Slack
PreviousNext
Powered by