https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# hobbyfarm
a
This message was deleted.
👍 1
w
I was working on this in December, and part of the issue is that I need to upgrade the KubeBuilder framework from I think v2 to v3 for this project.
a
I'm vaguely remembering having this similar issue in another project I worked back at PlatformOne
w
Changing a couple things, one sec
Okay waiting for build then I should be able to cut an RC for you
a
Cool
w
Done. Chart was released as well
v0.2.0-rc0
a
cool. I'm testing right now. Got new errors 🙂
w
Oh goody
a
missing my secret. One moment
Copy code
1.6790671281463888e+09    ERROR    controller.instance    Reconciler error    {"reconciler group": "<http://ec2.cattle.io|ec2.cattle.io>", "reconciler kind": "Instance", "name": "scheduled-b8042a1b-a3dc2f6c", "namespace": "hobbyfarm", "error": "No key aws_ac │
│ cess_key exists in Instance secret"}
I think I need to fix my key
w
Yeah I think It needs to be 
access_key
and 
secret_key
a
Whats this mean
Copy code
1.6790676233686204e+09    ERROR    controller.instance    Reconciler error    {"reconciler group": "<http://ec2.cattle.io|ec2.cattle.io>", "reconciler kind": "Instance", "name": "scheduled-b8042a1b-a3dc2f6c", "namespace": "hobbyfarm", "error": "EmptyStaticCreds: static credentials are emp │
│ <http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>                                                                                                                                                                                   │
│     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266                                                                                                                                                                           │
│ <http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>                                                                                                                                                                                         │
│     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227                                                                                                                                                                           │
│
w
That’s a new one to me. Let me investigate
a
ec2 operator importkeypair controller
starting around line 78
Might be something wrong with my creds. or entered them wrong or something. I'm not a ec2 expert by any stretch
w
Is that error the full error?
a
Copy code
1.6790682378071728e+09    INFO    controllers.ImportKeyPair    Fetching secret:     {"secret": "hobbyfarm/aws-secret"}                                                                                                                                                     │
│ 1.6790682378140862e+09    INFO    controllers.Instance    Fetching secret:     {"secret": "hobbyfarm/aws-secret"}                                                                                                                                                          │
│ 1.6790682378140905e+09    INFO    controllers.ImportKeyPair    Error during keypair creation    {"importkeypair": "hobbyfarm/scheduled-b8042a1b-a3dc2f6c"}                                                                                                                 │
│ 1.6790682378152735e+09    INFO    controllers.Instance    Creating instance    {"instance": "hobbyfarm/scheduled-b8042a1b-a3dc2f6c"}                                                                                                                                       │
│ 1.6790682378157363e+09    ERROR    controller.importkeypair    Reconciler error    {"reconciler group": "<http://ec2.cattle.io|ec2.cattle.io>", "reconciler kind": "ImportKeyPair", "name": "scheduled-b8042a1b-a3dc2f6c", "namespace": "hobbyfarm", "error": "EmptyStaticCreds: static credentia │
│ <http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>                                                                                                                                                                                   │
│     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266                                                                                                                                                                           │
│ <http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>                                                                                                                                                                                         │
│     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227                                                                                                                                                                           │
│ 1.67906823781659e+09    ERROR    controllers.Instance    scheduled-b8042a1b-a3dc2f6c    {"instance": "hobbyfarm/scheduled-b8042a1b-a3dc2f6c", "error": "Error during instance creation"}                                                                                   │
│ <http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile>                                                                                                                                                                                             │
│     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:114                                                                                                                                                                           │
│ <http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler>                                                                                                                                                                                      │
│     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:311                                                                                                                                                                           │
│ <http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>                                                                                                                                                                                   │
│     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266                                                                                                                                                                           │
│ <http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>                                                                                                                                                                                         │
│     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227                                                                                                                                                                           │
│ 1.6790682378170874e+09    ERROR    controller.instance    Reconciler error    {"reconciler group": "<http://ec2.cattle.io|ec2.cattle.io>", "reconciler kind": "Instance", "name": "scheduled-b8042a1b-a3dc2f6c", "namespace": "hobbyfarm", "error": "EmptyStaticCreds: static credentials are emp │
│ <http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>                                                                                                                                                                                   │
│     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266                                                                                                                                                                           │
│ <http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>                                                                                                                                                                                         │
│     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
w
can you delete the 
virtualmachine
and 
session
that you're using right now
let it all clean up and then try again. I'm wondering if this is some kind of weird half-state
a
yep
Need to run a quick errand. Be back in 10
Getting an imagepullbackoff on the rc image
I was running it locally before
It's "working" at least I'm getting unauthorized now
Closer, need to get public ips
w
How are things going now? Sorry had to step OOO for a min
a
Good timing what do y'all do with your amis to setup ssh on ec2. I have 🤏 experience with ec2 instances.
It's creating instances now
I'm still running hf-shim manually as I get an imagepullbackoff on the image with the rc chart
w
Nothing special to be done in the AMI - the Gargantua server is literally just SSHing to the public IP of the ec2 instance. So I guess in that manner you need network LOS
The image issue might be because of docker hub bullshit. pleasehold
hm docker hub shows it as being up there....
it's linux/amd64
a
That's my issue I run arm
w
how DARE you
Okay let me refuckulate the cicd
a
🤣
The M1 battery runs forever
w
It really does.
a
I do miss my linux laptop though
w
I got so damn tired of issues with pulling arm images that I set up a docker remote in the homelab
Okay added a 
v0.2.0-rc1
tag, give it a few to build and push
oh wait hf-shim my bad. well I added support for ec2 LOL
Okay momentarily you should see 
linux/arm64
support for both ec2-operator and hf-shim-operator
Both with the 
v0.2.0-rc1
tag
a
Still getting image pull backoff? Helms working fine
I can pull it manually
helm chart is looking for rc0
Failed to pull image "hobbyfarm/hf-shim-operator:v0.2.0-rc0": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/hobbyfarm/hf-shim-operator:v0.2.0-rc │ │ 0": no match for platform in manifest: not found
might be a me issue 1 sec
Yeah fat fingered it
Where can I diagnose gargantua connecting to the VM?
I have a instance with public IP and I'm stuck here
gargantua is saying this
Copy code
│ gargantua-ccdfc9f9f-jsk4f I0317 19:23:55.296930       1 vmclaimserver.go:121] retrieved vmc dynamic-t4o7nspu5k                                                                                                                                  │
│ gargantua-ccdfc9f9f-jsk4f I0317 19:24:00.333051       1 vmclaimserver.go:121] retrieved vmc dynamic-t4o7nspu5k
After that timed out and was reclaimed it appears things are in a bad state ``````
There's a logic error in here. Created a new dynamic and getting this after uninstalling an reinstalling hf
Tried kicking the pod
Keypair string is empty
Changed it to this since this secret exists
The SSH Check is failing. I can do it manually though
If I grab the private key and use it locally I can ssh in just fine
Ok, It's losing track of the KeyPair, so looks that's a big part of the problem
w
@astonishing-engine-86351 do we need a troubleshooting sesh on this?
a
That would be a good thing. I had hoped to grab you at this week's office hours, but got busy
I'm traveling tomorrow morning but will be in FL by 3 ET if you have a couple hours tomorrow afternoon
w
I miiight have time. Ping me when you’re free
a
Got in late we can pick this up next week
w
Sure thing
11 Views
Open in Slack
PreviousNext
Powered by