https://rancher.com/ logo
Powered by
Title
w

worried-fountain-60974

03/17/2023, 3:06 PM
Thread for this here. So we stop bothering people unnecessarily.
👍 1
I was working on this in December, and part of the issue is that I need to upgrade the KubeBuilder framework from I think v2 to v3 for this project.
a

astonishing-engine-86351

03/17/2023, 3:09 PM
I'm vaguely remembering having this similar issue in another project I worked back at PlatformOne
w

worried-fountain-60974

03/17/2023, 3:15 PM
Changing a couple things, one sec
Okay waiting for build then I should be able to cut an RC for you
a

astonishing-engine-86351

03/17/2023, 3:24 PM
Cool
w

worried-fountain-60974

03/17/2023, 3:28 PM
Done. Chart was released as well
v0.2.0-rc0
a

astonishing-engine-86351

03/17/2023, 3:28 PM
cool. I'm testing right now. Got new errors 🙂
w

worried-fountain-60974

03/17/2023, 3:28 PM
Oh goody
a

astonishing-engine-86351

03/17/2023, 3:28 PM
missing my secret. One moment
1.6790671281463888e+09    ERROR    controller.instance    Reconciler error    {"reconciler group": "<http://ec2.cattle.io|ec2.cattle.io>", "reconciler kind": "Instance", "name": "scheduled-b8042a1b-a3dc2f6c", "namespace": "hobbyfarm", "error": "No key aws_ac │
│ cess_key exists in Instance secret"}
I think I need to fix my key
w

worried-fountain-60974

03/17/2023, 3:34 PM
Yeah I think It needs to be 
access_key
and 
secret_key
a

astonishing-engine-86351

03/17/2023, 3:40 PM
Whats this mean 
1.6790676233686204e+09    ERROR    controller.instance    Reconciler error    {"reconciler group": "<http://ec2.cattle.io|ec2.cattle.io>", "reconciler kind": "Instance", "name": "scheduled-b8042a1b-a3dc2f6c", "namespace": "hobbyfarm", "error": "EmptyStaticCreds: static credentials are emp │
│ <http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>                                                                                                                                                                                   │
│     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266                                                                                                                                                                           │
│ <http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>                                                                                                                                                                                         │
│     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227                                                                                                                                                                           │
│
w

worried-fountain-60974

03/17/2023, 3:44 PM
That’s a new one to me. Let me investigate
a

astonishing-engine-86351

03/17/2023, 3:46 PM
ec2 operator importkeypair controller
starting around line 78
Might be something wrong with my creds. or entered them wrong or something. I'm not a ec2 expert by any stretch
w

worried-fountain-60974

03/17/2023, 3:51 PM
Is that error the full error?
a

astonishing-engine-86351

03/17/2023, 3:52 PM
1.6790682378071728e+09    INFO    controllers.ImportKeyPair    Fetching secret:     {"secret": "hobbyfarm/aws-secret"}                                                                                                                                                     │
│ 1.6790682378140862e+09    INFO    controllers.Instance    Fetching secret:     {"secret": "hobbyfarm/aws-secret"}                                                                                                                                                          │
│ 1.6790682378140905e+09    INFO    controllers.ImportKeyPair    Error during keypair creation    {"importkeypair": "hobbyfarm/scheduled-b8042a1b-a3dc2f6c"}                                                                                                                 │
│ 1.6790682378152735e+09    INFO    controllers.Instance    Creating instance    {"instance": "hobbyfarm/scheduled-b8042a1b-a3dc2f6c"}                                                                                                                                       │
│ 1.6790682378157363e+09    ERROR    controller.importkeypair    Reconciler error    {"reconciler group": "<http://ec2.cattle.io|ec2.cattle.io>", "reconciler kind": "ImportKeyPair", "name": "scheduled-b8042a1b-a3dc2f6c", "namespace": "hobbyfarm", "error": "EmptyStaticCreds: static credentia │
│ <http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>                                                                                                                                                                                   │
│     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266                                                                                                                                                                           │
│ <http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>                                                                                                                                                                                         │
│     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227                                                                                                                                                                           │
│ 1.67906823781659e+09    ERROR    controllers.Instance    scheduled-b8042a1b-a3dc2f6c    {"instance": "hobbyfarm/scheduled-b8042a1b-a3dc2f6c", "error": "Error during instance creation"}                                                                                   │
│ <http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile>                                                                                                                                                                                             │
│     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:114                                                                                                                                                                           │
│ <http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler>                                                                                                                                                                                      │
│     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:311                                                                                                                                                                           │
│ <http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>                                                                                                                                                                                   │
│     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266                                                                                                                                                                           │
│ <http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>                                                                                                                                                                                         │
│     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227                                                                                                                                                                           │
│ 1.6790682378170874e+09    ERROR    controller.instance    Reconciler error    {"reconciler group": "<http://ec2.cattle.io|ec2.cattle.io>", "reconciler kind": "Instance", "name": "scheduled-b8042a1b-a3dc2f6c", "namespace": "hobbyfarm", "error": "EmptyStaticCreds: static credentials are emp │
│ <http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>                                                                                                                                                                                   │
│     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266                                                                                                                                                                           │
│ <http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>                                                                                                                                                                                         │
│     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
Finding that here https://docs.aws.amazon.com/sdk-for-go/api/aws/credentials/
image.png
w

worried-fountain-60974

03/17/2023, 3:58 PM
can you delete the 
virtualmachine
and 
session
that you're using right now
let it all clean up and then try again. I'm wondering if this is some kind of weird half-state
a

astonishing-engine-86351

03/17/2023, 3:58 PM
yep
Need to run a quick errand. Be back in 10
Getting an imagepullbackoff on the rc image
I was running it locally before
It's "working" at least I'm getting unauthorized now
Closer, need to get public ips
w

worried-fountain-60974

03/17/2023, 5:55 PM
How are things going now? Sorry had to step OOO for a min
a

astonishing-engine-86351

03/17/2023, 5:55 PM
Good timing what do y'all do with your amis to setup ssh on ec2. I have 🤏 experience with ec2 instances.
It's creating instances now
I'm still running hf-shim manually as I get an imagepullbackoff on the image with the rc chart
w

worried-fountain-60974

03/17/2023, 5:56 PM
Nothing special to be done in the AMI - the Gargantua server is literally just SSHing to the public IP of the ec2 instance. So I guess in that manner you need network LOS
The image issue might be because of docker hub bullshit. pleasehold
hm docker hub shows it as being up there....
it's linux/amd64
a

astonishing-engine-86351

03/17/2023, 5:58 PM
That's my issue I run arm
w

worried-fountain-60974

03/17/2023, 5:58 PM
how DARE you
Okay let me refuckulate the cicd
a

astonishing-engine-86351

03/17/2023, 5:58 PM
🤣
The M1 battery runs forever
w

worried-fountain-60974

03/17/2023, 5:59 PM
It really does.
a

astonishing-engine-86351

03/17/2023, 5:59 PM
I do miss my linux laptop though
w

worried-fountain-60974

03/17/2023, 5:59 PM
I got so damn tired of issues with pulling arm images that I set up a docker remote in the homelab
Okay added a 
v0.2.0-rc1
tag, give it a few to build and push
oh wait hf-shim my bad. well I added support for ec2 LOL
Okay momentarily you should see 
linux/arm64
support for both ec2-operator and hf-shim-operator
Both with the 
v0.2.0-rc1
tag
a

astonishing-engine-86351

03/17/2023, 6:26 PM
Still getting image pull backoff? Helms working fine
I can pull it manually
helm chart is looking for rc0
Failed to pull image "hobbyfarm/hf-shim-operator:v0.2.0-rc0": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/hobbyfarm/hf-shim-operator:v0.2.0-rc │ │ 0": no match for platform in manifest: not found
might be a me issue 1 sec
Yeah fat fingered it
Where can I diagnose gargantua connecting to the VM?
I have a instance with public IP and I'm stuck here
gargantua is saying this 
│ gargantua-ccdfc9f9f-jsk4f I0317 19:23:55.296930       1 vmclaimserver.go:121] retrieved vmc dynamic-t4o7nspu5k                                                                                                                                  │
│ gargantua-ccdfc9f9f-jsk4f I0317 19:24:00.333051       1 vmclaimserver.go:121] retrieved vmc dynamic-t4o7nspu5k
After that timed out and was reclaimed it appears things are in a bad state ``````
There's a logic error in here. Created a new dynamic and getting this after uninstalling an reinstalling hf
Tried kicking the pod
image.png
Keypair string is empty
Changed it to this since this secret exists
The SSH Check is failing. I can do it manually though
If I grab the private key and use it locally I can ssh in just fine
Ok, It's losing track of the KeyPair, so looks that's a big part of the problem
image.png
w

worried-fountain-60974

03/23/2023, 11:52 PM
@astonishing-engine-86351 do we need a troubleshooting sesh on this?
a

astonishing-engine-86351

03/24/2023, 1:03 AM
That would be a good thing. I had hoped to grab you at this week's office hours, but got busy
I'm traveling tomorrow morning but will be in FL by 3 ET if you have a couple hours tomorrow afternoon
w

worried-fountain-60974

03/24/2023, 3:09 AM
I miiight have time. Ping me when you’re free
a

astonishing-engine-86351

03/24/2023, 8:34 PM
Got in late we can pick this up next week
w

worried-fountain-60974

03/24/2023, 8:38 PM
Sure thing
#hobbyfarmJoin Slack
Powered by