Does it look like I upgraded right But It s stil 1 23 I depl rancher-users #k3s

Title

fancy-jackal-96583

06/14/2022, 3:40 PM

Does it look like I upgraded right? But It's stil 1.23? I deployed update to newest stable version. I don't know if 1.24 is? Btw I am very green

gray-lawyer-73831

06/14/2022, 3:51 PM

kubectl version

k3s -v

, or

kubectl get nodes

are probably the best way to show the version. But it is most likely 1.23 yes. 1.24 isn’t marked as stable yet

👍 1

fancy-jackal-96583

06/14/2022, 8:45 PM

How bout these firings. Should I worry? I googled it without success….

gray-lawyer-73831

06/14/2022, 9:10 PM

That endpoint looks like

http

not

https

, and I believe only https is allowed now. See https://github.com/k3s-io/k3s/issues/3619#issuecomment-973188304 which might guide you in the right direction to scrape the correct scheme now

fancy-jackal-96583

06/14/2022, 9:49 PM

Man I'm screwed. Anyone wanna walk me threw it

These are my notes so far. Still haven't figured it out? Really not trying to start over from scratch. But I guess that how we learn…..

gray-lawyer-73831

06/15/2022, 4:27 PM

It’s okay, it’s all a learning experience! In this case, what will probably help you best/fastest is looking up how to configure prometheus metrics in your kubernetes cluster

fancy-jackal-96583

06/15/2022, 8:05 PM

Now grafana dash board doesn't come up and monitoring is in a loop

gray-lawyer-73831

06/15/2022, 8:25 PM

Probably misconfigured during an update. Was this a cluster that someone else setup and handed down to you?

fancy-jackal-96583

06/15/2022, 8:26 PM

Nope I set it up

Jeff herring and networkchuck Tutorials

Jeff geerling sorry

https://youtu.be/IafVCHkJbtI▾

And this one

https://youtu.be/X9fSMGkjtug▾

To be more specific

gray-lawyer-73831

06/15/2022, 8:29 PM

so, without explicitly watching those, I believe you should be able to do the same thing, but then just edit whatever config you used to deploy prometheus to use the https endpoint instead of http

fancy-jackal-96583

06/15/2022, 8:32 PM

They were make commands

These were the steps

I need to create a yaml and deploy it I think

Don't know my endpoints or ports

This is what I was going to put in the yaml. Like you suggested. But I know their end points aren't the same as mine and ports

Not sure what the call the yaml file either

gray-lawyer-73831

06/15/2022, 8:39 PM

Usually the name of the file doesn’t matter. I would check in

manifests/setup

and

manifests

if there’s any files for prometheus with some of this data. Also, I suspect the endpoints section in that data isn’t necessary, but I’m not an expert with prometheus so I’m not 100% certain about that

fancy-jackal-96583

06/15/2022, 8:41 PM

gray-lawyer-73831

06/15/2022, 9:16 PM

Those pretty aptly named! You can go through them and see what to edit. But likely the

prometheus-kubeControllerManager

and

prometheus-kubeScheduler

fancy-jackal-96583

06/15/2022, 9:21 PM

That look right

I'm getting closer I think

Don't wanna f it up. Too close I think

gray-lawyer-73831

06/15/2022, 9:22 PM

I think you are close too! Go ahead and f it up and keep learning. It might take some time, but you’ll figure it out 🙂

fancy-jackal-96583

06/15/2022, 9:22 PM

Lol

Thanks for all the help

gray-lawyer-73831

06/15/2022, 9:23 PM

That yaml looks like the endpoint. I think you want to get to the servicemonitor resource ideally

and no problem

fancy-jackal-96583

06/15/2022, 11:17 PM

How bout this info

Still trudging

gray-lawyer-73831

06/15/2022, 11:18 PM

You’ll want to describe that pod to see why it’s crashlooping like that

fancy-jackal-96583

06/16/2022, 10:15 AM

I'm hoping I going to run. Kubectl log deployment/ Prometheus-config-reloader -c Prometheus-K3S-0 ??? When I get home today for some more clues…..

Still looking for Guidance

gray-lawyer-73831

06/16/2022, 9:16 PM

what does

prometheus-kubeControllerManagerPrometheusDiscoveryService.yaml

contina?

fancy-jackal-96583

06/16/2022, 9:18 PM

I tried changing to https and did a reboot but still crashing

gray-lawyer-73831

06/16/2022, 9:21 PM

what about

prometheus-serviceMonitorKubeControllerManager.yaml

fancy-jackal-96583

06/16/2022, 9:24 PM

That looks like that

gray-lawyer-73831

06/16/2022, 9:24 PM

what about towards the bottom of that file? Same thing?

fancy-jackal-96583

06/16/2022, 9:25 PM

I could change that to https. Haven't tried that

Rebooting now

gray-lawyer-73831

06/16/2022, 9:27 PM

nah that’s not going to work. The name of that actually matches the name of the port in the Service definition

fancy-jackal-96583

06/16/2022, 9:29 PM

Yeah I didn't like that. Fan started fast

gray-lawyer-73831

06/16/2022, 9:29 PM

Will you put that back to http-metrics and actually just change the ports in

prometheus-kubeControllerManagerPrometheusDiscoveryService.yaml

to both use

? Probably put back whatever the other changes you made to these files as well. I think just updating those ports to be 10257 should work. Also update the ports in

prometheus-kubeSchedulerPrometheusDiscoveryService.yaml

to be

I expect in that file for scheduler they’re showing

currently

fancy-jackal-96583

06/16/2022, 9:35 PM

Yes

Do I still reboot?

gray-lawyer-73831

06/16/2022, 9:36 PM

Reboot the machine? Probably not necessary. But rebuild using those files. Might be able to just

kubectl apply -f <file_name>

them too

fancy-jackal-96583

06/16/2022, 9:40 PM

I don't think so

Unless it takes awhile

gray-lawyer-73831

06/16/2022, 9:42 PM

kubectl describe -n monitoring pod/prometheus-k8s-0

fancy-jackal-96583

06/16/2022, 9:43 PM

gray-lawyer-73831

06/16/2022, 9:44 PM

all the other files are set back to their original values right?

fancy-jackal-96583

06/16/2022, 9:44 PM

Correct I was very careful about that

Dam theirs so much

gray-lawyer-73831

06/16/2022, 9:46 PM

it’s okay you showed enough here

I’d like to get back to the point in this screenshot and then just edit those service ports 😕

fancy-jackal-96583

06/17/2022, 9:30 AM

https://github.com/carlosedp/cluster-monitoring This is what I did per Jeff geerling. Not sure it it helps

So I ran

kubectl delete -n monitoring pod/prometheus-k8s-0

Then it recreated and I had access to grafana again. Don't really know if it fixed it I doubt it

#k3s Join Slack