https://rancher.com/ logo
Title
f

fancy-jackal-96583

06/14/2022, 3:40 PM
Does it look like I upgraded right? But It's stil 1.23? I deployed update to newest stable version. I don't know if 1.24 is? Btw I am very green
g

gray-lawyer-73831

06/14/2022, 3:51 PM
kubectl version
,
k3s -v
, or
kubectl get nodes
are probably the best way to show the version. But it is most likely 1.23 yes. 1.24 isn’t marked as stable yet
👍 1
f

fancy-jackal-96583

06/14/2022, 8:45 PM
How bout these firings. Should I worry? I googled it without success….
g

gray-lawyer-73831

06/14/2022, 9:10 PM
That endpoint looks like
http
not
https
, and I believe only https is allowed now. See https://github.com/k3s-io/k3s/issues/3619#issuecomment-973188304 which might guide you in the right direction to scrape the correct scheme now
f

fancy-jackal-96583

06/14/2022, 9:49 PM
Man I'm screwed. Anyone wanna walk me threw it
These are my notes so far. Still haven't figured it out? Really not trying to start over from scratch. But I guess that how we learn…..
g

gray-lawyer-73831

06/15/2022, 4:27 PM
It’s okay, it’s all a learning experience! In this case, what will probably help you best/fastest is looking up how to configure prometheus metrics in your kubernetes cluster
f

fancy-jackal-96583

06/15/2022, 8:05 PM
Now grafana dash board doesn't come up and monitoring is in a loop
g

gray-lawyer-73831

06/15/2022, 8:25 PM
Probably misconfigured during an update. Was this a cluster that someone else setup and handed down to you?
f

fancy-jackal-96583

06/15/2022, 8:26 PM
Nope I set it up
Jeff herring and networkchuck Tutorials
Jeff geerling sorry

https://youtu.be/IafVCHkJbtI

And this one

https://youtu.be/X9fSMGkjtug

To be more specific
g

gray-lawyer-73831

06/15/2022, 8:29 PM
so, without explicitly watching those, I believe you should be able to do the same thing, but then just edit whatever config you used to deploy prometheus to use the https endpoint instead of http
f

fancy-jackal-96583

06/15/2022, 8:32 PM
They were make commands
These were the steps
I need to create a yaml and deploy it I think
Don't know my endpoints or ports
This is what I was going to put in the yaml. Like you suggested. But I know their end points aren't the same as mine and ports
Not sure what the call the yaml file either
g

gray-lawyer-73831

06/15/2022, 8:39 PM
Usually the name of the file doesn’t matter. I would check in
manifests/setup
and
manifests
if there’s any files for prometheus with some of this data. Also, I suspect the endpoints section in that data isn’t necessary, but I’m not an expert with prometheus so I’m not 100% certain about that
f

fancy-jackal-96583

06/15/2022, 8:41 PM
g

gray-lawyer-73831

06/15/2022, 9:16 PM
Those pretty aptly named! You can go through them and see what to edit. But likely the
prometheus-kubeControllerManager
and
prometheus-kubeScheduler
f

fancy-jackal-96583

06/15/2022, 9:21 PM
That look right
I'm getting closer I think
Don't wanna f it up. Too close I think
g

gray-lawyer-73831

06/15/2022, 9:22 PM
I think you are close too! Go ahead and f it up and keep learning. It might take some time, but you’ll figure it out 🙂
f

fancy-jackal-96583

06/15/2022, 9:22 PM
Lol
Thanks for all the help
g

gray-lawyer-73831

06/15/2022, 9:23 PM
That yaml looks like the endpoint. I think you want to get to the servicemonitor resource ideally
and no problem
f

fancy-jackal-96583

06/15/2022, 11:17 PM
How bout this info
Still trudging
g

gray-lawyer-73831

06/15/2022, 11:18 PM
You’ll want to describe that pod to see why it’s crashlooping like that
f

fancy-jackal-96583

06/16/2022, 10:15 AM
I'm hoping I going to run. Kubectl log deployment/ Prometheus-config-reloader -c Prometheus-K3S-0 ??? When I get home today for some more clues…..
Still looking for Guidance
g

gray-lawyer-73831

06/16/2022, 9:16 PM
what does
prometheus-kubeControllerManagerPrometheusDiscoveryService.yaml
contina?
f

fancy-jackal-96583

06/16/2022, 9:18 PM
I tried changing to https and did a reboot but still crashing
g

gray-lawyer-73831

06/16/2022, 9:21 PM
what about
prometheus-serviceMonitorKubeControllerManager.yaml
?
f

fancy-jackal-96583

06/16/2022, 9:24 PM
That looks like that
g

gray-lawyer-73831

06/16/2022, 9:24 PM
what about towards the bottom of that file? Same thing?
f

fancy-jackal-96583

06/16/2022, 9:25 PM
I could change that to https. Haven't tried that
Rebooting now
g

gray-lawyer-73831

06/16/2022, 9:27 PM
nah that’s not going to work. The name of that actually matches the name of the port in the Service definition
f

fancy-jackal-96583

06/16/2022, 9:29 PM
Yeah I didn't like that. Fan started fast
g

gray-lawyer-73831

06/16/2022, 9:29 PM
Will you put that back to http-metrics and actually just change the ports in
prometheus-kubeControllerManagerPrometheusDiscoveryService.yaml
to both use
10257
? Probably put back whatever the other changes you made to these files as well. I think just updating those ports to be 10257 should work. Also update the ports in
prometheus-kubeSchedulerPrometheusDiscoveryService.yaml
to be
10259
I expect in that file for scheduler they’re showing
10251
currently
f

fancy-jackal-96583

06/16/2022, 9:35 PM
Yes
Do I still reboot?
g

gray-lawyer-73831

06/16/2022, 9:36 PM
Reboot the machine? Probably not necessary. But rebuild using those files. Might be able to just
kubectl apply -f <file_name>
them too
f

fancy-jackal-96583

06/16/2022, 9:40 PM
I don't think so
Unless it takes awhile
g

gray-lawyer-73831

06/16/2022, 9:42 PM
kubectl describe -n monitoring pod/prometheus-k8s-0
f

fancy-jackal-96583

06/16/2022, 9:43 PM
g

gray-lawyer-73831

06/16/2022, 9:44 PM
all the other files are set back to their original values right?
f

fancy-jackal-96583

06/16/2022, 9:44 PM
Correct I was very careful about that
Dam theirs so much
g

gray-lawyer-73831

06/16/2022, 9:46 PM
it’s okay you showed enough here
I’d like to get back to the point in this screenshot and then just edit those service ports 😕
f

fancy-jackal-96583

06/17/2022, 9:30 AM
https://github.com/carlosedp/cluster-monitoring This is what I did per Jeff geerling. Not sure it it helps
So I ran
kubectl delete -n monitoring pod/prometheus-k8s-0
Then it recreated and I had access to grafana again. Don't really know if it fixed it I doubt it