https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# harvester
a
This message was deleted.
s
Hi @sticky-summer-13450, could you generate the SB for investigation? Thanks!
s
Sorry, I should have uploaded a support bundle earlier.
supportbundle_5bb44244-434e-4530-ad35-35c4ef1ff661_2024-07-06T19-09-14Z.zip
p
s
Thank you - yes, that was exactly the issue. Having followed those simple instructions the process has moved onto pre-draining the second node. Hopefully the process will complete without further issues 🙂
🙌 2
It did complete without further issue. Thank you so much 🙂 The next job is to get the monitoring / logging installed and enabled - which, if I understood the support I got last month, should be in here?
s
I am not sure about the monitoring/logging issue. Did you encounter any problems this time?
s
Hi Vicente - after freeing space be removing old container images which had not been removed by previous installs, then the cluster upgrade stuck after the first node is upgraded issue, there were no further issues. But Jian Wang says my 
cluster also does not have rancher-monitoring managed chart
and suggested that 
we can delete this managedchart first; after upgrade, then we create it back. (in v1.3.0, this is an addon <https://docs.harvesterhci.io/v1.3/advanced/addons> )
. I'm just after knowing how to create it back 🙂
s
Hi @ancient-pizza-13099, could you help with above question?
I thought @ancient-pizza-13099 meant you can enable it back by addons. But we can still wait the confirmation/suggestion from him
s
I thought @ancient-pizza-13099 meant you can enable it back by addons
Yeh - I hoped that the monitoring/logging addon might just appear as disabled in the addons, but as my picture showed above, it doesn't
a
@sticky-summer-13450 I guess your cluster is upgraded to v1.3.1 now. We can follow below steps to recover:
Copy code
(1) Confirm the CRD managedchart version, should be as such

$ kubectl get managedchart -n fleet-local rancher-monitoring-crd -ojsonpath="{.spec.version}"
103.0.3+up45.31.1

$ kubectl get managedchart -n fleet-local rancher-logging-crd -ojsonpath="{.spec.version}"
103.0.0+up3.17.10


(2) prepare two addon yaml files

<https://github.com/harvester/harvester-installer/blob/2d74427a8635b8a167a913c41252d9d902b4578e/pkg/config/templates/rancherd-22-addons.yaml#L38>


cat > logging.yam << 'EOF'
apiVersion: <http://harvesterhci.io/v1beta1|harvesterhci.io/v1beta1>
kind: Addon
metadata:
  name: rancher-logging
  namespace: cattle-logging-system
spec:
  repo: <http://harvester-cluster-repo.cattle-system.svc/charts>
  version: "103.0.0+up3.17.10"
  chart: rancher-logging
  enabled: false
  valuesContent: |
    systemdLogPath: /var/log/journal
    disablePvc: true
    additionalLoggingSources:
      rke2:
        enabled: true
      kubeAudit:
        enabled: true
        auditFilename: "audit.log"
        pathPrefix: "/var/lib/rancher/rke2/server/logs/"
        loggingRef: "harvester-kube-audit-log-ref"
        fluentbit:
          logTag: kube-audit
          tolerations:
          - key: <http://node-role.kubernetes.io/controlplane|node-role.kubernetes.io/controlplane>
            value: "true"
            effect: NoSchedule
          - key: <http://node-role.kubernetes.io/etcd|node-role.kubernetes.io/etcd>
            value: "true"
            effect: NoExecute
    fluentd:
      resources:
        limits:
          cpu: 1000m
          memory: 800Mi
        requests:
          cpu: 100m
          memory: 200Mi
    fluentbit:
      resources:
        limits:
          cpu: 200m
          memory: 200Mi
        requests:
          cpu: 50m
          memory: 50Mi
    eventTailer:
      name: harvester-default
      namespace: cattle-logging-system
      controlNamespace: cattle-logging-system
      workloadOverrides:
        containers:
        - image: rancher/harvester-eventrouter:v0.2.0
          name: event-tailer
          resources:
            limits:
              cpu: 100m
              memory: 100Mi
            requests:
              cpu: 20m
              memory: 20Mi

EOF


NOTE: replace below `YOUR_VIP` to the VIP/DNS you used
        
cat > monitoring.yaml << 'EOF'      
apiVersion: <http://harvesterhci.io/v1beta1|harvesterhci.io/v1beta1>
kind: Addon
metadata:
  name: rancher-monitoring
  namespace: cattle-monitoring-system
spec:
  repo: <http://harvester-cluster-repo.cattle-system.svc/charts>
  version: "103.0.3+up45.31.1"
  chart: rancher-monitoring
  enabled: false
  valuesContent: |
    alertmanager:
      enabled: true
      config:
        global:
          resolve_timeout: 5m
      service:
        port: 9093
      alertmanagerSpec:
        externalUrl: "<https://YOUR_VIP/api/v1/namespaces/cattle-monitoring-system/services/http:rancher-monitoring-alertmanager:9093/proxy/>"
        retention: 120h
        storage:
          volumeClaimTemplate:
            spec:
              storageClassName: harvester-longhorn
              accessModes:
              - ReadWriteOnce
              resources:
                requests:
                  storage: 5Gi
        resources:
          limits:
            memory: 600Mi
            cpu: 1000m
          requests:
            memory: 100Mi
            cpu: 100m
    grafana:
      persistence:
        enabled: true
        size: 2Gi
        storageClassName: harvester-longhorn
        type: pvc
        accessModes:
        - ReadWriteOnce
      resources:
        limits:
          cpu: 200m
          memory: 500Mi
        requests:
          cpu: 100m
          memory: 200Mi
    prometheus:
      service:
        port: 9090
      prometheusSpec:
        evaluationInterval: 1m
        resources:
          limits:
            cpu: 1000m
            memory: 2500Mi
          requests:
            cpu: 750m
            memory: 1750Mi
        retention: 5d
        retentionSize: 50GiB
        scrapeInterval: 1m
        storageSpec:
          volumeClaimTemplate:
            spec:
              accessModes:
              - ReadWriteOnce
              resources:
                requests:
                  storage: 50Gi
              storageClassName: harvester-longhorn
              volumeMode: Filesystem
        externalUrl: "<https://YOUR_VIP/api/v1/namespaces/cattle-monitoring-system/services/http:rancher-monitoring-prometheus:9090/proxy/>"
    prometheus-node-exporter:
      resources:
        limits:
          cpu: 200m
          memory: 180Mi
        requests:
          cpu: 100m
          memory: 30Mi
EOF

(3) create addon
$ kubectl apply -f logging.yaml
$ kubectl apply -f monitoring.yaml

(4) enable addon on Harvester UI and observe the status
👀 1
s
Excellent instructions, thanks @ancient-pizza-13099. Unfortunately the addon deployment says it has failed, the helm install jobs are failing - the last logs in the pods for those 
helm-install-rancher-logging
and 
helm-install-rancher-monitoring
jobs are erroring with this being the last lines of the logs.
Copy code
+ echo 'Installing helm_v3 chart'
+ helm_v3 install --version 103.0.0+up3.17.10 rancher-logging rancher-logging/rancher-logging --values /config/values-01_HelmChart.yaml
Error: INSTALLATION FAILED: cannot re-use a name that is still in use
Copy code
+ echo 'Installing helm_v3 chart'
+ helm_v3 install --version 103.0.3+up45.31.1 rancher-monitoring rancher-monitoring/rancher-monitoring --values /config/values-01_HelmChart.yaml
Error: INSTALLATION FAILED: cannot re-use a name that is still in use
Oddly, the metrics are now appearing in the dashboard.
a
Copy code
kubectl get secret -A | grep rancher-logging
kubectl get secret -A | grep rancher-monitoring
s
Copy code
kubectl get secret -A | grep rancher-logging
cattle-logging-system                    chart-values-rancher-logging                                        Opaque                                1      39m
cattle-logging-system                    rancher-logging-kube-audit-fluentbit                                Opaque                                1      625d
cattle-logging-system                    rancher-logging-kube-audit-fluentbit-token-bdvp4                    <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token>   3      625d
cattle-logging-system                    rancher-logging-kube-audit-fluentd                                  Opaque                                4      625d
cattle-logging-system                    rancher-logging-kube-audit-fluentd-app                              Opaque                                1      625d
cattle-logging-system                    rancher-logging-kube-audit-fluentd-configcheck-ac2d4553             Opaque                                4      625d
cattle-logging-system                    rancher-logging-kube-audit-fluentd-configcheck-output-ac2d4553      Opaque                                0      625d
cattle-logging-system                    rancher-logging-kube-audit-fluentd-output                           Opaque                                0      625d
cattle-logging-system                    rancher-logging-kube-audit-fluentd-token-2dwx6                      <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token>   3      625d
cattle-logging-system                    rancher-logging-rke2-journald-aggregator-token-g5zlx                <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token>   3      625d
cattle-logging-system                    rancher-logging-root-fluentbit                                      Opaque                                1      625d
cattle-logging-system                    rancher-logging-root-fluentbit-token-jqr6g                          <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token>   3      625d
cattle-logging-system                    rancher-logging-root-fluentd                                        Opaque                                4      625d
cattle-logging-system                    rancher-logging-root-fluentd-app                                    Opaque                                1      625d
cattle-logging-system                    rancher-logging-root-fluentd-configcheck-ac2d4553                   Opaque                                4      625d
cattle-logging-system                    rancher-logging-root-fluentd-configcheck-output-ac2d4553            Opaque                                0      625d
cattle-logging-system                    rancher-logging-root-fluentd-output                                 Opaque                                0      625d
cattle-logging-system                    rancher-logging-root-fluentd-token-4vbkn                            <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token>   3      625d
cattle-logging-system                    rancher-logging-token-fmn2k                                         <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token>   3      625d
cattle-logging-system                    sh.helm.release.v1.rancher-logging-crd.v1126                        <http://helm.sh/release.v1|helm.sh/release.v1>                    1      8d
cattle-logging-system                    sh.helm.release.v1.rancher-logging-crd.v1127                        <http://helm.sh/release.v1|helm.sh/release.v1>                    1      8d
cattle-logging-system                    sh.helm.release.v1.rancher-logging.v2147                            <http://helm.sh/release.v1|helm.sh/release.v1>                    1      308d
cattle-logging-system                    sh.helm.release.v1.rancher-logging.v2148                            <http://helm.sh/release.v1|helm.sh/release.v1>                    1      308d
cattle-logging-system                    sh.helm.release.v1.rancher-logging.v2149                            <http://helm.sh/release.v1|helm.sh/release.v1>                    1      308d
cattle-logging-system                    sh.helm.release.v1.rancher-logging.v2150                            <http://helm.sh/release.v1|helm.sh/release.v1>                    1      308d
cattle-logging-system                    sh.helm.release.v1.rancher-logging.v2151                            <http://helm.sh/release.v1|helm.sh/release.v1>                    1      308d
cattle-logging-system                    sh.helm.release.v1.rancher-logging.v2152                            <http://helm.sh/release.v1|helm.sh/release.v1>                    1      308d
cattle-logging-system                    sh.helm.release.v1.rancher-logging.v2153                            <http://helm.sh/release.v1|helm.sh/release.v1>                    1      308d
cattle-logging-system                    sh.helm.release.v1.rancher-logging.v2154                            <http://helm.sh/release.v1|helm.sh/release.v1>                    1      307d
cattle-logging-system                    sh.helm.release.v1.rancher-logging.v2155                            <http://helm.sh/release.v1|helm.sh/release.v1>                    1      307d
cattle-logging-system                    sh.helm.release.v1.rancher-logging.v2156                            <http://helm.sh/release.v1|helm.sh/release.v1>                    1      307d
Copy code
kubectl get secret -A | grep rancher-monitoring
cattle-monitoring-system                 alertmanager-rancher-monitoring-alertmanager                        Opaque                                2      625d
cattle-monitoring-system                 alertmanager-rancher-monitoring-alertmanager-generated              Opaque                                2      625d
cattle-monitoring-system                 alertmanager-rancher-monitoring-alertmanager-tls-assets             Opaque                                0      625d
cattle-monitoring-system                 chart-values-rancher-monitoring                                     Opaque                                1      40m
cattle-monitoring-system                 prometheus-rancher-monitoring-prometheus                            Opaque                                1      2y174d
cattle-monitoring-system                 prometheus-rancher-monitoring-prometheus-tls-assets                 Opaque                                1      2y174d
cattle-monitoring-system                 prometheus-rancher-monitoring-prometheus-web-config                 Opaque                                1      2y47d
cattle-monitoring-system                 rancher-monitoring-admission                                        Opaque                                3      2y174d
cattle-monitoring-system                 rancher-monitoring-alertmanager-token-vrs9w                         <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token>   3      625d
cattle-monitoring-system                 rancher-monitoring-crd-manager-token-4rdf5                          <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token>   3      2y174d
cattle-monitoring-system                 rancher-monitoring-grafana                                          Opaque                                3      2y174d
cattle-monitoring-system                 rancher-monitoring-grafana-token-dlmrr                              <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token>   3      2y174d
cattle-monitoring-system                 rancher-monitoring-kube-state-metrics-token-79t5l                   <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token>   3      2y174d
cattle-monitoring-system                 rancher-monitoring-operator-token-8kb2d                             <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token>   3      2y174d
cattle-monitoring-system                 rancher-monitoring-patch-sa-token-x7869                             <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token>   3      2y174d
cattle-monitoring-system                 rancher-monitoring-prometheus-adapter-token-58qgz                   <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token>   3      2y174d
cattle-monitoring-system                 rancher-monitoring-prometheus-node-exporter-token-fxn7g             <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token>   3      2y174d
cattle-monitoring-system                 rancher-monitoring-prometheus-token-sxxjl                           <http://kubernetes.io/service-account-token|kubernetes.io/service-account-token>   3      2y174d
cattle-monitoring-system                 sh.helm.release.v1.rancher-monitoring-crd.v1188                     <http://helm.sh/release.v1|helm.sh/release.v1>                    1      8d
cattle-monitoring-system                 sh.helm.release.v1.rancher-monitoring-crd.v1189                     <http://helm.sh/release.v1|helm.sh/release.v1>                    1      8d
cattle-monitoring-system                 sh.helm.release.v1.rancher-monitoring.v2239                         <http://helm.sh/release.v1|helm.sh/release.v1>                    1      309d
cattle-monitoring-system                 sh.helm.release.v1.rancher-monitoring.v2240                         <http://helm.sh/release.v1|helm.sh/release.v1>                    1      308d
cattle-monitoring-system                 sh.helm.release.v1.rancher-monitoring.v2241                         <http://helm.sh/release.v1|helm.sh/release.v1>                    1      308d
cattle-monitoring-system                 sh.helm.release.v1.rancher-monitoring.v2242                         <http://helm.sh/release.v1|helm.sh/release.v1>                    1      308d
cattle-monitoring-system                 sh.helm.release.v1.rancher-monitoring.v2243                         <http://helm.sh/release.v1|helm.sh/release.v1>                    1      308d
cattle-monitoring-system                 sh.helm.release.v1.rancher-monitoring.v2244                         <http://helm.sh/release.v1|helm.sh/release.v1>                    1      308d
cattle-monitoring-system                 sh.helm.release.v1.rancher-monitoring.v2245                         <http://helm.sh/release.v1|helm.sh/release.v1>                    1      308d
cattle-monitoring-system                 sh.helm.release.v1.rancher-monitoring.v2246                         <http://helm.sh/release.v1|helm.sh/release.v1>                    1      307d
cattle-monitoring-system                 sh.helm.release.v1.rancher-monitoring.v2247                         <http://helm.sh/release.v1|helm.sh/release.v1>                    1      307d
cattle-monitoring-system                 sh.helm.release.v1.rancher-monitoring.v2248                         <http://helm.sh/release.v1|helm.sh/release.v1>                    1      307d
a
1. Disable those 2 addons 2. Remove all secrets like `sh.helm.release.v1.rancher-logging.v21**`and 
sh.helm.release.v1.rancher-monitoring.v224**
3. Enable addon again
👀 1
those old 
sh.helm.release-*
are aged ~ 
307d
, old left resources
👍 1
don't remove 
-crd.v*
👍 1
s
Getting closer 🙂 • rancher-monitoring has deployed successfully ✅ • rancher-logging is stuck 
enabling
- last lines of helm-install-rancher-logging-* pod:
Copy code
+ echo 'Installing helm_v3 chart'
+ helm_v3 install --version 103.0.0+up3.17.10 rancher-logging rancher-logging/rancher-logging --values /config/values-01_HelmChart.yaml
Error: INSTALLATION FAILED: Unable to continue with install: EventTailer "harvester-default" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "<http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>": must be set to "Helm"; annotation validation error: missing key "<http://meta.helm.sh/release-name|meta.helm.sh/release-name>": must be set to "rancher-logging"; annotation validation error: missing key "<http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>": must be set to "cattle-logging-system"
a
a last obstacle
kubectl get eventtailer -A
, and manually delete the listed object, then enable 
rancher-logging
again
👀 1
s
• rancher-logging has deployed successfully ✅ You are amazing @ancient-pizza-13099 - thank you so much 🙂
🤝 1
79 Views
Open in Slack
PreviousNext
Powered by