https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# harvester
a
This message was deleted.
a
could you please share what you've reconfigured?
a
My full 
/oem/90_custom.yaml
(not sure it contains private keys?) or any specific section?
Looks like it's only a passwd in there, removed that 🙂
Copy code
# cat /oem/90_custom.yaml
name: Harvester Configuration
stages:
    initramfs:
        - commands:
            - modprobe kvm
            - modprobe vhost_net
            - rm -f /var/lib/kubelet/cpu_manager_state
            - sed -i 's/^NETCONFIG_DNS_STATIC_SERVERS.*/NETCONFIG_DNS_STATIC_SERVERS="192.168.154.150"/' /etc/sysconfig/network/config
          files:
            - path: /etc/multipath/conf.d/99-longhorn.conf
              permissions: 420
              owner: 0
              group: 0
              content: YmxhY2tsaXN0IHsgCiAgZGV2aWNlIHsgCiAgICB2ZW5kb3IgIklFVCIgCiAgICBwcm9kdWN0ICJWSVJUVUFMLURJU0siCiAgfQp9
              encoding: base64
              ownerstring: ""
            - path: /etc/systemd/system/multipathd.service
              permissions: 420
              owner: 0
              group: 0
              content: W1VuaXRdCkRlc2NyaXB0aW9uPURldmljZS1NYXBwZXIgTXVsdGlwYXRoIERldmljZSBDb250cm9sbGVyCkJlZm9yZT1sdm0yLWFjdGl2YXRpb24tZWFybHkuc2VydmljZQpCZWZvcmU9bG9jYWwtZnMtcHJlLnRhcmdldCBibGstYXZhaWxhYmlsaXR5LnNlcnZpY2Ugc2h1dGRvd24udGFyZ2V0CldhbnRzPXN5c3RlbWQtdWRldmQta2VybmVsLnNvY2tldApBZnRlcj1zeXN0ZW1kLXVkZXZkLWtlcm5lbC5zb2NrZXQKQWZ0ZXI9bXVsdGlwYXRoZC5zb2NrZXQgc3lzdGVtZC1yZW1vdW50LWZzLnNlcnZpY2UKQmVmb3JlPWluaXRyZC1jbGVhbnVwLnNlcnZpY2UKRGVmYXVsdERlcGVuZGVuY2llcz1ubwpDb25mbGljdHM9c2h1dGRvd24udGFyZ2V0CkNvbmZsaWN0cz1pbml0cmQtY2xlYW51cC5zZXJ2aWNlCkNvbmRpdGlvbktlcm5lbENvbW1hbmRMaW5lPSFub21wYXRoCkNvbmRpdGlvblZpcnR1YWxpemF0aW9uPSFjb250YWluZXIKCltTZXJ2aWNlXQpUeXBlPW5vdGlmeQpOb3RpZnlBY2Nlc3M9bWFpbgpFeGVjU3RhcnQ9L3NiaW4vbXVsdGlwYXRoZCAtZCAtcwpFeGVjUmVsb2FkPS9zYmluL211bHRpcGF0aGQgcmVjb25maWd1cmUKVGFza3NNYXg9aW5maW5pdHkKCltJbnN0YWxsXQpXYW50ZWRCeT1zeXNpbml0LnRhcmdldA==
              encoding: base64
              ownerstring: ""
            - path: /etc/rancher/rancherd/config.yaml
              permissions: 384
              owner: 0
              group: 0
              content: |
                server: <https://192.168.154.100:443>
                role: agent
                token: "resize-suggest-random"
                kubernetesVersion: v1.32.3+rke2r1
                rancherVersion: v2.11.0
                rancherInstallerImage: rancher/system-agent-installer-rancher:v2.11.0
                labels:
                 - <http://harvesterhci.io/managed=true|harvesterhci.io/managed=true>
                extraConfig:
                  disable:
                  - rke2-snapshot-controller
                  - rke2-snapshot-controller-crd
                  - rke2-snapshot-validation-webhook
              encoding: ""
              ownerstring: ""
            - path: /etc/rancher/rke2/config.yaml.d/91-harvester-cdi.yaml
              permissions: 384
              owner: 0
              group: 0
              content: |-
                # handle the permission issue of Longhorn for CDI
                "nonroot-devices": true
              encoding: ""
              ownerstring: ""
            - path: /etc/rancher/rke2/config.yaml.d/92-harvester-kube-audit-policy.yaml
              permissions: 384
              owner: 0
              group: 0
              content: |
                apiVersion: <http://audit.k8s.io/v1|audit.k8s.io/v1>
                kind: Policy
                omitStages:
                  - "ResponseStarted"
                  - "ResponseComplete"
                rules:
                  # Any include/exclude rules are added here

                  # A catch-all rule to log all other (create/delete/patch) requests at the Metadata level
                  - level: Metadata
                    verbs: ["create", "delete", "patch"]
                    omitStages:
                      - "ResponseStarted"
                      - "ResponseComplete"
              encoding: ""
              ownerstring: ""
            - path: /etc/rancher/rke2/config.yaml.d/90-harvester-agent.yaml
              permissions: 384
              owner: 0
              group: 0
              content: |-
                kubelet-arg:
                - "max-pods=200"
              encoding: ""
              ownerstring: ""
            - path: /etc/rancher/rke2/config.yaml.d/99-z00-harvester-reserved-resources.yaml
              permissions: 384
              owner: 0
              group: 0
              content: |-
                kubelet-arg+:
                - "system-reserved=cpu=400m"
                - "kube-reserved=cpu=600m"
              encoding: ""
              ownerstring: ""
            - path: /etc/wicked/scripts/setup_bond.sh
              permissions: 493
              owner: 0
              group: 0
              content: |
                #!/bin/sh

                ACTION=$1
                INTERFACE=$2

                case $ACTION in
                        post-up)
                                # inherit MAC address
                                ip link set dev mgmt-br address $(ip -json link show dev $INTERFACE | jq -j '.[0]["address"]')

                                # PVID=1 by default
                                ;;

                esac
              encoding: ""
              ownerstring: ""
            - path: /etc/sysconfig/network/ifcfg-mgmt-bo
              permissions: 384
              owner: 0
              group: 0
              content: |+
                STARTMODE='onboot'
                BONDING_MASTER='yes'
                BOOTPROTO='none'
                POST_UP_SCRIPT="wicked:setup_bond.sh"


                BONDING_SLAVE_0='enp2s0f0'
                BONDING_SLAVE_1='enp2s0f1'

                BONDING_MODULE_OPTS='miimon=100 mode=active-backup '

                DHCLIENT_SET_DEFAULT_ROUTE='no'

              encoding: ""
              ownerstring: ""
            - path: /etc/sysconfig/network/ifcfg-enp2s0f0
              permissions: 384
              owner: 0
              group: 0
              content: |
                STARTMODE='hotplug'
                BOOTPROTO='none'
              encoding: ""
              ownerstring: ""
            - path: /etc/sysconfig/network/ifcfg-enp2s0f1
              permissions: 384
              owner: 0
              group: 0
              content: |
                STARTMODE='hotplug'
                BOOTPROTO='none'
              encoding: ""
              ownerstring: ""
            - path: /etc/wicked/scripts/setup_bridge.sh
              permissions: 493
              owner: 0
              group: 0
              content: |
                #!/bin/sh

                ACTION=$1
                INTERFACE=$2

                case $ACTION in
                        pre-up)
                                # enable vlan-aware
                                ip link set $INTERFACE type bridge vlan_filtering 1
                                ;;

                        post-up)
                                # PVID=1 by default
                                ;;
                esac
              encoding: ""
              ownerstring: ""
            - path: /etc/sysconfig/network/ifcfg-mgmt-br
              permissions: 384
              owner: 0
              group: 0
              content: |+
                STARTMODE='onboot'
                BOOTPROTO='static'
                BRIDGE='yes'
                BRIDGE_STP='off'
                BRIDGE_FORWARDDELAY='0'
                BRIDGE_PORTS='mgmt-bo'
                PRE_UP_SCRIPT="wicked:setup_bridge.sh"
                POST_UP_SCRIPT="wicked:setup_bridge.sh"

                IPADDR=192.168.154.101
                NETMASK=255.255.255.0


                DHCLIENT_SET_DEFAULT_ROUTE='yes'


              encoding: ""
              ownerstring: ""
            - path: /etc/sysconfig/network/ifroute-mgmt-br
              permissions: 384
              owner: 0
              group: 0
              content: |
                default 192.168.154.1 - mgmt-br
              encoding: ""
              ownerstring: ""
          directories:
            - path: /etc/multipath/conf.d
              permissions: 420
              owner: 0
              group: 0
            - path: /etc/rancher/rke2/config.yaml.d
              permissions: 384
              owner: 0
              group: 0
            - path: /etc/wicked/scripts
              permissions: 420
              owner: 0
              group: 0
            - path: /etc/wicked/scripts
              permissions: 420
              owner: 0
              group: 0
          hostname: lab-hv-01
          users:
            rancher:
                passwd: <removed>
          systemctl:
            enable:
                - systemd-timesyncd
                - systemd-time-wait-sync
          timesyncd:
            NTP: <http://0.suse.pool.ntp.org|0.suse.pool.ntp.org>
    network:
        - authorized_keys:
            rancher: []
    rootfs:
        - if: '[ ! -f "/run/cos/recovery_mode" ]'
          name: Rootfs layout overwrite
          environment:
            OVERLAY: tmpfs:25%
            PERSISTENT_STATE_BIND: "true"
            PERSISTENT_STATE_PATHS: /etc/systemd /etc/rancher /etc/ssh /etc/iscsi /etc/nvme /etc/cni /etc/pki/trust/anchors /home /opt /root /usr/libexec /var/log /var/lib/rancher /var/lib/kubelet /var/lib/wicked /var/lib/cni /var/lib/third-party /var/crash /var/lib/longhorn
            RW_PATHS: /var /etc /srv /boot /lib/firmware
            VOLUMES: LABEL=COS_OEM:/oem LABEL=COS_PERSISTENT:/usr/local LABEL=HARV_LH_DEFAULT:/var/lib/harvester/defaultdisk
          environment_file: /run/cos/cos-layout.env
And
Copy code
# cat /oem/harvester.config
schemeversion: 1
serverurl: <https://192.168.154.100:443>
token: resize-suggest-random
os:
    afterinstallchrootcommands: []
    sshauthorizedkeys: []
    writefiles: []
    hostname: lab-hv-01
    modules:
        - kvm
        - vhost_net
    sysctls: {}
    ntpservers:
        - <http://0.suse.pool.ntp.org|0.suse.pool.ntp.org>
    dnsnameservers:
        - 192.168.154.150
    wifi: []
    password: <removed>
    environment: {}
    labels: {}
    sshd:
        sftp: false
    persistentstatepaths: []
    externalstorage:
        enabled: false
        multipathconfig: []
    additionalkernelarguments: ""
install:
    automatic: false
    skipchecks: true
    mode: join
    managementinterface:
        interfaces:
            - name: enp2s0f0
              hwaddr: 0c:c4:7a:bd:15:58
            - name: enp2s0f1
              hwaddr: 0c:c4:7a:bd:15:59
        method: static
        ip: 192.168.154.101
        subnet_mask: 255.255.255.0
        gateway: 192.168.154.1
        defaultroute: true
        bond_options:
            miimon: "100"
            mode: active-backup
        mtu: 0
        vlanid: 0
    vip: ""
    viphwaddr: ""
    vipmode: ""
    clusterdns: ""
    clusterpodcidr: ""
    clusterservicecidr: ""
    forceefi: false
    device: /dev/nvme1n1
    configurl: ""
    silent: false
    isourl: ""
    poweroff: false
    noformat: false
    debug: false
    tty: tty1
    forcegpt: true
    role: default
    withnetimages: false
    wipealldisks: false
    wipediskslist: []
    forcembr: false
    datadisk: /dev/nvme0n1
    webhooks: []
    addons: {}
    harvester:
        storageclass:
            replicacount: 0
        longhorn:
            defaultsettings:
                guaranteedenginemanagercpu: null
                guaranteedreplicamanagercpu: null
                guaranteedinstancemanagercpu: null
        enablegocoverdir: false
    rawdiskimagepath: ""
    persistentpartitionsize: ""
runtimeversion: v1.32.3+rke2r1
rancherversion: v2.11.0
harvesterchartversion: 1.5.0
monitoringchartversion: 105.1.2+up61.3.2
systemsettings:
    ntp-servers: '{"ntpServers":["<http://0.suse.pool.ntp.org|0.suse.pool.ntp.org>"]}'
loggingchartversion: 105.2.0+up4.10.0
a
regarding changing VIP in day2, I found this comment https://github.com/harvester/harvester/issues/4584#issuecomment-2046463200
btw, do u have a before & after for both 
/oem/90_custom.yaml
& 
/oem/harvester.config
?
Otherwise I can't tell what you have changed in the yaml
c.c. @red-king-19196
a
I already updated the 
/oem/harvester.config
the way I think it should be. So that's why you probably don't see any clear changes.
I went ahead and started the upgrade; it's at the node reboot stage and the first node seems to have survived the reboot; so maybe I was being too over-cautious... 🙂
6 Views
Open in Slack
PreviousNext
Powered by