https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# elemental
a
This message was deleted.
b
I just went through this, and I have some notes that might help
We're doing some fancy bonding, but the concepts are the same
c
Same here, but our switches are not ready yet. Dhcp is broken, so no IPs for the bond.
I wanted to debug that net.ifnames=0 that's part of the upstream image, but those reboots are getting in the way
b
UGH yes
same
I have part of that fix in my notes.
c
I've added a post-install hook for that to remove that setting from the bootargs.cfg - the hooks gets executed (verified), but the param is still there
b
I had to set it in a few places
c
I'll take every bit I can get ๐Ÿ˜… Our switches should be ready by tomorrow, I can verify our bond config by then hopefully. And all the other stuff that's currently missing
b
One of the things is that the stages SHOULD get thrown into the seedImage, but don't always, so that's something to check.
c
Copy code
config:
  cloud-config:
    stages:
      after-install-chroot:
        - commands:
            - sed -i "s/ net.ifnames=0//g" /etc/elemental/bootargs.cfg
            - >-
              echo "After install chroot executed" >
              /etc/elemental/after-install-chroot
          name: Enable predictable ifnames
that file gets created, it might be something with my sed
b
Copy code
stages:
  after-install-chroot:
  - name: "Set ifnames"
    commands:
    - sed -i 's/net.ifnames=0/net.ifnames=1/g' /etc/elemental/bootargs.cfg
  after-install:
  - name: "Longhorn Disk Setup"
    commands:
    - curl <http://server/elemental/elementaldisks.sh> | bash
    - curl <http://server/elemental/95_disks.yaml> -o /run/elemental/oem/95_disks.yaml
  - name: "Network Persistence Script"
    commands:
    - grub2-editenv /run/elemental/oem/grubenv set extra_cmdline=net.ifnames=1
    - curl <http://server/elemental/90_network.yaml> -o /run/elemental/oem/90_network.yaml
    - curl <http://server/elemental/bondsetup.sh> -o /run/elemental/oem/bondsetup.sh ; chmod +x /run/elemental/oem/bondsetup.sh
I'm not sure it's enough to just remove the 
ifnames=0
I think I had to specify 
ifnames=1
Also, the 
grub2-editenv /run/elemental/oem/grubenv set extra_cmdline=net.ifnames=1
will override it if/when the base image gets updated.
So I think you want both, but note that the stage and path changes for it to persist into that first boot.
c
https://github.com/rancher/elemental/pull/1581 introduced that setting, so I assumed it would be enough to "undo" that. That after-install hook... isn't that too late? Can you still modify the target system at that point?
b
in 
after-install
the oem is still mounted and writeable under 
/run/elemental/oem
it gets mounted under 
/oem
in the next boot
c
ah, right
b
Not good for the 
etc
stuff, but good for oem.
From our internal docs:
Copy code
The next several files are networkmanager connection files that will bring up the bond on the installation boot. In this example, there's only a few that it might bond together. Extra configs for devices that don't exist will be ignored. 

stages - There's a few portions here that were simplified through the use of scripts. 

    stages.after-install-chroot - this happens immediately after the installer finishes writing the files. The system/shell is still chrooted into the mounted systems and it remains writeable. We're swapping out grub lines here to get the unique names to populate in the next boot. We're also doing it later. Why in two places? Because computers suck and we're hosed if we don't get that parameter set for the subsequent boot. 
    stages.after-install - This happens right after the previous stage, but now we're no longer in chroot. Here there are two sections. One sets up the longhorn disks, and the other sets up the networking.  The partitions are still mounted in RW but the pathing is slightly different. We're using grub2-editenv to inject the net.ifnames=1 into grub via /oem, because if we don't get that kernel line in there, we're hosed on the next boots and the node will never join.  Next we curl the 90_network.yaml and put it in place so the bond will come up. It's very similar to the write_files: section, but is using `files` and the boot stage to put them in place. The idea of doing it this way (instead of using write files again) is to limit the duplication of effort needed to maintain different servers and server types and having to copy and paste this into as few places as possible when changes are needed. Last is a script that is downloaded and setup on the node so if for whatever reason the bond goes away, it can be recreated more quickly. (kinda a break glass in case of emergency).
lol
Copy code
We're also doing it later. Why in two places? Because computers suck and we're hosed if we don't get that parameter set for the subsequent boot.
๐Ÿ˜† 1
c
I'll try that tomorrow. ๐Ÿ˜„
Currently reinstalls take 20 minutes for me, we'll get a webserver for the ipmi image mount in the datacenter as well... that'll be less boring
b
I'm doing PXE and my installs went from 25-35 minutes to about 7.
c
much better, yes.
elemental is pxe-bootable?
b
Totally
Let me know if you want kernel parameters to inject different parameters for the seed image yaml and registration endpoint yaml.
c
I'll think about that. The dhcp servers on these dell os10 switches are pretty limited, can't even provide additional options. Not sure if they support next-server and filename options.
b
I'm not a network guy, but I'd be surprised if they didn't. PXE has been a standard for decades now, but I've also been surprised before. ยฏ\_(ใƒ„)_/ยฏ
c
It's that switch config system that's limiting us... they run debian underneath, but the configs are all generated. I'm glad we can do ip reservations in that dhcp, but even hostnames via dhcp is too much to ask for there. At least these switches support running docker containers, so we might replace the builtin dhcp server with our own - or we might do the bootstrap with temporary IPs, feeding that ipam provider with fitting network configs into the elemental setup... not sure yet.
well, masking that service helped, it silenced the output. Machine still went into reboot after ~3 minutes ๐Ÿ™ˆ
b
ha
I'm guessing you just need to find the other service that's triggering that
c
yup. That's gonna be fun ๐Ÿ˜ฎ
b
should be easier to tail journalctl and spot the one that's doing it.
c
mhm... is the journal from previous boot kept on disk? Thought it's readonly
b
It's not kept, you're right. I might suggest doing another boot and do a screen recording while you've got 
journalctl -f
running during that 3 min mark. Then you can review the logs that way.
c
classic ๐Ÿ˜‚
There's a boot-assessment checker in the elemental image, which sounds like it's responsible. It's made to reboot a previous snapshot if the boot failed - in my case it's rebooting the freshly installed system over and over again. I'll try to mask that one.
Well... I am 99% sure the sed-command to remove the 
net.ifnames=0
is actually enough to get the new nic names back. I might be able to explain why. But first: Any chance you have 2 root-disks in your servers? And specified both as valid install targets?
b
That's happened to us before
The rules match more than 1 disk.
And it picks 1, then you re-install and it picks the other
Then you bash your head against the keyboard trying to figure out why your new changes aren't there.
c
yup. And that's why that sed change didn't work for us as well. It simply loaded the grubenv.cfg from the other disk.
We'll add a blkdiscard execution for both disks into the setup hooks
b
I made a PXE entry that wips all the disks, and I typically run that before running the elemental installer.
c
I caught it redhanded with mounted partitions from both disks ๐Ÿ™ˆ
Just wasn't sure yet if its the installer randomly selecting a disk or if its the uefi randomly ordering the nvmes
But at least our bond configs are now working flawless, using the declarative network config. It's now just the switches no longer handing out dhcp responses during bootstrap while the lacp fallback is active. But vendor claims to have a solution for that...
4 Views
Open in Slack
PreviousNext
Powered by