Forums

Resolved
0 votes
We have ClearOS 7 Community systems configured to boot with UEFI that are failing to boot after the latest GRUB update.

Error:
System BootOrder not found. Initializing defaults.
Creating boot entry "Boot0002" with label "ClearOS" for file "\EFI\clearos\shimx64.efi"
Failed to open \EFI\clearos\grubx64.efi -- Not Found
Failed to load image \EFI\clearos\grubx64.efi: Not Found
start_image() returned Not Found
StartImage failed: Not Found

I've determined that one of the latest GRUB updates is moving grubx64.efi from /boot/efi/EFI/clearos/ to /boot/efi/EFI/centos/ Of course the boot configuration is looking for the grubx64.efi to be in the clearos directory, not the (new with the update) centos one. Therefore boot fails.

I don't know precisely which of the following packages broke it, but the problem appeared after version 2.02-0.87.v7.6 of all the following were installed during a regular yum run:
grub2
grub2-common
grub2-efi-x64
grub2-pc
grub2-pc-modules
grub2-tools
grub2-tools-extra
grub2-tools-minimal
Wednesday, June 30 2021, 09:14 PM
Share this post:

Accepted Answer

Friday, July 02 2021, 07:26 AM - #Permalink
Resolved
1 votes
Nick Howitt wrote:

Are you booting off a rescue/installation disk?


The threading on this forum is truly, um...

Anyway...

Being a bit of a Linux clutz, I missed step 2a on my first attempt!

1. Boot to Recovery environment (installation CD/Flashdrive, under the Troubleshooting menu)
2. proceed with the mount sysimage option

2a ENTER
3. chroot /mnt/sysimage
4. cp /boot/efi/EFI/centos/grubx64.efi /boot/efi/EFI/clearos/grubx64.efi
5. grub2-mkconfig -o /boot/efi/EFI/clearos/grub.cfg
6. exit
7. exit (again) -- system will reboot; remove recovery drive.

Cheers
The reply is currently minimized Show
Responses (66)
  • Accepted Answer

    Wednesday, June 30 2021, 09:24 PM - #Permalink
    Resolved
    1 votes
    If your system has installed this broken version, and you haven't rebooted, you can simply copy the file and you'll be fine to reboot:

    cp /boot/efi/EFI/centos/grubx64.efi /boot/efi/EFI/clearos/grubx64.efi


    However, if you've already rebooted and your system is in an error state, you may be be assisted with the following:

    DISCLAIMER: Use at your own risk: this is a reconstruction of some steps from testing in our lab, posting in case it helps someone. YMMV.

    1. Boot to Recovery environment (installation CD/Flashdrive, under the Troubleshooting menu)
    2. proceed with the mount sysimage option
    3. chroot /mnt/sysimage
    4. cp /boot/efi/EFI/centos/grubx64.efi /boot/efi/EFI/clearos/grubx64.efi
    5. grub2-mkconfig -o /boot/efi/EFI/clearos/grub.cfg
    6. exit
    7. exit (again) -- system will reboot; remove recovery drive.
    Like
    1
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, June 30 2021, 09:37 PM - #Permalink
    Resolved
    0 votes
    Thanks for the report and workaround. This was always my greatest fear and I know it happened a couple of years ago when it was last updated, but then it was patched. I've run it on a number of EFI systems,updating to it and rebooting, so I wonder why yours triggered it. All grub packages come from a single source so they are one big update.

    The other thing I don't understand is why your grub2-mkconfig command works. I was not able to get it to work with the old or new version as it always just put its output to screen so I had to redirect the output to /boot/efi/EFI/clearos/grubx64.efi
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, June 30 2021, 09:47 PM - #Permalink
    Resolved
    0 votes
    If anyone else gets this issue, please post a "me too" message to the thread. Looking at my primary UEFI system I don't see a problem, so I am not sure how to identify it.
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, June 30 2021, 11:39 PM - #Permalink
    Resolved
    0 votes
    Hmm interesting. Well perhaps it's not as a widespread problem as I thought. When I was testing, I was doing a fresh install of ClearOS 7 (might have been a ISO of 7.7; not sure -- I'm not at the office right now), and then running yum update. We were encountering the problem going from version 2.02.2-0.76.v7 to 2.02-0.87.v7.6, FWIW.
    The reply is currently minimized Show
  • Accepted Answer

    Thursday, July 01 2021, 04:15 AM - #Permalink
    Resolved
    0 votes
    Nick Howitt wrote:

    If anyone else gets this issue, please post a "me too" message to the thread. Looking at my primary UEFI system I don't see a problem, so I am not sure how to identify it.


    Me too. It's a 2011 vintage I3 HP desktop. Edit: Happy to advise that Marvin's method got me up and running once more. ? Kudos.
    The reply is currently minimized Show
  • Accepted Answer

    Thursday, July 01 2021, 01:22 PM - #Permalink
    Resolved
    0 votes
    I can see the problem. Mock is building differently from Koji and I can't get Koji to give the same results as mock. I'm just trying another build.
    The reply is currently minimized Show
  • Accepted Answer

    Thursday, July 01 2021, 02:33 PM - #Permalink
    Resolved
    0 votes
    ...which still didn't fix it. The try after that has now built correctly in GitLab so it is looking promising. Koji is always slower, but I hope to be testing within about 30min.
    The reply is currently minimized Show
  • Accepted Answer

    Thursday, July 01 2021, 03:17 PM - #Permalink
    Resolved
    0 votes
    OK, there is now a fix available for testing:
    rm -rf /var/cache/yum/
    yum update grub2* --disablerepo=* --enablerepo=clearos-updates-testing
    (you don't need the --disablerepo switch, it just makes it a bit faster).
    You are looking for a version number 2.02.2-0.87.v7.6.0.3 for all packages.

    So far I have tested it on 3 non-EFI machines and it now creates the /boot/efi/EFI/clearos folder instead of /boot/efi/EFI/centos. I've rebooted all three. I've also installed it on my one UEFI machine, but I can't reboot it until Sunday morning. My other UEFI is a Proxmox server and it needs me to swap the disk in it to do a clean install of ClearOS then a grub2 update. I did it before for testing but the server is not in a good location and realistically it is not going to happen today.

    I also have a UEFI machine in the US which I've updated and rebooted and it has come up correctly so it is looking good. :D

    If anyone with a UEFI machine can also confirm updating and rebooting is OK, I'll release the package immediately.
    The reply is currently minimized Show
  • Accepted Answer

    Thursday, July 01 2021, 04:12 PM - #Permalink
    Resolved
    0 votes
    Excellent! Great work on this, Nick!

    I can confirm your proposed fix does indeed work as advertised.

    Steps I took:
    1. Fresh installation of ClearOS 7.7 Community on the same hardware I was testing with yesterday.
    2. Brought all packages up to date and rebooted
    3. Installed grub update from testing repo per Nick's post
    4. Verified that a /boot/efi/EFI/centos directory did Not appear after the grub update
    5. Rebooted successfully.

    All is well. Many thanks!
    The reply is currently minimized Show
  • Accepted Answer

    Thursday, July 01 2021, 04:20 PM - #Permalink
    Resolved
    0 votes
    Thanks for the post back. I'll release it now.

    It was a horrible little one. I could see Dave had made a conditional change to a parameter > 2 years ago and it was not triggering. I made it absolute and tidied it up, moving it to near the top of the spec file with the rest of the parameters, not realising that there was a call to another program a bit later on in the spec file which overrode the setting. Moving it below the call to the other program fixed it.
    The reply is currently minimized Show
  • Accepted Answer

    Flash
    Flash
    Offline
    Friday, July 02 2021, 12:35 AM - #Permalink
    Resolved
    0 votes
    It happened to me too, but Marvins workaround doesn't work for me... :(
    After step 3, I have nothing in /boot directory... so I can't copy the grubx64.efi

    I have selected Troubleshooting
    Then Rescue a ClearOS system
    Then it asks me, to choose: 1 for Continue, 2 for Read-only mount, 3 for Skip to shell and 4 to Quit
    I tried 1 and 2, but /boot is empty... When I try 3, then it says: chroot: failed to run command "/bin/bash": No such file or directory

    Am I doing something wrong?
    What can I do? Must I reinstall everything??
    The reply is currently minimized Show
  • Accepted Answer

    Friday, July 02 2021, 07:08 AM - #Permalink
    Resolved
    0 votes
    Are you booting off a rescue/installation disk?
    The reply is currently minimized Show
  • Accepted Answer

    Flash
    Flash
    Offline
    Friday, July 02 2021, 07:22 AM - #Permalink
    Resolved
    0 votes
    I was booting from installation USB key.
    The reply is currently minimized Show
  • Accepted Answer

    Friday, July 02 2021, 07:48 AM - #Permalink
    Resolved
    0 votes
    I agree with Dave Pitman. If you skip the Enter (like I did as well) you end up somewhere on you installation USB (/boot had a vmlinuz file and one other bit no folders) rather than on the ClearOS Disk. I'll mark Dave's post as accepted but kudos to Marvin as well for finding the issue and posting a solution so quickly.

    There is a Rescue Mode HowTo but it looks like it needs to be updated with newer images and a small update to the instructions.
    The reply is currently minimized Show
  • Accepted Answer

    Flash
    Flash
    Offline
    Friday, July 02 2021, 08:01 AM - #Permalink
    Resolved
    0 votes
    Hmm, where should I hit ENTER? When I select Rescue a ClearOS system?
    I hit ENTER like 20 times, and nothing happens, it still gives me the same 4 options...
    Should I select 1, 2 or 3?
    The reply is currently minimized Show
  • Accepted Answer

    Friday, July 02 2021, 08:19 AM - #Permalink
    Resolved
    0 votes
    Flash wrote:

    Hmm, where should I hit ENTER? When I select Rescue a ClearOS system?
    I hit ENTER like 20 times, and nothing happens, it still gives me the same 4 options...
    Should I select 1, 2 or 3?


    It's the default choice, 1.

    Follow the amended instructions. Once you press enter, you get a different prompt and that's where the Marvin magic comes in.

    Cheers
    The reply is currently minimized Show
  • Accepted Answer

    Flash
    Flash
    Offline
    Friday, July 02 2021, 08:33 AM - #Permalink
    Resolved
    0 votes
    After I select option 1, all I get is this:
    ==================================================================================================
    ==================================================================================================
    ==================================================================================================
    ==================================================================================================
    And this goes on and on and on...
    If I hit Alt+F2 then I get:
    ClearOS 7 (Final)
    Kernel 3.10.0-1160.25.1.el7.x86_64 on an x86_64

    [anaconda root@localhost /]#

    Is it here where I should hit the ENTER? Because it doesn't do anything...

    If I hit Alt+F1 again, there are still just ====================================================================

    And still no /boot/efi/EFI/centos or clearsos... nothing... after I hit chroot /mnt/sysimage
    The reply is currently minimized Show
  • Accepted Answer

    Friday, July 02 2021, 08:57 AM - #Permalink
    Resolved
    0 votes
    Flash wrote:

    After I select option 1, all I get is this:
    ==================================================================================================
    ==================================================================================================
    ==================================================================================================
    ==================================================================================================
    And this goes on and on and on...
    If I hit Alt+F2 then I get:
    ClearOS 7 (Final)
    Kernel 3.10.0-1160.25.1.el7.x86_64 on an x86_64

    [anaconda root@localhost /]#

    Is it here where I should hit the ENTER? Because it doesn't do anything...

    If I hit Alt+F1 again, there are still just ====================================================================

    And still no /boot/efi/EFI/centos or clearsos... nothing... after I hit chroot /mnt/sysimage


    Sorry, beyond my skillset. Nick?
    The reply is currently minimized Show
  • Accepted Answer

    Friday, July 02 2021, 09:10 AM - #Permalink
    Resolved
    0 votes
    I've updated the howto at https://documentation.clearos.com/content:en_us:kb_rescue_mode with all the screenshots updated. Try that.
    The reply is currently minimized Show
  • Accepted Answer

    Flash
    Flash
    Offline
    Friday, July 02 2021, 11:27 AM - #Permalink
    Resolved
    0 votes
    Thank you for the updated how to.
    But after I hit 1 for Continue, at the picture in the how to there are only 2 rows of ==========================
    I get gazillion rows of ======================== and it doesn't end...
    I let it for 15 minutes and still only this ==========================
    It doesn't get past this...
    I don't see: Rescue mount
    Your system has been mounted under /mnt/sysimage.

    What's wrong?
    The reply is currently minimized Show
  • Accepted Answer

    Flash
    Flash
    Offline
    Friday, July 02 2021, 11:35 AM - #Permalink
    Resolved
    0 votes
    This is what I get, when I hit 1 for Continue, and the ventilator for the CPU goes almost full speed...

    Edit: Is this maybe I have a PCI-e NVMe disk where the ClearOS is installed? And it doesn't find it?
    Attachments:
    The reply is currently minimized Show
  • Accepted Answer

    Friday, July 02 2021, 12:17 PM - #Permalink
    Resolved
    0 votes
    That does not look good and I don't know what to recommend.

    If you do need to reinstall, if you have anything you need on the disk, use another distro's live cd to boot up, mount your ClearOS disk and pick off the files you want from ClearSOS to another disk before you reinstall. You may want a configuration backup from /var/clearos/configuration_backup. Flexshares are under /var/flexshare/shares. Is there anything else critical you have there which you want to recover?

    If you hold on for a few hours, Marvin may come back online with some ideas.
    The reply is currently minimized Show
  • Accepted Answer

    Friday, July 02 2021, 12:59 PM - #Permalink
    Resolved
    0 votes
    First, thanks Dave for updating the recovery instructions with the extra details. When I posted them, I was in a hurry to leave for an appointment, but wanted to get them out there quick. Glad it helped! I don't mind who gets the credit, Nick. :)

    Flash, as to your issue, I'd agree with your speculation that it's not seeing your NVMe drive, so I'm assuming it's just retrying infinitely or something. The reason you're not able to see any EFI directories is that it's never getting to the point of mounting your installation disk: any shell you get at this point is just in the live recovery environment you're booted to.

    Suggest that rather than having it "auto locate" your installation with option 1, you instead select option 3 for a shell in the recovery environment (again, this is Not a shell on your installation, just the live recovery OS you're running). Then run "lsblk" (without the quotes) to get a list of the storage devices the live environment can see.

    If lsblk lists out your NVMe drive, great! You should be able to proceed to mount it manually and then proceed with chroot (using whatever directory you mount it as -- won't be sysimage if you use another name) and continuing with the recovery steps.

    If lsblk does Not show your NVMe drive, you'll need to specify a driver file for it at the time of booting the recovery environment. See step 3 under Procedure 32.1 on this page in the RHEL Documentation for more information on this.

    A side note about mounting the drive to recover files in the event boot recovery fails: ClearOS by default uses LVM for the main partition, which can complicate things a bit as the Ubuntu live CD, for example, doesn't include LVM support by default (IIRC). It's not an intractable situation, however, one just needs to go to a bit of extra work to add that package to the live CD image. Here's a blog post about adding LVM support to a Debian live CD that may be helpful if you decide to go this route.

    I hope you're able to get the issue resolved!
    The reply is currently minimized Show
  • Accepted Answer

    Flash
    Flash
    Offline
    Friday, July 02 2021, 01:35 PM - #Permalink
    Resolved
    0 votes
    Thank you Marvin for the explanation, but I need a little more help.

    The command "lsblk" lists my NVMe drive.
    Which one should I mount? nvme0n1p1, nvme0n1p2, nvme0n1p3, nvme0n1p4 or nvme0n1p5?
    p1 and p2 are boot partitions and p3 is / partition.

    Is this the correct way: mount -t xfs /dev/nvme0n1p3 /mnt/sysimage ? Or must I mount p1 or p2?
    Because if I do that, I still don't have nothing in /boot directory.
    The reply is currently minimized Show
  • Accepted Answer

    Friday, July 02 2021, 01:42 PM - #Permalink
    Resolved
    0 votes
    It is probably p1 or p2. If you did a GPT installation it will be p2. Have a look and if nothing is there, come out of the drive then unmount it e.g:
    cd /mnt
    umount /mnt/sysimage
    Then you can try the next one.
    The reply is currently minimized Show
  • Accepted Answer

    Friday, July 02 2021, 01:56 PM - #Permalink
    Resolved
    0 votes
    Agree with Nick's comment on the partition.

    One redundant note just to be sure we're on the same page: after you successfully mount a partition in your recovery environment, you'll need to chroot /mnt/sysimage/ before you can go looking for boot/.
    The reply is currently minimized Show
  • Accepted Answer

    Flash
    Flash
    Offline
    Friday, July 02 2021, 02:02 PM - #Permalink
    Resolved
    0 votes
    If I mount p1 it says: wrong fs type, bad option, bad superblock on /dev/nvme0n1p1, missing codepage or helper program, or other error

    If I mount p2, then it mounts. But then, if I do chroot /mnt/sysimage it says: failed to run command "/bin/bash": No such file or directory

    If I go to /mnt/sysimage then there are files in there...
    The reply is currently minimized Show
  • Accepted Answer

    Friday, July 02 2021, 02:10 PM - #Permalink
    Resolved
    0 votes
    Marvin Martin wrote:
    One redundant note just to be sure we're on the same page: after you successfully mount a partition in your recovery environment, you'll need to chroot /mnt/sysimage/ before you can go looking for boot/.
    Out of interest, won't /boot show as /mnt/sysimage/boot without chrooting? Agree that you'll have to chroot before recovering.
    The reply is currently minimized Show
  • Accepted Answer

    Flash
    Flash
    Offline
    Friday, July 02 2021, 02:14 PM - #Permalink
    Resolved
    0 votes
    Should I try to chroot /boot ?
    The reply is currently minimized Show
  • Accepted Answer

    Friday, July 02 2021, 02:15 PM - #Permalink
    Resolved
    0 votes
    Flash wrote:

    If I mount p1 it says: wrong fs type, bad option, bad superblock on /dev/nvme0n1p1, missing codepage or helper program, or other error

    If I mount p2, then it mounts. But then, if I do chroot /mnt/sysimage it says: failed to run command "/bin/bash": No such file or directory

    If I go to /mnt/sysimage then there are files in there...
    p1 then is probably the msdos partition on a GPT labelled disk. Then p2 will be /boot.

    If it does not run bash, does it stay in sh? It should not matter as most commands are similar but I am worried.....

    ... because I wonder if you need to mount both; mount / first into /mnt/sysimage then perhaps chroot to it. Then mount p2 into /boot. Marvin may need to comment here.
    The reply is currently minimized Show
  • Accepted Answer

    Flash
    Flash
    Offline
    Friday, July 02 2021, 02:20 PM - #Permalink
    Resolved
    0 votes
    When I installed ClearOS it said that I need a /boot partition and a /efiboot partition or something like that. So one of them is /boot and one is /efiboot
    I forgot which is which :S
    Yes it stays in sh.
    The reply is currently minimized Show
  • Accepted Answer

    Friday, July 02 2021, 02:32 PM - #Permalink
    Resolved
    0 votes
    Out of interest, won't /boot show as /mnt/sysimage/boot without chrooting? Agree that you'll have to chroot before recovering.

    Yes, you're correct. I thought about mentioning this, but since I'm thinking you'd need to have changed root anyway to reconfigure grub, I had left it out to keep the discussion simple.

    Marvin may need to comment here.

    Hmm. I'll try to help, but must say I'm not super-experienced in "non-standard" boot recovery situations!

    Flash, when you're at the shell in your recovery environment (not chroot'd anywhere), run "lsblk -f" -- this should show the filesystem type for each partition, which will be helpful to specify the correct filesystem type when you mount a partition.

    As far as referencing shell utilities not in a given partition when using chroot, this IBM article may be helpful. That said, Nick's comment about changing directory vs chroot is a good one to bring up here: if you just want to "see what's there", while at your recovery environment shell, you can simply "cd /mnt/sysimage/" and that should get you to the root of the mounted partition.

    If this doesn't get you out of the bind, could you post the output of "ls -lh" of the root of the partitions after you mount them? (you'll either need to chroot or use the cd /mnt/sysimage/ method mentioned above).
    The reply is currently minimized Show
  • Accepted Answer

    Flash
    Flash
    Offline
    Friday, July 02 2021, 02:51 PM - #Permalink
    Resolved
    0 votes
    OK. I have now successfully mounted the p1 partition. It is vfat partition... don't now why.

    I have now a /mnt/sysimage/EFI directory. In there there are BOOT centos and clearos.
    Is this OK? Should I just copy grubx64.efi from centos to clearos?

    If I write chroot /mnt/sysimage I still get the same answer: failed to run command "/bin/bash": No such file or directory
    The reply is currently minimized Show
  • Accepted Answer

    Friday, July 02 2021, 03:08 PM - #Permalink
    Resolved
    0 votes
    While you were posting I was writing this:

    https://pario.no/2015/11/02/how-to-mount-lvm-partitions-from-rescue-mode/ and https://documentation.online.net/en/dedicated-server/rescue/mount-lvm-partition show how to mount an lvm main partition like / could be.

    On my system:
    [root@microserver ~]#  lvm vgscan -v
    Reading volume groups from cache.
    Found volume group "main" using metadata type lvm2

    [root@microserver ~]# lvm lvs
    LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
    root main -wi-ao---- 924.84g
    swap main -wi-ao---- 2.00g
    (it didn't like the --all switch)

    My /etc/fstab has:
    /dev/mapper/main-root   /                       ext4    defaults        1 1
    So, presumably:
    mount /dev/mapper/main-root   /mnt/sysimage
    will work but it is lightly inconsistent with the two examples I linked to which indicate:
    mount /dev/main/root   /mnt/sysimage
    should work, but, for me the two symlink together. Note there is no need to specify the filesystem type (xs, ext4 etc as it should be auto-detected)

    If that works, you can chroot then issue the command "mount-a" which shuold mount all the other partitions.
    The reply is currently minimized Show
  • Accepted Answer

    Flash
    Flash
    Offline
    Friday, July 02 2021, 03:37 PM - #Permalink
    Resolved
    0 votes
    I don't have /dev/main/root I'm lost... Who released this faulty GRUB for update???

    So can I just copy the grubx64.efi or not?

    The chroot command isn't working.
    The reply is currently minimized Show
  • Accepted Answer

    Friday, July 02 2021, 04:01 PM - #Permalink
    Resolved
    0 votes
    So can I just copy the grubx64.efi or not?

    Yes, IMO you can go for it.


    I take Nick's last post to be all about how to mount an LVM partition along with the others so you could have all of them mounted in your recovery environment. I'm not sure that this is needed, but perhaps Nick has a particular reason for recommending this. Maybe for the grub reconfiguration?
    The reply is currently minimized Show
  • Accepted Answer

    Flash
    Flash
    Offline
    Friday, July 02 2021, 04:10 PM - #Permalink
    Resolved
    0 votes
    It didn't work... :(
    This is what I get now at boot.
    Attachments:
    The reply is currently minimized Show
  • Accepted Answer

    Friday, July 02 2021, 04:24 PM - #Permalink
    Resolved
    0 votes
    Flash,
    I think that's some progress, though the "no such device" is a bit worrisome. Basically now the EFI system is starting up, but it's saying, hey, things aren't where I was expecting them. At this point, if we can trigger the grub2-mkconfig command on the right partition as per the recovery instructions, we might be getting places.

    To do this:
    Boot back into your recovery environment again as before, using the option 3 for the shell.
    Mount the main system partition.
    And chroot to that. (If this doesn't work, post the output of "lsblk -f" and "pwd")
    Pick up with the recovery steps at #5 and tell us how it goes. If you get errors, please post the output.

    EDIT: updated instructions. If I'm thinking correctly, you will need to be in the main system partition (not the boot one that I think you were in earlier).
    The reply is currently minimized Show
  • Accepted Answer

    Friday, July 02 2021, 04:31 PM - #Permalink
    Resolved
    0 votes
    Just a note that I made some fairly extensive edits to my previous post.
    The reply is currently minimized Show
  • Accepted Answer

    Friday, July 02 2021, 04:46 PM - #Permalink
    Resolved
    0 votes
    That is why I posted my thing about the LVM, because the command grub2-mkconfig and grub defaults are in / somewhere, so I suspect you need the whole filesystem mounted.
    The reply is currently minimized Show
Your Reply