Forums

Resolved
0 votes
The 3.10.0-1127.10.1.el7 kernel rolling out today is causing kernel panics on HPE ML10 Gen9 units that we have in the field:

Kernel panic; unable to mount root fs at block 0


Reverting to 3.10.0-1062.18.1.el7 enables the system to run fine again.

What additional details are needed?
Tuesday, June 16 2020, 07:08 PM
Share this post:
Responses (9)
  • Accepted Answer

    Tuesday, June 30 2020, 01:34 PM - #Permalink
    Resolved
    0 votes
    Brilliant. Thanks for letting me know. I had not found any noise on the internet about the update either so I'll push it today.
    The reply is currently minimized Show
  • Accepted Answer

    Tuesday, June 30 2020, 01:26 PM - #Permalink
    Resolved
    0 votes
    Ok, I installed microcode_ctl 2:2.1-61.10.el7_8 on several systems this morning that had been affected earlier, and so far, so good--no issues yet.

    The systems were running one of the following kernels at the time of microcode update install:
    3.10.0-1062.9.1.el7.x86_64
    3.10.0-1062.18.1.el7.x86_64

    I didn't think to wait to install the microcode update on a system running 3.10.0-1127.10.1.el7.x86_64, however upgrading to that kernel after a successful microcode update worked fine. So I think all is well.
    The reply is currently minimized Show
  • Accepted Answer

    Monday, June 29 2020, 09:33 PM - #Permalink
    Resolved
    0 votes
    Hi Marvin, it would be great if you could give feedback tomorrow. I'll do a search of other forums tomorrow before making a decision. I may also hold back waiting for your report depending on if I find anything on the internet about the upgrade.
    The reply is currently minimized Show
  • Accepted Answer

    Monday, June 29 2020, 09:13 PM - #Permalink
    Resolved
    0 votes
    Thanks for the update Nick! FWIW, I should have a chance to test the updated microcode package on an HPE unit tomorrow (Tuesday), if you were wanting to wait on me. If you'd rather deploy it, that's OK; I'll try to let you know how the updated microcode package install goes either way.
    The reply is currently minimized Show
  • Accepted Answer

    Saturday, June 27 2020, 08:10 AM - #Permalink
    Resolved
    0 votes
    I know I am double-posting, but there is an update to microcode_ctl available in the centos-updates-unverified repo which is designed to replace the faulty one released with ClearOS 7.8 (although it is not specifically a 7.8 package). If you would like to test it you can do a:
    yum update microcode_ctl --enablerepo=centos-updates-unverified

    I am planning to release it on Tuesday if I see no general noise on the internet or negative feedback from you or anyone else.
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, June 17 2020, 08:56 PM - #Permalink
    Resolved
    0 votes
    After further research and communication with Nick, I'm pretty certain that the (now) known microcode update problem was the root cause of my issues.

    I've successfully updated an HPE ML10 Gen 9 unit to the latest 3.10.0-1127.10.1.el7 kernel manually with no issues.

    At this point, I'm suspecting, as mentioned in this CentOS thread, that when the microcode update went sideways that it corrupted the new kernel it was installing at the same time.

    Many thanks to Nick and the ClearOS support folks for assistance throughout this.

    Also see this ClearOS 7.8 release thread.
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, June 17 2020, 10:29 AM - #Permalink
    Resolved
    0 votes
    Another report pointing to it being a microcode issue....
    The reply is currently minimized Show
  • Accepted Answer

    Tuesday, June 16 2020, 07:36 PM - #Permalink
    Resolved
    0 votes
    Looks like a bug in the microcode_ctl package. Please see https://bugs.centos.org/view.php?id=17452. I don't know if you can try downgrading microcode_ctl then blocking the update?
    The reply is currently minimized Show
  • Accepted Answer

    Tuesday, June 16 2020, 07:27 PM - #Permalink
    Resolved
    0 votes
    I'm really not sure what to do about this as all our kernels come directly from upstream untouched. I've been using the kernel while testing the beta for a week or so without issues.

    You could try downgrading to kernel-3.10.0-1127.8.2.el7.x86_64 which was the kernel available for 7.8 prior to last week. This may give you an idea if it is specific to this kernel or the whole 1127 series.
    The reply is currently minimized Show
Your Reply