Forums

Resolved
0 votes
Latest updates last night at 2AM EST pushed a new kernel "kernel-3.10.0-1160.42.2.el7" and the server was unresponsive this morning - I am assuming after reboot it hung at the emergency mode. How can I resolve this issue on a paid subscription version? I tried to reboot again and it hangs at the emergency mode every time.

Pressing Enter on the "no symbol table" moves to the Emergency Mode screen.

Pressing Control-D on the "Emergency Mode" continues to boot the OS without other errors/messages.

See attached screenshots.

Thanks!
Wednesday, September 22 2021, 03:28 PM
Share this post:
Responses (8)
  • Accepted Answer

    Wednesday, September 22 2021, 05:07 PM - #Permalink
    Resolved
    0 votes
    That kernel has been in to community for a week with no ill reports, so lets hope it is something particular to your system.

    During boot you will get a screen which allows you to select which kernel to boot from. You get 3-5 seconds to make your choice before it defaults to the top one. Can you move your cursor to the previous one (3.10.0-1160.41.1.el7.x86_64) and see if that will boot?
    The reply is currently minimized Show
  • Accepted Answer

    Thursday, September 23 2021, 03:10 PM - #Permalink
    Resolved
    0 votes
    Nick Howitt wrote:

    That kernel has been in to community for a week with no ill reports, so lets hope it is something particular to your system.

    During boot you will get a screen which allows you to select which kernel to boot from. You get 3-5 seconds to make your choice before it defaults to the top one. Can you move your cursor to the previous one (3.10.0-1160.41.1.el7.x86_64) and see if that will boot?


    After much poking around and trying different things, I keep getting core dumps in root which fill up the / 100%. I deleted them and immediately rebooted to kernel-3.10.0-1160.42.2.el7 and it booted without emergency mode.

    I am not able to get :81 GUI to open. From command, running firewall -start, I see and error:

    firewall : Running /etc/clearos/firewall.d/10-adamet
    Illegal target name 'ADAMNET'.
    Chain 'ADAMNET' does not exist.

    Would this cause the issue with not being able to access the :81 GUI? How would I go about fixing this?

    BTW, more core dumps after the boot, not as many this time though... 35 count I think...

    I appreciate any further assistance to get this thing running again.
    The reply is currently minimized Show
  • Accepted Answer

    Thursday, September 23 2021, 04:47 PM - #Permalink
    Resolved
    0 votes
    First thing to do is grab your configuration backups from /var/clearos/configuration_backup. It will simplify things if you have to reinstall.

    The ADAMNET reference is for Gateway Mangement/DNSThingy. Ignore it if you can. If the firewall gets to the end of start up you should be OK if it does not panic, even though "systemcrl status firewall" may show failure. If "iptables -nvL" looks OK then the firewall should be OK.

    When I googled your first error, it suggested it was not really a kernel error and that you may need a grub reinstall which I've never done.

    I suspect you may have run out of disk space sometime in the past and things have become corrupt.

    As a start, can you make sure you have enough space in all partitions ("df -h") then boot into an older kernel then try reinstalling the current one so you get a known goof installation?

    The firewall should not stop the webconfig starting. Do a "systemctl restart webconfig" and, if it fails, look at "systemctl status webconfig -l" and "journalctl -xe" and have a look in the logs in /var/log/webconfig for issues. I know when you are out of disk space the webconfig won't let you in.
    The reply is currently minimized Show
  • Accepted Answer

    Friday, September 24 2021, 06:12 AM - #Permalink
    Resolved
    0 votes
    Nick Howitt wrote:

    First thing to do is grab your configuration backups from /var/clearos/configuration_backup. It will simplify things if you have to reinstall.

    The ADAMNET reference is for Gateway Mangement/DNSThingy. Ignore it if you can. If the firewall gets to the end of start up you should be OK if it does not panic, even though "systemcrl status firewall" may show failure. If "iptables -nvL" looks OK then the firewall should be OK.

    When I googled your first error, it suggested it was not really a kernel error and that you may need a grub reinstall which I've never done.

    I suspect you may have run out of disk space sometime in the past and things have become corrupt.

    As a start, can you make sure you have enough space in all partitions ("df -h") then boot into an older kernel then try reinstalling the current one so you get a known goof installation?

    The firewall should not stop the webconfig starting. Do a "systemctl restart webconfig" and, if it fails, look at "systemctl status webconfig -l" and "journalctl -xe" and have a look in the logs in /var/log/webconfig for issues. I know when you are out of disk space the webconfig won't let you in.


    Well, I give up... this thing seems to be too far gone and I cant trace down the root issue(s). I am trying to move all the files off the server but the core dumps in root keep breaking everything...

    Do you know of a way to prevent core dumps entirely so I can save all the files and then reinstall?

    Thanks!
    The reply is currently minimized Show
  • Accepted Answer

    Friday, September 24 2021, 07:26 AM - #Permalink
    Resolved
    0 votes
    How about a link like https://hoststud.com/resources/how-to-disable-core-dumps-in-linux-centos-server.292/? Google will be your friend.

    I can't remember but don't the core dumps always have a consistent file name? If they do, as an alternative approach, you can try dropping an event into /etc/clearsync.d/. In it you'll need something like:

    <?xml version="1.0" encoding="ISO-8859-1"?>
    <!-- Watch for core dumps -->
    <plugin name="CoreDumpWatch" library="libcsplugin-filewatch.so" stack-size="65536">
    <action-group name="MyCoreDumps" delay="1">
    <action>sudo rm -f /core.*</action>
    </action-group>

    <on-create type="path" action-group="MyCoreDumps">/core.*</on-create>
    <on-modify type="path" action-group="MyCoreDumps">/core.*</on-modify>

    </plugin>
    I don't know the rules around the file but I just copy other files from the same place as a template. This assumes the core dump is called core.*. This should remove each core dump within a second of it being created. If it takes a while to create a core dump you may need to increase the delay a bit to avoid a file lock.
    The reply is currently minimized Show
  • Accepted Answer

    Friday, September 24 2021, 08:16 PM - #Permalink
    Resolved
    0 votes
    Nick Howitt wrote:

    How about a link like https://hoststud.com/resources/how-to-disable-core-dumps-in-linux-centos-server.292/? Google will be your friend.

    I can't remember but don't the core dumps always have a consistent file name? If they do, as an alternative approach, you can try dropping an event into /etc/clearsync.d/. In it you'll need something like:

    <?xml version="1.0" encoding="ISO-8859-1"?>
    <!-- Watch for core dumps -->
    <plugin name="CoreDumpWatch" library="libcsplugin-filewatch.so" stack-size="65536">
    <action-group name="MyCoreDumps" delay="1">
    <action>sudo rm -f /core.*</action>
    </action-group>

    <on-create type="path" action-group="MyCoreDumps">/core.*</on-create>
    <on-modify type="path" action-group="MyCoreDumps">/core.*</on-modify>

    </plugin>
    I don't know the rules around the file but I just copy other files from the same place as a template. This assumes the core dump is called core.*. This should remove each core dump within a second of it being created. If it takes a while to create a core dump you may need to increase the delay a bit to avoid a file lock.


    Thanks for the help Nick, I had already tried the google approach and had already added the hoststud.com method but that did not work... so I was thinking there was something specific for clearos... I just tried your plugin but that does not seem to work either... do I need to restart a service to make the plugin work? / actually just editing my previous reply > I rebooted and one of the two seems to be working preventing the core dumps from root... Interestingly, I was able to load the latest kernel without emergency mode...

    and the firewall status is not in panic anymore, the status is active exited....

    I cannot access the webconfig - even after "systemctl restart webconfig" - "systemctl status webconfig" shows active running.

    see this image for the firewall status, I see ibvpn which was previously uninstalled through the webconfig.... wondering if this is causing any issues?


    Thanks.
    The reply is currently minimized Show
  • Accepted Answer

    Friday, September 24 2021, 09:30 PM - #Permalink
    Resolved
    0 votes
    Al Catoe wrote:

    Nick Howitt wrote:

    How about a link like https://hoststud.com/resources/how-to-disable-core-dumps-in-linux-centos-server.292/? Google will be your friend.

    I can't remember but don't the core dumps always have a consistent file name? If they do, as an alternative approach, you can try dropping an event into /etc/clearsync.d/. In it you'll need something like:

    <?xml version="1.0" encoding="ISO-8859-1"?>
    <!-- Watch for core dumps -->
    <plugin name="CoreDumpWatch" library="libcsplugin-filewatch.so" stack-size="65536">
    <action-group name="MyCoreDumps" delay="1">
    <action>sudo rm -f /core.*</action>
    </action-group>

    <on-create type="path" action-group="MyCoreDumps">/core.*</on-create>
    <on-modify type="path" action-group="MyCoreDumps">/core.*</on-modify>

    </plugin>
    I don't know the rules around the file but I just copy other files from the same place as a template. This assumes the core dump is called core.*. This should remove each core dump within a second of it being created. If it takes a while to create a core dump you may need to increase the delay a bit to avoid a file lock.


    Thanks for the help Nick, I had already tried the google approach and had already added the hoststud.com method but that did not work... so I was thinking there was something specific for clearos... I just tried your plugin but that does not seem to work either... do I need to restart a service to make the plugin work? / actually just editing my previous reply > I rebooted and one of the two seems to be working preventing the core dumps from root... Interestingly, I was able to load the latest kernel without emergency mode...

    and the firewall status is not in panic anymore, the status is active exited....

    I cannot access the webconfig - even after "systemctl restart webconfig" - "systemctl status webconfig" shows active running.

    see this image for the firewall status, I see ibvpn which was previously uninstalled through the webconfig.... wondering if this is causing any issues?


    Thanks.


    Well, now it seems to be working again... thank you for all your help... I am going to pull all the files off the server then reinstall.

    Best regards!
    The reply is currently minimized Show
  • Accepted Answer

    Saturday, September 25 2021, 07:46 AM - #Permalink
    Resolved
    0 votes
    I was replying yesterday when you posted again to say that one of the solutions had worked.

    Re the plugin method, there is another plugin which sees any changes to the clearsync.d/ folder and restarts clearsync which will read the new plugin. The problem I have with this method is that in the past I've had to make a few attempts to get the file right, so what I posted may not work and I didn't have time to test. I suspect it was the other method which worked after rebooting. I was just putting up a little script with an infinite loop which deleted core.* every second when you posted that you'd had success. In reflection it would have been easier than the plugin as it only needed 4 lines of code.

    Longer term, I suggest you install logwatch. It runs nightly and e-mails you a report of various activities on your system and includes the output of "df -h".
    The reply is currently minimized Show
Your Reply