Is there any reference to the OOM Killer (out of memory killer) in the logs (not sure which one - probably messages or system).

Curiously, I had the same issue this morning with a server running as a VM (VirtualBox). Found it in a 'saved' state. On resume, I found the webconfig page unresponsive - that is to say that filling in login details and hitting enter just stayed put. I logged in via WinSCP/Putty quite happily, so did a reboot now. The VM hung 'stuck' - odd. Sent the power off signal to the VM and restarted it .. successfully. Haven't examined all the logs yet, but there's indications what went wrong.

Odd to say the least.

Oh and yes, there is plenty of diskspace available.

Just found this ...

Wed Sep 4 01:41:50 2019 info: system - syswatch terminated

I was in bed asleep ...

Michael,

Are you able to log in to Webconfig using the 'root' user? In your log, you have various errors, is there anyway you can correlate which actions on the UI give you which results? You can open a separate console window and use 'tail -f' to follow the log and determine this.

Leon,

What service is associated with your events? I'm not sure that your issue is correlated to Michaels. If you stop the httpd service (systemctl stop httpd) and the webconfig service (systemctl stop webconfig) does the problem go away?

@Michael,
I think I'll need more output than that. Also can you state if it is a root or user login? I am a little unclear if you now have root logins working.

If you have root logins working and not user logins, is LDAP running "service slapd status"?

Hi All

My unit started to show similar issues yesterday morning 09:00 SAST.
As the day progressed it became more and more unresponsive to the point where i could not access the system anymore at about 22:15 SAST

The only thing i saw was that there was a update that started at 02:10 SAST and that u had a high number of events, when i was still able to access the WEB interface.
I have now powered my system off - pulled the plug - as that was the only way to shut it down as i lost WEB and SSH access last night.

I hope that someone can assist before i attempt to do a clean install - not really looking forward to it...

Thanks to both of you, I have changed the password for root and this works fine, but there is no change for the webconfig. I tried to attach the debug.log but it will not let me. There are a few error lines but I cannot see anything that cries out as a massive error. Is there anything in particular I should be looking out for? I've trimmed a lot of debug an info lines, is the below any use?

Apr 7 9:20:18 UTC 2019: 0.1356: error: PHP notice: /usr/clearos/framework/application/libraries/Login_Session.php (344): Undefined index: lang_code

Apr 7 9:20:19 UTC 2019: 0.1493: exception: info: /usr/clearos/apps/base/libraries/File.php (426): File not found.

Apr 7 9:20:19 UTC 2019: 0.1830: error: PHP notice: /usr/clearos/framework/application/core/MY_Loader.php (125): Undefined property: CI::$user_agent

Apr 7 9:20:19 UTC 2019: 0.1938: error: PHP notice: /usr/clearos/framework/application/libraries/Login_Session.php (498): Trying to get property of non-object

Apr 7 9:20:19 UTC 2019: 0.1939: error: PHP notice: /usr/clearos/framework/application/libraries/Login_Session.php (498): Trying to get property of non-object

Apr 7 9:20:19 UTC 2019: 0.1940: error: PHP warning: /usr/clearos/framework/application/libraries/Login_Session.php (503): Invalid argument supplied for foreach()

Apr 7 9:20:19 UTC 2019: 0.1940: error: PHP notice: /usr/clearos/framework/application/libraries/Login_Session.php (508): Indirect modification of overloaded property CI_Session::$CI has no effect

Apr 7 9:20:19 UTC 2019: 0.1941: error: PHP warning: /usr/clearos/framework/application/libraries/Login_Session.php (508): Creating default object from empty value

Apr 7 9:20:19 UTC 2019: 0.5065: error: PHP notice: /usr/clearos/framework/application/libraries/Page.php (432): Undefined variable: app

Apr 7 9:20:19 UTC 2019: 0.5195: error: PHP notice: /usr/clearos/framework/application/libraries/Page.php (1021): Undefined index: nav_acl

Apr 7 9:20:20 UTC 2019: 0.1460: error: PHP notice: /usr/clearos/framework/application/core/MY_Loader.php (125): Undefined property: CI::$user_agent

Apr 7 9:20:20 UTC 2019: 0.4612: error: PHP notice: /usr/clearos/framework/application/libraries/Page.php (432): Undefined variable: app

Apr 7 9:20:20 UTC 2019: 0.4746: error: PHP notice: /usr/clearos/framework/application/libraries/Page.php (1021): Undefined index: nav_acl

Apr 7 9:20:24 UTC 2019: 2.7863: exception: info: /usr/clearos/apps/clearcenter/libraries/Web_Service.php (272): System is not registered.

Apr 7 9:25:02 UTC 2019: 0.0712: exception: info: /usr/clearos/apps/base/libraries/File.php (426): File not found.

Apr 7 10:25:02 BST 2019: 0.1530: profile: debug: clearos\apps\events\Events::get_instant_status (345): 

Apr 7 9:25:02 UTC 2019: 0.1088: profile: debug: clearos\apps\base\File::__construct (176): 

Apr 7 10:25:02 BST 2019: 0.1532: profile: debug: clearos\apps\events\Events::_load_config (878): 

Apr 7 9:25:02 UTC 2019: 0.1090: profile: debug: clearos\apps\base\File::exists (392): 

Apr 7 10:25:02 BST 2019: 0.1532: profile: debug: clearos\apps\base\Configuration_File::__construct (121): 

Apr 7 9:25:02 UTC 2019: 0.1091: profile: debug: clearos\apps\base\Shell::__construct (105): 

Apr 7 10:25:02 BST 2019: 0.1533: profile: debug: clearos\apps\base\Configuration_File::load (160): 

Apr 7 9:25:02 UTC 2019: 0.1091: profile: debug: clearos\apps\base\Shell::execute (132): /bin/ls '/var/run/webconfig/network2db.pid'

Apr 7 10:25:02 BST 2019: 0.1534: profile: debug: clearos\apps\base\File::__construct (176): 

Apr 7 10:25:02 BST 2019: 0.1534: profile: debug: clearos\apps\base\File::get_contents_as_array (224): 

Apr 7 10:25:02 BST 2019: 0.1535: profile: debug: clearos\apps\base\File::get_size (423): 

Apr 7 10:25:02 BST 2019: 0.1536: profile: debug: clearos\apps\base\File::exists (392): 

Apr 7 10:25:02 BST 2019: 0.1537: profile: debug: clearos\apps\base\File::_get_system_max_bytes (1599): 

Apr 7 10:25:02 BST 2019: 0.1539: profile: debug: clearos\apps\base\File::exists (392): 

Apr 7 10:25:02 BST 2019: 0.1542: error: PHP notice: /usr/clearos/apps/events/libraries/Events.php (350): Undefined index: instant_status

Thanks to both of you, I have changed the password for root and this works fine, but there is no change for the webconfig. I have attached the debug.log there are a few warning lines but I cannot see anything that cries out as a massive error. Is there anything in particular I should be looking out for?

Can you turn on webconfig debugging with:

touch /etc/clearos/webconfig.debug

Then try to log into the webconfig. After that please delete /etc/clearos/webconfig.debug then have a look in /var/log/webconfig/debug.log for errors.

To change root, do as Nick suggests. Try a simple password but a very long password just to validate that it is working. The 'passwd' command won't change user passwords because they are either in OpenLDAP or Active Directory. If in OpenLDAP, you can use the '/usr/sbin/userpasswd username' command from command line or make the change within webconfig.

Thanks Dave, space looks fine...
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/clearos-root 103G 12G 91G 12% /
devtmpfs 3.8G 0 3.8G 0% /dev
tmpfs 3.9G 8.0K 3.9G 1% /dev/shm
tmpfs 3.9G 8.9M 3.9G 1% /run
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/sdd1 1014M 167M 848M 17% /boot
/dev/md0 917G 77M 870G 1% /store/data1
/dev/sdc 3.6T 482G 3.0T 14% /store/data2
tmpfs 786M 0 786M 0% /run/user/993
tmpfs 786M 0 786M 0% /run/user/0

Make sure that your system has not run out of space.

df -h

Thanks Nick, I'm not sure exactly when the problem started, there is an off chance there was a power outage involved but as i said everything else works so I didn't notice.

My password has upper and lower case letters, numbers and an !. A user on the system had uppercase lowercase and numbers and the same behavior persists, I click Login and the page just refreshes. I used passwd and changed the root password and its made no change.

Do you know when the problem started? Do you have any special characters in your passwords? There are some guidelines and I have tried to pull them together:

Make sure that you set a strong password - i.e. it should not be based solely on a dictionary word and should contain numbers or other valid non-alpha characters. Also, the system will not stop you from using a bad password. It:

Must start with a letter or number
Must not contain the following characters: | ;*

In addition we suggest passwords that do not have a percent sign '%' which is then followed by a hexidecimal capable set of numbers. This can be interpreted by some web forms (including Webconfig) as a special character.

You can change the root pasword from the command line using the "passwd" command. It does not work with user passwords.