Some time last night our mail system stopped responding (ClearOS 6.8, Zarafa 7.1.5). The webapp loads, but any time it attempts to retrieve information from the server it gives an error, "could not load the contents of this folder," and we are no longer receiving push notifications. I tried stopping and restarting the zarafa services, and all stop and restart appropriately. Zarafa's services have not been touched in months and the only modifications to this system were the monthly antispam and antimalware updates last week.

The only error I can find is in the server.log:

Mon Apr 10 12:40:49 2017: Shutting down.

Mon Apr 10 12:40:50 2017: Still waiting for 8 threads to exit

Mon Apr 10 12:40:52 2017: Server shutdown complete.

Mon Apr 10 12:44:44 2017: Starting zarafa-server version 7,1,5,42059, pid 12017

Mon Apr 10 12:44:44 2017: Listening for priority pipe connections on /var/run/zarafa-prio

Mon Apr 10 12:44:44 2017: Listening for pipe connections on /var/run/zarafa

Mon Apr 10 12:44:44 2017: Listening for TCP connections on port 236

Mon Apr 10 12:44:44 2017: Connection to database 'zarafa' succeeded

Mon Apr 10 12:44:44 2017: Using commercial license serial '----------------'

Mon Apr 10 12:44:46 2017: Loading searchfolders

Mon Apr 10 12:44:46 2017: Startup succeeded on pid 12022

Mon Apr 10 12:44:46 2017: Not all objects in relation found for object '-------'

Mon Apr 10 12:44:46 2017: Not all objects in relation found for object '-------'

Mon Apr 10 12:45:00 2017: Not all objects in relation found for object '-------'

Mon Apr 10 12:45:00 2017: Not all objects in relation found for object '-------'

Mon Apr 10 12:45:00 2017: Not all objects in relation found for object '-------'

Mon Apr 10 12:45:02 2017: SQL [00000030] Failed: Got error 28 from storage engine, Query Size: 1496, Query: "SELECT 0,properties.tag,properties.type,properties.val_ulong, LEFT(properties.val_string,255),LEFT(properties.val_binary,511),properties.val_double,properties.val_longint,properties.val_hi,properties.val_lo FROM properties WHERE hierarchyid=579753 AND properties.tag IN (23,26,41,48,54,55,55,57,59,63,64,65,66,67,68,81,82,99,100,101,117,118,119,120,3095,3097,3098,3101,3102,3103,3586,3587,3588,3591,3592,3611,4224,4225,4226,4240,4241,4245,4246,12296,16355,26255,32769,32803,32855,33186,33187,33204,33232,33280,34051) UNION SELECT count(*),mvproperties.tag,mvproperties.type,group_concat(length(mvproperties.val_ulong),':', mvproperties.val_ulong ORDER BY mvproperties.orderid SEPARATOR ''), group_concat(length(mvproperties.val_string),':', mvproperties.val_string ORDER BY mvproperties.orderid SEPARATOR ''), group_concat(length(mvproperties.val_binary),':', mvproperties.val_binary ORDER BY mvproperties.orderid SEPARATOR ''), group_concat(length(mvproperties.val_double),':', mvproperties.val_double ORDER BY mvproperties.orderid SEPARATOR ''), group_concat(length(mvproperties.val_longint),':', mvproperties.val_longint ORDER BY mvproperties.orderid SEPARATOR ''), group_concat(length(mvproperties.val_hi),':', mvproperties.val_hi ORDER BY mvproperties.orderid SEPARATOR ''), group_concat(length(mvproperties.val_lo),':', mvproperties.val_lo ORDER BY mvproperties.orderid SEPARATOR '') FROM mvproperties WHERE hierarchyid=579753 AND mvproperties.tag IN (4248,34050)  GROUP BY hierarchyid, tag"

Mon Apr 10 12:45:03 2017: SQL [00000027] Failed: Got error 28 from storage engine, Query Size: 1496, Query: "SELECT 0,properties.tag,properties.type,properties.val_ulong, LEFT(properties.val_string,255),LEFT(properties.val_binary,511),properties.val_double,properties.val_longint,properties.val_hi,properties.val_lo FROM properties WHERE hierarchyid=579753 AND properties.tag IN (23,26,41,48,54,55,55,57,59,63,64,65,66,67,68,81,82,99,100,101,117,118,119,120,3095,3097,3098,3101,3102,3103,3586,3587,3588,3591,3592,3611,4224,4225,4226,4240,4241,4245,4246,12296,16355,26255,32769,32803,32855,33186,33187,33204,33232,33280,34051) UNION SELECT count(*),mvproperties.tag,mvproperties.type,group_concat(length(mvproperties.val_ulong),':', mvproperties.val_ulong ORDER BY mvproperties.orderid SEPARATOR ''), group_concat(length(mvproperties.val_string),':', mvproperties.val_string ORDER BY mvproperties.orderid SEPARATOR ''), group_concat(length(mvproperties.val_binary),':', mvproperties.val_binary ORDER BY mvproperties.orderid SEPARATOR ''), group_concat(length(mvproperties.val_double),':', mvproperties.val_double ORDER BY mvproperties.orderid SEPARATOR ''), group_concat(length(mvproperties.val_longint),':', mvproperties.val_longint ORDER BY mvproperties.orderid SEPARATOR ''), group_concat(length(mvproperties.val_hi),':', mvproperties.val_hi ORDER BY mvproperties.orderid SEPARATOR ''), group_concat(length(mvproperties.val_lo),':', mvproperties.val_lo ORDER BY mvproperties.orderid SEPARATOR '') FROM mvproperties WHERE hierarchyid=579753 AND mvproperties.tag IN (4248,34050)  GROUP BY hierarchyid, tag"

Error 28 is a space error, so I checked the partitions:

Filesystem            Size  Used Avail Use% Mounted on

/dev/mapper/main-root

                      4.8G  4.6G  492K 100% /

tmpfs                 855M     0  855M   0% /dev/shm

/dev/sda1             477M  136M  316M  31% /boot

/dev/mapper/data-data0

                      1.4T   53G  1.3T   4% /store/data0

/dev/mapper/main-var   20G  962M   18G   6% /var

/dev/mapper/main-log  7.8G  515M  6.9G   7% /var/log

Which seems like it would be cause for alarm, but it was my understanding was that the sql system lived on data0 and not root, and that 492k would be large enough for an sql query in any case. Am I incorrect in that assumption? Is this a zarafa-specific issue? If it is an issue with the space on root, what is the best way to clear or add space?