Forums

Resolved
0 votes
Hi,

I am relatively new to Linux Servers and mostly work in the Windows Server area but increasingly I am wanting to move over to Linux servers and I need to be clear in my head how things work with clearOS so that when I set one up I get things right.

In the Windows Server world particularly in my experience using Small Business Servers, the way Microsoft configure them is to have an internal domain usually ending .local that is the same as your external email domain name. For example your external email addresses would be someone@xyz.co.uk and your internal domain would be xyz.local.

When setting up remote access using say VPN or remote desktop your external DNS settings would use say "remote.xyz.co.uk" to access the IP address of your server (I usually have my NAT router connected directly to the internet and the server connected directly to that by LAN connection). I only ever use a single net card that connects the Server to the router via a switch box and to the other machines on the network. In SBS boxes the server is the DHCP and DNS Server for the machines connected to it and usually the server is the domain controller that all workstations on the internal network are connected to, I usually also have "mail.xyz.co.uk" pointing to the same IP address and use that for email settings and MX record on the ISP's DNS settings.

I would like to do the same with my clearOS box but want to know if for example this is not the correct way to do it with a Linux server. I.e. should the Linux Server be named as it is seen from the internet, so in my example should I name the clearOS box as remote.xyz.co.uk and not have an internal domain name ending in .local. Indeed can you scrap the whole idea of having the Windows PCs being domain joined and just have them in the "Workgroup" Workgroup??

In the scenario I am thinking of I will have WIndows PCs for use by the users connecting to shares on the clearOS server and also using IMAP email on the clearOS server. I would also want to set up a daily backup that backs up the user's shared folders and the server configuration to a set of external USB drives that would be changed daily and taken off site for disaster recovery. Usually I have 1 USB drive for each day of the working week and these would be switched each morning when the users come in as long as some notification has been received to indicate the backup has finished and is not still in progress.

Could someone advise me the best way to a) name the server and what domain names to use and b) what of the various setup options I should use and c) How to achieve the backup process I am wanting to use as it seems that clearOS doesn't offer the kind of backup I am envisioning (my concern is how the end users will know it's safe to remove the USB drives and how you eject a USB drive on a clearOS box safely and how to make the system know it should backup on whichever drive is plugged in even though they will be different every day??

Sorry for the long question but I am hopeful someone here has used the clearOS system as a replacement for Windows Servers and has managed to figure out these issues.

Siv
Thursday, December 28 2017, 01:41 PM

Location [ View Larger Map ]

http://maps.googleapis.com/maps/api/staticmap?center=52.2354752,-2.3625468000000183&language=en&maptype=roadmap&zoom=5&size=450x300&sensor=true&markers=color:red|label:S|52.2354752,-2.3625468000000183
Share this post:
Responses (16)
  • Accepted Answer

    Thursday, December 28 2017, 06:09 PM - #Permalink
    Resolved
    0 votes
    I don't know if there is a right and wrong way for domain naming. If you use the same internal and external domain then any laptop/mobile configured to use your mail server will work directly with the "external" domain name. If you have them different then you need to use your hosts file (DNS Server) to map the external name to an internal IP and the clients *must* use ClearOS/Other Internal DNS server when connected to the LAN to get the right IP addresses. I use the same internally and externally but I never used to when I first started.

    You probably know more about domains than me so I am not particularly qualified to give best fit.

    Re Workgroup, I think the main thing is that it matches on all your machines for M$ Network Browsing to work.

    I don't use domains, so have Windows Networking in "simple" mode. This is fine in my family environment, but if you want people to log in and do things like authenticate to a proxy then PDC is probably a better option.

    I also use IMAP internally and IMAPS externally. For that you can stick with the free mail stack - IMAP and POP server, Mail Anti-virus and Anti-spam, Greylisting, SMTP server and Mail Settings. Optionally you may want the mail retrieval app if you are picking up your mail from external mailboxes rather than open up ClearOS as a fully-fledged mail server. Your external DNS settings should be fine if they resolve to your external IP and you then port forward to ClearOS. I keep the mail server name the same as the external MX record. If you want to use SMTP externally for sending mail, my suggestion is to keep authentication off and use SMTPS on port 465 which is configured by default and will use authentication anyway. If you are lazy like me you use trusted networks for your LAN or else you have to configure your static PC's to use port 465. If you open anything externally, also use the Attack Detector app for a bit of extra protection.

    I use a homebrew backup solution, backing up mail and the server config daily to offsite by upload. I also do a weekly data backup, but no bare metal backup - I rely on reinstalling and restoring the config for the server. You can do Bare Metal Backup but I don't know if this backs up flexshares. You should be able to delegate admin rights to various screens but I've never done it and I don't know how you eject a USB drive from the Webconfig. I'll have to give BMB a try sometime. You should be able to see when backups are complete, either by e-mail of from the Webconfig interface. Perhaps it is best if you just give BMB a try out.
    The reply is currently minimized Show
  • Accepted Answer

    Thursday, December 28 2017, 06:38 PM - #Permalink
    Resolved
    0 votes
    Nick,
    Thanks for your reply. I am beginning to think I should use the same name as the external name, so that would be host name "remote" and domain would be "xyz.co.uk" and then use that for email and remote access via VPN.

    I already did a test install and did use BMBackup and it does seem to back up what I need however it don't think it allows me to have multiple drives that it uses and I can't see a way to swap the disks I may just have to configure more than one and see if I can switch between them. I am a bit uncomfortable about how you eject the disk as there is nothing in the BMBackup page in the Dashboard that seems to allow for ejection of the disk? How is Linux if you eject the disk without doing it through the O/S will it screw the drive up like Windows will if there are any pending writes or does Linux always write directly to the disk without caching so that once it's completed a backup you can remove it?

    I would be interested to know how your homebrew backups work as this may be a better option for me what I want to do is make something that the end users can manage the backup process themselves so maybe have some limited commands they can run on the server console that allows them to eject the disk.
    The reply is currently minimized Show
  • Accepted Answer

    Thursday, December 28 2017, 07:08 PM - #Permalink
    Resolved
    0 votes
    One thing to bear in mind is that name resolution does not work well through OpenVPN unless you pull a couple of manual tricks in the config. This is a fault of Win10.

    I'll have to post more about the backups some other time as some house guests have just arrived.
    The reply is currently minimized Show
  • Accepted Answer

    Thursday, December 28 2017, 07:18 PM - #Permalink
    Resolved
    0 votes
    Nick,
    Thanks for the update I use PPTPD anyway as that is what I am used to with Windows and is working reliably with ClearOS.
    I'll await your reply re backups when convenient appreciate you have a life away from this place.

    Graham Sivill
    The reply is currently minimized Show
  • Accepted Answer

    Thursday, December 28 2017, 09:53 PM - #Permalink
    Resolved
    0 votes
    Graham wrote

    How is Linux if you eject the disk without doing it through the O/S will it screw the drive up like Windows will if there are any pending writes or does Linux always write directly to the disk without caching so that once it's completed a backup you can remove it?

    Don't use USB drives with ClearOS - but do with Fedora. It caches. When copying large media files notice that the copy can take up to several minutes to complete after the system indicates it is ready for more commands. Removing the USB drive before the cache is empty obviously creates incomplete file(s) and corruption. Suspect ClearOS could be the same...

    Know nothing about BMBackup as custom rsync scripts are used here. Maybe all the drives could have the same label and mounted using the "LABEL=xxxxx" option? Not sure of the wisdom of this... you would need to think that through.

    Tony www.sraellis.tk
    The reply is currently minimized Show
  • Accepted Answer

    Friday, December 29 2017, 02:54 PM - #Permalink
    Resolved
    0 votes
    Tony,
    Thanks for your advice I will bear that in mind.
    Are you aware of a way of forcing the USB drive to flush and report safe to remove?
    This is the only weakspot with ClearOS in that if you are going to try and use it as serious alternative to MS SBS there should be a reliable USB backup option. I am in the UK and the thought of backing up to a cloud service is a non-starter, even using differential backups, as the internet connections in the places where I would use these servers are often in the 2 to 3 mbps area and backups would never complete. Also the threat of these ransomware viruses escaping from a Windows client PC onto the shared folders and encrypting them means that permanently USB connected storage is non-starter.

    Siv
    The reply is currently minimized Show
  • Accepted Answer

    Friday, December 29 2017, 03:16 PM - #Permalink
    Resolved
    0 votes
    Just a small thought before I can give more of an idea of what I do. I have a Raspberry Pi running OSMC on it as a video player and I had a spare USB disk. What I do for some of my backups is rsync them to the disk connected to the Pi (with rsync running as a daemon on the Pi so passwords are not needed). This has the advantage that it is not any form of network file share so would not be attacked by current ransomware. It is also in a completely different part of the house so there is some separation between the server and the backup.
    The reply is currently minimized Show
  • Accepted Answer

    Friday, December 29 2017, 05:11 PM - #Permalink
    Resolved
    0 votes
    Nick,

    Certainly a neat idea. I do need to take the backups off site in case the office catches fire so I would still want to use some form of removable media. If I could get decent broadband rates I would consider going the backup to cloud route as that would make sense but with some of my clients who I may start building clearOS servers for, they like the idea of an off site backup. A lot of my clients do not trust cloud services both from a reliabilty and privacy of sensitive data perspective.

    It is interesting to see how other people handle things like backup. I am surprised that Linux Admins seem to prefer backing up to another machine or to cloud as most Linux guys in the forums seem to be quite conservative in their thinking and not trusting new technologies like cloud until it's been proven safe and reliable. Also I would have thought as experienced admins they would want off site backups as a matter of course to prevent against a disaster like fire or flood taking out your equipment.

    Siv
    The reply is currently minimized Show
  • Accepted Answer

    Friday, December 29 2017, 05:34 PM - #Permalink
    Resolved
    0 votes
    Graham,

    Based on your use case we would like to privately engage and review some disruptive offerings we are working on. Would you email mproper[at]clearcenter.com to connect and review these new offerings and see about getting you added to some of the open programs in your region? Based on your thought process and questions, we feel you would be a solid fit.

    Looking forward to connecting. :)

    M~
    The reply is currently minimized Show
  • Accepted Answer

    Friday, December 29 2017, 11:37 PM - #Permalink
    Resolved
    0 votes
    Graham wrote

    Are you aware of a way of forcing the USB drive to flush and report safe to remove?

    Good Question... paranoid answers :)
    The safest way is to..
    Issue the "sync" command ***
    Monitor the "/sys/block/<device>/stat" file for a 0. It counts down as the buffered data is flushed.
    (Note that if the USB drive has more than one partition - you have multiple <devices> to monitor
    When the file(s) report 0, umount the device(s)

    A number of other options are often used...
    1. If the the USB device has a blinking light to indicate activity - wait until it stops blinking, then umount.
    2. Monitor disk activity in real time. Since Fedora is used here - have gkrellmd running on all systems and monitored on a Fedora virtual desktop using gkrellm. Therefore can monitor disk activity for any partition on any drive on any system visually. Umount after activity stops.
    3. Determine some arbitrary time that is longer than what should be necessary for the flush to complete and wait for that time period... umount
    4. Issue the umount command and monitor the devices mounted and wait until the USB drive disappears before removing it.

    *** There is conflicting information on the web regarding the sync command .Some say it returns immediately - others that is only returns once the sync is complete. Will test to see what Fedora does and update here...

    Tony www.sraellis.tk
    The reply is currently minimized Show
  • Accepted Answer

    Saturday, December 30 2017, 04:27 PM - #Permalink
    Resolved
    0 votes
    Michael,

    I will connect to you via email.

    Graham.
    The reply is currently minimized Show
  • Accepted Answer

    Saturday, December 30 2017, 04:32 PM - #Permalink
    Resolved
    0 votes
    Tony,

    Thanks for all your unmount options and I am interested to hear what your findings are. Ideally I would like something that reports back and tells me for definite that it's safe to remove so anything that does that would be brilliant. Ideally I would like a backup solution that I can set that it occurs every day at a certain time or times and that there is a pool of up to 5 different backup USB drives that can be used. Additionally it would email a report to say that a) the backup is complete and b) that the drive is safe to remove now. I would like to be able to specify who that email goes to (usually me and another end user who is responsible for managing backups at the site where the server is set up.

    Graham Sivill
    The reply is currently minimized Show
  • Accepted Answer

    Sunday, December 31 2017, 04:22 AM - #Permalink
    Resolved
    0 votes
    OK Graham - sync did wait on Fedora for the data to finish flushing to disk - its an older Fedora and the sync command is version "sync (coreutils) 8.15". Both ClearOS 6 and 7 have newer versions so would expect them to execute in the same manner...

    Monitored data transfer activity both by watching my gkrellm displays as well as field 9 in the output of "watch -n 2 cat /sys/block/sdc/sdc1/stat" run in a terminal. Field 9 counted down to '0', the gkrellm indicated transfer had finished and the sync command returned all at virtually the same time.

    Incidentally - if using a device with partitions then the stat file to monitor would be in this format "/sys/block/sdc/sdc1/stat" - in this example partition 1 on device sdc...

    So it should be safe to umount and unplug the USB device after the sync command returns - assuming no more write have been subsequently issued to the USB drive.

    Tony www.sraellis.tk
    The reply is currently minimized Show
  • Accepted Answer

    Sunday, December 31 2017, 10:54 AM - #Permalink
    Resolved
    0 votes
    Tony,

    Thanks for the results of your testing. That does seem to provide a reliable way of ejecting the disk and if I can come up with a bash script that initially mounts the disk, then does the backup and finally ejects the disk and when the sync returns after countdown I can get it to send an email to say the backup is complete and the disk can be unplugged,

    The tricky bit will be getting the script to work with multiple disks i.e. one per day.

    Thanks again,

    Graham Sivill
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, January 03 2018, 07:35 PM - #Permalink
    Resolved
    1 votes
    For backups I have the following strategy:
    1 - Copy key files to a second disk nightly some of which I also rotate:

    • /boot archived to tar.bz2, 14 days rotated (don't ask why 14 days)
    • /etc archived to tar.bz2, (excluding /etc/postfix/filters, /etc/selinux and /etc/cups), 14 days rotated
    • cyrus-imap files (/var/lib/imap and /var/spool/imap excluding /var/lib/imap/socket/*) to a tar file (because later I 7z it and it takes more space if compressed first), 14 days rotated
    • the config backup folder from /var/clearos/configuration_backup/backup*.tgz

    2 - Copy the most recent file from the top three above and the whole of the config folder to a bit of a samba share.
    3 - rsync the files in 2 plus the important shared file of my wife's and mine to a Pi at my mother's house nightly
    4 - rsync my non-critical stuff to my Pi USB Disk which is connected to the Pi media player running OSMC in the playroom, diametrically the other side of the house from my server
    5 - Weekly back up, key stuff on my PC and 3 above to a USB pen, my PC and the server (because it contains a combination of PC and server files so back it up to both places), and rotate these backups.

    On the Pi's rsync runs as a daemon so I can rsync like:
    /usr/bin/rsync -az --delete /BackupDisc/etc/etc.tar.bz2 	pi::Mums_pi/etc/etc.tar.bz2 &
    /usr/bin/rsync -az --delete /BackupDisc/config/ pi::Mums_pi/config/ &
    /usr/bin/rsync -az --delete /BackupDisc/mail/mail.tar pi::Mums_pi/mail/mail.tar &
    /usr/bin/rsync -az --delete /BackupDisc/boot/boot.tar.bz2 pi::Mums_pi/boot/boot.tar.bz2 &
    /usr/bin/rsync -az --delete /shares/private/ pi::Mums_pi/T/ --exclude=*.trash* &


    For mail I do:
    #!/bin/bash

    declare -i COUNTER1=0
    declare -i COUNTER2=0
    declare ArchiveType
    declare TarSwitches

    ArchiveType="tar"

    sleep 300

    cd /BackupDisc/mail

    for MYFILE in "mail.$ArchiveType" ; do
    COUNTER1=13
    COUNTER2=14
    while [ $COUNTER1 -gt 0 ] ; do
    if [ -e $MYFILE.$COUNTER1 ] ; then
    mv -f $MYFILE.$COUNTER1 $MYFILE.$COUNTER2
    fi;
    COUNTER2=$COUNTER1
    COUNTER1=$[$COUNTER1-1]
    done
    cp -f -p $MYFILE $MYFILE.1
    done

    # stop cyrus
    systemctl stop cyrus-imapd.service > /dev/null

    if [ $ArchiveType = "tar" ] ; then
    TarSwitches="-cvPf"
    else
    TarSwitches="-P -cjf"
    fi

    # create new archive; try adding 2>&1 to surpress errors
    tar $TarSwitches /shares/private/ServerBackup/mail.$ArchiveType /var/spool/imap /var/lib/imap --exclude=/var/lib/imap/socket/* > /dev/null
    cp /shares/private/ServerBackup/mail.$ArchiveType /BackupDisc/mail/

    logger -t Mail backed up

    # start cyrus
    systemctl start cyrus-imapd.service > /dev/null
    I'm not proud of the rotation and it may be easier to use (abuse?) the logrotate function.

    I put the routines in cron.daily or cron.weekly. Note the files run alphabetically in each folder so you can control the order by taking care with the names. In cron.daily the last one I run is /etc/cron.daily/z_ServerBackup_to_Pi named as such to keep it at the end of the run so all the other backups are done before I rsync them to my mother's house.
    The reply is currently minimized Show
  • Accepted Answer

    Thursday, January 04 2018, 12:21 AM - #Permalink
    Resolved
    0 votes
    Nick Howitt wrote:

    For backups I have the following strategy:
    1 - Copy key files to a second disk nightly some of which I also rotate:
    <ul>
    /boot archived to tar.bz2, 14 days rotated (don't ask why 14 days)
    /etc archived to tar.bz2, (excluding /etc/postfix/filters, /etc/selinux and /etc/cups), 14 days rotated
    cyrus-imap files (/var/lib/imap and /var/spool/imap excluding /var/lib/imap/socket/*) to a tar file (because later I 7z it and it takes more space if compressed first), 14 days rotated
    the config backup folder from /var/clearos/configuration_backup/backup*.tgz
    </ul>
    2 - Copy the most recent file from the top three above and the whole of the config folder to a bit of a samba share.
    3 - rsync the files in 2 plus the important shared file of my wife's and mine to a Pi at my mother's house nightly
    4 - rsync my non-critical stuff to my Pi USB Disk which is connected to the Pi media player running OSMC in the playroom, diametrically the other side of the house from my server
    5 - Weekly back up, key stuff on my PC and 3 above to a USB pen, my PC and the server (because it contains a combination of PC and server files so back it up to both places), and rotate these backups.

    On the Pi's rsync runs as a daemon so I can rsync like:
    /usr/bin/rsync -az --delete /BackupDisc/etc/etc.tar.bz2 	pi::Mums_pi/etc/etc.tar.bz2 &
    /usr/bin/rsync -az --delete /BackupDisc/config/ pi::Mums_pi/config/ &
    /usr/bin/rsync -az --delete /BackupDisc/mail/mail.tar pi::Mums_pi/mail/mail.tar &
    /usr/bin/rsync -az --delete /BackupDisc/boot/boot.tar.bz2 pi::Mums_pi/boot/boot.tar.bz2 &
    /usr/bin/rsync -az --delete /shares/private/ pi::Mums_pi/T/ --exclude=*.trash* &


    For mail I do:
    #!/bin/bash

    declare -i COUNTER1=0
    declare -i COUNTER2=0
    declare ArchiveType
    declare TarSwitches

    ArchiveType="tar"

    sleep 300

    cd /BackupDisc/mail

    for MYFILE in "mail.$ArchiveType" ; do
    COUNTER1=13
    COUNTER2=14
    while [ $COUNTER1 -gt 0 ] ; do
    if [ -e $MYFILE.$COUNTER1 ] ; then
    mv -f $MYFILE.$COUNTER1 $MYFILE.$COUNTER2
    fi;
    COUNTER2=$COUNTER1
    COUNTER1=$[$COUNTER1-1]
    done
    cp -f -p $MYFILE $MYFILE.1
    done

    # stop cyrus
    systemctl stop cyrus-imapd.service > /dev/null

    if [ $ArchiveType = "tar" ] ; then
    TarSwitches="-cvPf"
    else
    TarSwitches="-P -cjf"
    fi

    # create new archive; try adding 2>&1 to surpress errors
    tar $TarSwitches /shares/private/ServerBackup/mail.$ArchiveType /var/spool/imap /var/lib/imap --exclude=/var/lib/imap/socket/* > /dev/null
    cp /shares/private/ServerBackup/mail.$ArchiveType /BackupDisc/mail/

    logger -t Mail backed up

    # start cyrus
    systemctl start cyrus-imapd.service > /dev/null
    I'm not proud of the rotation and it may be easier to use (abuse?) the logrotate function.

    I put the routines in cron.daily or cron.weekly. Note the files run alphabetically in each folder so you can control the order by taking care with the names. In cron.daily the last one I run is /etc/cron.daily/z_ServerBackup_to_Pi named as such to keep it at the end of the run so all the other backups are done before I rsync them to my mother's house.


    Wow!
    Thanks for taking the the time to post this I think I could use some of your techniques to do a backup by scripting. My issue at the moment is notifying the end user responsible for changing the backup disks that a) the backup has completed successfully and b) that it is safe to eject the disk.

    Graham Sivill
    The reply is currently minimized Show
Your Reply