Backing Up With Rsync

Before you try any backups from the NAS4Free system, you may wish to see what files and directories will actually be backed up from each system on your network. After you've installed DeltaCopy on Windoze, or rsyncd on your Linux boxes, you can use this command from the console on the NAS4Free system or a logged-in terminal to see what rsync modules are available for backup:

     /usr/local/bin/rsync rsync://mysys

For each of the modules listed, you can verify what files and directories will be backed up by passing the individual module names to rsync, one at a time, like this:

     /usr/local/bin/rsync --recursive --list-only rsync://mysys/modname

Once everything looks good, the following script can be used to back up files from Windoze or Linux remote systems using rsync. Copy it to your System share and then, using a copy command from the console or a logged-in terminal, move it to the OS volume, where nobody will monkey with it:

     su
     mkdir /etc/rsync
     cp /mnt/System/backup /etc/rsync
     chown root:wheel /etc/rsync/backup
     chmod u=rwx,go=rx /etc/rsync/backup

Here is the backup script. You should change the email variables, FromAddr and ToAddr, before you install the script. You may also wish to change the defalut ZFS pool, if you don't plan on setting it via the command line. And, the default backup file ownership and permissions should be set to something other than "root:wheel", etc., if you don't like those settings and don't wish to set them for each backed up directory, in the config file.

/etc/rsync/backup:

     #!/bin/bash
     #
     # Shell script to back up files from remote systems, using rsync, to the
     # ZFS pool on NAS4Free.
     #
     # This script should be run at regular intervals (typically hourly), by
     # cron, under the root userid.  It can be run whenever backups should be
     # done with the times set in the config file determining when the backups
     # are actually done.  The command line looks like this:
     #
     #      backup [-d|--debug] [-h|--help] [-n|--noemail] [-v|--verbose]
     #             configfile [zfspool]
     #
     # The parameters are:
     #
     #      -d|--debug             Set this debug flag to test the config file,
     #                             etc.  Doesn't actually run rsync.
     #
     #      -h|--help              Setting this flag displays help information.
     #
     #      -n|--noemail           If this flag is set, don't send an e-mail with
     #                             the results.  Normally, if the email variables
     #                             are set (below), email is sent to the specified
     #                             address, giving the results of the backup.
     #
     #      -v|--verbose           If this flag is present, output status
     #                             information to stdout, as the backup
     #                             progresses.
     #
     #      configfile             The configuration file that drives the backups
     #                             to be done (see below).
     #
     #      zfspool                Optional ZFS pool name, if you wish to
     #                             override the internal value (set below).
     #
     # The configuration file that is read by this backup script is used to
     # decide which Rsync modules (share points) on one or more remote systems
     # should be backed up.
     #
     # Note that the configuration file must not contain any of the evil,
     # MS-generated carriage returns.  If it does, this script will undoubtedly
     # function incorrectly.  This ain't f**king Windoze, Sparky.
     #
     # The config file contains one or more source systems, specified by the
     # "Source" keyword, followed by an equal sign and the system's name or IP
     # address.  If a machine name is given, it must be able to be resolved via
     # DNS or the hostname must be added to the /etc/hosts file via the
     # Network/Hosts tab on the NAS4Free Web page.
     #
     # After that, you may supply one or more destination directories (in the
     # ZFS pool) and one or more time strings.  These are specified by the "Dest"
     # and "Time" keywords respectively, followed by an equal sign and the
     # destination directory or time string.
     #
     # The destination directory is given as a relative subdirectory underneath
     # the ZFS pool that is either implied or specified on the command line.
     # If the directory doesn't exist in the ZFS pool, it will be created.
     #
     # The time string is given as one or more hours, separated by spaces, in a
     # simple list.  The hour numbers use the 24-hour clock.  If this backup
     # script is run at an hour that matches one of those in the list, the
     # backup will be done.  A time string of "" will match all backup times.
     #
     # The ownership and permissions of the backed up directories and files will
     # be set to the default values set within this script.  If you don't like
     # this arrangement (e.g. if you are trying to adhere to a more diverse
     # security scheme), you may set the permissions on an individual backup
     # directory basis using the BackupOwner, DirPerms and FilePerms parameters,
     # which must occur before the module list for the directory that they are to
     # apply to.
     #
     # The values for these parameters must comply with the rules for the chown
     # and chmod commands.  The ownership parameter will be applied to both the
     # directories and files that are backed up.  The DirPerms parameter will
     # give the permissions only for directories.  The FilePerms parameter will
     # give the permissions only for files.  Sample values might be: "root:wheel";
     # "u=rwx,go=rx" for directories; and "u=rw,go=r" for files.
     #
     # Following the keywords, lists of Rsync modules to be backed up to the
     # backup directory are given to direct the actual backups.  The list can
     # consist of the string "", in which case all of the Rsync modules found
     # on the remote system will be backed up.  If the list begins with a minus
     # sign, followed by a space, all of the Rsync modules found on the remote
     # system will be backed up, except for those modules listed.  Otherwise,
     # only those modules listed will be backed up.  The list of modules should
     # be separated by blanks.
     #
      Blank lines and lines that begin with "" can be liberally interspersed
     # throughout the config file.  Comments may not be applied to the parameter
      or list lines, even if they do start with "".  Note that the keywords
     # accumulate and the current value for all three is used whenever a backup
     # list is found.
     #
     # Here is a sample config file:
     #
     #      # Backup the Profiles directory on Audrey.
     #      Source = audrey
     #
     #      Dest = Backup_Audrey
     #      Time = 0
     #        Profiles
     #
     #      # Backup all of the directories on Gabriella.
     #      Source = gabriella
     #
     #      Dest = Backup_Gabby
     #      Time = 0
     #      BackupOwner = root:gabby
     #      DirPerms = ug=rwx,o=
     #      FilePerms = ug=rw,o=
     #        *
     #
     #      # Backup all of the directories on Video-Game (at diffent times).
     #      Source = video-game
     #
     #      Dest = Backup_Windoze
     #      Time = 2,10,14,18
     #        - Profiles
     #
     #      Dest = Backup_Video-Game
     #      Time = 0
     #        Profiles
     #
     # Note that this script creates a lock file for each destination directory
     # to prevent multiple backups from running at the same time.  If the system
     # crashes while the lock is held or you need to break the lock for any other
     # reason, simply delete the file "/var/run/rsync_client_running_backupdir",
     # replacing the string "backupdir" with the actual name of the top level
     # backup directory.
     #
     ############################################################################
     #
     # Default ZFS pool where all backup files are to be stored.  You can modify
     # this parameter to reflect your NAS4Free configuration, if you don't use the
     # zfspool positional on the command line.
     #
     DefZFSPool="/mnt/Pool1/Datastore1"
     #
     # File ownership.  Rsync has its own ideas about who should own what.  We
     # may have different ideas.
     #
     DefBackupOwner="root:wheel"
     DefDirPerms="u=rwx,go=rx"
     DefFilePerms="u=rw,go=r"
     #
     # Email variables.  Set these up, if you want to have the script email you
     # backup status.
     #
     FromAddr=NAS4Free.Alexandria@gntrains.com
     ToAddr=ewilde@gntrains.com
     #
     # Email commands.
     #
     Printf=/usr/bin/printf
     MailerSMTP=/usr/local/bin/msmtp
     MailerSMTPConf=/var/etc/msmtp.conf
     #
     # Arguments.
     #
     Debug=0
     WantMail=1
     Verbose=0
     #
     # Work areas.
     #
     Body=""
     Backups=0
     Errors=0
     LineNum=0
     DestDir=""
     SourceSys=""
     TimeStr=""
     ############################################################################
     #
     # Function to do an rsync backup from the source system to the destination
     # directory.
     #
     function BackupSource
         {
         #
         # Make sure that the user has indicated the source system whose files
         # are to be backed up.
         #
         if [ x"$SourceSys" = x ]; then
             if [ $Debug = 1 ]; then
                 echo "  No source host specified for backup"
             else
                 let Errors=Errors+1
                 LogMsg "No source host specified for backup at config file \
                         line $LineNum"
             fi
             return
         fi
         #
         # Make sure that the user has specified the destination
         # directory where backups are to be put.
         #
         if [ x"$DestDir" = x ]; then
             if [ $Debug = 1 ]; then
                 echo "  No destination specified for backup"
             else
                 let Errors=Errors+1
                 LogMsg "No destination specified for backup at config file \
                         line $LineNum"
             fi
             return
         fi
         #
         # If we aren't debugging the config file, we should check that the
         # time we were given is now, before we do any work.  Otherwise, there's
         # nothing to do.
         #
         if [ x"$TimeStr" != x ] && [ "$TimeStr" != "*" ] && [ $Debug != 1 ]; then
             #
             # See if the current hour is in the run time string.
             #
             CurrHour=$CurrentHour
          for RunTime in $TimeStr; do
              if [ $RunTime = $CurrHour ]; then
                  CurrHour=""
                  break
              fi
          done
          #
          # If the current hour wasn't found in the list, we're done.
          #
          if [ x"$CurrHour" != x ]; then
              return
          fi
      fi
      #
      # If the module list that we were given is one that requires the full
      # list of modules (share points) from the remote system, or one that
      # operates on the full list, let's get it from the remote system now.
      #
      if [ "$1" = "" ] || [ "$1" = "-" ]; then
          SourceModules=`/usr/local/bin/rsync rsync://${SourceSys}`
          #
          # If the list isn't the full list, but one that we modify, let's
          # do that now.
          #
          if [ "$1" = "-" ]; then
              shift 1
              IncludedModules=""
              #
              # Compare the list from the source against the exclude list.
              #
              for Module in $SourceModules; do
                  for Exclude in "$"; do
                      if [ "$Module" = "$Exclude" ]; then
                          Module=""
                          break
                      fi
                  done
                  #
                  # This particular module isn't excluded so add it to the
                  # list.
                  #
                  if [ x"$Module" != x ]; then
                      if [ x"$IncludedModules" = x ]; then
                          IncludedModules=$Module
                      else
                          IncludedModules="${IncludedModules} ${Module}"
                      fi
                  fi
              done
              SourceModules=$IncludedModules
          fi
      else
          SourceModules="$*"
      fi
      #
      # If we're debugging the config file, we now have the list of modules
      # to be backed up so we can dump that list and get out.
      #
      if [ $Debug = 1 ]; then
          echo "Backing up \"$SourceModules\""
          return
      fi
      let Backups=Backups+1
      #
      # If the destination directory doesn't exist, create it now.
      #
      if [ ! -d ${ZFSPool}/${DestDir} ]; then
          mkdir ${ZFSPool}/${DestDir}
          chown $BackupOwner ${ZFSPool}/${DestDir}
          chmod $DirPerms ${ZFSPool}/${DestDir}
      fi
      #
      # Log the beginning of our work.
      #
      LogMsg "Starting rsync backup from ${SourceSys} to ${DestDir}"
      #
      # Check if we're already backing up the selected modules.
      #
      if [ -r /var/run/rsync_client_running_${DestDir} ]; then
          LogMsg "  Previous backup still running, skipping this one"
          return
      fi
      #
      # Set the lock file.
      #
      /usr/bin/touch /var/run/rsync_client_running_${DestDir}
      #
      # Copy all of the modules on the remote system.
      #
      for Module in $SourceModules; do
          LogMsg "  Backing up module ${Module}"
          /usr/local/bin/rsync --log-file=/var/log/rsync_client.log \
              --recursive --times --compress --archive \
              --delete --delete-delay --quiet --perms \
              "rsync://${SourceSys}/${Module}" \
              "${ZFSPool}/${DestDir}/${Module}"
          RetCode=$?
          if [ $RetCode != 0 ]; then
              let Errors=Errors+1
              LogMsg "    Backup failed with return code: $RetCode"
          fi
          #
          # Set the ownership and permissions.
          #
          # We start by setting the file permissions on everything.  This is
          # pretty fast, since we use the "-R" option on chmod.  Then, we
          # come back and set the directory permissions using find, which is
          # also pretty fast, since it only works on the directories.  Note
          # that this works, even if the file permissions take away the
          # execute (i.e. list) permission of the directory because the
          # recursive find will add the execute permission back to the
          # directory before it dives into it.
          #
          chown -R $BackupOwner ${ZFSPool}/${DestDir}/
          chmod -R $FilePerms ${ZFSPool}/${DestDir}/
          find ${ZFSPool}/${DestDir} -type d -exec chmod $DirPerms \{\} \;
      done
      #
      # Clear the lock file.
      #
      /bin/rm -f /var/run/rsync_client_running_${DestDir}
      #
      # Log the end of our work.
      #
      LogMsg "Completed backup from ${SourceSys} to ${DestDir}"
      }
     ############################################################################
     #
     # Function to add a message to the email body and/or log it to stdout.
     #
     function LogMsg
         {
         #
         # Add the message to the email body.
         #
         if [ $WantMail = 1 ]; then
             Body="${Body}${1}\n"
         fi
         #
         # Output the message to stdout, if in verbose mode.
         #
         if [ $Verbose = 1 ]; then
             echo $1
         fi
         }
     ############################################################################
     #
     # Function to send an email status message, if the user has requested it.
     #
     function SendMail
         {
         if [ $WantMail = 1 ]; then
             #
             # Create a subject.
             #
             Subject="NAS4Free rsync backup - `date +"%Y %b %d %H:%M:%S"`"
          if [ $1 -gt 0 ]; then
              Subject="${Subject}: Error Detected"
          fi
          #
          # Send the message.
          #
          $Printf "From:$FromAddr\nTo:$ToAddr\nSubject:$Subject\n\n$Body" | \
                  $MailerSMTP --file=$MailerSMTPConf -t
      fi
      }
     ############################################################################
     #
     # Process all of the arguments (flags and positionals).
     #
     Args=("$*")
     for Arg in $Args; do
         case $Arg in
             #
             # Flags.
             #
             "-d" | "--debug")
                 Debug=1
                 ;;
             "-n" | "--noemail")
                 WantMail=0
                 ;;
             "-v" | "--verbose")
                 Verbose=1
                 ;;
             "-h" | "--help")
                 echo <<-ENDHELP
     usage: $0 [-d|--debug] [-h|--help] [-n|--noemail] [-v|--verbose]
         configfile [zfspool]
      -d|--debug      Debug the config file but don't run rsync.
      -h|--help       Display this help.
      -n|--noemail    Don't send an e-mail with the results.
      -v|--verbose    Output status information to stdout.
      configfile      Config file with backup tasks defined therein.
      zfspool         ZFS pool name, if iternal name is not acceptable.
  ENDHELP
              exit
              ;;
          #
          # Positionals.
          #
          *)
              if [ x"$ConfigFile" = x ]; then
                  ConfigFile=$Arg
              else
                  if [ x"$ZFSPool" = x ]; then
                      ZFSPool=$Arg
                  fi
              fi
              ;;
      esac

done
#
# If there are no email addresses, we'll turn off email. #
if [ x"$FromAddr" = x ] || [ x"$ToAddr" = x ]; then

      WantMail=0

fi
#
# Make sure that the user has indicated where the config file is. #
if [ x"$ConfigFile" = x ]; then

      Verbose=1
      LogMsg "No config file specified listing the backup tasks"
      SendMail 1
      exit 1

fi
#
# See if the user has specified the ZFS pool where backups are to be put. #
if [ x"$ZFSPool" = x ]; then

      ZFSPool=$DefZFSPool

fi
#
# Set the default ownership and permissions. #
BackupOwner=$DefBackupOwner
DirPerms=$DefDirPerms
FilePerms=$DefFilePerms
#
# Get the current hour. Used to compare against the time string parameter # in the config file.
#
CurrentHour=`date +%k`
#
# Read the config file and process all of the backups therein. #
while read Line; do

      #
      # Count lines for error reportage.
      #
      let LineNum=LineNum+1
      if [ $Debug = 1 ]; then
          echo Line $LineNum: $Line
      fi
      #
      # If the line is not a comment, process it.
      
      if [ "`echo $Line | grep '^[^]'`"x != x ]; then
          #
          # If the line is a parameter and its value, process it.  The
          # parameter will be turned into a shell variable.  Here's what we
          # are looking for:
          #
          #      BackupOwner    The ownership string to apply to the backed up
          #                     directories and files (e.g. root:wheel).
          #
          #      DestDir        The name of the top level directory in the
          #                     ZFS pool where the remote system's files are
          #                     to be backed up.
          #
          #      DirPerms       The directory permission string (e.g.
          #                     u=rwx,go=rx).  Only applies to backed up
          #                     directories.
          #
          #      FilePerms      The file permission string (e.g. u=rw,go=r).
          #                     Only applies to backed up files.
          #
          #      SourceSys      The source remote system whose files are to
          #                     be backed up.  This is given as either an IP
          #                     address or a machine name.
          #
          #      TimeStr        The time string, containing a list of all
          #                     times when the backup should be done.  Only
          #                     the hours are given, in 24-hour format, and
          #                     separated from one another by spaces.
          #
          # Note that you are supposed to be able to turn values read into
          # variables into actual shell variables like this:
          #
          #      eval ${ParmName}=`echo -ne \""$ParmVal"\"`
          #
          # This doesn't work on the NAS4Free version of bash.  Furthermore,
          # the shell variable name that is set depends on what name the
          # user uses in the config file.
          #
          # Instead, we look for what we want and set it.  This lets us
          # control what we get and give error feedback if it ain't what we
          # want.
          #
          if [ "`echo $Line | grep -E '^[^=]+=[^=]+$'`"x != x ]; then
              #
              # Pick up the parameter name and value.  Note that the two
              # values in the square brackets of the sed lines (below) are
              # <space> and <tab>.  You cannot use "\t", "\\t", "\\\\t",
              # "\\\\\\t" or any other combination of backslashes and the
              # letter "t" that you might think of.  Too baddie!  You need
              # to use a real, hard tab.
              #
              ParmName=`echo "$Line" | awk -F"=" '{print $1}' | sed "s/[ ]\$//"`
              ParmVal=`echo "$Line" | awk -F"=" '{print $2}' | sed "s/^[ ]//"`
              if [ $Debug = 1 ]; then
                  echo "  Parm $ParmName = \"$ParmVal\""
              fi
              #
              # Look for all of our variables.
              #
              case "$ParmName" in
                  #
                  # Backup owner.
                  #
                  backupowner|owner|BackUpOwner|Owner)
                      BackupOwner=$ParmVal
                      ;;
                  #
                  # Destination.
                  #
                  dest|destination|Dest|Destination)
                      DestDir=$ParmVal
                      ;;
                  #
                  # Directory permissions.
                  #
                  dirperms|DirPerms)
                      DirPerms=$ParmVal
                      ;;
                  #
                  # File permissions.
                  #
                  fileperms|FilePerms)
                      FilePerms=$ParmVal
                      ;;
                  #
                  # Source.  Note that this parameter resets all of the
                  # other parameters so that stuff left over from the
                  # previous source isn't used with this source.
                  #
                  source|Source)
                      SourceSys=$ParmVal
                      BackupOwner=$DefBackupOwner
                      DestDir=""
                      DirPerms=$DefDirPerms
                      FilePerms=$DefFilePerms
                      TimeStr=""
                      ;;
                  #
                  # Time string.
                  #
                  time|timestr|Time|Timestr|TimeStr)
                      TimeStr="$ParmVal"
                      ;;
                  #
                  # Unknown parameter.
                  #
                  )
                      if [ $Debug = 1 ]; then
                          echo "  Invalid parameter $ParmName"
                      else
                          let Errors=Errors+1
                          LogMsg "Invalid parameter $ParmName in config file \
                                  at line $LineNum"
                      fi
              esac
          #
          # Otherwise, the line is treated is a list, unless it is blank.
          #
          else
              if [ "`echo $Line | grep -c '^\\s$'`" = 0 ]; then
                  if [ $Debug = 1 ]; then
                      echo "  List = \"$Line\""
                  fi
                  #
                  # Looks like we're good to go.
                  #
                  if [ "$Line" = "" ]; then
                      BackupSource ""
                  else
                      BackupSource $Line
                  fi
              fi
          fi
      fi

done < $ConfigFile
#
# Give a report of the number of errors. #
if [ $Errors -gt 0 ]; then

      LogMsg "$Errors error(s) occurred during backup"

fi
#
# Send email, if required.
#
if [ $Errors -gt 0 ] || [ $Backups -gt 0 ] ; then

      SendMail $Errors

fi

Before you run this script, you'll need to create a config file to tell it which sources and modules (share points) are to be backed up and where the backups are to be stored. The script itself has quite a bit of documentation about how to do this so we'll just show a sample config file here. We put it in the System share and then, using a copy command from the console or a logged-in terminal, move it to the OS volume, where nobody will monkey with it:

     su
     cp /mnt/System/backup.cfg /etc/rsync
     chown root:wheel /etc/rsync/backup.cfg
     chmod u=rwx,go=rx /etc/rsync/backup.cfg

/etc/rsync/backup.cfg:

     # This configuration file is read by the NAS4Free backup script to decide
     # which Rsync modules (share points) on one or more remote systems should
     # be backed up.
     #
     # Note that this file must not contain any of the evil, MS-generated
     # carriage returns.  This ain't f**king Windoze, Sparky.
     # Backup all of the directories on Audrey.
     Source = audrey
     Dest = Backup_Audrey
     Time = 0
       *
     # Backup all of the directories on Video-Game.
     Source = video-game
     Dest = Backup_Windoze
     Time = 2 10 14 18
       - Profiles
     Dest = Backup_Video-Game
     Time = 0
       Profiles
     # Backup the global directories on the gateway server.
     Source = gateway
     Dest = Backup_Gateway
     Time = 0
       mail mysql
     # Backup user directories on the gateway server.
     Source = gateway
     Dest = Backup_HomeDirs
     Time = 2 10 14 18
       home-jsmith home-jblow

Before you do any actual damage, you can test your config file by running the script with the debug flag:

     /etc/rsync/backup -d /etc/rsync/backup.cfg

Once you're happy with how your config file is being handled, you can test the operation of the script like this:

     su
     /etc/rsync/backup -n -v /etc/rsync/backup.cfg

If any of the backups specified in the config file occur at the current time (i.e. the current hour), you should see the results of the backup on your terminal. If the time isn't correct, you can come back later or just modify the config file to use the current times. And, if you want to test that the backup job's status message is being delivered properly, you can test it like this:

     su
     /etc/rsync/backup /etc/rsync/backup.cfg

When you are happy with the script and the config file, you should add a cron table entry to run the backup script at hourly intervals (the config file will decide which backups actually run during each hour).

System/Advanced/Cron

     Begin by clicking the "Plus" button to add a new entry.
     Command: /etc/rsync/backup /etc/rsync/backup.cfg
     Who: root
     Description: Back up source_host on a regular basis.
     Schedule time: Minutes - 0, Hours - All, Days - All, Months - All,
                    Weekdays - All
     If you want to test the script, you can click the "Run now" button.  This
     is probably a good idea because the cron job may fail silently later on.
     Click the "Add" button to save the crontab entry and then click to "Apply
     Changes" button to actually save the crontab entry.