LinHES Forums
http://forums.linhes.org/

What a cool app NCDU
http://forums.linhes.org/viewtopic.php?f=24&t=24047
Page 1 of 1

Author:  mattbatt [ Mon Sep 01, 2014 10:11 pm ]
Post subject:  What a cool app NCDU

I got a warning that my Harddrive was 100% full! Since my storage Directories say that I'm only about 50% full I was confused. I was able to install NCDU and figure out that my redundant database backup folder on a 15 GB partition had grown out of control.
NCDU was awesome at pointing me to the largest directories.

Thanks admins for putting it in the Repo.

Author:  bgrant3406 [ Wed Sep 03, 2014 6:50 am ]
Post subject:  Re: What a cool app NCDU

I had noticed that while my main backup directory was getting pruned, my remote backup directory wasn't, not a huge issue since it is on a 1TB drive, but knowing me, I would forget about it and in a few year run into the same issue you ran into.

My MBE has two Harddrives, with the second drive setup via the LinHES setup menu->advanced settings->screen 2, remote backup enabled as:
Code:
/data/storage/disk1/backup/system_backups


I wanted 2 main things, for my remote backup directory to be updated the same as my main backup directory and for there to be a message in "myth_mtc.log" indicating if files had been removed. It would be nice if the log message was towards the end of the "myth_mtc.log" file, rather than somewhere in the middle.

So being a total hack at programming, I made the following changes to:
Code:
 /usr/LH/bin/lh_system_backup_job

I replaced the entire
Code:
function remove_old_backups()

with the following:
Code:
 function remove_old_backups(){                                                                                             
    #remove old backups                                                                                                   
    NumBackups=`ls $BACKUPDIR/backup*.tgz|wc -l`                                                                           
    if [[ $NumBackups -gt $KeepBackups ]]; then                                                                           
        numdel=$(($NumBackups-$KeepBackups))                                                                               
        echo                                                                                                               
        echo " Removed Oldest Backup File from $BACKUPDIR: "                                                               
        rm -f `ls $BACKUPDIR/backup*.tgz -tr1|head -$numdel`
    else                                                                                                                   
        echo
        echo "    No Backups removed, less than "$KeepBackups" backup files in directory. "
    fi                                                                                                                     
    if [ x$RemoteBackup = x1 ]
    then                                                                                                                   
        localRemoteCheck=`echo $RemoteBackupDir | cut -d: -f1`                                                             
        if [ x$localRemoteCheck = xdir ]                                                                                   
        then                                                                                                               
            localRemotedir=`echo $RemoteBackupDir | cut -d: -f2`
            RemoteNumBackups=`ls $localRemotedir|wc -l`
                if [[ $RemoteNumBackups -gt $KeepBackups ]]; then
                    remnumdel=$(($RemoteNumBackups-$KeepBackups))
                    echo                                                                                                   
                    echo "Removed Oldest Remote Backup File: "                                                             
                    rm -f `ls $localRemotedir/backup*.tgz -tr1|head -$remnumdel`                                           
                else                                                                                                       
                    echo                                                                                                   
                    echo "    No Backups removed, less than "$KeepBackups" backup files in directory. "                           
                fi                                                                                                         
        else                                                                                                               
            /usr/bin/func ${RemoteBackupDir} ping| grep -q "FAILED"                                                       
            rc=$?                                                                                                         
            if [ $rc = 0 ]                                                                                                 
            then                                                                                                           
                #this is here to mark failed removal of the backup.                                                       
                echo " Remote drive failed, Drive appears to be off-line. "                                               
            else                                                                                                           
                echo " Reminder to manually cleanup older backups from /usr/bin/func ${RemoteBackupDir} "                 
            fi                                                                                                             
       fi                                                                                                                                                                                                                       
    fi                                                                                                                     
}   


To move the message to the end of the log, I needed to move when the removal function was run. Since the script above had a check to make sure there is a minimum number of backups before it will remove any, I felt it was pretty safe to move the "remove_old_backups" call. I commented out the original location as shown:
Code:
 #------------------------------------                                                                                     
lock_myth                                                                                                                 
if [ "x$1" = "x" ]                                                                                                         
then                                                                                                                       
    #remove_old_backups                                                                                                   
    backup                                                                                                                 
    update_backup_status                                                                                                   
    remote_backup                                                                                                         
    add_link                                                                                                               
    remove_old_backups                                                                                                     
else                                                                                                                       
    #this is where we attempt to transfer again as part of the cronjob                                                     
    if [ -f $BACKUPDIR/remote_backup_failed.txt ]                                                                         
    then                                                                                                                   
        mv -f $BACKUPDIR/remote_backup_failed.txt /tmp                                                                     
        while read line                                                                                                   
        do                                                                                                                 
            echo $line                                                                                                     
            remote_transfer $line                                                                                         
        done < /tmp/remote_backup_failed.txt                                                                               
        rm -f /tmp/remote_backup_failed.txt                                                                               
    fi                                                                                                                     
    remove_link                                                                                                           
fi                                                                                                                         
unlock_myth                                                                                                               
echo $backup_status > /var/run/systembackup.rc                                                                             
exit $backup_status 


The main issue with the moving "remove_old_backups" is that you will end up with 13 backups in rotation instead of 14. To change the number to something other than 13, locate the following line (should be about the 6th line from the top of the file) and change the 13 to what ever number you wish, I changed mine to 14 to maintain two weeks worth of backup files:
Code:
KeepBackups=13

The nice thing about the way the original pruning works is it will remove any files over the number specified in "KeepBackups=", so if you have had a bunch of files built up in your remote backup drive, after the change is made to the file and "myth_mtc.py" runs, all of the excess files should be removed.

Hopefully if someone has a similar issue and needs a starting point on where to look/what to change, this will give them a quick idea of what to look for/at.

These changes have only been tested on my specific setup, since I don't want to potentially mess up my system, I have no way to test the other scenario's or extend the scripts so that it will correctly prune either a remote network drive, or the secondary backup location when remote backup is not enabled via LinHES setup Advanced menu option.

HTH anyone wanting to automate cleaning up the remote backup directory.
Bruce.

Author:  brfransen [ Wed Sep 03, 2014 9:08 am ]
Post subject:  Re: What a cool app NCDU

Bruce,

Nice job figuring out a solution to your problem. A couple of things to note. First, be aware that the next time the LinHES-system package is updated/reinstalled all your changes to lh_system_backup_job will be overwritten. Second, there is no need for the reminder to manually clean up files in SECBACKUPLINK as it should be a straight clone (using rsync) of the BACKUPDIR.

Author:  bgrant3406 [ Wed Sep 03, 2014 10:14 am ]
Post subject:  Re: What a cool app NCDU

brfransen wrote:
Bruce,

Nice job figuring out a solution to your problem. A couple of things to note. First, be aware that the next time the LinHES-system package is updated/reinstalled all your changes to lh_system_backup_job will be overwritten. Second, there is no need for the reminder to manually clean up files in SECBACKUPLINK as it should be a straight clone (using rsync) of the BACKUPDIR.


Thanks for pointing out the SECBACKUPLINK, I've removed it from code above (and my system).

Also great reminder about losing the file on an update, I have a separate directory where I keep all files I have modified, so when I do update I can just compare old/new file and see if I do a direct replacement, or if I need to modify the new file.

Thanks again!
Bruce.

Author:  mattbatt [ Wed Sep 03, 2014 3:23 pm ]
Post subject:  Re: What a cool app NCDU

I only had 10 backups so I think the pruning process is working.

Page 1 of 1 All times are UTC - 6 hours
Powered by phpBB® Forum Software © phpBB Group
http://www.phpbb.com/