LinHES Forums
http://forums.linhes.org/

Synchronize two directories
http://forums.linhes.org/viewtopic.php?f=14&t=22285
Page 1 of 1

Author:  gatorback [ Sat Oct 15, 2011 10:38 am ]
Post subject:  Synchronize two directories

A source and a target NTFS drive and they are mounted to a LinHES box. The drives are SAMBA accessable. Archlinux automagically mounted the USB HDDs to : /media/DEVICE_VOLUMENAME.

I tried to recursively copy via UNIX CLI , however, only 80% of the files came through. I imagine this is the fastest way to copy because it is USB <=> SAMBA <=> USB.

Identifying the missing 20% was assigned to WinSCP's synchronication tool. Samba allows the source drive to be a 'local' drive.

This process is slow because the data is circulating from LinHES through WinSCP along with encryption overhead. If encryption was disabled or FTP was used, then that might speed it up: I dislike the idea of adding an FTP server just for this exercise.

This seems to be an effective, but kludgey way of doing things. 'There's got to be something better than this!' Suggestions are appreciated. It would preferable to use the existing resources on LinHES and avoid installing additional services \ utilities on LinHES.

Author:  tjc [ Sat Oct 15, 2011 2:25 pm ]
Post subject: 

My first choice tool for any kind of disk to disk directory cloning like this is usually rsync. I've used it to to copy stuff to locally mounted MS-DOS formatted USB drives before, although never to a remote share that I can remember, but it should work.

Author:  graysky [ Sat Oct 15, 2011 4:14 pm ]
Post subject: 

+1 for rsync. It will do what you want so long as you have the shares mounted.

Code:
rsync -avxu --progress --delete-after --exclude "RECYCLER/" --exclude "System Volume Information/" /media/data/ /media/aa_backup/drive_data/

Author:  gatorback [ Sat Oct 15, 2011 7:12 pm ]
Post subject: 

Thanks gentlemen. As usual: excellent contributions.

After reading about the -a (archive) option, I decided to recopy 400 GB tonight and will check the results. Speed is really nice: 14.16MB/s (Megabits / Sec?) => 1.06 GB / Min.

Author:  graysky [ Sun Oct 16, 2011 7:49 am ]
Post subject: 

gatorback wrote:
Speed is really nice: 14.16MB/s (Megabits / Sec?) => 1.06 GB / Min.


MB/s = megabytes per second
Mb/s = megabits per second

So, you're doing about 0.83 GB/min. If it were in Mb you'd be getting 0.10 GB/min. Anyway, see, http://en.wikipedia.org/wiki/Data_rate_units

I'm guessing you're on a 100baseT(X) backbone for your LAN or is that wireless? If your LAN is wired and has GLAN potential in your NICs (any modern PC does), a $45 switch can upgrade your to GigaLAN speeds. Assuming the HDDs are >=SATA2 and assuming the PCs are relative modern, you increase the bottleneck from the LAN to the HDD's data read/write. I usually get 80-90 MB/s on drive-to-drive xfers (no LAN) and the same on PC-to-PC xfers using the LAN.

Author:  gatorback [ Sun Oct 16, 2011 9:40 am ]
Post subject: 

@graysky, thanks for clearing up the data rate convention.

The data rate above is within the context of two USB devices connected to the same MythTV box (no networking, which is what I wanted).

For the casual reader, I have added logging of transferred files and timed the transfer:

Code:
date > TicToc_label.txt; rsync -avxu --progress --delete-after --exclude "RECYCLER/" --exclude "System Volume Information/" /media/SOURCE /media/TARGET   | tee ~/rsync_log_label.txt; date >> TicToc_label.txt

Author:  tjc [ Mon Oct 17, 2011 8:11 pm ]
Post subject: 

graysky wrote:
a $45 switch can upgrade your to GigaLAN speeds.

Heck $40 will get you a commercial grade managed switch. ;-)

http://www.roc-noc.com/mikrotik/routerb ... 250gs.html

Or an unmanaged Netgear switch.

http://www.newegg.com/Product/Product.a ... 6833122128

Page 1 of 1 All times are UTC - 6 hours
Powered by phpBB® Forum Software © phpBB Group
http://www.phpbb.com/