Syncing very large number of files to another server

  • humbletech99
  • Proficient
  • Proficient
  • User avatar
  • Posts: 300

Post 3+ Months Ago

I've got a folder with a very large number of files and subdirectories (it's full of maildir folders for my company actually) which has files that rank in the millions. I need to replicate this to another server as a backup.

The number of files is not the only consideration, the volume is quite significant, even over gigabit.

I've tried using rsync to try to avoid re-copying a large volume of data by only taking the differences across to the backup server, but the extremely large number of files tends to hurt rsync and take forever while it's building up file lists.

Does anybody have a better idea or another tool for replicating a directory structure with both large volume and a large number of files?
  • Anonymous
  • Bot
  • No Avatar
  • Posts: ?
  • Loc: Ozzuland
  • Status: Online

Post 3+ Months Ago

  • Don2007
  • Web Master
  • Web Master
  • Don2007
  • Posts: 4924
  • Loc: NY

Post 3+ Months Ago

http://www.cis.upenn.edu/%7Ebcpierce/unison/

I never used it. It's not a recommendation.
  • kc0tma
  • o|||||||o
  • Web Master
  • User avatar
  • Posts: 3318
  • Loc: Trout Creek, MT

Post 3+ Months Ago

I would say for something that huge and important, you might want to use tar and back it up to tape (if you've got a tape drive) and then move it to the other server. That way you also have a hard copy backup if something bad happens.

Post Information

  • Total Posts in this topic: 3 posts
  • Users browsing this forum: No registered users and 79 guests
  • You cannot post new topics in this forum
  • You cannot reply to topics in this forum
  • You cannot edit your posts in this forum
  • You cannot delete your posts in this forum
  • You cannot post attachments in this forum
 
 

© 1998-2014. Ozzu® is a registered trademark of Unmelted, LLC.