Rsync Examples over SSH
Rsync is arguably one of the most powerful tools you can have in
your arsenal as a systems administrator or user. It does not matter if
you take care of one system or thousands, rsync can make your life
easier. You can backup files, transfer data from one system to another
or make sure the data in two locations are identical. Rsync can do this
for you without you worrying about what data has changed; it will take
care of the details.
History of rsync...
rsync is a software application for Unix systems which synchronizes files and directories from one location to another while minimizing data transfer using delta encoding when appropriate. An important feature of rsync not found in most similar programs/protocols is that the mirroring takes place with only one transmission in each direction. rsync can copy or display directory contents and copy files, optionally using compression and recursion. The rsync utility uses an algorithm (invented by Australian computer programmer Andrew Tridgell) for efficiently transmitting a structure (such as a file) across a communications link when the receiving computer already has a different version of the same structure.
The recipient splits its copy of the file into fixed-size non-overlapping chunks, of size S, and computes two checksums for each chunk: the MD4 hash, and a weaker 'rolling checksum'. It sends these checksums to the sender.
The sender then compares its rolling checksums with the set sent by the recipient to determine if any matches exist. If they do, it verifies the match by computing the MD4 checksum for the matching block and by comparing it with the MD4 checksum sent by the recipient.
The sender then sends the recipient those parts of its file that didn't match any of the recipient's blocks, along with assembly instructions on how to merge these blocks into the recipient's version to create a file identical to the sender's copy.
If the sender's and recipient's versions of the file have many sections in common, the utility needs to transfer relatively little data to synchronize the files. Wikipedia, rsync
How does rsync help me?
What
this means in layman's terms is rsync will only copy the data that is
different on the two systems. If there is no data on the target machine
then rsync will simple copy the data. But, if there was a copy of the
data and only 1% of the file has changed there is no need to copy the
entire file again. For example, if you had a 1 gig ISO file and only 1%
of the file has changed then rsync will only copy the 1% that has
changed thus saving time and bandwidth. This is incredibly efficient in
time savings and essential if you pay for your Internet data rate.
Lets take a look at some examples...
Using rsync instead of copy (cp)
The
simplest use of rsync is to copy data from on place to another on the
same machine. You can copy entire disks, directories or just files with
rsync. Lets says we have a external usb disk mount at /usbdisk/ that we
are going to copy data from. We want to make sure the copy can resume
even if someone unplugs the usb disk or we loose power to the unit. The
following command will copy the files on the usb disk from the directory
/usbdisk/ to the local disk in /localdisk/.
rsync -av /usbdisk/ /localdisk/
Now, what if you lost connectivity to the usb device and
the sync stopped for some reason? You would then run the same rsync
command a second time. Rsync would look at the files in both locations
and copy the difference. In effect it would continue the copy from the
point the sync was interrupted.
Using rsync over ssh instead of scp
Lets
say we wanted to copy all of the files under our local directory,
"/usbdisk/" and place a copy on the machine "somemachine" under the
remote directory "/backups/". The name of the user on the remote machine
is "calomel". Notice we added the "-z" argument so all data will be
compressed over the network. We could use the following line:
rsync -avz /usbdisk/ calomel@somemachine:/backups/
What if you wanted to delete any files on the _remote_
machine that are no longer on the source machine? We can use the same
command above and simply add the "--del" argument. This will tell rsync
to delete any files from "/backups/" that are no longer on the source
directory, "/usbdisk/".
rsync -avz --del /usbdisk/ calomel@somemachine:/backups/
What if you wanted to not just copy all of the
files under /usbdisk/, but also the /usbdisk/ directory name. Just leave
off the last "/" after the "/usbdisk" directory name. This will tell
rsync to copy the directory name and all files under that directory. On
the remote server you will then see the directory structure
/backups/usbdisk/.
rsync -avz /usbdisk calomel@somemachine:/backups/
You can also rsync multiple files or directories using a
single rsync line. This is useful when coping over the network as you
only need to enter your ssh password once. Here we will copy the source
directories /data1 and /data2 to the remote machine "somemachine" in the
/backups/ directory.
rsync -avz /data1 /data2 calomel@somemachine:/backups/
Here we are pulling multiple files from a remote server
to our local machine. We have to use curly braces to tell rsync we want
only the files "file1", "file2" and "file3." Using this method you only
have to enter your ssh password once to collect all the files.
rsync -avz calomel@somemachine:/remote_machine/{file1,file2,file3} /local/disk/
You can also pull the data from a remote machine
to your local box. To rsync all of the files from the same
"somemachine" to our local box just reverse the target and destination
of the previous example:
rsync -avz calomel@somemachine:/backups/ /usbdisk/
Rsync when remote files or directories contain spaces
Spaces
cause all sorts of problems. To rsync when you have spaces in the files
and/or directories you need to use a single tick to encompass the
entire string and then escape the spaces. The following line shows the
remote directory name "/I Hate Spaces" and the file name "some
File.avi", both contain spaces. We will be rsync'ing the file to our
local directory "/current_dir/".
rsync -av calomel@somemachine:'/I\ Hate\ Spaces/some\ File.avi' /current_dir/
Execute remote shell command to rsync files
It
is important to note rsync can also execute commands on the remote
machine to help you generate a list of files copy. The shell command is
expanded by your remote shell before rsync is called.
The
following line will run the find command on the remote machine in the
video directory and rsync all "avi" files it finds to our machine in the
/download directory.
rsync -avR calomel@somemachine:'`find /data/video -name "*.[avi]"`' /download/
Pull data from a remote machine to local server using ssh
The
following command will pull the data from "remote_machine" in
/stuff/data/ and place it on the local system in
/BACKUP/remote_machine/. The arguments "-avx" will set archive mode (-a)
equivalent to -rlptgoD, be verbose (-v) and will not cross file system
boundaries (-x) like NFS or samba. The timeout command makes sure rsync
will not hang if the remote system is unreachable after 30 seconds. We
will be deleting any files on the target directory
(/BACKUP/remote_machine/) that are _not_ found in the source directory
(data/). If you do not want to allow rsync to delete any files then take
out "--delete-excluded".
The directory structure of the target
machine will look like /BACKUP/remote_machine/data/ and this is
considered a non-relative path option. Notice /stuff/ is not in the
path.
rsync -avx --timeout=30 --delete-excluded backupuser@remote_machine:/stuff/data/ /BACKUP/remote_machine/
If you wanted the target directory structure to be
relative you can add the argument "-R". The directory structure would
then look like /BACKUP/remote_machine/stuff/data/ as the sync path name
starts / on the source machine. The command with "-R" added looks like:
rsync -Ravx --timeout=30 --delete-excluded backupuser@remote_machine:/stuff/data/ /BACKUP/remote_machine/
Incremental backups using rsync
The
following example will make an incremental backup of the directory
/data/working/ and put a copy of any file that changes into a dated
directory in /BACKUP/ . This can be used to keep a daily backup tree of
any changed files and not have to overwrite the previous days files.
Note that this method does need to copy the entire file if it changes as
the new files are made in the directory named under current day.
rsync --backup --backup-dir=`date +%Y.%m.%d` -a /data/working/ /BACKUP/
If you have a file under /BACKUP/ called file1 and that
file has changed on the source machine in /data/working/ then a new
directory will be made. The incremental dir would be named `date
+%Y.%m.%d` or the numerical values for YEAR.MONTH.DAY and put under
/BACKUP/. The changed data "file1" would be put under that directory.
Using rsync through ssh with keys? Validate that only rsync can be run
Lets
say you need to setup an automated rsync between two machines. Lets
also say that you are using ssh with ssh keys to authenticate and you
only want rsync used. This mean you do not want the ssh connection used
to bring up an interactive shell terminal.
The following script can be put anywhere, but we will put it in /usr/local/bin and called it validate_rsync.sh
and make sure it is executable by running "chmod 755
/usr/local/bin/validate_rsync.sh". Then put your ssh key in
~/.ssh/id_dsa.pub from user1@machine1 into the ~/.ssh/authorized_keys
file on user2@machine2. Now, at the beginning of the ssh key in
~/.ssh/authorized_keys on user2@machine2 add the string, command="/usr/local/bin/validate_rsync.sh"
. This will mean that any machine using the ssh key from user1@machine1
will run the script validate_rsync.sh and will only be allow to execute
rsync.
## Calomel.org -- validate_rsync.sh # #!/bin/sh case "$SSH_ORIGINAL_COMMAND" in *\&*) echo "Rejected" ;; *\(*) echo "Rejected" ;; *\{*) echo "Rejected" ;; *\;*) echo "Rejected" ;; *\>*) echo "Rejected" ;; *\`*) echo "Rejected" ;; *\|*) echo "Rejected" ;; rsync\ --server*) $SSH_ORIGINAL_COMMAND ;; *) echo "Rejected" ;; esac
Questions?
How can I change the encryption method of ssh which rsync is using?
Using the "--rsh" directive you can specify any encryption algorithm you want to. Lets use the "arcfour" encryption as it is the least CPU intensive and works well for very slow systems like 386's and Sun's SPARC 20's.rsync -avx --timeout=30 --delete-excluded --rsh="ssh -c arcfour -l backupuser" remote_machine:/stuff/data/ /BACKUP/remote_machine/
What should I be careful with when using Rsync?
- Be careful with the --delete command. If your using a source dir that is not mounted (nfs,cifs,etc) but the mount for the dir is still there then you will sync your blank dir. All remote files will be deleted.
- Be careful with slashes after the dir names on the source. A slash after the dir name compared to no slash after the dir name will do 2 totally different things.
- Running out of memory with the older version of Rsync when doing a huge amount of files. Use the newer version due to its ability to do incremental file lists.
الرائع تعتبر من افضل الشركات التي تقدم لكم خدمات التنظيف الكاملة و ايضا خدمات الترميم و نقل الاثاث حيث اننا نعتبر افضل شركة تنظيف بالمملكة العربية السعوديةافضل شركة تنظيف بالرياض
ReplyDeleteاذا اردت شركة تنظيف بالرياض فبالتالي سوف تقوم بالبحث علي شبكات الانترنت عن المراد و سوف يظهر امامك علي الشاشة الكثير و الكثير من ارقام شركات التنظيف الموجودة في الرياض فشركتنا تعتبر الافضل من بين تلك الشركات بالكامل
شركة الرائع
شركة الرائع تقدم لكم المزيد من الخدمات المنزلية حيث اننا نعتبر افضل شركة تنظيف منازل بالرياض نقوم بتنظيف المنازل الكبيرة و الصغيرة و المفروشة و الجديدة من كافة الوساخ الموجودة فيها - افضل شركة تنظيف منازل بالرياض
اما عن خدمة نقل الاثاث فنحن شركة الرائع نعتبر افضل شركة نقل اثاث من الرياض الي الدمام نقوم بنقل الاثاث من الرياض الي كافة انحاء المملكة العربية السعودية
شركة نقل اثاث من الرياض الي الدمام
فاذا كنت تود الذهاب الي اي مكان و لكن تحتاج الي من يقوم بنقل الاثاث الخاص بك فعليك ان تتواصل معنا علي شركة الرائع عبر الهاتف 0500440185
تنظيف منازل بالرياض
اما عن خدمة الترميم فتعتبر افضل الخدمات المنزلية حيث ان هناك الكثير من المنازل القديمة و المفروشة تحتاج الي التجديد فنحن نعتبرافضل شركة ترميم منازل بالرياضنقوم بترميم الفيلا و الشقق و المنازل و البيوت
ebcue.org
ebcue.org
ebcue.org
ebcue.org
المنارة افضل شركة تقدم لكم جميع الخدمات المنزلية حيث اننا نعتبر الشركة الاولي في تخزين الاثاثافضل شركة تخزين عفش بالرياض نقوم بتخزين العفش في مستودعات تخزين علي اعلي مستوي ممكن كما اننا نعتبر الشركة الولي في خدمة تنظيف الاثاث اتصل علي شركة الطيب افضل شركة تخزين اثاث بالرياض الافضل هي شركة المنارة المتدربة علي التقنيات الحديثة و العاملة علي تحقيق افضل الاهداف في تلك المجال
ReplyDelete