rsync

Most basic transfer options

Example: --no-devices, --no-perms have different defaults in various circumstances
Example: --no-whole-file, --no-blocking-io, --no-dirs.

The order of the options is important. The result of
--no--recursive --archive is that --recursive would end up being turned on, the opposite of
--archive --no--recursive.

rsync --list-only uname@Real-World-Systems.com:*.txt # list all txt files on that host's root directory
-rw-r----- 160 2017/05/18 20:31:30 apacheNotes.txt
-rw-r--r-- 1490 2018/10/22 16:57:32 pepper-181022.txt
-rw-r----- 3679 2017/12/29 19:04:47 smacproPart4.txt

rsync --list-only /Volumes/big/*.MOV # list all the MOV files on that volume/dirctory
-rw-r--r-- 308842544 2018/05/16 07:56:53 CamperGrandTour_2064.MOV
-rw-r--r-- 12014106 2018/06/23 15:40:36 DoorClosingWithGap188.MOV
-rw-r--r-- 185538306 2017/02/18 13:29:27 Hummers_0655.MOV
-rw-r--r-- 19434966 2019/01/19 09:28:38 IMG_0332.MOV

Filenames are displayed in --out-format of %n%L by default, ( name and if it is a link, where it points).

No mention is made if a file had its attributes changed.

rsync works silently by default.

.d        var/
.f        var/logPrune.sh 

Each file, dir, etc. that gets updated in a significant way (a transferred file, a recreated symlink/device, or a touched directory) will be output.
(e.g. With --itemize-changes the logging of names includes any item changed.

The out-formated string is output prior to a file's transfer.
With transfer-statistic, logging is done at the end of the file's transfer. When this late logging is in effect and --progress is specified, the name of the file being transferred is output prior to its progress information followed by the out-format output.

rsync --archive --verbose --rsync-path="rsync --log-file=rlog" src/ dest/

Number of files: 88417
Number of files transferred: 0
Total file size: 5008416760 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 2710302
File list generation time: 1.156 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 20
Total bytes received: 2710322

sent 20 bytes  received 2710322 bytes  285299.16 bytes/sec
total size is 5008416760  speedup is 1847.89

782448  63%  110.64kB/s    0:00:04

Statistics can be misleading with the delta-transfer algorithm. For example, if the file is a log previously sent the reported rate will drop dramatically when the receiver gets to the new data, and the transfer will take much longer to finish than predicted.

When a file transfer finishes, the progress line is replaced with a summary line like :

1238099 100%  146.38kB/s    0:00:08  (xfer#5, to-check=169/396)

Does not affect the copying of symlinks or other special files.

transfer handling

Important for transfers which need to be restarted.

Useful when

• There is insufficient free space on the destination file system to contain the existing file and the new temp version.
• transfers with large files that may to be interrupted.
• transfers include large files with block-based changes or appended data,
• systems are disk bound, not network bound.

Effects:
1. During the transfer, file will be inconsistent
2. source files being updated will not be transfered.
3. destination files being read cannot be updated.
4. If the transfer is interrupted or fails, file may be left in an inconsistent state
5. A file that does not have write permissions can not be updated, and
6. The efficiency of the delta-transfer algorithm may be reduced if some data in the destination file is overwritten before it can be copied to a position later in the file
   Combine with --backup, since rsync will use the backup file as the basis file for the transfer to prevent this.

WARNING: Do not use this to update files being accessed by others.

Implies --partial (an interrupted transfer does not delete the file),
Conflicts with --partial-dir and --delay-updates.

If partdir is an absolute path it is consideered an existing working directory.

Do not use an absolute path unless
(1) there are no files in the transfer having the same name (since all the updated files will be put into a single directory )
and
(2) there are no mount points in the hierarchy (the delayed updates will fail if they can't be renamed into place).

If partdir is a relative path, it will be made and excluded preventing sending partdir files on the sending side, and prevent untimely deletion of partdir files on the receiving side.
An example: the above --partial-dir option would add the equivalent of -f '-p .rsync-partial/' at the end of any other filter rules.

Add an exclude/hide/protect rule for the partdir to

• override default.

Conflicts with --inplace and --append.

Requires enough free disk space to hold an additional copy of all the updated files.

See the "atomic-rsync" perl script in the "support" subdir for an update algorithm that is more atomic (it uses --link-dest and a parallel hierarchy of files).

> rsync -av /foo/bar/baz.c remote:/tmp/ 

> rsync -avR /foo/bar/baz.c remote:/tmp/ 

These extra path elements are called "implied directories" ("foo" and the "foo/bar" directories in the above example).
rsync sends these implied directories as real directories in the file list, even if a path element is really a symlink on the sending side. This prevents unexpected behaviors when copying the full path of a file that you didn't realize had a symlink in its path.
To duplicate a server-side symlink, include both the symlink via its path, and referent directory via its real path.

To limit the amount of path information sent as implied directories for each path, insert a dot and a slash into the source path.

>rsync -avR /foo/./bar/baz.c remote:/tmp/

For instance, if a command-line arg or a files-from entry told rsync to transfer the file "path/foo/file", the directories "path" and "path/foo" are implied when --relative is used. If "path/foo" is a symlink to "bar" on the destination system, the receiving rsync would ordinarily delete "path/foo", recreate it as a directory, and receive the file into the new directory. With --no-implied-dirs, the receiving rsync updates "path/foo/file" using the existing path elements, which means that the file ends up being created in "path/bar".
Another way to accomplish this link preservation is to use the --keep-dirlinks option (which will also affect symlinks to directories in the rest of the transfer).

without --backup-dir,

1. --omit-dir-times is implied, and
2. if --delete is in effect (without --delete-excluded), rsync will add a "protect" filter-rule for the backup suffix to the end of all existing excludes (e.g. -f "P *~").

--dirs is implied by --files-from or --list-only (including an implied --list-only usage) if wasn't specified (so that directories are seen in the listing).
Specify --no-dirs to turn this off.

Link handling

Without this , if the sending side has replaced a directory with a symlink to a directory, the receiving side will delete anything that is in the way of the new symlink, including a directory hierarchy (as long as --force or --delete is in effect).

See --keep-dirlinks for an analogous option for the receiving side.

For example, transfer a directory "foo" that contains a file "file", but "foo" is a symlink to directory "bar" on the receiver.
Without --keep-dirlinks, the receiver deletes symlink "foo", recreates it as a directory, and receives the file into the new directory.
With --keep-dirlinks, the receiver keeps the symlink and "file" ends up in "bar". caution: when using --keep-dirlinks, you must trust all the symlinks in the copy!
If it is possible for an untrusted user to create their own symlink to any directory, the user could then (on a subsequent copy) replace the symlink with a real directory and affect the content of whatever directory the symlink references.
For backup copies, use a bind mount instead of a symlink to modify your receiving hierarchy.

See also --copy-dirlinks for an analogous option for the sending side.

When updating a non-empty destination, ensures that files that are hard-linked together on the source are hard-linked together on the destination.
Does NOT break existing hard links on the destination that do not exist between the source files.
If one or more extra-linked files have content changes, they will become unlinked when updated (if not using --inplace ).

Rsync can only detect hard links between files that are in the transfer set. If rsync updates a file that has extra hard-link connections to files outside the transfer, that linkage will be broken.
Using --inplace to avoid this breakage, there may be unintended changes due to lingering hard links (see --inplace ).

If incremental recursion is active rsync may transfer a missing hard-linked file before it finds that another link for that contents exists elsewhere in the hierarchy.
This does not affect the accuracy of the transfer, just its efficiency, To avoid this disable incremental recursion with --no-inc-recursive

Attributes, Meta-data, i.e. file permissions, Access Control Lists, owner, group …

With --no--perms:

An item that should only apply to a directory is prefixed with D', or to a file if prefixed with 'F'.
For example:

--chmod=Dg+s,ug+w,Fo-w,+X

See --perms and --executability for how the resulting permission value can be applied to the files in the transfer.

only if the receiver is super-user (see --super and --fake-super ).
Caution: The preservation of ownership will associate matching names but if it falls back to using the ID number (see --numeric-ids it will surely be incorrect!!.

The preservation of group information will associate matching names by default, but may fall back to using the ID number

 rsync -av --rsync-path="rsync --fake-super" /src/ host:/dest/

overridden by both --super and --no-super.

See fake super in rsyncd.conf.

If repeated, rsync omits all mount-point directories.
Otherwise, it includes an empty directory at each mount-point it encounters (using the attributes of the mounted directory because those of the underlying mount-point directory are inaccessible).

If collapse symlinks (via --copy-links or --copy-unsafe-links), a symlink to a directory on another device is treated like a mount-point. Symlinks to non-directories are unaffected by this option.

This is a transfer rule, not an exclude, it doesn't affect the data that goes into the file-lists, and thus it doesn't affect deletions. It just limits the files that the receiver requests to be transferred.

Deletions will also occur when --dirs is enabled for directories whose contents are being copied.

Including --dry-run shows files that would be deleted without deleting them.

--delete-during(default)
--delete-before
--delete-after
--delete-delay May be a better choice than after)
--prune-empty-dirs, and --delay-updates.
(use --del --delete-during to request this deletion mode explicitly).
Incremental recursion can be disabled using --no-inc-recursive.

If the sending side detects I/O errors, deletion of any files will be disabled,
useful with NFS errors on the sending side)

May be combined with one of the --delete-when as well as --delete-excluded.
See --delete-delay and --delete-after.

Helpful when the destination filesystem is low on space.
Introduces a delay before the start of the transfer which might cause the transfer to timeout
Forces non-incremental recursion that requires a scan of the files for the transfer into memory at once
(see --recursive).

See the FILTER RULES section for a way to make individual exclusions behave this way on the receiver, and for a way to protect files from --delete-excluded

--max-delete=0 warning without removing

Does not preserve hardlinks, ---hard-links.

If -archive follows --compress it will reenable the "--no-" options

The side-effects of the --files-from option are NOT positional, as it affects the default state of several options and slightly changes the meaning of --archive.

• KB (1,000) : K KiB kilibyte (1024):
• MB ( 1,000,000) :M MiB megibyte (1024*1024)
• GB (1,000 MB) :G GiB gibibyte (1024*1024*1024)
• +1 or -1 to offset by one byte.

Examples:
--max-size=1.5mb-1 is 1,499,999 bytes
--max-size=2g+1 is 2,147,483,649 bytes.

1. The exclude list initialized with:

   RCS  SCCS  CVS  CVS.adm  RCSLOG  cvslog.*  tags  TAGS  .make.state  .nse_depinfo  *~  #*  .#*  ,*  _$*  *$  *.old  *.bak  *.BAK  *.orig  *.rej  .del-*  *.a  *.olb  *.o  *.obj  *.so  *.exe  *.Z  *.elc  *.ln  core  .svn/  .git/  .bzr/

2. Files in $CVSIGNORE (delimited by whitespace). See CVS
3. Files listed in $HOME/.cvsignore
4. Files in a directory containing a .cvsignore
5. Files matching a pattern in .cvsignore (delimited by whitespace).

When combining --filter rules with -C CVS excludes are appended regardless of where the -C was placed on the commandline.
This makes them a lower priority than any rules specified explicitly.
To control where these get inserted into filter rules, omit -C as a command-line option and use a combination of --filter=:C and --filter=-C (either on the command-line or by putting the ":C" and "-C" rules into a filter file with other rules). :C turns on the per-directory scanning for the .cvsignore file. -C does a one-time import of the CVS excludes.

if used with --iconv, the args related to the remote side will also be translated to the remote character-set. The translation happens before wild-cards are expanded. See --files-from .

Useful when there is not enough free space to hold a copy of the largest file in the destination filesystem. This requires rsync to it copy the file into place when the file is transfered.

Combined with --delay-updates , which will ensure that all copied files get put into subdirectories in the destination hierarchy, awaiting the end of the transfer.
If there is not enough room to duplicate all the arriving files on the destination filesystem, use --partial-dir with a relative path; because it is OK to stash off a copy of a single file in a subdir in the destination hierarchy, rsync will use the partial-dir as a staging area to bring over the copied file, and then rename it into place from there. (Specifying a --partial-dir with an absolute path does not have this side-effect.)

Use of --delete might get rid of any potential fuzzy-match files, so either use --delete-after or specify some filename exclusions to prevent this.

multiple --compare-dest directories may be provided, which will cause rsync to search the list in the order specified for an exact match. If a match is found that differs only in attributes, a local copy is made and the attributes updated. If a match is not found, a basis file from one of the DIRs will be selected to try to speed up the transfer.

If DIR is a relative path, it is relative to the destination directory. See also --copy-dest and --link-dest.

Multiple --copy-dest directories may be provided, which will cause rsync to search the list in the order specified for an unchanged file. If a match is not found, a basis file from one of the DIRs will be selected to try to speed up the transfer.

If DIR is a relative path, it is relative to the destination directory. See also --compare-dest and --link-dest.

rsync -av --link-dest=$PWD/prior_dir host:src_dir/ new_dir/

multiple --link-dest directories may be provided, which will cause rsync to search the list in the order specified for an exact match. If a match is found that differs only in attributes, a local copy is made and the attributes updated. If a match is not found, a basis file from one of the DIR will be selected to try to speed up the transfer.

works best when copying into an empty destination hierarchy, as rsync treats existing files as definitive (so it never looks in the link-dest dirs when a destination file already exists), and as malleable (so it might change the attributes of a destination file, which affects all the hardlinked versions).

combine this option with --ignore-times, rsync will not link any files together because it only links identical files together as a substitute for transferring the file, never as an additional check after the file is updated.

If DIR is a relative path, it is relative to the destination directory. See also --compare-dest and --copy-dest.

With [user@]host::module/path, the remote shell to run an rsync daemon on the remote host, and all data will be transmitted through that remote shell connection, not a socket connection to a running rsync daemon on the remote host.
See using rsync-daemon features via a remote-shell connection

Command-line arguments are permitted in rmsh provided that is presented to rsync as a single argument.
Use spaces (not tabs or other whitespace) to separate rmsh into the command and arguments.
Use apostrophe and/or quotes to preserve spaces in an argument (not backslashes).
Doubling an apostrophe inside a apostrophe'd string results in apostrophe ( likewise for quotes ) (attention to which quotes the shell is parsing and which quotes rsync is parsing).

Examples:

 -e 'ssh -p 2234'
 -e 'ssh -o "ProxyCommand nohup ssh firewall nc -w1 %h %p"' 

$RSYNC_RSH specifies remote shell program, which accepts the same range of values as -e.
See --blocking-io which is affected by this .

                  rsync -avR --rsync-path="cd /a/b && rsync" host:c/d /e/  

Simple character-class matching is supported: consisting of a list of letters inside the square brackets (e.g. no special classes, such as "[:alpha:]", are supported).
asterisk (*) and question-mark (?) have no special meaning.

Example specifies 6 suffixes to skip (since 1 of the 5 rules matches 2 suffixes):

--skip-compress=gz/jpg/mp[34]/7z/bz2

rsyncd will add skipped suffixes

For shell transfers use Public key authentication.

By default username and groupname are used to establish ownership.
If a user or group on the source system does not match the destination, the numeric ID from the source system is used.

If on the source system, user jones is user ID 507 and on destination is user ID 509 and bsmith is user ID 507, files on the the destination will belong to bsmith !

See "use chroot" in rsyncd.conf on how it affects rsync's ability to look up the names of the users and groups The special user and groups ID 0 are never mapped names even if the --numeric-ids option is not specified.
0 Only applicable when transfering files with various owners.

the use of transfer rules, such as --min-size , does not affect what goes into the file list, and thus does not leave directories empty, even if none of the files in a directory match the transfer rule.

Because the file-list is being pruned, affects what directories get deleted when a delete is active.
Excluded files and directories can prevent existing items from being deleted due to an exclude both hiding source files and protecting destination files. See perishable filter-rule for how to avoid this.

Prevent the pruning of certain empty directories from the file-list by using a global "protect" filter.
For instance, this option would ensure that the directory "emptydir" was kept in the file-list:

       --filter 'protect emptydir/'

example that copies all .pdf files in a hierarchy, only creating the necessary destination directories to hold the .pdf files. Superfluous files and directories in the destination are removed (note the hide filter of non-directories being used instead of an exclude):

 rsync -avm --del --include='*.pdf' -f 'hide,! */' src/ dest 

Write the batch directly to portable media: if this media fills before the end of the transfer, apply that partial transfer to the destination and repeat the process to get the rest of the changes (this will use a partially updated destination system while the multi-update cycle is happening).

you only save bandwidth when pushing changes to a remote system because this allows the batched data to be diverted from the sender into the batch file without having to flow over the wire to the receiver (when pulling, the sender is remote, and thus can't write the batch).

Output a literal backslash (\) and a hash (#), and 3 octal digits. For example, a newline would output as "\#012". A literal backslash that is in a filename is not escaped unless it is followed by a hash and 3 digits (0-9).

   rsync -a --info=progress2 src/ dest/
    rsync -avv --info=stats2,misc1,flist0 src/ dest/ 

some debug messages will only be output when --msgs2stderr is specified, especially those pertaining to I/O and buffer debugging.

If standard input is a socket then rsync will assume that it is being run via inetd, otherwise
it will detach from the current terminal and become a background daemon, reading rsyncd.conf on each connect made by a client. config

"exported" directories (modules) must be specified in rsyncd.conf

For a list of charset names local iconv library supports, run "iconv --list".

The following list contain all the coded character sets:
 8859_1, 8859_2, 8859_3, 8859_4, 8859_5, 8859_6, 8859_7, 8859_8, 8859_9, 
 10646-1:1993, 10646-1:1993/UCS4,
  ANSI_X3.4-1968, ANSI_X3.4-1986, ANSI_X3.4, ANSI_X3.110-1983, ANSI_X3.110, 
  ARABIC, ARABIC7, ARMSCII-8, ASCII, 
…
 US-ASCII, US, UTF-7, UTF-8, UTF-16, UTF-16BE, UTF-16LE, UTF-32, UTF-32BE, UTF-32LE, 
 UTF7, UTF8, …

With --protect-args filenames for transfer on the command-line are translated. See --files-from .

No conversion of names in filters (including include/exclude files) is done!.
Specify extra include/exclude rules if there are filename differences on the two sides.

With --iconv the daemon uses the charset specified in its "charset" configuration

 echo "++++++++++++++++++" >> rsync.err  # initalize rsync.err so tail is ok
 tail -f rsync.err &     # continue to show errors as rsync runs †
 rsync --progress --archive --hard-links --compress --update     --stats               \
                  --log-file=`date +rsync_%y%m%d.log` --log-format='%t %o %B %l %M %f' \
       uname@webSite.com:public_html backupDir   2>> rsync.err 

Expect to see

uname@webSite.com password:

Use ^Z to push the job into the backround.

The progress of rsync will be displayed.

2011/10/26 21:14:30 recv rwx------ 20480 2011/10/26-21:14:10 jupiter/downloads
2011/10/26 21:14:30 recv rw------- 4520453 2011/09/23-13:00:56 jupiter/downloads/ACPIspec40a.pdf
     4520453 100%    2.70MB/s    0:00:01 (xfer#1, to-check=47541/88416)
2011/10/26 21:14:31 recv rw-r--r-- 1364852 2011/10/18-10:18:34 jupiter/downloads/nuketheappstore.zip
     1364852 100%    1.42MB/s    0:00:00 (xfer#2, to-check=47246/88416)
…

Number of files: 88416
Number of files transferred: 2
Total file size: 5004066643 bytes
Total transferred file size: 5885305 bytes
Literal data: 5885305 bytes
Matched data: 0 bytes
File list size: 2710269
File list generation time: 1.120 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 70
Total bytes received: 7497967

sent 70 bytes  received 7497967 bytes  714098.76 bytes/sec
total size is 5004066643  speedup is 667.38 

 fg to bring the tail process to the foreground.

 then ^C to interrupt the tail.

rsync --daemon --help    
rsync  version 3.0.7  protocol version 30
Copyright (C) 1996-2009 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 32-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace, append, no ACLs, xattrs, iconv, no symtimes

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU General Public Licence for details.

Usage: rsync daemon [OPTION]…

     --address=ADDRESS       bind to the specified address (like when  the server has multiple addresses)
     --bwlimit=KBPS          limit I/O bandwidth; KBytes per second
     --config=FILE           default: /etc/rsyncd.conf 
     --no-detach             do not detach from the parent
     --port=PORT             port number (Instead of 873 check firewall)
     --log-file=FILE         
     --log-file-format=FMT   
     --sockopts=OPTIONS      custom TCP options like:
 -v, --verbose               increase verbosity
 -4, --ipv4                  prefer 
 -6, --ipv6