rsync

Severely tersified by DGG, original Here

High performance, file copying tool, used locally or to/from another host, for backup and mirroring. Can perform most ftp tasks.

Local side is refered to as the "client" and the remote as the "server".
rsyncd, the (optional) daemon is always a server. A remote server can be either a daemon or a remote-shell spawned process.

Local: rsync [option …] src… [dest]

Access via remote shell:
Pull: rsync [option …] [user@]host:src… [dest]
Push: rsync [option …] src… [user@]host:dest

Access via rsync daemon:
Pull: rsync [option …] [user@]host::src… [dest]
rsync [option …] rsync://[user@]host[:port]/src… [dest]
Push: rsync [option …] src… [user@]host::dest
rsync [option …] src… rsync://[user@]host[:port]/dest

Copies locally If neither the source or destination path are a host,

rsync --progress --recursive user@fromHost:directory/sub . --list-only

Most basic transfer options:

--list-only don't do anything (can be at the end on the command alowing easy removal.)
--progress displays activity
--log-file=0rsync.log
--archive preserve basic attributes (permissions, owner, type symbolic links ]).
--xattrs preserve eXtended attributes
--acls preserve Access Control Lists
--hard-links preserve them, see symbolic links
--compress in flight data ( no effect on destination)
--update if existing files are current, skip transferring them again
--exclude-from=rsync-excludes patterns (one per line).

Is your shell clean? abort message.

Examples

There is a lot to know about rsync ( and I only know a little), here's what you HAVE TO know.

If a single source arg is specified without a destination, the files are listed in an output format similar to ls -l.


The "=" is optional; whitespace can be used.

--no-option Turn off implied options
For example --archive is a synonym for -recursive --links etc. follow it with --no-links to not recreate links.

Example: --no-devices, --no-perms have different defaults in various circumstances
Example: --no-whole-file, --no-blocking-io, --no-dirs.

The order of the options is important. The result of
--no--recursive --archive is that --recursive would end up being turned on, the opposite of
--archive --no--recursive.

--list-only Useful when:
  • a copy command that includes a destination is be forced to only list. This displays the files to be copied
  • to specify more than one source arg (note: be sure to include the destination).
  • a source with a wild-card is expanded by the shell into multiple args,
rsync --archive --verbose --list-only *.sh
-n
--dry-run
Doesn't perform any transfers but produces a similar report as a real run.
-a
--archive
synonym for -recursive --links --perms --times --group --owner --devices --specials
( ACL and eXtended attributes may not be preserved(?) ed)
When --files-from is specified --recursive is not implied.

Does not preserve hardlinks, ---hard-links.

If -archive follows --compress it will reenable the "--no-" options

The side-effects of the --files-from option are NOT positional, as it affects the default state of several options and slightly changes the meaning of --archive.

Options that effect how files are determined to be the same and don't need to be transfered.
Different type of object (for example: file vs directory) always requires an update,
Default: skip files with the same size and modification timestamp.
-I
--ignore-times
 
--size-only Useful when a system which does not preserve timestamps well.
--modify-window=mwseconds timestamps considered equal if they differ by no more than the mwseconds.
Default 0 (exact match).
MS Windows FAT filesystem represents times with a 2-second resolution --modify-window=1 allows times to differ by up to 1 second.
-u
--update
If the destination has a newer modification time it is not overwritten.

Does not affect the copying of symlinks or other special files.

-c
--checksum
compare a MD5 128-bit checksums for files that have matching sizes.
Performance degradation affecting both sides. Requires reading all the data in the files with matching size in addition to reading if the file is transfered.
 
--append update by appending data to the file when data exists on the destination and the source begins with the identical data.
If the size of the destination is the same or longer, the file is not overwirtten.

This does not affect updating a file's attributes (e.g. permissions, ownership, etc.) when the file does not need to be transferred, nor does it affect the updating of any non-regular files.
Implies --inplace

--append-verify Like --append but the existing data on the receiving side is included in the full-file checksum verification, which will cause a file to be resent if the final verification fails (rsync uses a normal, non-appending --inplace transfer for the resend).
 
-v
--verbose
Increases the amount of information displayed during the transfer.
By default, rsync works silently.
A single -verbose gives information about what files are being transferred and a brief summary at the end.
-verbose -verbose adds information on what files are being skipped .

Following only seen with --verbose --verbose
xxx is over max-size
[sender] hiding file email/Trash because of pattern Trash*.

Filenames are displayed in --out-format of %n%L by default, the name and if it is a link, where it points.
At the first level verbosity no mention is made if a file had its attributes changed..
More than two are used for debugging

-q
--quiet
decreases the amount of information reported.
Useful when invoking from cron.
Use with --verbose and/or --itemize-changes to see what a command is going to do.
options affecting report
-i
--itemize-changes
Increases verbosity. Default format --out-format='%i %n%L'.
Repeating the option causes unchanged files to be output,.

.d        var/
.f        var/logPrune.sh 
%i string YXcstpoguax, where
Y . item is not being updated (attributes may be modified).
< file is being sent.
> file is being received
c change/creation is occurring for the item
(such as the creation of a directory or the changing of a symlink, etc.).
h hard link to another item (requires --hard-links).
* itemized-output area contains a message (example: "deleting").
X file-type
f file
d directory
L symlink
D device
S special file (examples: named sockets and fifos).

The other letters in the the associated attribute being updated
. no change.
+ new item ++++++++++
space identical item
?unknown attribute
reasons for transfer
c regular file has a different checksum (requires --checksum) or
symlink, device, or special file has a changed value.
s size of a regular file
t time of last modification (requires --times).
Ttime of last modification will be set to the transfer time
if a file/symlink/device is updated without --times or
   a symlink is changed and the receiver can't set its time.
p permissions are different (requires --perms).
o owner is different (requires --owner and super-user privileges).
g group is different (requires --group and the authority to set the group).
u future use.
a ACL (Access Control List) information changed.
x extended attribute information changed.
When deleting files, %i" will output the string "*deleting" if approiate.
--out-format=format string of field specifiers.
Default with --verbose is %n%L (name and, if the item is a link, where it points). See "log format" in rsyncd.conf .

Each file, dir, etc. that gets updated in a significant way (a transferred file, a recreated symlink/device, or a touched directory) will be output.
(e.g. With --itemize-changes the logging of names includes any item changed.

The out-format string is output prior to a file's transfer.
If a transfer-statistic option is requested, logging is done at the end of the file's transfer. When this late logging is in effect and --progress is specified, the name of the file being transferred is output prior to its progress information followed by the out-format output.

--log-file=file For a non-daemon transfer.
If specified as a client option, transfer logging will be enabled with a default format of %i %n%L. Example: Remote side logs what is happening:
rsync --archive --verbose --rsync-path="rsync --log-file=/tmp/rlog" src/ dest/
Useful to diagnose why a connection is closing unexpectedly.
--log-file-format=format With an empty string, updated files will not be mentioned in the log file.
enables "transfer logging" unless the string is empty, in which case transfer logging is turned off. See log format in rsyncd.conf .
Default with --log-file %i %n%L i.e. name link.
-h
--human-readable
K, M, or G. units are K (1000), M (1000*1000), and G (1000*1000*1000);
-h -h , units are powers of 1024 .
--stats output statistics
Number of files: 88417
Number of files transferred: 0
Total file size: 5008416760 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 2710302
File list generation time: 1.156 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 20
Total bytes received: 2710322

sent 20 bytes  received 2710322 bytes  285299.16 bytes/sec
total size is 5008416760  speedup is 1847.89
--progress Implies --verbose
While transferring a regular file, updates a progress line like :
782448  63%  110.64kB/s    0:00:04
The transfer might finish in :04 seconds

Statistics can be misleading with the delta-transfer algorithm. For example, if the file is a log previously sent the reported rate will drop dramatically when the receiver gets to the new data, and the transfer will take much longer to finish than predicted.

When a file transfer finishes, the progress line is replaced with a summary line like :

1238099 100%  146.38kB/s    0:00:08  (xfer#5, to-check=169/396)
This was the 5th regular file transfered, 169 more files for the receiver to check out of the 396 files --delete-before, --delete-after, --prune-empty-dirs, and --delay-updates.

The default with --delete is --delete-during (use --del --delete-during to request this deletion mode explicitly).
--delete-delay is a better choice than using --delete-after.
Incremental recursion can be disabled using --no-inc-recursive.

-R,
--relative
Use relative paths.
The full path names specified on the command line are sent to the server rather than just the last parts of the filenames.
Useful when sending several different directories at the same time. For example:
rsync -av /foo/bar/baz.c remote:/tmp/
creates file /tmp/baz.c.
rsync -avR /foo/bar/baz.c remote:/tmp/
creates /tmp/foo/bar/baz.c preserving its full path.
These extra path elements are called "implied directories" (i.e. the "foo" and the "foo/bar" directories in the above example). rsync sends these implied directories as real directories in the file list, even if a path element is really a symlink on the sending side. This prevents unexpected behaviors when copying the full path of a file that you didn't realize had a symlink in its path.
To duplicate a server-side symlink, include both the symlink via its path, and referent directory via its real path.

To limit the amount of path information that is sent as implied directories for each path, insert a dot and a slash into the source path.

rsync -avR /foo/./bar/baz.c remote:/tmp/
That would create /tmp/bar/baz.c on the remote machine. (Note that the dot must be followed by a slash, so "/foo/." would not
--no-implied-dirs affects the default behavior of --relative .
The attributes of the implied directories from the source names are not included in the transfer. This means that the corresponding path elements on the destination system are left unchanged if they exist, and any missing implied directories are created with default attributes. This allows these implied path elements to have big differences, such as being a symlink to a directory on the receiving side.

For instance, if a command-line arg or a files-from entry told rsync to transfer the file "path/foo/file", the directories "path" and "path/foo" are implied when --relative is used. If "path/foo" is a symlink to "bar" on the destination system, the receiving rsync would ordinarily delete "path/foo", recreate it as a directory, and receive the file into the new directory. With --no-implied-dirs, the receiving rsync updates "path/foo/file" using the existing path elements, which means that the file ends up being created in "path/bar".
Another way to accomplish this link preservation is to use the --keep-dirlinks option (which will also affect symlinks to directories in the rest of the transfer).

-b,
--backup
Preexisting destination files are renamed as each file is transferred or deleted. Controls where the backup file goes and what (if any) suffix gets appended. Use --backup-dir and --suffix (watch free space on destination).

without --backup-dir,

  1. --omit-dir-times will be implied, and
  2. if --delete is also in effect (without --delete-excluded), rsync will add a "protect" filter-rule for the backup suffix to the end of all existing excludes (e.g. -f "P *~").
Prevents previously backed-up files from being deleted. If you are supplying filter rules, insert own exclude/protect rule higher up in the list so that it has a high enough priority to be effective (e.g., if your rules specify a trailing inclusion/exclusion of '*', the auto-added rule would never be reached).
--backup-dir=DIR In combination with --backup , to store all backups in the specified directory on the receiving side. This can be used for incremental backups.
--suffix=SUFFIX override the default backup suffix used with --backup.
Default suffix is ~ if no --backup-dir was specified, otherwise it is an empty string.
--inplace When a files data needs to be updated:
Writes the updated data directly to the destination file instead of the Normal creating a copy of the file and moved into place when it is complete.

    Effects:
  1. in-use binaries cannot be updated (either the OS will prevent this from happening, or binaries that attempt to swap-in their data will misbehave or crash),
  2. file data will inconsistent during the transfer,
  3. file data may be left in an inconsistent state after the transfer if the transfer is interrupted or fails,
  4. a file that does not have write permissions can not be updated, and
  5. the efficiency of rsync's delta-transfer algorithm may be reduced if some data in the destination file is overwritten before it can be copied to a position later in the file
    Combine with --backup, since rsync will use the backup file as the basis file for the transfer to prevent this.

WARNING: Do not use this to update files being accessed by others.

Useful for transferring large files with block-based changes or appended data, and also on systems that are disk bound, not network bound.

Implies --partial (since an interrupted transfer does not delete the file),
Conflicts with --partial-dir and --delay-updates.

-P equivalent to --partial --progress. Make it much easier to specify these two options for a long transfer that may be interrupted.
-d,
--dirs
Tell the sending side to include any directories that are encountered. Unlike --recursive, a directory's contents are not copied unless the directory name specified is "." or ends with a trailing slash (e.g. ".", "dir/.", "dir/", etc.). Without this option or --recursive , rsync will skip all directories it encounters (and output a message to that effect for each one).
Specifing both --dirs and --recursive, --recursive takes precedence.

--dirs is implied by --files-from or --list-only (including an implied --list-only usage) if wasn't specified (so that directories are seen in the listing).
Specify --no-dirs to turn this off.

Link handling

skip to attribute handling
-l
--links
recreate symlinks on the destination.
-L
--copy-links
for symlinks, the item that they point to (the referent) is copied.
--copy-unsafe-links copy the referent of symlinks that point outside the copied tree.
Absolute symlinks are treated like ordinary files, and as are symlinks in the source path itself when --relative is used.
Has no additional effect if --copy-links was also specified.
--safe-links ignore any symbolic links which point outside the copied tree and absolute symlinks
With --relative may give unexpected results.
-k
--copy-dirlinks
sending side treats a symlink to a directory as though it were a real directory.
Useful if you don't want symlinks to non-directories to be affected, as they would be using --copy-links.

Without this , if the sending side has replaced a directory with a symlink to a directory, the receiving side will delete anything that is in the way of the new symlink, including a directory hierarchy (as long as --force or --delete is in effect).

See --keep-dirlinks for an analogous option for the receiving side.

-K
--keep-dirlinks
receiving side treats a symlink to a directory as though it were a real directory, only if it matches a real directory from the sender.
Without this the receiver's symlink would be deleted and replaced with a real directory.

For example, transfer a directory "foo" that contains a file "file", but "foo" is a symlink to directory "bar" on the receiver.
Without --keep-dirlinks, the receiver deletes symlink "foo", recreates it as a directory, and receives the file into the new directory.
With --keep-dirlinks, the receiver keeps the symlink and "file" ends up in "bar". caution: when using --keep-dirlinks, you must trust all the symlinks in the copy!
If it is possible for an untrusted user to create their own symlink to any directory, the user could then (on a subsequent copy) replace the symlink with a real directory and affect the content of whatever directory the symlink references.
For backup copies, use a bind mount instead of a symlink to modify your receiving hierarchy.

See also --copy-dirlinks for an analogous option for the sending side.

Attributes, Meta-data, i.e. file permissions, Access Control Lists, owner, group …

-p
--perms
set the destination permissions to be the same as the source permissions.
See --chmod to modify what are considered to be the source permissions.

With --no--perms:

  • Existing files retain permissions, ( --executability might change the execute permission.)
  • New files Have access permissions set to the source permissions masked with the receiving directory's default permissions (either the receiving process's umask, or the permissions specified via the destination directory's default AccessControlList.
    Special mod bits are disabled except where a new directory inherits setgid from its parent.

    To give new files the destination-default permissions (while leaving existing files unchanged), use --no--perm and --chmod=ugo=rwX

    rsync alias --compress --no--perms --no--group --chmod=ugo=rwX

    then use : rsync -avZ src/ dest/

  • --chmod apply comma-separated chmod strings to the permission of the files in the transfer.
    as if they were the permissions supplied by the sender.
    Can seem to have no effect on existing files if --perms is not enabled.

    An item that should only apply to a directory is prefixed with D', or to a file if prefixed with 'F'.
    For example:

    --chmod=Dg+s,ug+w,Fo-w,+X
    Multiple --chmods are permitted.

    See --perms and --executability for how the resulting permission value can be applied to the files in the transfer.

    -E
    --executability
    preserve the executability (or nonexecutability) of regular files when --perms is not enabled.
    A regular file is considered to be executable if at least one 'x' is turned on in its permissions. When an existing destination file's executability differs from that of the corresponding source file, rsync modifies the destination file's permissions as follows:
  • To make a file non-executable, rsync turns off all its 'x' permissions.
  • To make a file executable, rsync turns on each 'x' permission that has a corresponding 'r' permission enabled.
  • -A
    --acls
    update the ACLs, implies --perms.
    See --fake-super to backup and restore ACLs that are not compatible.
    -X
    --xattrs
    update the remote extended attributes.
    When done by super-user, copies all namespaces except system.*.
    A normal user only copies the user.* namespace.
    To to backup and restore non-user namespaces as a normal user, see --fake-super .
    -o
    --owner
    preserve owner (other than receiving user)
    Default: the owner is set to the user on the receiving side.

    only if the receiver is super-user (see --super and --fake-super ).
    Caution: The preservation of ownership will associate matching names but if it falls back to using the ID number (see --numeric-ids it will surely be incorrect!!.

    -g
    --group
    preserve group
    If the receiving program is not running as the super-user (or if --no-super was specified).
    Only groups that the invoking user on the receiving side is a member of will be preserved.
    Without this option, the group is set to the default group of the invoking user on the receiving side.

    The preservation of group information will associate matching names by default, but may fall back to using the ID number

    -t
    --times
    preserve modification times (implied with --archive).
    If times are not preserved (in addition to corrupting the modification time on the destination) the next time a transfer is performed, the evaluation which excludes files that have not been modified will be incorrect. The deltatransfer algorithm will minimize the update.
    -O
    --omit-dir-times
    omit directories when preserving modification times (see --times).
    Suggested if NFS is sharing the directories on the receiving side.
    Implied with --backup without --backup-dir.
     
    --super receiving side attempts to execute super-user activities even if it wasn't run by the super-user.
    Include: preserving users via the --owner option,
    preserving all groups (not just the current user's groups) via --groups , and
    copying devices via --devices .
    Useful for systems that allow such activities without being the super-user and
    ensuring errors if the receiving side isn't being run as the super-user.
    To turn off super-user activities, the super-user can use --no-super.
    --fake-super simulate super-user activities by saving/restoring the privileged attributes via special extended attributes.
    Includes the file's owner and group (if it is not the default), the file's device info (device & special files are created as empty text files), and any permission bits that won't be allowed to be set on the real file (e.g. the real file gets u-s,g-s,o-t for safety) or that would limit the owner's access (since the real super-user can always access/change a file, the files we create can always be accessed/changed by the creating user).
    Handles ACLs (if --acls was specified) and non-user extended attributes (if --xattrs was specified).
    backup data without using a super-user, and to store ACLs from incompatible systems.
    Only affects the side where the option is used. To affect the remote side of a remote-shell connection, specify an rsync path:
    rsync -av --rsync-path="rsync --fake-super" /src/ host:/dest/

    Since there is only one "side" in a local copy, this option affects both the sending and receiving of files. specify a copy using "localhost" to avoid this, possibly using the "lsh" shell script (from the support directory) as a substitute for an actual remote shell (see --rsh).

    overridden by both --super and --no-super.

    See "fake super" in rsyncd.conf .

    -S
    --sparse
    sparse files efficiently so they take up less space on the destination.
    Conflicts with --inplace as it is not possible to overwrite data in a sparse fashion.

    NOTE: Don't use if destination is Solaris tmpfs . There are problems seeking over null regions, and corrupts the files.

    -W
    --whole-file
    delta-transfer algorithm is not used.
    The transfer may be faster if this option is used when the bandwidth between the source and destination machines is higher than the bandwidth to disk (example: when the "disk" is actually a networked filesystem).
    Default when source and destination are local paths, if no batch writing option is in effect.
    -x
    --one-file-system
    avoid crossing a filesystem boundary when recursing.
    This does not limit the ability to specify items to copy from multiple filesystems, just rsync's recursion through the hierarchy of each directory that the user specified, and also the analogous recursion on the receiving side during deletion.
    A "bind" mount to the same device is considered being on the same filesystem.

    If repeated, rsync omits all mount-point directories.
    Otherwise, it includes an empty directory at each mount-point it encounters (using the attributes of the mounted directory because those of the underlying mount-point directory are inaccessible).

    If collapse symlinks (via --copy-links or --copy-unsafe-links), a symlink to a directory on another device is treated like a mount-point. Symlinks to non-directories are unaffected by this option.

    --existing,
    --ignore-non-existing
    skip creating files and directories that do not exist on the destination.
    If combined with the --ignore-existing , no files will be updated (useful to delete extraneous files).

    This is a transfer rule, not an exclude, it doesn't affect the data that goes into the file-lists, and thus it doesn't affect deletions. It just limits the files that the receiver requests to be transferred.

    --ignore-existing skip updating files that already exist on the destination (this does NOT ignore existing directories, or nothing would get done). See also --existing. This is a transfer rule, not an exclude, so it doesn't affect the data that goes into the file-lists, and doesn't affect deletions.
    It limits the files that the receiver requests to be transferred. Useful for doing backups using --link-dest when continung a n interrupted backup. Since a --link-dest run is copied into a new directory hierarchy (when it is used properly), using --ignore existing will ensure that the already-handled files don't get tweaked (which avoids a change in permissions on the hard-linked files).
    This does mean that this option is only looking at the existing files in the destination hierarchy itself.
    Removing Files
    --max-delete=N Attempts to delete more than N files or directories results in a warning and exit with an error code 25 .

    --max-delete=0 generates warning about files in the destination without removing them.

    Removing Files from the receiving side
    --delete files > that aren't on the sending side for directories being synchronized.
    It is expected that rsync previously sent the whole directory ( using dir or dir/) without using a wildcard for the directory's contents (e.g. dir/*) (A wildcard is expanded by the shell and rsync gets a list of files, not the files' parent directory. )
    Files excluded from the transfer are also excluded from being deleted unless --delete-excluded or mark the rules as only matching on the sending side See include/exclude modifiers .

    Deletions will also occur when --dirs is enabled for directories whose contents are being copied.

    Use --dry-run option to see what files are going to be deleted.

    If the sending side detects I/O errors, deletion of any files will be disabled,
    Especially useful with NFS errors on the sending side)

    May be combined with one of the --delete-WHEN options as well as --delete-excluded.
    Default is --delete-during .
    See --delete-delay and --delete-after.

    --delete-before all deletions are done before any transfer starts.
    See --delete.

    Helpful when the destination filesystem is low on space.
    Introduces a delay before the start of the transfer which might cause the transfer to timeout
    Forces non-incremental recursion algorithm that requires rsync to scan all the files in the transfer into memory at once (see --recursive).

    --delete-during
    --del
    File deletions are done as the transfer occurs.
    The per-directory delete scan is done before each directory is checked for updates,
    --delete-delay File deletions are done after the transfer completes.
    Useful when combined with --delay-updates and/or --fuzzy, and is more efficient than using --delete-after (can behave differently, since --delete-after computes the deletions in a separate pass after all updates are done).
    If the number of removed files is very large a temporary file is created with the list. If it cannot be created rsync will use --delete-after (unless --recursive is doing an incremental scan). See --delete.
    --delete-after file deletions are done after the transfer has completed.
    Useful when sending new per-directory merge files as a part of the transfer and you want their exclusions to take effect for the delete phase of the current transfer.
    Forces rsync to use non-incremental recursion algorithm that requires rsync to scan all the file names in the transfer into memory at once.
    --delete-excluded In addition to deleting the files not present on the sending side,
    delete files on the receiving side that are excluded (see --exclude). See the FILTER RULES section for a way to make individual exclusions behave this way on the receiver, and for a way to protect files from --delete-excluded
    --force Delete a directory when it is to be replaced by a file, `even if the directory contains files. Only relevant if deletions are not active (see --delete for details).
    --ignore-errors --delete deletes files even when there are I/O errors.Danger
    --remove-source-files remove from the sending side files successfully duplicated . (non-directories)
     
    options to restrict activities
    --max-size=bytes Supress transferring files larger than bytes.
    may be a fractional (e.g. --max-size=1.5m).
    A transfer rule limits the files that the receiver requests to be transferred
    not an exclude, doesn't affect the file-lists or deletions.
    Suffixes (case insensitive)
    • KB (1,000) : K KiB kilibyte (1024):
    • MB ( 1,000,000) :M MiB megibyte (1024*1024)
    • GB (1,000 MB) :G GiB gibibyte (1024*1024*1024)
    • +1 or -1 to offset by one byte.

    Examples: --max-size=1.5mb-1 is 1,499,999 bytes, and
    --max-size=2g+1 is 2,147,483,649 bytes.

    --min-size=mb avoid transferring any file that is smaller than mb, which can help in not transferring small, junk files. See the --max-size option for a description of SIZE and other information.
    -B
    --block-size=blksz
    forces the block size in delta-transfer algorithm to a fixed value.
    Selected based on the size of each file being updated. See technical report.
    -e
    --rsh=rmsh
    Remote shell for communication. default: ssh prefer to use rsh on a local network.

    With [user@]host::module/path, the remote shell will be used to run an rsync daemon on the remote host, and all data will be transmitted through that remote shell connection, not a socket connection to a running rsync daemon on the remote host.
    See using rsync-daemon features via a remote-shell connection

    Command-line arguments are permitted in rmsh provided that is presented to rsync as a single argument.
    Use spaces (not tabs or other whitespace) to separate rmsh into the command and arguments.
    Use apostrophe and/or quotes to preserve spaces in an argument (not backslashes).
    Doubling an apostrophe inside a apostrophe'd string results in apostrophe ( likewise for quotes ) (attention to which quotes the shell is parsing and which quotes rsync is parsing).

    Examples:

     -e 'ssh -p 2234'
      -e 'ssh -o "ProxyCommand nohup ssh firewall nc -w1 %h %p"' 
    
    ssh uses .ssh/config for site specific options.

    RSYNC_RSH environment variable specifies remote shell program, which accepts the same range of values as -e.
    See --blocking-io which is affected by this option.

    --devicescharacter and block device files are created on remote system.
    receiving rsync must be super-user (see also the --super and --fake-super options).
    --specials transfer special files such as named sockets and fifos.
    -D equivalent to --devices --specials
    --rsync-path=fullName When rsync is not in the remote path
    e.g. --rsync-path=/usr/local/bin/rsync .
    fullName is run with a shell and can be any program, script, or command sequence , so long as it does not corrupt the standard-in & standard-out that rsync is using to communicate. To set a different default directory on the remote for use with --relative option. example:
                      rsync -avR --rsync-path="cd /a/b && rsync" host:c/d /e/  
    options controling files to be transfered
    -r,
    --recursive
    copy directories recursively. See --dirs .
    -f
    --filter=rule
    add rules to selectively include/exclude files. Useful in combination with a recursive transfer.
    Multiple --filter options are permitted.
    If rule contains whitespace, quote it.
    An underscore can be used to replace the space that separates a rule from its arg. See FILTER RULES
    -F

    -F-F

    Exclude .rsync-filter files sprinkled through the hierarchy and use their rules to filter the files.

    Filters out the .rsync-filter files themselves .

    --exclude=pattern
    --exclude-from=file file contains exclude patterns (one per line).

    Blank lines and lines starting with ; or # are ignored.
    If FILE is -, the list will be read from standard input.
    -C
    --cvs-exclude
    Exclude files that shouldn't be transfered, similar to CVS
    These are marked perishable see filter rules
    1. The exclude list is initialized with :
         RCS   SCCS   CVS   CVS.adm   RCSLOG   cvslog.*   tags   TAGS    .make.state   .nse_depinfo   *~   #*   .#*   ,*   _$*   *$   *.old   *.bak    *.BAK   *.orig   *.rej   .del-*   *.a   *.olb   *.o   *.obj   *.so   *.exe    *.Z   *.elc   *.ln   core   .svn/   .git/   .bzr/
    2. Files in the CVSIGNORE environment variable (delimited by whitespace).
    3. Files listed in $HOME/.cvsignore
    4. Files in a directory containing a .cvsignore
    5. Files matching a pattern in .cvsignore These patterns are split on whitespace. See CVS

    When combining --filter rules with -C CVS excludes are appended regardless of where the -C was placed on the commandline.
    This makes them a lower priority than any rules specified explicitly.
    To control where these get inserted into filter rules, omit -C as a command-line option and use a combination of --filter=:C and --filter=-C (either on the command-line or by putting the ":C" and "-C" rules into a filter file with other rules). :C turns on the per-directory scanning for the .cvsignore file. -C does a one-time import of the CVS excludes.

    --include=pattern
    --include-from= file as with --exclude-from
    --files-from=file Specify the exact list of files (as read from file or - for standard input).
    Adjusts the default behavior to make transferring the files and directories easier:
  • --relative is implied, presering path information
  • --dirs is implied, creating directories
  • --archive does not imply --recursive so specify it.

    These side-effects change the default state of rsync.
    The position of --files-from on the command line has no bearing on how other options are parsed (i.e. --archive works the same before or after --files-from, as does --no---relative ). The filenames that are read from file are all relative to the source dir -- any leading slashes are removed and no ".." references are allowed to go higher than the source dir. For example,

    rsync -a --files-from=rsyncfiles /usr remote:/backup
    If rsyncfiles contains the string "bin" (or even "/bin"), the /usr/bin directory will be created as /backup/bin on the destination.
    If it contains "bin/" (note the trailing slash), the immediate contents of the directory would also be sent (without needing to be explicitly mentioned in the file
    In both cases, if --recursive was enabled, that dir's entire hierarchy would also be transferred (keep in mind that --recursive needs to be specified explicitly with --files-from, since it is not implied by --archive). Note that the effect of (enabled by default) --relative is to duplicate only the path info that is read from the file -- it does not force the duplication of the source-spec path (/usr in this case).

    In addition, the --files-from file can be read from the remote host instead of the local host if you specify a "host:" in front of the file (the host must match one end of the transfer). As a short-cut, you can specify just a prefix of ":" to mean "use the remote end of the transfer". For example:

    rsync -a --files-from=:/path/file-list src:/ /tmp/copy
    Copy all the files specified in the /path/file-list file that was located on the remote "src" host.

    If --iconv and --protect-args are specified and the --files-from filenames are being sent from one host to another, the filenames will be translated from the sending host's charset to the receiving host's charset.

  • -0
    --from0
    rules/filenames are terminated by a null ('00') ,
    This affects --exclude-from, --include-from, --files-from, and any merged files specified in a --filter rule. Does not affect --cvs-exclude
    -s
    --protect-args
    sends filenames and most options without allowing the shell to interpret them. This means that spaces are not split in names, and any non-wildcard special characters are not translated (such as ~, $, ;, &, etc.).
    Wildcards are expanded by rsync (not the shell).

    if used with --iconv, the args related to the remote side will also be translated to the remote character-set. The translation happens before wild-cards are expanded. See --files-from .

    -T
    --temp-dir=dir
    Use dir as a scratch directory when creating temporary copies of the files transferred on the receiving side.
    Default: create each temporary file in the same directory as the associated destination file.

    Useful when there is not enough free space to hold a copy of the largest file in the destination filesystem. This requires rsync to it copy the file into place when the file is transfered.

    Combined with --delay-updates , which will ensure that all copied files get put into subdirectories in the destination hierarchy, awaiting the end of the transfer.
    If there is not enough room to duplicate all the arriving files on the destination filesystem, use --partial-dir with a relative path; because it is OK to stash off a copy of a single file in a subdir in the destination hierarchy, rsync will use the partial-dir as a staging area to bring over the copied file, and then rename it into place from there. (Specifying a --partial-dir with an absolute path does not have this side-effect.)

    -y
    --fuzzy
    basis file for any destination file that is missing. The current algorithm looks in the same directory as the destination file for either a file that has an identical size and modified-time, or a similarly-named file. If found, rsync uses the fuzzy basis file to try to speed up the transfer.

    Use of --delete might get rid of any potential fuzzy-match files, so either use --delete-after or specify some filename exclusions to prevent this.

    --compare-dest=DIR use DIR on the destination machine as an additional hierarchy to compare destination files against doing transfers (if the files are missing in the destination directory). If a file is found in DIR that is identical to the sender's file, the file will NOT be transferred to the destination directory. This is useful for creating a sparse backup of just files that have changed from an earlier backup.

    multiple --compare-dest directories may be provided, which will cause rsync to search the list in the order specified for an exact match. If a match is found that differs only in attributes, a local copy is made and the attributes updated. If a match is not found, a basis file from one of the DIRs will be selected to try to speed up the transfer.

    If DIR is a relative path, it is relative to the destination directory. See also --copy-dest and --link-dest.

    --copy-dest=dir like --compare-dest, also copy unchanged files found in dir using a local copy.
    Useful for doing transfers to a new destination while leaving existing files intact, and then doing a flash-cutover when all files have been successfully transferred.

    Multiple --copy-dest directories may be provided, which will cause rsync to search the list in the order specified for an unchanged file. If a match is not found, a basis file from one of the DIRs will be selected to try to speed up the transfer.

    If DIR is a relative path, it is relative to the destination directory. See also --compare-dest and --link-dest.

    --link-dest=DIR like --copy-dest, but unchanged files are hard linked from DIR to the destination directory.
    The files must be identical in all preserved attributes (e.g. permissions, possibly ownership) in order for the files to be linked together. An example:
    rsync -av --link-dest=$PWD/prior_dir host:src_dir/ new_dir/
    If file's aren't linking, double-check their attributes. Also check if some attributes are getting forced outside of rsync's control, such a mount option that squishes root to a single user, or mounts a removable drive with generic ownership (such as OS X's "Ignore ownership on this volume" option).

    multiple --link-dest directories may be provided, which will cause rsync to search the list in the order specified for an exact match. If a match is found that differs only in attributes, a local copy is made and the attributes updated. If a match is not found, a basis file from one of the DIR will be selected to try to speed up the transfer.

    works best when copying into an empty destination hierarchy, as rsync treats existing files as definitive (so it never looks in the link-dest dirs when a destination file already exists), and as malleable (so it might change the attributes of a destination file, which affects all the hardlinked versions).

    combine this option with --ignore-times, rsync will not link any files together because it only links identical files together as a substitute for transferring the file, never as an additional check after the file is updated.

    If DIR is a relative path, it is relative to the destination directory. See also --compare-dest and --copy-dest.

    -z
    --compress
    compresses data as it is sent. ( better compression than a compressing transport )
    --skip-compress=suf[[/…] Override the list of file suffixes that will not be compressed.
    An empty string to indicates that no file should be skipped.

    Simple character-class matching is supported: each must consist of a list of letters inside the square brackets (e.g. no special classes, such as "[:alpha:]", are supported).

    asterisk (*) and question-mark (?) have no special meaning.

    Example specifies 6 suffixes to skip (since 1 of the 5 rules matches 2 suffixes):

    --skip-compress=gz/jpg/mp[34]/7z/bz2
    Default:
    gz/zip/z/rpm/deb/iso/bz2/t[gb]z/7z/mp[34]/mov/avi/ogg/jpg/jpeg/

    A copy from a daemon rsync will adds skipped suffixes to its list of non-compressing files (and its list may be configured to a different default).

    --compress-level=n compression level. If n is non-zero, --compress is implied.
    --password-file=file for accessing an rsync daemon, must not be world readable.

    When accessing a daemon using a remote shell, this option only comes into effect after the remote shell finishes its authentication (i.e. if a password is specified in the config file).

    For shell transfers use "Public key authentication".

    --numeric-ids Use numeric user and group IDs rather than using user and group names. DANGER! Only applicable when transfering files with various owners.

    By default username and groupname are used to establish ownership.
    If a user or group on the source system does not match the destination, the numeric ID from the source system is used.

    If on the source system, user jsmith is user ID 507 and on destination is user ID 509 and bsmith is user ID 507, files on the the destination will belong to bsmith !

    See the comments on the "use chroot" setting in the rsyncd.conf manpage for information on how the chroot setting affects rsync's ability to look up the names of the users and groups and what you can do about it.
    The special user and groups ID 0 are never mapped names even if the --numeric-ids option is not specified.

    Options related to networking
    --bwlimit=kbps BandWidth limit, kilobytes per second. Only effective with files greater than 3 MB.
    Zero specifies no limit.
    --timeout=seconds If no data is transferred for seconds rsync exits with rc=12.
    Default:0, no timeout.bad
    --contimeout=seconds Wait seconds for a connection to the rsync daemon, ( host::dir) afterwards exit with rc=35.
    --address address Specific IP address (or hostname) to bind to for virtual hosting. See daemon mode .
    --port=port Only useful connecting with an rsync daemon using non default port. Default:873.
    URL for connection via shell can include a port.
    -4, --ipv4
    -6, --ipv6
    Only affects sockets that rsync has direct control over, such as the outgoing socket when directly contacting an rsync daemon
    --sockopts tune systems to the utmost degree. See rsync docs.
    --blocking-io when launching a remote shell transport.
    If the remote shell is rsh or remsh defaults to using blocking I/O.
    -8
    --8-bit-output
    leave all high-bit characters unescaped in the output (report? like ugly filenames?). Control characters (except tabs) are always escaped, regardless of this option's setting.

    Output a literal backslash (\) and a hash (#), and 3 octal digits. For example, a newline would output as "\#012". A literal backslash that is in a filename is not escaped unless it is followed by a hash and 3 digits (0-9).

    --partial keep partial file which should make a subsequent transfer of the rest of the file much faster.
    By default, rsync will delete any partially transferred file if the transfer is interrupted.
    --partial-dir=dir to hold the partial data (instead of to the destination file).
    On the next transfer, rsync will use a file found in this dir as data to speed up the resumption of the transfer and then delete it

    If --whole-file is specified (or implied), any partial-dir file that is found for a file that is being updated will be removed (since rsync is sending files without using rsync's delta-transfer algorithm).

    Rsync creates dir if necessary (just the last dir -- not the whole path).
    Allowing use of relative path (such as "--partial-dir=.rsync-partial") to have rsync create the partial-directory in the destination file's directory when needed, and then remove it again when the partial file is deleted.


    If dir is not an absolute path, rsync will add an exclude rule at the end of all existing excludes. preventing the sending of any partial-dir files that may exist on the sending side, and will also prevent the untimely deletion of partial-dir items on the receiving side.
    An example: the above --partial-dir option would add the equivalent of -f '-p .rsync-partial/' at the end of any other filter rules.

    When supplying exclude rules, you may need to add your own exclude/hide/protect rule for the partial-dir because
    (1) the auto-added rule may be ineffective at the end of your other rules, or
    (2) you may wish to override rsync's exclude choice. For instance, make rsync clean-up any left-over partial-dirs that may be lying around, specify --delete-after and add a "risk" filter rule, eg. -f 'R .rsync-partial/'. (Avoid using --delete-before or --delete-du ing unless you don't need rsync to use any of the left-over partial-dir data during the current run.)

    IMPORTANT: the --partial-dir should not be writable by other users or it is a security risk.
    AVOID "/tmp".

    You can also set the partial-dir value the RSYNC_PARTIAL_DIR environment variable. Setting this in the environment does not force --partial to be enabled, but rather it affects where partial files go when --partial is specified. For instance, instead of using --partial-dir=.rsync-tmp along with --progress, Set RSYNC_PARTIAL_DIR=.rsync-tmp environment variable and then use -P to turn on the use of the .rsync-tmp dir for partial transfers.
    The only times that --partial does not look for this environment value are (1) when --inplace was specified (since --inplace conflicts with --partial-dir), and (2) when --delay-updates was specified (see below).

    For the purposes of the daemon-config's "refuse options" setting, --partial-dir does NOT imply --partial. This is so that a refusal of the --partial option can be used to disallow the overwriting of destination files with a partial transfer, while still allowing the safer idiom provided by --partial-dir.

    --delay-updates puts the temporary file from each updated file into a holding directory until the end of the transfer, at which time all the files are renamed into place in rapid succession. This attempts to make the updating of the files a little more atomic.
    By default the files are placed into a directory named .~tmp~ in each file's destination directory, but if you've specified the --partial-dir option, that directory will be used instead.
    See the comments in the --partial-dir section for a discussion of how this ".~tmp~" dir will be excluded from the transfer, and what you can do if you want rsync to cleanup old tmp~ dirs that might be lying around.
    Conflicts with --inplace and --append.

    uses more memory on the receiving side (one bit per file transferred) and also requires enough free disk space on the receiving side to hold an additional copy of all the updated files.

    Do not use an absolute path to --partial-dir unless
    (1) there is no chance of any of the files in the transfer having the same name (since all the updated files will be put into a single directory if the path is absolute)
    and
    (2) there are no mount points in the hierarchy (since the delayed updates will fail if they can't be renamed into place).

    See also the "atomic-rsync" perl script in the "support" subdir for an update algorithm that is even more atomic (it uses --link-dest and a parallel hierarchy of files).

    -m
    --prune-empty-dirs
    receiving rsync removes empty directories from the file-list, including nested directories that have no non-directory children.
    Useful for avoiding the creation of a bunch of useless directories when the sending rsync is recursively scanning a hierarchy of files using include/exclude/filter rules.

    the use of transfer rules, such as the --min-size option, does not affect what goes into the file list, and thus does not leave directories empty, even if none of the files in a directory match the transfer rule.

    Because the file-list is actually being pruned, also affects what directories get deleted when a delete is active. However, keep in mind that excluded files and directories can prevent existing items from being deleted due to an exclude both hiding source files and protecting destination files. See the perishable filter-rule option for how to avoid this.

    prevent the pruning of certain empty directories from the file-list by using a global "protect" filter.
    For instance, this option would ensure that the directory "emptydir" was kept in the file-list:

    --filter 'protect emptydir/'

    Here's an example that copies all .pdf files in a hierarchy, only creating the necessary destination directories to hold the .pdf files, and ensures that any superfluous files and directories in the destination are removed (note the hide filter of non-directories being used instead of an exclude): rsync -avm --del --include='*.pdf' -f 'hide,! */' src/ dest TO retain superfluous destination files, the "--include='*/' --exclude='*'"

    --write-batch=FILE create a file that can later be applied to another identical destination with --read-batch . See the "BATCH MODE" section for details, and also the --only-write-batch option.
    --only-write-batch=file like --write-batch, except that no updates are made on the destination system when creating the batch allowing transporting changes to the destination system via some other means and then apply the changes via --read-batch.

    Write the batch directly to portable media: if this media fills before the end of the transfer, apply that partial transfer to the destination and repeat the process to get the rest of the changes (this will use a partially updated destination system while the multi-update cycle is happening).

    you only save bandwidth when pushing changes to a remote system because this allows the batched data to be diverted from the sender into the batch file without having to flow over the wire to the receiver (when pulling, the sender is remote, and thus can't write the batch).

    --read-batch=FILE Apply changes stored in , a file previously generated by --write-batch.
    If file is -, the batch data will be read from standard input. See the "BATCH MODE" section for details.
    --protocol=NUM Force an older protocol version to be used. This is useful for creating a batch file that is compatible with an older version of rsync. For instance, if rsync 2.6.4 is being used with --write-batch , but rsync 2.6.3 is what will be used to run the --read-batch option, you should use "--protocol=28" when creating the batch file to force the older protocol version to be used in the batch file (assuming you can't upgrade the rsync on the reading system).
    --iconv=convert_spec convert filenames between character sets
    Use "." for character-set via the locale setting.
    --iconv=local,remote For example: --iconv=utf8,iso88591. This order ensures that the option will stay the same whether pushing or pulling files.
    --no-iconv or a convert_spec of "-" to turn off conversion.
    Default is site-specific, and can be affected via $RSYNC_ICONV

    For a list of charset names local iconv library supports, run "iconv --list".

    The following list contain all the coded character …
      , 8859_1, 8859_2, 8859_3, 8859_4, 8859_5, 8859_6, 8859_7, 8859_8, 8859_9, 10646-1:1993, 10646-1:1993/UCS4,
      ANSI_X3.4-1968, ANSI_X3.4-1986, ANSI_X3.4, ANSI_X3.110-1983, ANSI_X3.110, ARABIC, ARABIC7, ARMSCII-8, ASCII, 
    …
     US-ASCII, US, UTF-7, UTF-8, UTF-16, UTF-16BE, UTF-16LE, UTF-32, UTF-32BE, UTF-32LE, UTF7, UTF8, …
    

    With --protect-args filenames for transfer on the command-line are translated. See --files-from .

    No conversion of names in filters (including include/exclude files) is done!.
    Specify extra include/exclude rules if there are filename differences on the two sides.

    With --iconv the daemon uses the charset specified in its "charset" configuration

    --info=flags fine-grained control over the information output . An individual flag name may be followed by a level number, with 0 meaning to silence that output, 1 being the default output level, and higher numbers increasing the output of that flag (for those that support higher levels). Use --info=help to see all the available flag names, what they output, and what flag names are added for each increase in the verbose level. Some examples: rsync -a --info=progress2 src/ dest/ rsync -avv --info=stats2,misc1,flist0 src/ dest/ Note that --info=name's output is affected by the --out-format and --itemize-changes (-i) options. See those options for more information on what is output and when.
    --debug=FLAGS fine-grained control over the debug output. An individual flag name may be followed by a level number, with 0 meaning to silence that output, 1 being the default output level, and higher numbers increasing the output of that flag (for those that support higher levels).
    --debug=help displays flag names, what they output, and what flag names are added for each increase in the verbose level. Some examples: rsync -avvv --debug=none src/ dest/ rsync -avA --del --debug=del2,acl src/ dest/

    some debug messages will only be output when --msgs2stderr is specified, especially those pertaining to I/O and buffer debugging. This option was added to 3.1.0, so an older rsync on the server side might reject your attempts at fine-grained control (if one or more flags needed to be send to the server and the server was too old to understand them). See also the "max verbosity" caveat above when dealing with a daemon.

    --msgs2stderr All output to stderr rather than to send messages to the client side via the protocol (which normally outputs info messages via stdout). Intended for debugging in order to avoid changing the data sent via the protocol, since the extra protocol data can change what is being tested. The option does not affect the remote side of a transfer without using --remote-option -- e.g. -M--msgs2stderr. Daemon connection does not have a stderr channel to send messages back to the client side, so if you are doing any daemon-transfer debugging using this option, you should start up a daemon using --no-detach so that you can see the stderr output on the daemon side. This option has the side-effect of making stderr output get line-buffered so that the merging of the output of 3 programs happens in a more readable manner. -
    --checksum-seed=NUM This 4 byte checksum seed is included in each block and file checksum calculation.
    By default the checksum seed is generated by the server and defaults to the current time() .
    used to set a specific checksum seed, for repeatable block and file checksums, or in the case where the user wants a more random checksum seed.
    Setting NUM to 0 causes rsync to use the default of time() for checksum seed.
    --help
    --version


    Daemon Options

    --daemon Run as a daemon. accessed using a client with host::module or rsync://host/module/.

    If standard input is a socket then rsync will assume that it is being run via inetd, otherwise
    it will detach from the current terminal and become a background daemon, reading rsyncd.conf on each connect made by a client. config

    "exported" directories (modules) must be specified in rsyncd.conf

    --address addr allows to specify a specific IP address (or hostname) to bind to.
    By default rsync will bind to the wildcard address when run as a daemon with --daemon . This makes virtual hosting possible in conjunction with --config . See "address" global in rsyncd.conf
    --bwlimit=kbps BandWith limit. for daemon sends.
    The client can specify a smaller value, but their requested value will be rounded down if they try to exceed it. See the client version of this option (above) for some extra details.
    --log-file=file over-ride setting in the config file.
    --log-file-format=FORMAT override the FORMAT in the config file.
    --config=file alternate config file. default /etc/rsyncd.conf
    When running over a remote shell and the remote user is not the super-user use rsyncd.conf in the current directory ($HOME).
    --no-detach nor become a background job. a service on Cygwin, When rsync is run under a debugger. No effect if rsync is run from inetd or sshd.
    --port=port Default: 873 is priviledged so used over 1000 for non root user.
    See port rsyncd.conf.1.html,
    --sockopts overrides the socket setting in rsyncd.conf
    -v
    --verbose
    increases the amount of information the daemon logs during its startup phase.
    After the client connects, the verbosity level will be controlled by the options the client used and the "max verbosity" setting in the module's config section.
    -4
    --ipv4
    -6
    --ipv6
    prefered transport when creating the incoming sockets
    -h
    --help --daemon,
    output a short help page describing the options available for starting daemon.
    --no-motd suppresses the Message-Of-The-Day (MOTD) text, and the list of modules that the daemon sends in response to the "rsync dahost::"


    The client can connect to a host using either:

    In daemon mode an exported directory is called a module which has specific permissions and restrictions.
    This allows control over which directories are transferable.

    To list the modules available
    > rsync hostname::

    To list the files in a module :
    > rsync hostname::module

    Example

     > rsync jupiter::rsync
    drwxr-xr-x         102 2010/11/19 18:40:23 001_PHOTOS
    drwxr-xr-x         238 2010/11/25 15:46:02 p8_2006 

    Once rsync starts it traps sigINT which means the usual ^c will not interrupt rsync.
                      Use ^\ which is sigQUIT or another screen to quit with

    kill -quit `/bin/ps | /usr/bin/grep " rsync" |/usr/bin/head -1|/usr/bin/cut -f2 -d" "`

    Example

    Backup files from a web server to the local system, updating files previously saved,
    monitoring progress and error (saving a log to a file for later review).
    
     echo "++++++++++++++++++++++++++" >> rsync.err
     tail -f rsync.err&                  
     rsync --progress --archive --hard-links --compress --update     --stats   \
                      --log-file=`date +rsync_%y%m%d.log` --log-format='%t %o %B %l %M %f' \
                                                                                  
           uname@webSite.com:public_html backupDir   2>> rsync.err 
    
    

    Expect to see

    uname@webSite.com password:
    Enter the password which will not show ( If the password shows rsync has exited see diagnose).

    Use ^Z to push the job into the backround.

    This uses the remote shell transport as indicated by the single colon, : between the hostname and the directory.
    This means that rsync will login and start rsync at the other end. This precludes having the rsync daeom running.

    The progress of rsync will be displayed.

    2011/10/26 21:14:30 recv rwx------ 20480 2011/10/26-21:14:10 jupiter/downloads
    2011/10/26 21:14:30 recv rw------- 4520453 2011/09/23-13:00:56 jupiter/downloads/ACPIspec40a.pdf
         4520453 100%    2.70MB/s    0:00:01 (xfer#1, to-check=47541/88416)
    2011/10/26 21:14:31 recv rw-r--r-- 1364852 2011/10/18-10:18:34 jupiter/downloads/nuketheappstore.zip
         1364852 100%    1.42MB/s    0:00:00 (xfer#2, to-check=47246/88416)
    
    After all the files are transfered hard links are created.
    Number of files: 88416
    Number of files transferred: 2
    Total file size: 5004066643 bytes
    Total transferred file size: 5885305 bytes
    Literal data: 5885305 bytes
    Matched data: 0 bytes
    File list size: 2710269
    File list generation time: 1.120 seconds
    File list transfer time: 0.000 seconds
    Total bytes sent: 70
    Total bytes received: 7497967
    
    sent 70 bytes  received 7497967 bytes  714098.76 bytes/sec
    total size is 5004066643  speedup is 667.38
    
    

    When the transfer is complete

     fg to bring the tail process to the foreground.
     then ^C to interrupt the tail.

    time rsync --verbose --progress gername@webtems.com:* .

    --xattrs --acls 

    Symbolic links

    Symbolic links are directory entries that contain a string which is the name of the file they link to(referant), perhaps in another directory or even another filesystem.

    If the referant file is not included then the symlink might require the referent to be included.
    If the symlink refers to a directory on another filesystem the referent probably should not be included!

    During any given transfer some symlinks might need to be handled differently from others!

    default: not transferred, outputs: "skipping non-regular" file.
    --links are recreated with the same target on the destination
    --archive implies --links.
    --copy-links are "collapsed" by copying their referent.(requires additional space)

    safe and unsafe symbolic links:
    Symbolic links are unsafe if they are absolute symlinks (start with /), empty, or if they contain enough ".." components to ascend from the directory being copied.

    Example: web site mirror wishes to ensure that the module that is copied does not include symbolic links to /etc/passwd in the public section of the site.

    1. --copy-unsafe-links will cause any links to be copied as the file they point to on the destination.
    2. --safe-links will cause unsafe links to be omitted altogether.
      (specify --links for --safe-links be effective.)

    In order of precedence
    (from most likely to cause trouble, i.e. skip referent file,
     to most likely to cause excesive data)

    default ignored.
    --copy-links become files.
    safe symlinksunsafe symlinks
    --links
    --archive
    --copy-unsafe-links
    are duplicated become files
    --copy-unsafe-links skipedbecome files, noisily.
    --links --safe-links
    --archive
    duplicated skiped.
    --links
    --archive
    Duplicate all

    Problems

    The startup script(.profile, .bashrc, …) or remote shell is producing output. This could include motd (Message Of The Day), lastlog date as well as host fingerprint produced with the ssh -o VisualHostKey=yes

    To diagnose this problem run:

    /usr/bin/ssh uname@remotehost /bin/true > out.dat
    if [ -s out.dat ] ; then echo "-- should be size 0; here it is:";cat out.dat
    else                     echo "No output(good). Check the firewall rules";fi

    Shell startup scripts (ex: .cshrc, .profile) which produce output must be renamed temporarily.

    Filter patterns

    specify -vv to show why each file is included or excluded.

    Environment Variables

    CVSIGNORE supplements any ignore patterns in .cvsignore files. --cvs-exclude
    RSYNC_ICONV defaults --iconv
    RSYNC_RSHoverrides shell used as the transport . Command line options are permitted after the command name, as in -e.
    RSYNC_PROXY redirects client to use a web proxy when connecting, specify:hostname:port pair.
    RSYNC_PASSWORD runs authenticated connections without intervention.
    Does not supply a password to a remote shell transport such as ssh.
    USER or LOGNAME defaults username sent to daemon. default: nobody.
    HOMEdefault directory for .cvsignore.

    Examples

    > sudo  rsync --progress --archive --hard-links --compress \
                         meuname@host::sources mysrcs >>0_0rsync.log &
    
    > tail -f 0_0rsynd.log   # this shows output
    receiving file list ... 
    46108 files to consider
    …
    iPhoto/iPod Photo Cache/F50/T9226.ithmb
          671600 100%  636.14kB/s    0:00:01 (xfer#7321, to-check=7/46108)
    iPhotoHidden/iPhoto
          443076 100%    6.50MB/s    0:00:00 (xfer#7322, to-check=5/46108)
    lt/Originals/0104_BirdsAtHome/OurOwl_5063.AVI
        74781936 100%    9.78MB/s    0:00:07 (xfer#7323, to-check=0/46108)
    
    sent 161377 bytes  received 7744458181 bytes  7114946.77 bytes/sec
    total size is 25579631224  speedup is 3.30
    
    

    Installing

    Download install config and start rsync

    1. MS windows version from SourceForge
    Uncompress/utar it: tar -xvf rsync-3.0.7.tar.gz

    Now it's in rsync-3.0.7/


    rsync --daemon --help    
    rsync  version 3.0.7  protocol version 30
    Copyright (C) 1996-2009 by Andrew Tridgell, Wayne Davison, and others.
    Web site: http://rsync.samba.org/
    Capabilities:
        64-bit files, 64-bit inums, 32-bit timestamps, 64-bit long ints,
        socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
        append, no ACLs, xattrs, iconv, no symtimes
    
    rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
    are welcome to redistribute it under certain conditions.  See the GNU
    General Public Licence for details.
    
    Usage: rsync daemon [OPTION]...
    
         --address=ADDRESS       bind to the specified address (like when  the server has multiple addresses)
         --bwlimit=KBPS          limit I/O bandwidth; KBytes per second
         --config=FILE           default: /etc/rsyncd.conf 
         --no-detach             do not detach from the parent
         --port=PORT             port number (Instead of 873 check firewall)
         --log-file=FILE         
         --log-file-format=FMT   
         --sockopts=OPTIONS      custom TCP options like:
     -v, --verbose               increase verbosity
     -4, --ipv4                  prefer 
     -6, --ipv6                  
    
    When not invoking rsync as a daemon, avoid daemon-specific options.  
    

    FILES

    /etc/rsyncd.conf or rsyncd.conf

    Genuine documentation: Samba.org/rsync

    
           Rsync is a fast and versatile file  copying  tool.   It can  copy  locally,  to/from  another  host  over  any remote shell, or
           to/from a remote rsync daemon.  It offers a  large  number  of  options that  control  its behavior and permit flexible
           specification of the set of files to be copied.  Its delta-transfer  algorithm  reduces the amount of data sent over
           the network by sending only the differences between  the  source  files and  the  existing  files in the destination.  Rsync is used for
           backups and mirroring.
    

    Rsync finds files that need to be transferred using a "quick check" algorithm (by default): that have changed in size or last-modified time. Changes in the preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the file's data does not need to be updated.

    Some of the additional features of rsync are:

    1. support for copying links, devices, owners, groups, and permissions
    2. exclude and exclude-from options similar to GNU tar
    3. a CVS exclude mode for ignoring the same files that CVS would ignore ???????
    4. can use transparent remote shell, including ssh or rsh
    5. does not require super-user privileges
    6. pipelining of file transfers to minimize latency
    7. support for anonymous or authenticated rsync daemons (ideal for mirroring)
    Rsync copies files to or from a remote host, or locally (it does not copying files between two remote hosts).

    The different ways for rsync to contact a remote system:

    1. using a remote-shell program as the transport (such as ssh or rsh) when the source or destination path contains a single colon (:) separator after a host specification or
    2. contacting an rsync daemon directly via TCP double colon (::) separator after a host specification, OR when an rsync:// URL is specified (see also the "USING RSYNC-DAEMON FEATURES VIA A REMOTE-SHELL CONNECTION" section for an exception to this latter rule).

    If neither the source or destination path specify a remote host, the copy occurs locally (see also the --list-only option).

    Rsync refers to the local side as the "client" and the remote side as the "server" (an rsync daemon is always a server, but a server can be either a daemon or a remoteshell spawned process.)

    Setup

    See the file README for installation instructions.

    rsync transfers between any systems that you can access via a remote shell (as well as some that you can access using the rsync daemon) that have rsync installed For remote transfers, ssh is used. it may have been configured to use a different remote shell by default, such as rsh or remsh.

    specify remote shell using -e or by setting the RSYNC_RSH environment variable.

    Usage

    Examples:

    To transfer files in the current directory matching the pattern *.jpg to the system OutThere.com into the directory pictures .

    rsync *.jpg OutThere.com:pictures/


    Recursively (requires --archive) transfer files on Elsewhere.com in the directory pictures/party into /data/images/bar on the local machine.
    Preserving symbolic links, devices, attributes, permissions, ownerships, etc.(included with --archive)
    Output more descriptive messages then the default and
    Use compression to reduce the size of the transfer(not the resultant files).
    rsync --archive --verbose --compress Elsewhere.com:pictures/party /data/images

    A trailing slash on the source will use the same directory level at the destination. (a trailing / on a source means "copy the contents of this directory" as opposed to "copy the directory by name", but in both cases the attributes of the containing directory are transferred to the containing directory on the destination.

    rsync -avz foo:src/bar/ /data/tmp
    the following copy the files in the same way, including their setting of the attributes of /dest/foo:

    rsync --archive /src/foo  /dest
    rsync --archive /src/foo/ /dest/foo

    Host and module references don't require a trailing slash to copy the contents of the default directory.
    These are equivelevent:

    rsync -av host:        /dest
    rsync -av host::module /dest

    Using rsync in local-only mode, (neither the source nor destination contain a :) as an improved copy command.

    list all the (listable) modules a particular rsync daemon by leaving off the module name:

    rsync somehost.mydomain.com::

    Files that exist in the destination will be updated (if necessary) using remote-update protocol by sending only the differences. See the tech report for details.

    Advanced usage

    To transfer multiple files from a remote host specify additional remote-host args in the same style as the first, or with the hostname omitted. For instance

    rsync host:file1 :file2 host:file{3,4} /dest/
    rsync host::modname/file{1,2} host::modname/file3 /dest/
    rsync host::modname/file1 ::modname/file{3,4}
    .

    To transfer a filename that contains whitespace specify --protect-args (-s) , or escape the whitespace in a way that the remote shell will understand.
    For example:

    rsync host:'file\ name\ with\ spaces' /dest

    Running via a remote-shell connection

    Usefull where rsyncd is not running or available as a system daemon or if the server version is incompatible with client.
    This uses a remote shell and spawns a single-use server that reads its config file from the users' home directory.

    Unavailable features include chroot or change the uid used by the daemon.

    Differences between shell mode and daemon mode include:

    1. to separate the hostname from the path, or rsync:// URL.
      use a double colon :: for daemon mode, a single colon for shell mode
    2. first word of the "path" is a module in daemon mode, a directory in shell mode.
    3. ommiting path name on the remote daemon lists accessible paths on the daemon will be shown.
    4. ommiting the local destination a listing of the specified files on the remote daemon is provided.
    5. not specify --rsh (-e) for daemon mode

    copy all the files in a remote module named "src":

    rsync -av host::src /dest

    Modules can be configured to require password . To avoid the password prompt, set $RSYNC_PASSWORD or use --password-file

    Establish the connection via a web proxy by setting $RSYNC_PROXY to a hostname:port of the web proxy which must accept connections to port 873(default).

    Establish a daemon connection using a program as a proxy setting $RSYNC_CONNECT_PROG to the commands in place of making a direct socket connection.
    The string may contain %H to represent the hostname specified in the rsync command (use %% to specify a single % ).
    For example: Use ssh to run nc (netcat) on a proxyhost, which forwards all data to port 873 (the rsync daemon) on the targethost (%H).

    export RSYNC_CONNECT_PROG='ssh proxyhost nc %H 873'
    rsync -av targethost1::module/src/ /dest/
    rsync -av rsync:://targethost2/module/src/ /dest/

    A transfer via a remote-shell connection uses the same command-line except the remote program command-line must use --rsh=command
    . (Setting $RSYNC_RSH will not turn on this.)
    For example:

    rsync -av --rsh=ssh host::module /dest
    To specify a different remote user, use rsync-user@ prefix of the host (for a module that requires user-based authentication). Also use -l user to ssh when specifying the remote-shell, as in this example that uses the short version of --rsh :
    rsync -av -e "ssh -l ssh-user" rsync-user@host::module /dest

    ssh-user will be used at the ssh level; the rsync-user to log-in to the module.

    Examples

    To periodically backup the home directory, to jupiter use a cron job :
    rsync -Cavz . jupiter:backup

    To synchronize source trees use the following Makefile targets:

               get:
                       rsync -avuzb --exclude '*~' samba:samba/ .
               put:
                       rsync -Cavuzb . samba:samba/
               sync: get put
    
    
    allows syncing with a CVS directory . then do CVS operations on the remote machine, which saves a lot of time as the remote CVS protocol isn't very efficient.

    Mirror a directory between "old" and "new" ftp sites launched from cron every few hours. :

    rsync -az -e ssh --delete ~ftp/pub/samba nimbus:"~ftp/pub/tridge"


    --help
    
    -v, --verbose           increase verbosity
    -q, --quiet             suppress non-error messages
    --no-motd               suppress daemon-mode MOTD (see caveat)
    -c, --checksum          skip based on checksum, not mod-time & size
    -a, --archive           archive mode; equals -rlptgoD (no -H,-A,-X)
    --no-OPTION             turn off an implied OPTION (e.g. --no-D)
    -r, --recursive         recurse into directories
    -R, --relative          use relative path names
    --no-implied-dirs       don't send implied dirs with --relative
    -b, --backup            make backups (see --suffix & --backup-dir)
    --backup-dir=DIR        make backups into hierarchy based in DIR
    --suffix=SUFFIX         backup suffix (default ~ w/o --backup-dir)
    -u, --update            skip files that are newer on the receiver
    --inplace               update destination files in-place
    --append                append data onto shorter files
    --append-verify         --append w/old data in file checksum
    -d, --dirs              transfer directories without recursing
    -l, --links             copy symlinks as symlinks
    -L, --copy-links        transform symlink into referent file/dir
    --copy-unsafe-links     only "unsafe" symlinks are transformed
    --safe-links            ignore symlinks that point outside the tree
    -k, --copy-dirlinks     transform symlink to dir into referent dir
    -K, --keep-dirlinks     treat symlinked dir on receiver as dir
    -H, --hard-links        preserve hard links
    -p, --perms             preserve permissions
    -E, --executability     preserve executability
    --chmod=CHMOD           affect file and/or directory permissions
    -A, --acls              preserve ACLs (implies -p)
    -X, --xattrs            preserve extended attributes
    -o, --owner             preserve owner (super-user only)
    -g, --group             preserve group
    --devices               preserve device files (super-user only)
    --specials              preserve special files
    -D                      same as --devices --specials
    -t, --times             preserve modification times
    -O, --omit-dir-times    omit directories from --times
    --super                 receiver attempts super-user activities
    --fake-super            store/recover privileged attrs using xattrs
    -S, --sparse            handle sparse files efficiently
    -n, --dry-run           perform a trial run with no changes made
    -W, --whole-file        copy files whole (w/o delta-xfer algorithm)
    -x, --one-file-system   don't cross filesystem boundaries
    -B, --block-size=SIZE   force a fixed checksum block-size
    -e, --rsh=COMMAND       specify the remote shell to use
    --rsync-path=PROGRAM    specify the rsync to run on remote machine
    --existing              skip creating new files on receiver
    --ignore-existing       skip updating files that exist on receiver
    --remove-source-files   sender removes synchronized files (non-dir)
    --del                   an alias for --delete-during
    --delete                delete extraneous files from dest dirs
    --delete-before         receiver deletes before transfer (default)
    --delete-during         receiver deletes during xfer, not before
    --delete-delay          find deletions during, delete after
    --delete-after          receiver deletes after transfer, not before
    --delete-excluded       also delete excluded files from dest dirs
    --ignore-errors         delete even if there are I/O errors
    --force                 force deletion of dirs even if not empty
    --max-delete=NUM        don't delete more than NUM files
    --max-size=SIZE         don't transfer any file larger than SIZE
    --min-size=SIZE         don't transfer any file smaller than SIZE
    --partial               keep partially transferred files
    --partial-dir=DIR       put a partially transferred file into DIR
    --delay-updates         put all updated files into place at end
    
    
    
    
    
    -m, --prune-empty-dirs      prune empty directory chains from file-list
    --numeric-ids           don't map uid/gid values by user/group name
    --timeout=SECONDS       set I/O timeout in seconds
    --contimeout=SECONDS    set daemon connection timeout in seconds
    -I, --ignore-times          don't skip files that match size and time
    --size-only             skip files that match in size
    --modify-window=NUM     compare mod-times with reduced accuracy
    -T, --temp-dir=DIR          create temporary files in directory DIR
    -y, --fuzzy                 find similar file for basis if no dest file
    --compare-dest=DIR      also compare received files relative to DIR
    --copy-dest=DIR         ... and include copies of unchanged files
    --link-dest=DIR         hardlink to files in DIR when unchanged
    -z, --compress              compress file data during the transfer
    --compress-level=NUM    explicitly set compression level
    --skip-compress=LIST    skip compressing files with suffix in LIST
    -C, --cvs-exclude           auto-ignore files in the same way CVS does
    -f, --filter=RULE           add a file-filtering RULE
    -F                          same as --filter='dir-merge /.rsync-filter'
                            repeated: --filter='- .rsync-filter'
    --exclude=PATTERN       exclude files matching PATTERN
    --exclude-from=FILE     read exclude patterns from FILE
    --include=PATTERN       don't exclude files matching PATTERN
    --include-from=FILE     read include patterns from FILE
    --files-from=FILE       read list of source-file names from FILE
    -0, --from0                 all *from/filter files are delimited by 0s
    -s, --protect-args          no space-splitting; wildcard chars only
    --address=ADDRESS       bind address for outgoing socket to daemon
    --port=PORT             specify double-colon alternate port number
    --sockopts=OPTIONS      specify custom TCP options
    --blocking-io           use blocking I/O for the remote shell
    --stats                 give some file-transfer stats
    -8, --8-bit-output          leave high-bit chars unescaped in output
    -h, --human-readable        output numbers in a human-readable format
    --progress              show progress during transfer
    -P                          same as --partial --progress
    -i, --itemize-changes       output a change-summary for all updates
    --out-format=FORMAT     output updates using the specified FORMAT
    --log-file=FILE         log what we're doing to the specified FILE
    --log-file-format=FMT   log updates using the specified FMT
    --password-file=FILE    read daemon-access password from FILE
    --list-only             list the files instead of copying them
    --bwlimit=KBPS          limit I/O bandwidth; KBytes per second
    --write-batch=FILE      write a batched update to FILE
    --only-write-batch=FILE like --write-batch but w/o updating dest
    --read-batch=FILE       read a batched update from FILE
    --protocol=NUM          force an older protocol version to be used
    --iconv=CONVERT_SPEC    request charset conversion of filenames
    --checksum-seed=NUM     set block/file checksum seed (advanced)
    -4, --ipv4                  prefer IPv4
    -6, --ipv6                  prefer IPv6
    --version               print version number
       (-h) --help                  show this help (see below for -h comment)
    
    

    As a daemon
    --daemon                run as a daemon
    --address=ADDRESS       bind to the specified address
    --bwlimit=KBPS          limit I/O bandwidth; KBytes per second
    --config=FILE           alternate rsyncd.conf 
    --no-detach             
    --port=PORT             alternate port default 873
    --log-file=FILE         override the "log file" setting
    --log-file-format=FMT   override the "log format" setting
    --sockopts=OPTIONS      specify custom TCP options
    -v, --verbose               increase verbosity
    -4, --ipv4                  prefer IPv4
    -6, --ipv6                  prefer IPv6
    -h, --help                  show this help (if used after --daemon)
    

    FILTER RULES

    allow for flexible selection of which files to transfer (include) and skip (exclude).
    either directly specifng patterns or a way to acquire patterns (eg to read them from a file).

    As the list of files/directories to transfer is built, rsync checks each name to be transferred against the list of include/exclude patterns in turn, and the first matching pattern is acted on: if it is an if no matching pattern is found, then the filename is not skipped.

    Rsync builds an ordered list of filter rules as specified on the command-line.

    RULE [PATTERN_OR_FILENAME]
    RULE,MODIFIERS [PATTERN_OR_FILENAME]

    choose of using either short or long RULE names
    Short-named rule, the ',' separating the RULE from the MODIFIERS is optional.
    The PATTERN or FILENAME that follows (when present) must come after either a single space or an underscore (_).

    1. exclude, - specifies an exclude pattern.
    2. include, + specifies an include pattern.
    3. merge, . specifies a merge-file to read for more rules.
    4. dir-merge, : specifies a per-directory merge-file.
    5. hide, H specifies a pattern for hiding files from the transfer.
    6. show, S files that match the pattern are not hidden.
    7. protect, P specifies a pattern for protecting files from deletion.
    8. risk, R files that match the pattern are not protected.
    9. clear, ! clears the current include/exclude list (takes no arg)
    Empty lines are ignored, as are comment lines that start with a "#".

    --include/--exclude do not allow the full range of rule parsing, only specification of include/exclude patterns plus a "!" token to clear the list (and the normal comment parsing when rules are read from a file).
    If a pattern does not begin with "- " (dash, space) or "+ " (plus, space), then the rule will be interpreted as if "+ " (for an include option) or "- " (for an exclude option) were prefixed to the string.
    A --filter option, must always contain either a short or long rule name at the start of the rule.

    --filter, --include, and --exclude take one rule/pattern each. To add multiple ones, repeat the options on the command-line, use the merge-file syntax of the --filter option, or the --include-from/--exclude-from options.

    INCLUDE/EXCLUDE PATTERN RULES

    include and exclude files by specifying patterns using the "+", "-", etc. filter rules .
    The include/exclude rules each specify a pattern that is matched against the names of the files that are going to be transferred.

    Patterns forms:

    When using the --recursive (-r) option (implied by -a), every subcomponent of every path is visited from the top down, so include/exclude patterns get applied recursively to each subcomponent's full name (e.g. to include "/foo/bar/baz" the subcomponents "/foo" and "/foo/bar" must not be excluded).
    The exclude patterns shortcircuit the directory traversal stage when rsync finds the files to send.
    If a pattern excludes a particular parent directory, it can render a deeper include pattern ineffectual because rsync did not descend through that excluded section of the hierarchy.
    This is particularly important when using a trailing '*' rule.
    For instance, this won't work:

                  + /some/path/this-file-will-not-be-found
                  + /file-is-included
                  - * 
    Because the parent directory "some" is excluded by the '*' rule, so rsync never visits any of the files in the "some" or "some/path" directories. One solution is to ask for all directories in the hierarchy to be included by using a single rule: "+ */" (put it somewhere before the "- *" rule), and perhaps use the --prune-empty-dirs option. Another solution is to add specific include rules for all the parent dirs that need to be visited. For instance, this set of rules works fine:
                  + /some/
                  + /some/path/
                  + /some/path/this-file-is-found
                  + /file-also-included
                  - * 

    Examples of exclude/include matching:

    Modifiers are accepted after a + or -

    / the include/exclude rule should be matched against the absolute pathname of the current item.
    For example, "-/ /etc/passwd" would exclude the passwd file any time the transfer was sending files from the "/etc" directory, and "-/ subdir/foo" would always exclude "foo" when it is in a dir named "subdir", even if "foo" is at the root of the current transfer.
    ! the include/exclude should take effect if the pattern fails to match. For instance, "-! */" would exclude all non-directories.
    C indicates that all the global CVS-exclude rules should be inserted as excludes in place of the "-C". No arg should follow.
    s rule applies to the sending side. When a rule affects the sending side, it prevents files from being transferred. The default is for a rule to affect both sides unless --delete-excluded was specified, in which case default rules become sender-side only. See also the hide (H) and show (S) rules, which are an alternate way to specify sending-side includes/excludes.
    r rule applies to the receiving side. When a rule affects the receiving side, it prevents files from being deleted. See the s modifier for more info. See also the protect (P) and risk (R) rules, which are an alternate way to specify receiver-side includes/excludes.
    p rule is perishable, meaning that it is ignored in directories that are being deleted. For instance, the -C option's default rules that exclude things like "CVS" and "*.o" are marked as perishable, and will not prevent a directory that was removed on the source from being deleted on the destination.

    MERGE-FILE FILTER RULES

    Merge files into rules by specifying either a merge (.) or a dir-merge (:) filter rule

    There are two kinds of merged files -- single-instance ('.') and perdirectory (':'). A single-instance merge file is read one time, and its rules are incorporated into the filter list in the place of the "." rule. For per-directory merge files, rsync will scan every directory that it traverses for the named file, merging its contents when the file exists into the current list of inherited rules. These per-directory rule files must be created on the sending side because it is the sending side that is being scanned for the available files to transfer. These rule files may also need to be transferred to the receiving side if you want them to affect what files don't get deleted (see PER-DIRECTORY RULES AND DELETE below).

    Some examples:

                  merge /etc/rsync/default.rules
                  dir-merge .per-dir-filter
                  dir-merge,n- .non-inherited-per-dir-excludes
                  :n- .non-inherited-per-dir-excludes
    
    
    The following modifiers are accepted after a merge or dir-merge rule:
    - file should consist of only exclude patterns, with no other rule-parsing except for in-file comments.
    + file should consist of only include patterns, with no other rule-parsing except for in-file comments.
    C file should be read in a CVS compatible manner. This turns on 'n', 'w', and '-', allows the list-clearing token (!) to be specified. If no filename is provided, ".cvsignore" is assumed.
    e exclude the merge-file name from the transfer; e.g. "dir-merge,e .rules" is like "dir-merge .rules" and "- .rules".
    n rules are not inherited by subdirectories.
    w rules are word-split on whitespace instead of the normal line-splitting. also turns off comments.
    the space that separates the prefix from the rule is treated specially, so "- foo + bar" is parsed as two rules (assuming that prefix-parsing wasn't also disabled).
    specify any of the modifiers for the + or - rules to have the rules read in from the file default to having that modifier set. For instance, "merge,-/ .excl" would treat the contents of .excl as absolute path excludes, while "dir-merge,s .filt" and ":sC" would each make all their per-directory rules apply only on the sending side.
    Per-directory rules are inherited in all subdirectories of the directory where the merge-file was found unless the 'n' modifier was used. Each subdirectory's rules are prefixed to the inherited per-directory rules from its parents, which gives the newest rules a higher priority than the inherited rules. The entire set of dir-merge rules are grouped together in the spot where the merge-file was specified, so it is possible to override dir-merge rules via a rule that got specified earlier in the list of global rules. When the list-clearing rule ("!") is read from a per-directory file, it only clears the inherited rules for the current merge file.

    Another way to prevent a single rule from a dir-merge file from being inherited is to anchor it with a leading slash. Anchored rules in a per-directory merge-file are relative to the merge-file's directory, so a pattern "/foo" would only match the file "foo" in the directory where the dir-merge filter file was found. Here's an example filter file which you'd specify via --filter=". file":

                  merge /home/user/.global-filter
                  - *.gz
                  dir-merge .rules
                  + *.[ch]
                  - *.o
    
    
    This will merge the contents of the /home/user/.global-filter file at the start of the list and also turns the ".rules" filename into a per- directory filter file. All rules read in prior to the start of the directory scan follow the global anchoring rules (i.e. a leading slash matches at the root of the transfer).

    If a per-directory merge-file is specified with a path that is a parent directory of the first transfer directory, rsync will scan all the parent dirs from that starting point to the transfer directory for the indicated per-directory file. For instance, here is a common filter (see -F): --filter=': /.rsync-filter'

    That rule tells rsync to scan for the file .rsync-filter in all directories from the root down through the parent directory of the transfer prior to the start of the normal directory scan of the file in the directories that are sent as a part of the transfer. (Note: for an rsync daemon, the root is always the same as the module's "path".) Some examples of this pre-scanning for per-directory files:

                  rsync -avF /src/path/ /dest/dir
                  rsync -av --filter=': ../../.rsync-filter' /src/path/ /dest/dir
                  rsync -av --filter=': .rsync-filter' /src/path/ /dest/dir
    
    
    The first two commands above will look for ".rsync-filter" in "/" and "/src" before the normal scan begins looking for the file in "/src/path" and its subdirectories. The last command avoids the parent-dir scan and only looks for the ".rsync-filter" files in each directory that is a part of the transfer

    . If you want to include the contents of a ".cvsignore" in your patterns, you should use the rule ":C", which creates a dir-merge of the .cvsig- nore file, but parsed in a CVS-compatible manner. You can use this to affect where the --cvs-exclude (-C) option's inclusion of the per- directory .cvsignore file gets placed into your rules by putting the ":C" wherever you like in your filter rules. Without this, rsync would add the dir-merge rule for the .cvsignore file at the end of all your other rules (giving it a lower priority than your command-line rules). For example:

                  cat <<EOT | rsync -avC --filter='. -' a/ b
                  + foo.o
                  :C
                  - *.old
                  EOT
                  rsync -avC --include=foo.o -f :C --exclude='*.old' a/ b
    
    Above commands are identical. Each one will merge all the per-directory .cvsignore rules in the middle of the list rather than at the end. allows dir-specific rules to supersede the rules that follow the :C instead of being subservient to specified rules. To affect the other CVS exclude rules (i.e. the default list of exclusions, the contents of $HOME/.cvsignore, and the value of $CVSIGNORE) you should omit the -C command-line option and instead insert a "-C" rule into your filter rules; e.g. "--filter=-C".

    LIST-CLEARING filter rule

    Clear the current include/exclude list by using "!" filter rule. The "current" list is either the global list of rules (if the rule is encountered while parsing the filter options) or a set of per-directory rules (which are inherited in their own sub-list, so a subdirectory can use this to clear out the parent's rules).

    Anchoring INCLUDE/EXCLUDE patterns

    global include/exclude patterns are anchored at the "root of the transfer" (as opposed to per-directory patterns, which are anchored at the merge-file's directory). Consider of the transfer as a subtree of names , the transfer-root is where the tree starts to be duplicated in the destination directory. This root governs where patterns that start with a / match.

    Matching is relative to the transfer-root, changing the trailing slash on a source path or changing use of --relative affects the path to use in matching (in addition to changing how much of the file tree is duplicated on the destination host).

    To match two source files, one with an absolute path of "/home/me/foo/bar", and one with a path of "/home/you/bar/baz". Here is how the various command choices differ for a 2-source transfer:

     
    Example cmd: rsync -a /home/me /home/you /dest
                  +/- pattern: /me/foo/bar
                  +/- pattern: /you/bar/baz
                  Target file: /dest/me/foo/bar
                  Target file: /dest/you/bar/baz 
    
    Example cmd: rsync -a /home/me/ /home/you/ /dest
                  +/- pattern: /foo/bar               (note missing "me")
                  +/- pattern: /bar/baz               (note missing "you")
                  Target file: /dest/foo/bar
                  Target file: /dest/bar/baz
    
    Example cmd: rsync -a --relative /home/me/ /home/you /dest
                  +/- pattern: /home/me/foo/bar       (note full path)
                  +/- pattern: /home/you/bar/baz      (ditto)
                  Target file: /dest/home/me/foo/bar
                  Target file: /dest/home/you/bar/baz
    
    
    Example cmd: cd /home; rsync -a --relative me/foo you/ /dest
                  +/- pattern: /me/foo/bar      (starts at specified path)
                  +/- pattern: /you/bar/baz     (ditto)
                  Target file: /dest/me/foo/bar
                  Target file: /dest/you/bar/baz 
    
    To see what name to filter see the output with --verbose and put a / in front of the name (use --dry-run )

    Per-directory rules and delete

    Without a delete option, per-directory rules are only relevant on the sending side, so you can feel free to exclude the merge files themselves without affecting the transfer. To make this easy, the 'e' modifier adds this exclude , as seen in these two equivalent commands:
                  rsync -av --filter=': .excl' --exclude=.excl host:src/dir /dest
                  rsync -av --filter=':e .excl' host:src/dir /dest 
    
    To do a delete on the receiving side AND some files are to be excluded from being deleted, Include the per-directory merge files in the transfer and use --delete-after, this ensures that the receiving side gets all the same exclude rules as the sending side before it tries to delete anything:
                  rsync -avF --delete-after host:src/dir /dest 
    
    if the merge files are not a part of the transfer, either specify some global exclude rules (i.e. specified on the com- mand line), or maintain per-directory merge files on the receiving side. An example of the first is this (assume that the remote .rules files exclude themselves):
           rsync -av --filter=': .rules' --filter='. /my/extra.rules' --delete host:src/dir /dest
    
    In the above example extra.rules can affect both sides of the transfer, but (on the sending side) the rules are subservient to the rules merged from .rules because they were specified after the per-directory merge rule.

    The remote side is excluding the .rsync-filter files from the transfer, but to use our .rsync-filter to control what gets deleted on the receiving side. To do this specifically exclude the per-directory merge files (so that they don't get deleted) and then put rules into the local files to control what else should not get deleted.

     rsync -av --filter=':e /.rsync-filter' --delete host:src/dir /dest
     rsync -avFF --delete host:src/dir /dest 

    BATCH MODE

    Useful to apply the same set of updates to multiple systems, for example to replicated a tree on a number of hosts. If changes have been made to this source tree and those changes need to be propagated to the other hosts write-batch causes the client to store in a "batch file" all the information needed to repeat this operation against other, identical destination trees.

    Generating the batch file once, saves having to perform the file status, checksum, and data block generation more than once when updating multiple destination trees. Multicast transport protocols can be used to transfer the batch update files in parallel to many hosts at once, instead of sending the same data to every host individually.

    To apply the recorded changes to another destination tree, run rsync with the read-batch option, specifying the name of the same batch file, and the destination tree. Rsync updates the destination tree using the information stored in the batch file.

    A script file is created when the write batch option is used, named the same as the batch file with ".sh" appended. This script file contains a command-line suitable for updating a destination tree using the associated batch file. It can be executed using a Bourne (or Bourne-like) shell, optionally passing in an alternate destination tree pathname which is then used instead of the original destination path. This is useful when the destination tree path on the current host differs from the one used to create the batch file.

    Examples:

     $ rsync --write-batch=foo -a host:/source/dir/ /adest/dir/
                  $ scp foo* remote:
                  $ ssh remote ./foo.sh /bdest/dir/
    
                  $ rsync --write-batch=foo -a /source/dir/ /adest/dir/
                  $ ssh remote rsync --read-batch=- -a /bdest/dir/ <foo
    
    In these examples, rsync is used to update /adest/dir/ from /source/dir/ and the information to repeat this operation is stored in "foo" and "foo.sh". The host "remote" is then updated with the batched data going into the directory /bdest/dir. The differences between the two examples reveals some of the flexibility you have in how you deal with batches:
  • The first example the initial copy doesn't have to be local -- you can push or pull data to/from a remote host
  • The first example uses the created "foo.sh" file to get the right rsync options when running the read-batch command on the remote host.
  • The second example reads the batch data via standard input so that the batch file doesn't need to be copied to the remote machine first. This example avoids the foo.sh script because it needed to use a modified --read-batch , but you could edit the script file if you wished to make use of it (no other option is trying to use standard input, such as "--exclude-from=-" ).

    Caveats:

    read-batch expects the destination tree that it is updating to be identical to the destination tree that was used to create the batch update fileset. When a difference between the destination trees is encountered the update might be discarded with a warning (if the file appears to be up-to-date already) or the file-update may be attempted and then, if the file fails to verify, the update discarded with an error. This means that it should be safe to re-run a read- batch operation if the command got interrupted. If you wish to force the batched-update to always be attempted regardless of the file's size and date, use the -I option (when reading the batch). If an error occurs, the destination tree will probably be in a partially updated state. In that case, rsync can be used in its regular (non-batch) mode of operation to fix up the destination tree.

    The version used on all destinations must be at least as new as the one used to generate the batch file. See also the --protocol option for a way to have the creating rsync generate a batch file that an older rsync can understand. (Note that batch files changed format in version 2.6.3, so mixing versions older than that with newer versions will not work.)

    When reading a batch file, rsync will force the value of certain options to match the data in the batch file if you didn't set them to the same as the batch-writing command. Other options can (and should) be changed. For instance --write-batch changes to --read-batch, --files-from is dropped, and the --filter/--include/--exclude options are not needed unless one of the --delete options is specified.

    The code that creates the BATCH.sh file transforms any filter/include/exclude options into a single list that is appended as a "here" document to the shell script file. An advanced user can use this to modify the exclude list if a change in what gets deleted by --delete is desired.
    A normal user can ignore this detail and just use the shell script as an easy way to run the appropriate --read-batch command for the batched data.

    Symbolic Links

    Default: symlinks are not transferred with message: "skipping non-regular" file

    With --links symlinks are recreated with the same target on the destination. ( --archive implies --links.

    With --copy-links symlinks are "collapsed" by copying their referent

    Rsync can also distinguish "safe" and "unsafe" symbolic links. For example a web site mirror with symbolic links to /etc/passwd in the public section of the site.
    --copy-unsafe-links will cause any links to be copied as the file they point to on the destination.
    --safe-links will cause unsafe links to be omitted . specify --links as well.

    Symbolic links are unsafe if they are
      absolute symlinks (start with /),
      empty, or
      if they contain enough ".." components to ascend from the directory being copied.

    Summary of how the symlink options are interpreted. In order of precedence.

    --copy-links replace symlinks into normal files (leaving no symlinks for any other options to affect).
    --links --copy-unsafe-links Turn all unsafe symlinks into files and duplicate all safe symlinks.
    --copy-unsafe-links replace unsafe symlinks into files, noisily skip all safe symlinks.
    --links --safe-links Duplicate safe symlinks and skip unsafe ones.
    --links Duplicate all symlinks.
    Regarding --perms to make this latter behavior easier to type, define a popt alias for it, such as putting this line in the file ~/.popt (the -Z option, and includes --no-g to use the default group of the destination dir):

    "skipping non-regular" file : symlinks are not transferred

    EXIT VALUES

           0      Success 
           1      Syntax or usage error 
           2      Protocol incompatibility 
           3      Errors selecting input/output files, dirs 
           4      Requested action not supported: an attempt was made  to  manipulate  64-bit files on a platform that cannot support them; or an
                  option was specified that is supported by the client and not  by the server.  
           5      Error starting client-server protocol 
           6      Daemon unable to append to log-file 
           10     Error in socket I/O 
           11     Error in file I/O 
           12     Error in rsync protocol data stream 
           13     Errors with program diagnostics 
           14     Error in IPC code 
           20     Received SIGUSR1 or SIGINT 
           21     Some error returned by waitpid() 
           22     Error allocating core memory buffers 
           23     Partial transfer due to error 
           24     Partial transfer due to vanished source files 
           25     The --max-delete limit stopped deletions 
           30     Timeout in data send/receive 
           35     Timeout waiting for daemon connection
    

    ENVIRONMENT VARIABLES

    CVSIGNORE supplements any ignore patterns .cvsignore . See --cvs-exclude
    RSYNC_ICONV default --iconv setting
    RSYNC_RSH override the default shell used as the transport for rsync. Command line options are permitted after the command name, as in -e .
    RSYNC_PROXY client use a web proxy when connecting to a hostname:port
    RSYNC_PASSWORD allow running authenticated connections to an daemon without user intervention. This does not supply a password to a remote shell transport such as ssh; , consult the remote shell's documentation.
    USER or LOGNAME used to determine the default username sent to an rsync daemon. Default: nobody
    HOME location of user's default .cvsignore .

    FileS

    /etc/rsyncd.conf or rsyncd.conf

    See

    rsyncd.conf(5)

    Bugs/unexpected behavories

    Times are transferred as *nix time_t values

    When transferring to FAT filesystems rsync may re-sync unmodified files. See code>--modify-window .

    File permissions, devices, etc. are transferred as native numerical values see --delete

    Version

    This man page is for 3.0.7 ; as of 11/22/17 the current version is 3.1.2

    Internal Options

    --server and --sender are used internally by rsync, and should never be typed by a user under normal circumstances. Some awareness of these options may be needed in certain scenarios, such as when setting up a login that can only run an rsync command. For instance, the support directory of the rsync distribution has an example script named rrsync (for restricted rsync) that can be used with a restricted ssh login.

    Go to the /http://rsync.samba.org/ for the Credits, Thanks and Author..

    Mailing lists.samba.org

    Interesting notes from : How Rsync Works A Practical Overview

    1. The sender creates the file list including pathnames, ownership, mode, permissions, size, modtime and checksum (If --checksum has been specified).
      The list is transmitted as it is created.
    2. both sides sort the list and files are then refered to by index.
    3. sender transfers idname tables for users and groups
    4. The receiver traverses the existing file directory, comparing files to the filelist, creating directorys and identifing entries that will not need to be transfered
    5. The receiver creates block checksums of files needing to be transfered and provides them to the sender
    6. The sender generates a "rolling checksum" of a file and determins if a block already exists on the receiver. This provides for dynamic block creation on the sender which is effective when updates insert/delete information into the file. Consider the case of the addition of 1 byte near the beginning of the file. Fixed block algorithm to determine that all blockd would need to be transfered. 31 Dec 2009 rsync(1)

      versions

      It is unfortunate that many hosting services do not maintain current versions of many programs.
      as of 01/04/18 the current version is from 12/21/15 i.e. 2 years ago.
      Mac os
      rsync version 2.6.9 protocol version 29 (who would have guessed !) Copyright (C) 1996-2006 by Andrew Tridgell, Wayne Davison, and others. Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles, inplace, IPv6, 64-bit system inums, 64-bit internal inums
      GoDaddy 
      rsync version 3.0.6 protocol version 30 (5/8/09 8 ½ years old) Copyright (C) 1996-2009 by Andrew Tridgell, Wayne Davison, and others. Web site: http://rsync.samba.org/ Capabilities: 64-bit files, 64-bit inums, 32-bit timestamps, 64-bit long ints, socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace, append, ACLs, xattrs, iconv, no symtimes
      midphase
      rsync  version 3.0.6  protocol version 30(5/8/09 8 ½ years old)
      Copyright (C) 1996-2009 by Andrew Tridgell, Wayne Davison, and others.
      Web site: http://rsync.samba.org/
      Capabilities:
          64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
          socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
          append, ACLs, xattrs, iconv, symtimes
      
      onion omega
      LEDE
      WRT
       rsync  version 3.1.2  protocol version 31
      Copyright (C) 1996-2015 by Andrew Tridgell, Wayne Davison, and others.
      Web site: http://rsync.samba.org/
      Capabilities:
          64-bit files, 64-bit inums, 32-bit timestamps, 64-bit long ints,
          no socketpairs, hardlinks, symlinks, no IPv6, batchfiles, inplace,
          append, no ACLs, no xattrs, iconv, symtimes, prealloc
      
      
      
      A very complete "cronable" script
      script, ftp rename profiles, ftp restore profiles