ellsync
Version 0.6 | Entry @ catseye.tc | See also: yastasoti ∘ tagfarm ∘ shelf
ellsync
is an opinionated poka-yoke for [rsync
][].
- opinionated: it was designed for a particular use case for
rsync
(offline backups). - poka-yoke: it exposes a restricted interface to
rsync
, which prevents using it in dangerous ways.
Because the restricted interface that ellsync
presents can be accessed
by shorthand form, it also happens to provide some convenience over
using rsync
directly — but its real purpose is to increase safety.
(I've been burned more than once when I've made a mistake using rsync
.)
Quick start
Make sure you have Python (2.7 or 3.x) installed, clone this repository,
and put its bin
directory on your executable search path. You will
then be able to run ellsync
from your terminal.
Usage guide
Backup router
ellsync
's operation is based on a backup router which is a JSON file
that looks like this:
{
"art": {
"from": "/media/user/External1/art/",
"to": "/home/user/art/"
}
}
In this, art
is the name of a backup stream, in which files in
/media/user/External1/art/
(called the canonical) are periodically
synced to /home/user/art/
(called the cache).
The idea is that all changes to the contents of the canonical directory are bona fide changes, but any change to the contents of the cache can be discarded.
sync
command
With the above router saved as router.json
we can then say
ellsync router.json sync art
and this will in effect run
rsync --archive --verbose --delete --dry-run /home/user/art/ /media/user/External1/art/
Note that by default it only runs a --dry-run
. It's a good practice to
do a dry run first, to see what will be changed. As a bonus, the files
involved will often remain in the filesystem cache, meaning a subsequent
actual run will go quite quickly. To do that actual run, use --apply
:
ellsync router.json sync art --apply
Note that, since the contents of the canonical and the cache normally
have the same directory structure, ellsync
allows specifying that
only a subdirectory of a stream is to be synced:
ellsync router.json sync art:painting/oil/ --apply
While rsync
is sensitive about whether a directory name ends in a slash or
not, ellsync
detects when a trailing slash is missing and adds it. Thus
ellsync router.json sync art:painting/oil --apply
will work as well as the above. (But note that the directories specified in the router do need to have the trailing slashes.)
--thorough
option
By default, rsync
does not attempt to sync the contents of an existing file
if the destination file has a same-or-newer timestamp as the source file.
However, this means that if the destination file has become corrupted (a not-
uncommon occurrence on inexpensive removable media), rsync
will not attempt
to repair the corruption, as the timestamp of the corrupted file did not change.
To compensate for this, ellsync
provides the --thorough
option:
ellsync router.json sync art:painting/oil --thorough
This invokes rsync
with the --checksum
flag, to force it to do a thorough
check of the files. See man rsync
for more details.
--reverse
option
This flag causes the meaning of the from
and the to
of the selected
route to be flipped. However, to prevent this from being done accidentally
(which could be catastrophic, file-backup-wise), this action must be confirmed
by the user, and this confirmation takes the form of requiring that a file
called .reverse-to-here
must be present in the (non-usual) destination
directory before the reverse sync takes place. In practice, the user runs
touch destination/.reverse-to-here
before running the reverse sync, and
the reverse sync deletes the .reverse-to-here
file as part of its operation.
verify
command
This provides a more robust and flexible option than the --thorough
option
to sync
for verifying that the contents on both sides of the stream are
identical.
ellsync router.json sync art:painting
This creates a list of all files under /media/user/External1/art/painting/
in a text file in your temporary directory. It then goes through each file in
that list and runs diff
on it, comparing it to its corresponding file in
/home/user/art/painting/
. If there are any differences, the name of the
file is printed out.
The reason the filenames are written to a text file is that for huge filesets,
or on slow media (or both), verification of all files can take a long, long
time. The user may wish to interrupt it before it is done, and continue it
later. Persisting the list of files to a text file supports this. If the
text file already exists, the user can supply the --continue-from
option
naming a file to continue the verification process from. All files up until
that file in the existing text file will be skipped.
list
command
Either the canonical or the cache (or both) may be offline storage (removable
media), therefore neither directory is assumed to exist (it might not exist
if the volume is not mounted.) If either of the directories does not exist,
ellsync
will refuse to use this backup stream. Based on this, there is a
subcommand to list which streams are, at the moment, backupable:
ellsync router.json list
In practice, this command is often the command you run first, before running
sync
, as it will let you know what backup streams are available to sync.
rename
command
Sometimes you want to rename a subdirectory somewhere under the canonical of
one of the streams. It's completely fine to do this, but the next time it is synced,
rsync
will treat it, in the cache, as the old subdirectory being deleted and
a new subdirectory being created. If there are a large number of files in the
subdirectory, this delete-and-create sync can take a long time. It's also not
obvious from rsync
's logging output that everything being deleted is also being
created somewhere else.
To ease this situation, ellsync
has a rename
command that works like so:
ellsync router.json rename art: sclupture sculpture
This renames the /media/user/External1/art/sclupture
directory to
/media/user/External1/art/sculpture
and also renames the /home/user/art/sclupture
directory to /home/user/art/sculpture
. If the contents of the source and
destination directories were in sync before this rename occurred, they will
continue to be in sync after the rename happens.
Hints and Tips
You might have a router you use almost always, in which case you might want to establish an alias like
alias myellsync ellsync $HOME/my-standard-router.json
(or whatever.)
TODO
- If
rsync
encounters an error, it will abort, having only partially completed. In particular, if it encounters a directory which it cannot read, because it is for example owned by another user and not world-readable, it will abort.ellsync
does not currently detect this properly. It should be made to handle it gracefully, if possible. - (Aspirational) Ability to convert the backup router to a
dot
file (graphviz
) so that the relationships between the streams can be easily visualized.
History
0.6
Added the verify
subcommand.
Added the --reverse
mode of operation to the sync
subcommand.
0.5
The output of the list
subcommand is now sorted by stream name.
The sync
subcommand now supports multiple streams. Each stream will be synced
in the order they are given on the command line. OS-level sync
will only be
performed once, at the very end.
A bash tab-completion script is included in the script
directory. It enables
tab-completion of both subcommand names, and stream names in the sync
subcommand.
Internally, shell expansion is no longer used when executing system commands, and several new tests have been added to the test suite.
0.4
The :
in a backup stream identifier is optional, when no subdirectory is being
specified.
0.3
Argument parser was refactored to use subparsers, improving usage info and usage error output.
Removed syncdirs
as it introduces some redundancy and I never use it.
After sync
is performed, the system sync
command is run, to ensure all buffers
are flushed to devices before the ellsync
tool actually exits.
The --thorough
options now invokes rsync
with --checksum
flag, to cause it
to thoroughly check if files differ, even if their datestamps have not changed.
Added --stream-name-only
option to list
command.
0.2
Every ellsync
functionality has an explicit subcommand (list
and sync
to
start.)
sync
was split into sync
(takes a stream) and syncdirs
(takes to and
from dirs).
Added rename
command.
0.1
Initial release.
Commit History
@0.6
git clone https://git.catseye.tc/ellsync/
- Use system's temp dir name. Chris Pressey 6 months ago
- List verification failures and delete manifest file at end of run. Chris Pressey 7 months ago
- Add documentation for `verify` subcommand. Chris Pressey 7 months ago
- Derive tmp filename better and fix up running under Python 2.7. Chris Pressey 7 months ago
- Improve tests and allow run_command to be run quietly Chris Pressey 7 months ago
- Fix traverse_directories, refactor, actually shell out to diff Chris Pressey 7 months ago
- Add --manifest command-line option for ellsync verify. Chris Pressey 7 months ago
- Write a unit test to exercise this new code. Chris Pressey 7 months ago
- Allow tests to pass on Python 2.7. Chris Pressey 7 months ago
- Keep on sketchin' Chris Pressey 7 months ago