git @ Cat's Eye Technologies ellsync / 6d48dbd
Remove `syncdirs` command. Chris Pressey 2 years ago
3 changed file(s) with 28 addition(s) and 95 deletion(s). Raw diff Collapse all Expand all
5252 are bona fide changes, but any change to the contents of the cache can be
5353 discarded.
5454
55 ### `syncdirs` command
55 ### `sync` command
5656
5757 With the above router saved as `router.json` we can then say
5858
59 ellsync router.json syncdirs /home/user/art/ /media/user/External1/art/
59 ellsync router.json sync art:
6060
6161 and this will in effect run
6262
63 rsync --archive --verbose --delete --checksum --dry-run /home/user/art/ /media/user/External1/art/
63 rsync --archive --verbose --delete --dry-run /home/user/art/ /media/user/External1/art/
6464
6565 Note that by default it only runs a `--dry-run`. It's a good practice to
6666 do a dry run first, to see what will be changed. As a bonus, the files
6767 involved will often remain in the filesystem cache, meaning a subsequent
6868 actual run will go quite quickly. To do that actual run, use `--apply`:
6969
70 ellsync router.json syncdirs /home/user/art/ /media/user/External1/art/ --apply
70 ellsync router.json sync art: --apply
7171
72 Note that if we try
73
74 ellsync router.json syncdirs /media/user/External1/art/ /home/user/art/
75
76 we will be prevented, because it is an error, because the direction of
77 the backup stream is always from canonical to cache.
78
79 Various other configurations are prevented. You may have noticed that `rsync`
80 is sensitive about whether a directory name ends in a slash or not. `ellsync`
81 detects when a trailing slash is missing and adds it. Thus
82
83 ellsync router.json syncdirs /media/user/External1/art /home/user/art/
84
85 is still interpreted as
86
87 rsync --archive --verbose --delete --checksum --dry-run /home/user/art/ /media/user/External1/art/
88
89 (but note that the directories in the router do need to have the
90 trailing slashes.)
91
92 Also, ince the contents of the canonical and the cache normally
72 Note that, since the contents of the canonical and the cache normally
9373 have the same directory structure, `ellsync` allows specifying that
9474 only a subdirectory of a stream is to be synced:
9575
96 ellsync router.json syncdirs /home/user/art/painting/ /media/user/External1/art/painting/
76 ellsync router.json sync art:painting/oil/ --apply
9777
98 This is of course allowed only as long as it is the same subdirectory.
99 This will fail:
78 While `rsync` is sensitive about whether a directory name ends in a slash or
79 not, `ellsync` detects when a trailing slash is missing and adds it. Thus
10080
101 ellsync router.json syncdirs /home/user/art/painting/ /media/user/External1/art/sculpture/
81 ellsync router.json sync art:painting/oil --apply
82
83 will work as well as the above. (But note that the directories specified
84 in the router *do* need to have the trailing slashes.)
85
86 #### --thorough
10287
10388 By default, `rsync` does not attempt to sync the contents of an existing file
10489 if the destination file has a same-or-newer timestamp as the source file.
10792 uncommon occurrence on inexpensive removable media), `rsync` will not attempt
10893 to repair the corruption, as the timestamp of the corrupted file did not change.
10994
110 To compensate for this, `ellsync` invokes `rsync` with the `--checksum` flag, to
111 force it to do a thorough check of the files. See `man rsync` for more details.
95 To compensate for this, `ellsync` provides the `--thorough` option:
96
97 ellsync router.json sync art:painting/oil --thorough
98
99 This invokes `rsync` with the `--checksum` flag, to force it to do a thorough
100 check of the files. See `man rsync` for more details.
112101
113102 ### `list` command
114103
119108 subcommand to list which streams are, at the moment, backupable:
120109
121110 ellsync router.json list
122
123 ### `sync` command
124
125 Since each stream configuration is named in the router, we don't even have to
126 give these directory names. We can use the `sync` command where we give
127 just the name of the stream, followed by a colon:
128
129 ellsync router.json sync art:
130
131 The `sync` command syntax allows specifying that only a subdirectory of a
132 stream is to be synced, by giving the subdirectory name after the colon:
133
134 ellsync router.json sync art:painting/
135
136 Like `syncdirs`, `sync` understands the `--apply` and `--thorough` options.
137111
138112 ### `rename` command
139113
187161 Argument parser was refactored to use subparsers, improving usage info and usage
188162 error output.
189163
164 Removed `syncdirs` as it introduces some redundancy and I never use it.
165
190166 After `sync` is performed, the system `sync` command is run, to ensure all buffers
191167 are flushed to devices before the `ellsync` tool actually exits.
192168
193 `rsync` is now invoked with `--checksum` flag to cause it to thoroughly check if
194 files differ, even if their datestamps have not changed.
169 The `--thorough` options now invokes `rsync` with `--checksum` flag, to cause it
170 to thoroughly check if files differ, even if their datestamps have not changed.
195171
196172 Added `--stream-name-only` option to `list` command.
197173
7171 perform_sync(from_dir, to_dir, dry_run=(not options.apply), checksum=(options.thorough))
7272
7373
74 def syncdirs(router, options):
75 from_dir = clean_dir(options.from_dir)
76 to_dir = clean_dir(options.to_dir)
77 selected_stream_name = None
78 for stream_name, stream in router.items():
79 if from_dir.startswith(stream['from']) and to_dir.startswith(stream['to']):
80 from_suffix = from_dir[len(stream['from']):]
81 to_suffix = to_dir[len(stream['to']):]
82 if from_suffix != to_suffix:
83 raise ValueError( (from_suffix, to_suffix) )
84 selected_stream_name = stream_name
85 break
86 if selected_stream_name is None:
87 raise ValueError("Stream {} => {} was not found in router".format(from_dir, to_dir))
88
89 perform_sync(from_dir, to_dir, dry_run=(not options.apply))
90
91
9274 def rename(router, options):
9375 stream_name = options.stream_name
9476 if ':' in stream_name:
154136 )
155137 parser_sync.set_defaults(func=sync)
156138
157 # - - - - syncdirs - - - -
158 parser_syncdirs = subparsers.add_parser(
159 'syncdirs', help='Sync contents across a sync stream specified by source and dest directories'
160 )
161 parser_syncdirs.add_argument('from_dir', metavar='FROM_DIR', type=str,
162 help='Canonical directory to sync contents from'
163 )
164 parser_syncdirs.add_argument('to_dir', metavar='TO_DIR', type=str,
165 help='Cache directory to sync contents to'
166 )
167 parser_syncdirs.add_argument('--apply', default=False, action='store_true',
168 help='Actually run the rsync command'
169 )
170 parser_syncdirs.set_defaults(func=syncdirs)
171
172139 # - - - - rename - - - -
173140 parser_rename = subparsers.add_parser(
174141 'rename', help='Rename a subdirectory in both source and dest of sync stream'
5252 with self.assertRaises(SystemExit):
5353 main(['backup.json'])
5454
55 def test_dry_run(self):
56 main(['backup.json', 'syncdirs', 'canonical', 'cache'])
55 def test_sync_dry_run(self):
56 main(['backup.json', 'sync', 'basic:'])
5757 self.assertFalse(os.path.exists('cache/thing'))
5858 output = sys.stdout.getvalue()
5959 self.assertEqual(output.split('\n')[0], 'rsync --dry-run --archive --verbose --delete "canonical/" "cache/"')
6060 self.assertIn('DRY RUN', output)
6161
62 def test_apply(self):
63 main(['backup.json', 'syncdirs', 'canonical', 'cache', '--apply'])
62 def test_sync_apply(self):
63 main(['backup.json', 'sync', 'basic:', '--apply'])
6464 self.assertTrue(os.path.exists('cache/thing'))
65 output = sys.stdout.getvalue()
66 self.assertEqual(output.split('\n')[:4], [
67 'rsync --archive --verbose --delete "canonical/" "cache/"',
68 'sending incremental file list',
69 'thing',
70 ''
71 ])
72
73 def test_stream(self):
74 main(['backup.json', 'sync', 'basic:', '--apply'])
7565 output = sys.stdout.getvalue()
7666 self.assertEqual(output.split('\n')[:4], [
7767 'rsync --archive --verbose --delete "canonical/" "cache/"',