git @ Cat's Eye Technologies ellsync / 0.3
Merge pull request #2 from cpressey/develop-0.3 Develop 0.3 Chris Pressey authored 10 months ago GitHub committed 10 months ago
3 changed file(s) with 119 addition(s) and 139 deletion(s). Raw diff Collapse all Expand all
00 `ellsync`
11 =========
22
3 _Version 0.2_
3 _Version 0.3_
44 | _Entry_ [@ catseye.tc](https://catseye.tc/node/ellsync)
55 | _See also:_ [yastasoti](https://github.com/catseye/yastasoti#readme)
66 ∘ [tagfarm](https://github.com/catseye/tagfarm#readme)
5252 are bona fide changes, but any change to the contents of the cache can be
5353 discarded.
5454
55 ### `syncdirs` command
55 ### `sync` command
5656
5757 With the above router saved as `router.json` we can then say
5858
59 ellsync router.json syncdirs /home/user/art/ /media/user/External1/art/
59 ellsync router.json sync art:
6060
6161 and this will in effect run
6262
6767 involved will often remain in the filesystem cache, meaning a subsequent
6868 actual run will go quite quickly. To do that actual run, use `--apply`:
6969
70 ellsync router.json syncdirs /home/user/art/ /media/user/External1/art/ --apply
70 ellsync router.json sync art: --apply
7171
72 Note that if we try
73
74 ellsync router.json syncdirs /media/user/External1/art/ /home/user/art/
75
76 we will be prevented, because it is an error, because the direction of
77 the backup stream is always from canonical to cache.
78
79 Various other configurations are prevented. You may have noticed that `rsync`
80 is sensitive about whether a directory name ends in a slash or not. `ellsync`
81 detects when a trailing slash is missing and adds it. Thus
82
83 ellsync router.json syncdirs /media/user/External1/art /home/user/art/
84
85 is still interpreted as
86
87 rsync --archive --verbose --delete /home/user/art/ /media/user/External1/art/
88
89 (but note that the directories in the router do need to have the
90 trailing slashes.)
91
92 Also, ince the contents of the canonical and the cache normally
72 Note that, since the contents of the canonical and the cache normally
9373 have the same directory structure, `ellsync` allows specifying that
9474 only a subdirectory of a stream is to be synced:
9575
96 ellsync router.json syncdirs /home/user/art/painting/ /media/user/External1/art/painting/
76 ellsync router.json sync art:painting/oil/ --apply
9777
98 This is of course allowed only as long as it is the same subdirectory.
99 This will fail:
78 While `rsync` is sensitive about whether a directory name ends in a slash or
79 not, `ellsync` detects when a trailing slash is missing and adds it. Thus
10080
101 ellsync router.json syncdirs /home/user/art/painting/ /media/user/External1/art/sculpture/
81 ellsync router.json sync art:painting/oil --apply
82
83 will work as well as the above. (But note that the directories specified
84 in the router *do* need to have the trailing slashes.)
85
86 #### --thorough
87
88 By default, `rsync` does not attempt to sync the contents of an existing file
89 if the destination file has a same-or-newer timestamp as the source file.
90
91 However, this means that if the destination file has become corrupted (a not-
92 uncommon occurrence on inexpensive removable media), `rsync` will not attempt
93 to repair the corruption, as the timestamp of the corrupted file did not change.
94
95 To compensate for this, `ellsync` provides the `--thorough` option:
96
97 ellsync router.json sync art:painting/oil --thorough
98
99 This invokes `rsync` with the `--checksum` flag, to force it to do a thorough
100 check of the files. See `man rsync` for more details.
102101
103102 ### `list` command
104103
109108 subcommand to list which streams are, at the moment, backupable:
110109
111110 ellsync router.json list
112
113 ### `sync` command
114
115 Since each stream configuration is named in the router, we don't even have to
116 give these directory names. We can use the `sync` command where we give
117 just the name of the stream, followed by a colon:
118
119 ellsync router.json sync art:
120
121 The `sync` command syntax allows specifying that only a subdirectory of a
122 stream is to be synced, by giving the subdirectory name after the colon:
123
124 ellsync router.json sync art:painting/
125111
126112 ### `rename` command
127113
153139
154140 (or whatever.)
155141
156 Notes
157 -----
142 TODO
143 ----
158144
159 If `rsync` encounters an error, it will abort, having only partially completed.
160 In particular, if it encounters a directory which it cannot read, because it
161 is for example owned by another user and not world-readable, it will abort.
162 `ellsync` does not currently detect this properly (if it is detectable (I hope
163 that it is!))
145 * If `rsync` encounters an error, it will abort, having only partially completed.
146 In particular, if it encounters a directory which it cannot read, because it
147 is for example owned by another user and not world-readable, it will abort.
148 `ellsync` does not currently detect this properly. It should be made to handle
149 it gracefully, if possible.
150 * Tab-completion of stream names.
151 * Better test case for `--thorough`.
152 * When executing system commands, don't use shell expansion.
153 * (Aspirational) Ability to convert the backup router to a `dot` file (`graphviz`)
154 so that the relationships between the streams can be easily visualized.
164155
165156 History
166157 -------
158
159 ### 0.3
160
161 Argument parser was refactored to use subparsers, improving usage info and usage
162 error output.
163
164 Removed `syncdirs` as it introduces some redundancy and I never use it.
165
166 After `sync` is performed, the system `sync` command is run, to ensure all buffers
167 are flushed to devices before the `ellsync` tool actually exits.
168
169 The `--thorough` options now invokes `rsync` with `--checksum` flag, to cause it
170 to thoroughly check if files differ, even if their datestamps have not changed.
171
172 Added `--stream-name-only` option to `list` command.
167173
168174 ### 0.2
169175
1313 return dirname
1414
1515
16 def perform_sync(from_dir, to_dir, dry_run=True):
17 for d in (from_dir, to_dir):
18 if not os.path.isdir(d):
19 raise ValueError("Directory '{}' is not present".format(d))
20 rsync_options = '--dry-run ' if dry_run else ''
21 cmd = 'rsync {}--archive --verbose --delete "{}" "{}"'.format(rsync_options, from_dir, to_dir)
16 def run_command(cmd):
2217 sys.stdout.write(cmd + '\n')
2318 try:
2419 p = Popen(cmd, shell=True, stderr=STDOUT, stdout=PIPE, encoding='utf-8')
3732 # - - - - commands - - - -
3833
3934
40 def list_(router, args):
35 def list_(router, options):
4136 for stream_name, stream in router.items():
4237 if os.path.isdir(stream['from']) and os.path.isdir(stream['to']):
43 print("{}: {} => {}".format(stream_name, stream['from'], stream['to']))
38 if options.stream_name_only:
39 print(stream_name)
40 else:
41 print("{}: {} => {}".format(stream_name, stream['from'], stream['to']))
4442
4543
46 def sync(router, args):
47 argparser = ArgumentParser()
48 argparser.add_argument('stream_name', metavar='STREAM', type=str,
49 help='Name of stream (or stream:subdirectory) to sync contents across'
50 )
51 argparser.add_argument('--apply', default=False, action='store_true',
52 help='Actually run the rsync command'
53 )
54 options = argparser.parse_args(args)
55
44 def sync(router, options):
5645 if ':' in options.stream_name:
5746 stream_name, subdir = options.stream_name.split(':')
5847 else:
6756 from_dir = clean_dir(from_dir)
6857 to_dir = clean_dir(to_dir)
6958
70 perform_sync(from_dir, to_dir, dry_run=(not options.apply))
59 for d in (from_dir, to_dir):
60 if not os.path.isdir(d):
61 raise ValueError("Directory '{}' is not present".format(d))
62
63 dry_run = not options.apply
64 dry_run_option = '--dry-run ' if dry_run else ''
65 checksum_option = '--checksum ' if options.thorough else ''
66 cmd = 'rsync {}{}--archive --verbose --delete "{}" "{}"'.format(dry_run_option, checksum_option, from_dir, to_dir)
67 run_command(cmd)
68 if not dry_run:
69 run_command('sync')
7170
7271
73 def syncdirs(router, args):
74 argparser = ArgumentParser()
75 argparser.add_argument('from_dir', metavar='FROM_DIR', type=str,
76 help='Canonical directory to sync contents from, or name of stream to use'
77 )
78 argparser.add_argument('to_dir', metavar='TO_DIR', nargs='?', default=None, type=str,
79 help='Cache directory to sync contents to (only required when canonical dir, '
80 'not stream, was specified)'
81 )
82 argparser.add_argument('--apply', default=False, action='store_true',
83 help='Actually run the rsync command'
84 )
85 options = argparser.parse_args(args)
86
87 from_dir = clean_dir(options.from_dir)
88 to_dir = clean_dir(options.to_dir)
89 selected_stream_name = None
90 for stream_name, stream in router.items():
91 if from_dir.startswith(stream['from']) and to_dir.startswith(stream['to']):
92 from_suffix = from_dir[len(stream['from']):]
93 to_suffix = to_dir[len(stream['to']):]
94 if from_suffix != to_suffix:
95 raise ValueError( (from_suffix, to_suffix) )
96 selected_stream_name = stream_name
97 break
98 if selected_stream_name is None:
99 raise ValueError("Stream {} => {} was not found in router".format(from_dir, to_dir))
100
101 perform_sync(from_dir, to_dir, dry_run=(not options.apply))
102
103
104 def rename(router, args):
105 argparser = ArgumentParser()
106 argparser.add_argument('stream_name', metavar='STREAM', type=str,
107 help='Name of stream to operate under'
108 )
109 argparser.add_argument('existing_subdir_name', metavar='DIRNAME', type=str,
110 help='Existing subdirectory to be renamed'
111 )
112 argparser.add_argument('new_subdir_name', metavar='DIRNAME', type=str,
113 help='New name for subdirectory'
114 )
115 options = argparser.parse_args(args)
116
72 def rename(router, options):
11773 stream_name = options.stream_name
11874 if ':' in stream_name:
11975 stream_name, subdir = options.stream_name.split(':')
154110 argparser.add_argument('router', metavar='ROUTER', type=str,
155111 help='JSON file containing the backup router description'
156112 )
157 argparser.add_argument('command', metavar='COMMAND', type=str,
158 help='The action to take. One of: list, sync, syncdirs, rename'
113 argparser.add_argument('--version', action='version', version="%(prog)s 0.3")
114
115 subparsers = argparser.add_subparsers()
116
117 # - - - - list - - - -
118 parser_list = subparsers.add_parser('list', help='List available sync streams')
119 parser_list.add_argument('--stream-name-only', default=False, action='store_true',
120 help='Output only the names of the available streams'
159121 )
122 parser_list.set_defaults(func=list_)
160123
161 options, remaining_args = argparser.parse_known_args(args)
124 # - - - - sync - - - -
125 parser_sync = subparsers.add_parser('sync', help='Sync contents across a sync stream specified by name')
126 parser_sync.add_argument('stream_name', metavar='STREAM', type=str,
127 help='Name of stream (or stream:subdirectory) to sync contents across'
128 )
129 parser_sync.add_argument('--apply', default=False, action='store_true',
130 help='Actually run the rsync command'
131 )
132 parser_sync.add_argument('--thorough', default=False, action='store_true',
133 help='Ignore the timestamp on all destination files, to ensure content is synced'
134 )
135 parser_sync.set_defaults(func=sync)
162136
137 # - - - - rename - - - -
138 parser_rename = subparsers.add_parser(
139 'rename', help='Rename a subdirectory in both source and dest of sync stream'
140 )
141 parser_rename.add_argument('stream_name', metavar='STREAM', type=str,
142 help='Name of stream to operate under'
143 )
144 parser_rename.add_argument('existing_subdir_name', metavar='DIRNAME', type=str,
145 help='Existing subdirectory to be renamed'
146 )
147 parser_rename.add_argument('new_subdir_name', metavar='DIRNAME', type=str,
148 help='New name for subdirectory'
149 )
150 parser_rename.set_defaults(func=rename)
151
152 options = argparser.parse_args(args)
163153 with open(options.router, 'r') as f:
164154 router = json.loads(f.read())
165
166 if options.command == 'list':
167 list_(router, remaining_args)
168 elif options.command == 'sync':
169 sync(router, remaining_args)
170 elif options.command == 'syncdirs':
171 syncdirs(router, remaining_args)
172 elif options.command == 'rename':
173 rename(router, remaining_args)
174 else:
155 try:
156 func = options.func
157 except AttributeError:
175158 argparser.print_usage()
176159 sys.exit(1)
160 func(router, options)
5252 with self.assertRaises(SystemExit):
5353 main(['backup.json'])
5454
55 def test_dry_run(self):
56 main(['backup.json', 'syncdirs', 'canonical', 'cache'])
55 def test_sync_dry_run(self):
56 main(['backup.json', 'sync', 'basic:'])
5757 self.assertFalse(os.path.exists('cache/thing'))
5858 output = sys.stdout.getvalue()
5959 self.assertEqual(output.split('\n')[0], 'rsync --dry-run --archive --verbose --delete "canonical/" "cache/"')
6060 self.assertIn('DRY RUN', output)
6161
62 def test_apply(self):
63 main(['backup.json', 'syncdirs', 'canonical', 'cache', '--apply'])
62 def test_sync_apply(self):
63 main(['backup.json', 'sync', 'basic:', '--apply'])
6464 self.assertTrue(os.path.exists('cache/thing'))
65 output = sys.stdout.getvalue()
66 self.assertEqual(output.split('\n')[:4], [
67 'rsync --archive --verbose --delete "canonical/" "cache/"',
68 'sending incremental file list',
69 'thing',
70 ''
71 ])
72
73 def test_stream(self):
74 main(['backup.json', 'sync', 'basic:', '--apply'])
7565 output = sys.stdout.getvalue()
7666 self.assertEqual(output.split('\n')[:4], [
7767 'rsync --archive --verbose --delete "canonical/" "cache/"',