git @ Cat's Eye Technologies yastasoti / a1f006f
Rewrite README. Chris Pressey 1 year, 10 months ago
1 changed file(s) with 28 addition(s) and 23 deletion(s). Raw diff Collapse all Expand all
1616 * tries to be idempotent and not create a new local file if the remote file hasn't changed
1717 * handles links that are local files; checks if the file exists locally
1818
19 #### Archive routers ####
20
21 An archive router (used with `--archive-via`) is a JSON file that looks like this:
22
23 {
24 "http://catseye.tc/*": "/dev/null",
25 "https://footu.be/*": "footube/",
26 "*": "archive/"
27 }
28
29 Three guesses as to what these parts mean.
30
31 #### Planned features ####
32
33 * Archive youtube links with youtube-dl.
34 * Handle failures (redirects, etc) better (detect 503 / "connection refused" better.)
35 * Allow use of an external tool like `wget` or `curl` to do fetching.
36 * If the same link occurs more than once in the input, don't request it more than once.
37
3819 ### Examples ###
3920
40 Check that the links in a set of Feedmark documents all resolve:
21 #### Check all links in a set of Feedmark documents ####
4122
4223 feedmark --output-links article/*.md | yastasoti --article-root=article/ - | tee results.json
4324
44 Since `--archive-to` was not specified, this will make only `HEAD`
45 requests to check that the resources exist. It will not fetch them.
25 This will make only `HEAD` requests to check that the resources exist.
26 It will not fetch them. The ones that could not be fetches will appear
27 in `results.json`, and you can run yastasoti on that again to re-try:
4628
47 Archive stuff off teh internets:
29 yastasoti --article-root=article/ results.json | tee results2.json
30
31 #### Archive stuff off teh internets ####
4832
4933 cat >links.json << EOF
5034 [
5539 EOF
5640 yastasoti --archive-to=downloads links.json
5741
42 #### Categorize archived materials with a router ####
43
44 An archive router (used with `--archive-via`) is a JSON file that looks like this:
45
46 {
47 "http://catseye.tc/*": "/dev/null",
48 "https://footu.be/*": "footube/",
49 "*": "archive/"
50 }
51
52 Three guesses as to what these parts mean. Then you use it like
53
54 yastasoti --archive-via=router.json links.json
55
5856 ### Requirements ###
5957
6058 Tested under Python 2.7.12. Seems to work under Python 3.5.2 as well,
6462 with version 2.17.3.
6563
6664 If `tqdm` Python library is installed, will display a nice progress bar.
65
66 ### TODO ####
67
68 * Archive youtube links with youtube-dl.
69 * Handle failures (redirects, etc) better (detect 503 / "connection refused" better.)
70 * Allow use of an external tool like `wget` or `curl` to do fetching.
71 * If the same link occurs more than once in the input, don't request it more than once.