git @ Cat's Eye Technologies Space-Madness / master
master

Tree @master (Download .tar.gz)

Space Madness

Space Madness is a backup system. It is evolving. It is my attempt to describe the backup system that I try to use. This repository contains those notes, and some tools that may or may not help in impementing this backup system.

The main tool used here is rsync. If it is not installed as a standard part of your operating system, install it first.

Depositories

The main unit of organizing files in Space Madness is the depository, which is a directory tree. The subdirectories in a depository are called subjects. Subjects are hierarchical (for example, you might have art and art/paintings and art/paintings/watercolours).

A subject is expected to have the same content in all the different depositories in which it appears. (This is not 100% true, but it's useful to think of them this way.)

There are three kinds of depositories:

With these definitions, we can say that a canonical depository is backed up if and only if every subject in it exists in at least one cache depository which resides on a different physical medium than the canonical depository.

Note that two partitions on the same hard disk are not different physical media!

The goal of the backup system is that every interesting file is in a canonical depository, and that every canonical depository is backed up.

To accomplish this goal, there are two important actions on depositories.

Update a cache depository from a canonical depository

To update a cache depository from a canonical depository,

rsync --archive --verbose --delete $CANONICAL/subject/ $CACHE/subject/

To check if a cache depository is up-to-date with a canonical depository,

rsync --archive --verbose --delete --dry-run $CANONICAL/subject/ $CACHE/subject/

We might provide a lightweight wrapper for those.

Add the contents of an incoming depository to a canonical depository

To update a canonical depository from an incoming depository,

deposit $CANONICAL $INCOMING

In fact this is virtually the same as saying

rsync --archive --verbose $INCOMING/ $CANONICAL/

but incorporates some consistency checking, i.e. that you don't trample something in the canonical depository with something in the incoming with the same name. (Also, rsync is pretty picky about directory names, and if you forget the trailing slash, bad things can happen. So it helps there too.)

The deposit tool takes an additional flag, --clean, which deletes all the files (but not the directories) in the incoming depository after they are copied over. However, as a safety check, it will not function unless the files have already been copied into the canonical depository, and are identical.

It is an excellent idea to update a cache depository from the canonical depository after running deposit (and before running deposit --clean.)

Advanced Topics in Backup Subjects

Because subjects are hierarchical, it is also possible to think of (and treat) a subject as a sort of sub-depository.

It is up to you to decide if you have any backup use cases that are complex enough to justify doing that.

It is also useful to keep certain backups in version-controlled repositories, e.g. git repos. In this case, a repo directory can usually be treated as a subject. Instead of depositing files with deposit, one can push changes to the repo-subject residing in the canonical depository.

Read-only Depositories

Need to research this, but the basic idea would be to chmod -R a cache depository so that you don't accidentally change files in it or add files to it. (But then changing the permissions back when you need to update it from a canonical depository.)

Additional tools

It is unlikely that you will always use a backup system perfectly, and even less likely that you started backing up your files with a perfect backup system. It is in fact likely that at some point you just copied important files from one place to another to ensure that you had a backup copy.

Thus, Space Madness includes some tools to assist with cleaning up backups and making them meaningful.

They are:

We should eventually document them here.