diff --git a/docs/content/_index.md b/docs/content/_index.md index 2a7886f..ca80a62 100644 --- a/docs/content/_index.md +++ b/docs/content/_index.md @@ -4,39 +4,46 @@ title = "zrepl - ZFS replication" # zrepl - ZFS replication -`zrepl` is a tool for replicating ZFS filesystems. +zrepl is a tool for replicating ZFS filesystems. -{{% panel theme="danger" header="Important" %}} -`zrepl` as well as this documentation is still under active -development. Some of the features below are not implemented yet. Use & test at your own risk ;) -{{% / panel %}} +{{% notice info %}} +`zrepl` as well as this documentation is still under active development. +Use & test at your own risk ;) +{{% /notice %}} + +## Getting started + +The [5 minute tutorial setup](/tutorial/) gives you a first impression. ## Main Features -* filesystem replication - * local & over network (SSH) - * push & pull mode - * snapshots & bookmarks support - * feature-negotiation for - * resumable `send & receive` - * compressed `send & receive` - * raw encrypted `send & receive` (as soon as it is available) - * access control checks when pulling datasets - * [flexible mappings]({{< ref "configuration/overview.md#mapping-filter-syntax" >}}) for filesystems -* automatic snapshot creation - * periodic interval -* automatic snapshot pruning - * [Retention Grid]({{< ref "configuration/snapshots.md#retention-grid" >}}) - +* Filesystem Replication + * [x] Local & Remote + * [x] Pull mode + * [ ] Push mode + * [x] Access control checks when pulling datasets + * [x] [Flexible mapping]({{< ref "configuration/map_filter_syntax.md" >}}) rules + * [ ] Bookmarks support + * [ ] Feature-negotiation for + * Resumable `send & receive` + * Compressed `send & receive` + * Raw encrypted `send & receive` (as soon as it is available) +* Automatic snapshot creation + * [x] Ensure fixed time interval between snapshots +* Automatic snapshot pruning + * [x] [Retention Grid]({{< ref "configuration/prune.md#retention-grid" >}}) +* Maintainable implementation in Go + * [x] Cross platform + * [x] Type safe & testable code + ## Contributing -`zrepl` is usable but nowhere near a stable release right now - we are happy -about contributors! +zrepl is usable but nowhere near a stable release right now - we are happy about contributors! * Explore the codebase * These docs live in the `docs/` subdirectory -* Document non-obvious / confusing / plain broken things you encounter when using `zrepl` for the first time -* Check the *Issues* and *Projects* sections for things to do ;) +* Document any non-obvious / confusing / plain broken behavior you encounter when setting up zrepl for the first time +* Check the *Issues* and *Projects* sections for things to do {{% panel header=" Getting your code merged"%}} [The GitHub repository](https://github.com/zrepl/zrepl) is where all development happens. diff --git a/docs/content/configuration/_index.md b/docs/content/configuration/_index.md index 791bcd1..dff78cf 100644 --- a/docs/content/configuration/_index.md +++ b/docs/content/configuration/_index.md @@ -3,4 +3,6 @@ title = "Configuration" alwaysopen = true +++ +{{% alert theme="warning" %}}Under Construction{{% /alert %}} + {{% children description="true" %}} diff --git a/docs/content/configuration/example_pull_backup.md b/docs/content/configuration/example_pull_backup.md deleted file mode 100644 index 7283f6e..0000000 --- a/docs/content/configuration/example_pull_backup.md +++ /dev/null @@ -1,9 +0,0 @@ -+++ -title = "Example: Pull Backup" -description = "Example configuration for a typical pull-backup scenario, e.g. server to server" -+++ - -Example configuration for a typical pull-backup scenario, e.g. server to server - -{{% alert theme="warning"%}}TBD{{% /alert %}} - diff --git a/docs/content/configuration/example_push_backup.md b/docs/content/configuration/example_push_backup.md deleted file mode 100644 index ffb70ef..0000000 --- a/docs/content/configuration/example_push_backup.md +++ /dev/null @@ -1,8 +0,0 @@ -+++ -title = "Example: Push Backup" -description = "Example configuration for a typical push-backup scenario, e.g. laptop to NAS" -+++ - -Example configuration for a typical push-backup scenario, e.g. laptop to NAS - -{{% alert theme="warning"%}}TBD{{% /alert %}} diff --git a/docs/content/configuration/jobs.md b/docs/content/configuration/jobs.md new file mode 100644 index 0000000..4f1ee4f --- /dev/null +++ b/docs/content/configuration/jobs.md @@ -0,0 +1,12 @@ ++++ +title = "Job Types" +weight = 10 ++++ + +{{% alert theme="warning" %}}Under Construction{{% /alert %}} + +## Source + +## Pull + +## Local \ No newline at end of file diff --git a/docs/content/configuration/overview.md b/docs/content/configuration/map_filter_syntax.md similarity index 59% rename from docs/content/configuration/overview.md rename to docs/content/configuration/map_filter_syntax.md index bea8685..c5204e2 100644 --- a/docs/content/configuration/overview.md +++ b/docs/content/configuration/map_filter_syntax.md @@ -1,60 +1,10 @@ +++ -title = "Overview" -weight = 100 -description = "Configuration format, SSH authentication, etc." +title = "Mapping & Filter Syntax" +weight = 20 +description = "How to specify mappings & filters" +++ -{{% panel header="Recommendation" %}} -Keep the [sample configuration file](https://github.com/zrepl/zrepl/blob/master/cmd/sampleconf/zrepl.yml) open on the side while reading this document! -{{% / panel %}} - -All configuration is managed in a single YAML file.
-It is structured by sections roughly corresponding to `zrepl` subcommands: - -```yaml -# REPLICATION -# Remote zrepl instances where pull and push jobs connect to -remotes: - name_of_remote: #... -# Push jobs (replication from local to remote) -pushs: - name_of_push_job: #... - name_of_other_push_job: #... -# pull jobs (replication from remote to local & local to local) -pulls: - name_of_pull_job: #... -# mapping incoming pushs to local datasets -sinks: - client_identity: #... -# access control for remote pull jobs -pull_acls: - client_identity: #... - -# SNAPSHOT MANAGEMENT -# Automatic snapshotting of filesystems -autosnap: - name_of_autosnap_job: #... -# Automatic pruning of snapshots based on creation date -prune: - name_of_prune_job: #... -``` - -When using `zrepl(8)`, a *subcommand* is passed the *job name* as a positional argument: - -```yaml -autosnap: # subcommand - db: # job name - prefix: zrepl_ - interval: 10m - dataset_filter: { - "tank/db<": ok - } -``` -```bash -$ zrepl autosnap --config zrepl.yml db -``` - -Run `zrepl --help` for a list of subcommands and options. +{{% alert theme="warning" %}}Under Construction{{% /alert %}} ## Mapping & Filter Syntax @@ -140,10 +90,3 @@ pull_acls: "tank/usr/home<": ok, } ``` - -## Next up - -* [Automating snapshot creation & pruning]({{< ref "configuration/snapshots.md" >}}) - -* [Replicating filesystems]({{< ref "configuration/replication.md" >}}) - diff --git a/docs/content/configuration/prune.md b/docs/content/configuration/prune.md new file mode 100644 index 0000000..7a04098 --- /dev/null +++ b/docs/content/configuration/prune.md @@ -0,0 +1,9 @@ ++++ +title = "Pruning" +description = "Automated pruning of snapshots" +weight = 200 ++++ + +{{% alert theme="warning" %}}Under Construction{{% /alert %}} + +## Retention Grid diff --git a/docs/content/configuration/snapshots.md b/docs/content/configuration/snapshots.md deleted file mode 100644 index abf73a4..0000000 --- a/docs/content/configuration/snapshots.md +++ /dev/null @@ -1,9 +0,0 @@ -+++ -title = "Snapshot Management" -description = "Automated snapshot creation & pruning" -weight = 200 -+++ - -{{% alert theme="warning"%}}TBD{{% /alert %}} - -## Retention Grid diff --git a/docs/content/configuration/replication.md b/docs/content/configuration/transports.md similarity index 57% rename from docs/content/configuration/replication.md rename to docs/content/configuration/transports.md index 81d6998..bb52e44 100644 --- a/docs/content/configuration/replication.md +++ b/docs/content/configuration/transports.md @@ -1,27 +1,11 @@ +++ -title = "Filesystem Replication" -description = "Replicating filesystems with existing bookmarks & snapshots" -weight = 300 +title = "Transports" +++ -{{% alert theme="warning"%}}Under Construction{{% /alert %}} +{{% alert theme="warning" %}}Under Construction{{% /alert %}} -### Remotes +## Stdinserver -The `remotes` section specifies remote `zrepl` instances from which to pull from / push backups to: - -```yaml -remotes: - offsite_backups: - transport: - ssh: - host: 192.168.122.6 - user: root - port: 22 - identity_file: /etc/zrepl/identities/offsite_backups -``` - -#### SSH Transport The SSH transport connects to the remote server using the SSH binary in `$PATH` and the parameters specified in the `zrepl` config file. @@ -31,10 +15,9 @@ another instance of `zrepl` on the other side of the connection; You may be familiar with this concept from [git shell](https://git-scm.com/docs/git-shell) or [Borg Backup](https://borgbackup.readthedocs.io/en/stable/deployment.html). -Check the examples for instructions on how to set this up on your machines! +Check the examples for instructions on how to set this up on your machines! {{% panel %}} The environment variables of the underlying SSH process are cleared. `$SSH_AUTH_SOCK` will not be available. We suggest creating a separate, unencrypted SSH key. {{% / panel %}} - diff --git a/docs/content/impl/_index.md b/docs/content/impl/_index.md index 973ba71..a373564 100644 --- a/docs/content/impl/_index.md +++ b/docs/content/impl/_index.md @@ -2,6 +2,8 @@ title = "Implementation Overview" +++ +{{% alert theme="warning" %}}Under Construction{{% /alert %}} + The following design aspects may convince you that `zrepl` is superior to a hacked-together shell script solution. ## Language diff --git a/docs/content/install/_index.md b/docs/content/install/_index.md new file mode 100644 index 0000000..0356d41 --- /dev/null +++ b/docs/content/install/_index.md @@ -0,0 +1,94 @@ ++++ +title = "Installation" +weight = 20 ++++ + +{{% notice note %}} +Note: check out the [tutorial]({{< relref "tutorial/_index.md" >}}) if you want a first impression of zrepl. +{{% /notice %}} + +## User Privileges + +It is possible to run zrepl as an unprivileged user in combination with +[ZFS delegation](https://www.freebsd.org/doc/handbook/zfs-zfs-allow.html). + +Also, there is the possibility to run it in a jail on FreeBSD by delegating a dataset to the jail. + +However, until we get around documenting those setups, you will have to run zrepl as root or experiment yourself :) + +## Installation + +zrepl is currently not packaged on any operating system. Signed & versioned releases are planned but not available yet. + +Check out the sources yourself, compile and install to the zrepl user's `$PATH`.
+**Note**: if the zrepl binary is not in `$PATH`, you will have to adjust the examples in the [tutorial]({{< relref "tutorial/_index.md" >}}). + +```bash +# NOTE: you may want to checkout & build as an unprivileged user +cd /root +git clone https://github.com/zrepl/zrepl.git +cd zrepl +go build -o zrepl +cp zrepl /usr/local/bin/zrepl +rehash +# see if it worked +zrepl help +``` + + + +## Configuration & Runtime Directories + +### Main Configuration File + +The main configuration file lives in either + +* `/etc/zrepl/zrepl.yml` +* `/usr/local/etc/zrepl/zrepl.yml` + +The file locations are check in that order. Alternative locations must be specified using a command line flag. + +### Runtime Directories + +**Source jobs** with `serve.type = stdinserver` require a private runtime directory. ([Why?]({{< relref "configuration/transports.md#stdinserver" >}})) + +The default is `/var/run/zrepl/stdinserver` and it must only be accessible by the user +that runs `zrepl daemon` and `zrepl stdinserver`: + +```bash +mkdir -p /var/run/zrepl/stdinserver +chmod -R 0700 /var/run/zrepl +``` + +## Running the Daemon + +All work zrepl done is performed by a daemon process. + +There are no *rc(8)* or *systemd.service(5)* service definitions yet. + +The daemon does not fork and writes all log output to stderr. + +```bash +zrepl --config /etc/zrepl/zrepl.yml daemon +``` + +FreeBSD ships with the *daemon(8)* utility which is also a good start for writing an *rc(8)* file: + +```bash +daemon -o /var/log/zrepl.log \ + -p /var/run/zrepl/daemon.pid \ + zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon +``` + +{{% notice info %}} +Make sure to read the first lines of log output after the daemon starts: if the daemon cannot create the [stdinserver]({{< relref "configuration/transports.md#stdinserver" >}}) sockets +in the runtime directory, it will complain but not terminate as other tasks such as taking periodic snapshots might still work. +{{% / notice %}} + +### Restarting + +The daemon handles SIGINT and SIGTERM for graceful shutdown. + +Graceful shutdown means at worst that a job will not be rescheduled for the next interval. + +The daemon exits as soon as all jobs have reported shut down. \ No newline at end of file diff --git a/docs/content/tutorial/_index.md b/docs/content/tutorial/_index.md new file mode 100644 index 0000000..7483fe6 --- /dev/null +++ b/docs/content/tutorial/_index.md @@ -0,0 +1,173 @@ +--- +title: "Tutorial" +weight: 1 +--- + +This tutorial shows how zrepl can be used to implement a ZFS-based pull backup. + +We assume the following scenario + +* Production server `prod1` with filesystems to back up + * `zroot/var/db` + * `zroot/usr/home` and all its child filesystems + * **except** `zroot/usr/home/paranoid` belonging to a user doing backups themselves +* Backup server `backups` with + * Filesystem `storage/zrepl/pull/prod1` + children dedicated to backups of `prod1` + +Our backup solution should fulfill the following requirements: + +* Periodically snapshot the filesystems on `prod1` *every 10 minutes* +* Incrementally replicate these snapshots to `storage/zrepl/pull/prod1/*` on `backups` +* Keep only very few snapshots on `prod1` to save disk space +* Keep a fading history (24 hourly, 30 daily, 6 monthly) of snapshots on `backups` + +## Analysis + +We can model this situation as two jobs: + +* A **source job** on `prod1` + * Creates the snapshots + * Keeps a few snapshots that are also on `prod1` to enable incremental replication +* A **pull job** on `prod1` + * Pulls the snapshots + * Fades out snapshots as they age + +{{%expand "Side note: why doesn't `backups` take the snapshots right before replication?" %}} +After all, a little `ssh prod1 'zfs snapshot...'` wouldn't be so bad, right? + +As is the case with all distributed systems, the link between `prod1` and `backups` might be down for an hour or two. +We do not want to sacrifice our required backup resolution of 10 minute intervals for a temporary connection outage. + +When the link comes up again, `backups` will happily catch up the 12 snapshots taken by `prod1` in the meantime, without +a gap in our backup history. +{{%/expand%}} + +## Install zrepl + +Follow the [OS-specific installation instructions](/install/) and come back here. + +## Configure `backups` + +We define a **pull job** named `pull_prod1` in the [main configuration file](/install/#main-configuration-file): + +```yaml +jobs: +- name: pull_prod1 + type: pull + connect: + type: ssh+stdinserver + host: prod1.example.com + user: root + port: 22 + identity_file: /etc/zrepl/ssh/prod1 + interval: 10m + mapping: { + "<":"storage/zrepl/pull/prod1" + } + initial_repl_policy: most_recent + snapshot_prefix: zrepl_pull_backup_ + prune: + policy: grid + grid: 1x1h(keep=all) | 24x1h | 35x1d | 6x30d +``` + +The `connect` section instructs zrepl to use the `stdinserver` transport: instead of directly exposing zrepl on `prod1` +to the internet, `backups` starts the `zrepl stdinserver` on `prod1` via SSH. +(You can learn more about what happens [here]({{< relref "configuration/transports.md#stdinserver" >}}), or just continue following this tutorial.) + +Thus, we need to create the SSH key pair `/etc/zrepl/ssh/prod1{,.pub}` and later pass the public part to `prod1` +which will use it to authenticate `backups`. Execute the following commands on `backups` as the root user: + +```bash +cd /etc/zrepl +mkdir -p ssh +chmod 0700 ssh +ssh-keygen -t ed25519 -N '' -f /etc/zrepl/ssh/prod1 +``` +You can learn more about the [**pull job** format here]({{< relref "configuration/jobs.md#pull" >}}) but for now we are good to go. + +## Configure `prod1` + +We define a corresponding **source job** named `pull_backup` in the [main configuration file](/install/#main-configuration-file) +`zrepl.yml`: + +```yaml +jobs: + +- name: pull_backup + type: source + serve: + type: stdinserver + client_identity: backups.example.com + datasets: { + "zroot/var/db": "ok", + "zroot/usr/home<": "ok", + "zroot/usr/home/paranoid": "!", + } + snapshot_prefix: zrepl_pull_backup_ + interval: 10m + prune: + policy: grid + grid: 1x1d(keep=all) + +``` + +The `serve` section corresponds to the `connect` section in the configuration of `backups`. + +We need to allow the SSH key on `backups` to execute `zrepl stdinserver backups.example.com` on +`prod1`. For good measure, we will in fact enforce that only this command can be executed. + +Open `/root/.ssh/authorized_keys` and add either of the the following lines, replacing BACKUPS_SSH_PUBKEY at the end +of the line with the contents of `/etc/zrepl/ssh/prod1.pub` (note the **.pub** !) from `backups`. + +``` +# for OpenSSH >= 7.2 +command="zrepl stdinserver backups.example.com",restrict BACKUPS_SSH_PUBKEY +# for older OpenSSH versions +command="zrepl stdinserver backups.example.com",no-port-forwarding,no-X11-forwarding,no-pty,no-agent-forwarding,no-user-rc BACKUPS_SSH_PUBKEY +``` + +{{% alert theme="info" %}}The entries **must** be on a single line, including the replaced BACKUPS_SSH_PUBKEY{{% /alert %}} + +Again, you can learn more about the [**source job** format here]({{< ref "configuration/jobs.md#source" >}}). + +## Apply Configuration Changes + +We need to restart the zrepl daemon on **both** `prod1` and `backups`. + +This is [OS-specific](/install/#restarting). + +## Watch it Work + +A common setup is to watch the log output and zfs list of snapshots on both machines. + +If you like tmux, here is a handy script that works on FreeBSD: + +```bash +pkg install gnu-watch tmux +tmux new-window +tmux split-window "tail -f /var/log/zrepl.log" +tmux split-window "gnu-watch 'zfs list -t snapshot -o name,creation -s creation | grep zrepl_pull_backup_'" +tmux select-layout tiled +``` + +The Linux equivalent might look like this + +```bash +# make sure tmux is installed & let's assume you use systemd + journald +tmux new-window +tmux split-window "journalctl -f -u zrepl.service" +tmux split-window "watch 'zfs list -t snapshot -o name,creation -s creation | grep zrepl_pull_backup_'" +tmux select-layout tiled +``` + +## Summary + +Congratulations, you have a working pull backup. Where to go next? + +* Read more about [configuration format, options & job types](/configuration/) +* Learn about [implementation details](/impl/) of zrepl. + + + + diff --git a/docs/themes/docdock b/docs/themes/docdock index f1a902a..9e429b2 160000 --- a/docs/themes/docdock +++ b/docs/themes/docdock @@ -1 +1 @@ -Subproject commit f1a902a088f925e1478062d5327aa8e897752b13 +Subproject commit 9e429b2c209e6062e7bd6f53ced0a31d3b624448