From a4963cecb7168d055b3e71c9a1c1e2d1d8f4b97f Mon Sep 17 00:00:00 2001 From: Christian Schwarz Date: Tue, 3 Oct 2017 14:21:10 +0200 Subject: [PATCH] docs: document job types The documentation describes intended behavior. Apparently, there are some bugs regarding *patient* tasks. refs #8 refs #13 --- docs/content/configuration/jobs.md | 105 +++++++++++++++++- docs/layouts/shortcodes/zrepl-filter.html | 6 + .../layouts/shortcodes/zrepl-job-patient.html | 1 + docs/layouts/shortcodes/zrepl-mapping.html | 6 + docs/layouts/shortcodes/zrepl-prune.html | 6 + docs/layouts/shortcodes/zrepl-transport.html | 6 + docs/themes/docdock | 2 +- 7 files changed, 126 insertions(+), 6 deletions(-) create mode 100644 docs/layouts/shortcodes/zrepl-filter.html create mode 100644 docs/layouts/shortcodes/zrepl-job-patient.html create mode 100644 docs/layouts/shortcodes/zrepl-mapping.html create mode 100644 docs/layouts/shortcodes/zrepl-prune.html create mode 100644 docs/layouts/shortcodes/zrepl-transport.html diff --git a/docs/content/configuration/jobs.md b/docs/content/configuration/jobs.md index 6b0b48a..80b19a7 100644 --- a/docs/content/configuration/jobs.md +++ b/docs/content/configuration/jobs.md @@ -3,12 +3,107 @@ title = "Job Types" weight = 10 +++ -{{% alert theme="warning" %}}Under Construction{{% /alert %}} +A *job* is the unit of activity tracked by the zrepl daemon and configured in the [configuration file]({{< relref "install/_index.md#configuration-files" >}}). -See {{< sampleconflink >}} for examples. +Every job has a unique `name`, a `type` and type-dependent fields which are documented on this page. -## Source +Check out the [tutorial]({{< relref "tutorial/_index.md" >}}) and {{< sampleconflink >}} for examples on how job types are actually used. -## Pull +## Job Types -## Local \ No newline at end of file +### Source + +|Parameter|Default|Description / Example| +|-----|-------|-------| +|`type`||`source`| +|`name`||unique name of the job| +|`serve`||{{< zrepl-transport "serve transport" >}} specification| +|`datasets`||{{< zrepl-filter >}} for datasets to expose to client| +|`snapshot_prefix`||prefix for ZFS snapshots taken by this job| +|`interval`||snapshotting interval| +|`prune`||{{< zrepl-prune >}} policy for datasets in `datasets` with prefix `snapshot_prefix`| + +* Snapshotting Task (every `interval`, {{% zrepl-job-patient %}}) + 1. A snapshot of filesystems matched by `datasets` is taken every `interval` with prefix `snapshot_prefix`. + 1. The `prune` policy is triggered on datasets matched by `datasets` with snapshots matched by `snapshot_prefix`. +* Serve Task + * Wait for connections from pull job using `serve` + +A source job is the counterpart to a [pull job]({{< relref "#pull" >}}). + +Example: {{< sampleconflink "pullbackup/productionhost.yml" >}} + +### Pull + +|Parameter|Default|Description / Example| +|-----|-------|-------| +|`type`||`pull`| +|`name`||unique name of the job| +|`connect`||{{< zrepl-transport "connect transport" >}} specification| +|`interval`||Interval between pull attempts| +|`mapping`||{{< zrepl-mapping >}} for remote to local filesystems| +|`initial_repl_policy`|`most_recent`|initial replication policy| +|`snapshot_prefix`||prefix filter used for replication & pruning| +|`prune`||{{< zrepl-prune >}} policy for local filesystems reachable by `mapping`| + +* Main Task (every `interval`, {{% zrepl-job-patient %}}) + 1. A connection to the remote source job is established using the strategy in `connect` + 1. `mapping` maps filesystems presented by the remote side to local *target filesystems* + 1. Those remote filesystems with a local *target filesystem* are replicated: + 1. Only snapshots with prefix `snapshot_prefix` are replicated. + 1. If possible, incremental replication takes place. + 1. If the local target filesystem does not exist, `initial_repl_policy` is used. + 1. On conflicts, an error is logged but replication of other filesystems with mapping continues. + 1. The `prune` policy is triggered for all *target filesystems* + + +A pull job is the counterpart to a [source job]({{< relref "#source" >}}). + +Example: {{< sampleconflink "pullbackup/backuphost.yml" >}} + +### Local + +|Parameter|Default|Description / Example| +|-----|-------|-------| +|`type`||`local`| +|`name`||unique name of the job| +|`mapping`||{{}} from source to target filesystem (both local)| +|`snapshot_prefix`||prefix for ZFS snapshots taken by this job| +|`interval`|snapshotting & replication interval| +|`initial_repl_policy`|`most_recent`|initial replication policy| +|`prune_lhs`||pruning policy on left-hand-side (source)| +|`prune_rhs`||pruning policy on right-hand-side (target)| + +* Main Task (every `interval`, {{% zrepl-job-patient %}}) + 1. Evaluate `mapping` for local filesystems, those with a *target filesystem* are called *mapped filesystems*. + 1. Snapshot *mapped filesystems* with `snapshot_prefix`. + 1. Replicate *mapped filesystems* to their respective *target filesystems*: + 1. Only snapshots with prefix `snapshot_prefix` are replicated. + 1. If possible, incremental replication takes place. + 1. If the *target filesystem* does not exist, `initial_repl_policy` is used. + 1. On conflicts, an error is logged but replication of other *mapped filesystems* continues. + 1. The `prune_lhs` policy is triggered for all *mapped filesystems* + 1. The `prune_rhs` policy is triggered for all *target filesystems* + +A local job is combination of source & pull job executed on the same machine. + +Example: {{< sampleconflink "localbackup/host1.yml" >}} + +## Glossary + +#### Task + +A job consists of one or more tasks and a task consists of one or more steps. +Some tasks may be periodic while others wait for an event to occur. + +#### patient + +it is supposed to execute some task every `interval`. +We call the start of the task an *invocation*. + +* If the task completes in less than `interval`, the task is restarted at `last_invocation + interval`. +* Otherwise, a patient job + * logs a warning as soon as a task exceeds its configured `interval` + * waits for the last invocation to finish + * logs a warning with the effective task duration + * immediately starts a new invocation of the task diff --git a/docs/layouts/shortcodes/zrepl-filter.html b/docs/layouts/shortcodes/zrepl-filter.html new file mode 100644 index 0000000..80383de --- /dev/null +++ b/docs/layouts/shortcodes/zrepl-filter.html @@ -0,0 +1,6 @@ +{{ $this := .}} +{{with .Site.GetPage "page" "configuration" "map_filter_syntax.md" }} +{{default "filter" ($this.Get 0) }} +{{else}} +{{.PageDoesNotExistError}} +{{end}} diff --git a/docs/layouts/shortcodes/zrepl-job-patient.html b/docs/layouts/shortcodes/zrepl-job-patient.html new file mode 100644 index 0000000..acf624e --- /dev/null +++ b/docs/layouts/shortcodes/zrepl-job-patient.html @@ -0,0 +1 @@ +{{ "[patient](#patient)" | markdownify }} diff --git a/docs/layouts/shortcodes/zrepl-mapping.html b/docs/layouts/shortcodes/zrepl-mapping.html new file mode 100644 index 0000000..97e22e0 --- /dev/null +++ b/docs/layouts/shortcodes/zrepl-mapping.html @@ -0,0 +1,6 @@ +{{ $this := .}} +{{with .Site.GetPage "page" "configuration" "map_filter_syntax.md" }} +{{default "mapping" ($this.Get 0) }} +{{else}} +{{.PageDoesNotExistError}} +{{end}} diff --git a/docs/layouts/shortcodes/zrepl-prune.html b/docs/layouts/shortcodes/zrepl-prune.html new file mode 100644 index 0000000..812d764 --- /dev/null +++ b/docs/layouts/shortcodes/zrepl-prune.html @@ -0,0 +1,6 @@ +{{ $this := .}} +{{with .Site.GetPage "page" "configuration" "prune.md" }} +{{default "prune policy" ($this.Get 0) }} +{{else}} +{{.PageDoesNotExistError}} +{{end}} diff --git a/docs/layouts/shortcodes/zrepl-transport.html b/docs/layouts/shortcodes/zrepl-transport.html new file mode 100644 index 0000000..69db4f9 --- /dev/null +++ b/docs/layouts/shortcodes/zrepl-transport.html @@ -0,0 +1,6 @@ +{{ $this := .}} +{{with .Site.GetPage "page" "configuration" "transports.md" }} +{{default "transport" ($this.Get 0) }} +{{else}} +{{.PageDoesNotExistError}} +{{end}} diff --git a/docs/themes/docdock b/docs/themes/docdock index 1e35d34..1ee4435 160000 --- a/docs/themes/docdock +++ b/docs/themes/docdock @@ -1 +1 @@ -Subproject commit 1e35d34235f01ada2fae96802510fd3937dfdb54 +Subproject commit 1ee44353686c9efd84509ef38e384e603661d3a5