adminer/zrepl_patched
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: reflect changes in replication_rewrite branch

Browse Source
This commit is contained in:
Christian Schwarz
2018-10-11 17:46:26 +02:00
parent 125b561df3
commit 1643198713
16 changed files with 823 additions and 554 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/configuration/filter_syntax.rst
+67
View File
@@ -0,0 +1,67 @@
.. include:: ../global.rst.inc
.. _pattern-filter:
Filter Syntax
=============
For :ref:`source jobs <job-source>` and :ref:`push jobs <job-push>`, a filesystem filter must be defined (field ``filesystems``).
A filter takes a filesystem path (in the ZFS filesystem hierarchy) as parameter and returns ``true`` (pass) or ``false`` (block).
A filter is specified as a **YAML dictionary** with patterns as keys and booleans as values.
The following rules determine which result is chosen for a given filesystem path:
* More specific path patterns win over less specific ones
* Non-wildcard patterns (full path patterns) win over *subtree wildcards* (`<` at end of pattern)
* If the path in question does not match any pattern, the result is ``false``.
The **subtree wildcard** ``<`` means "the dataset left of ``<`` and all its children".
.. TIP::
You can try out patterns for a configured job using the ``zrepl test`` subcommand.
Examples
--------
Full Access
~~~~~~~~~~~
The following configuration will allow access to all filesystems.
::
jobs:
- type: source
filesystems: {
"<": true,
}
...
Fine-grained
~~~~~~~~~~~~
The following configuration demonstrates all rules presented above.
::
jobs:
- type: source
filesystems: {
"tank<": true, # rule 1
"tank/foo<": false, # rule 2
"tank/foo/bar": true, # rule 3
}
...
Which rule applies to given path, and what is the result?
::
tank/foo/bar/loo => 2 false
tank/bar => 1 true
tank/foo/bar => 3 true
zroot => NONE false
tank/var/log => 1 true
docs/configuration/jobs.rst
+234 -150
View File
@@ -1,20 +1,106 @@
.. include:: ../global.rst.inc
.. |patient| replace:: :ref:`patient <job-term-patient>`
.. |serve-transport| replace:: :ref:`serve transport<transport-ssh+stdinserver-serve>`
.. |connect-transport| replace:: :ref:`connect transport<transport-ssh+stdinserver-connect>`
.. |mapping| replace:: :ref:`mapping <pattern-mapping>`
.. |filter| replace:: :ref:`filter <pattern-filter>`
.. |prune| replace:: :ref:`prune <prune>`
.. |serve-transport| replace:: :ref:`serve specification<transport>`
.. |connect-transport| replace:: :ref:`connect specification<transport>`
.. |snapshotting-spec| replace:: :ref:`snapshotting specification <job-snapshotting-spec>`
.. |pruning-spec| replace:: :ref:`pruning specification <prune>`
.. |filter-spec| replace:: :ref:`filter specification<pattern-filter>`
.. _job:
Job Types
=========
Job Types & Replication
=======================
Overview & Terminology
----------------------
A *job* is the unit of activity tracked by the zrepl daemon and configured in the |mainconfig|.
Every job has a unique ``name``, a ``type`` and type-dependent fields which are documented on this page.
Check out the :ref:`tutorial` and :sampleconf:`/` for examples on how job types are actually used.
Replication always happens between a pair of jobs: one is the **active side**, and one the **passive side**.
The active side executes the replication logic whereas the passive side responds to requests after checking the active side's permissions.
For communication, the active side connects to the passive side using a :ref:`transport <transport>` and starts issuing remote procedure calls (RPCs).
The following table shows how different job types can be combined to achieve both push and pull mode setups:
+-----------------------+--------------+----------------------------------+-----------------------------------------------+
| Setup name | active side | passive side | use case |
+=======================+==============+==================================+===============================================+
| Push mode | ``push`` | ``sink`` | * Laptop backup |
| | | | * NAS behind NAT to offsite |
+-----------------------+--------------+----------------------------------+-----------------------------------------------+
| Pull mode | ``pull`` | ``source`` | * Central backup-server for many nodes |
| | | | * Remote server to NAS behind NAT |
+-----------------------+--------------+----------------------------------+-----------------------------------------------+
| Local replication | | ``push`` + ``sink`` in one config | * Backup FreeBSD boot pool |
| | | with :ref:`local transport <transport-local>` | |
+-----------------------+--------------+----------------------------------+-----------------------------------------------+
How the Active Side Works
~~~~~~~~~~~~~~~~~~~~~~~~~
The active side (:ref:`push <job-push>` and :ref:`pull <job-pull>` job) executes the replication and pruning logic:
* Wakeup because of finished snapshotting (``push`` job) or pull interval ticker (``pull`` job).
* Connect to the corresponding passive side using a :ref:`transport <transport>` and instantiate an RPC client.
* Replicate data from the sending to the receiving side.
* Prune on sender & receiver.
.. TIP::
The progress of the active side can be watched live using the ``zrepl status`` subcommand.
How the Passive Side Works
~~~~~~~~~~~~~~~~~~~~~~~~~~
The passive side (:ref:`sink <job-sink>` and :ref:`source <job-source>`) waits for connections from the corresponding active side,
using the transport listener type specified in the ``serve`` field of the job configuration.
Each transport listener provides a client's identity to the passive side job.
It uses the client identity for access control:
* The ``sink`` job only allows pushes to those ZFS filesystems to the active side that are located below ``root_fs/${client_identity}``.
* The ``source`` job has a whitelist of client identities that are allowed pull access.
.. TIP::
The implementation of the ``sink`` job requires that the connecting client identities be a valid ZFS filesystem name components.
How Replication Works
~~~~~~~~~~~~~~~~~~~~~
One of the major design goals of the replication module is to avoid any duplication of the nontrivial logic.
As such, the code works on abstract senders and receiver **endpoints**, where typically one will be implemented by a local program object and the other is an RPC client instance.
Regardless of push- or pull-style setup, the logic executes on the active side, i.e. in the ``push`` or ``pull`` job.
The following steps take place during replication and can be monitored using the ``zrepl status`` subcommand:
* Plan the replication:
* Compare sender and receiver filesystem snapshots
* Build the **replication plan**
* Per filesystem, compute a diff between sender and receiver snapshots
* Build a list of replication steps
* If possible, use incremental sends (``zfs send -i``)
* Otherwise, use full send of most recent snapshot on sender
* Give up on filesystems that cannot be replicated without data loss
* Retry on errors that are likely temporary (i.e. network failures).
* Give up on filesystems where a permanent error was received over RPC.
* Execute the plan
* Perform replication steps in the following order:
Among all filesystems with pending replication steps, pick the filesystem whose next replication step's snapshot is the oldest.
* After a successful replication step, update the replication cursor bookmark (see below)
The idea behind the execution order of replication steps is that if the sender snapshots all filesystems simultaneously at fixed intervals, the receiver will have all filesystems snapshotted at time ``T1`` before the first snapshot at ``T2 = T1 + $interval`` is replicated.
.. _replication-cursor-bookmark:
The **replication cursor bookmark** ``#zrepl_replication_cursor`` is kept per filesystem on the sending side of a replication setup:
It is a bookmark of the most recent successfully replicated snapshot to the receiving side.
It is is used by the :ref:`not_replicated <prune-keep-not-replicated>` keep rule to identify all snapshots that have not yet been replicated to the receiving side.
Regardless of whether that keep rule is used, the bookmark ensures that replication can always continue incrementally.
.. ATTENTION::
@@ -22,12 +108,133 @@ Check out the :ref:`tutorial` and :sampleconf:`/` for examples on how job types
Whe receiving a filesystem, it is never mounted (`-u` flag) and `mountpoint=none` is set.
This is temporary and being worked on :issue:`24`.
.. _job-snapshotting-spec:
Taking Snaphots
---------------
The ``push`` and ``source`` jobs can automatically take periodic snapshots of the filesystems matched by the ``filesystems`` filter field.
The snapshot names are composed of a user-defined prefix followed by a UTC date formatted like ``20060102_150405_000``.
We use UTC because it will avoid name conflicts when switching time zones or between summer and winter time.
For ``push`` jobs, replication is automatically triggered after all filesystems have been snapshotted.
::
jobs:
- type: push
filesystems: {
"<": true,
"tmp": false
}
snapshotting:
type: periodic
prefix: zrepl_
interval: 10m
...
There is also a ``manual`` snapshotting type, which covers the following use cases:
* Existing infrastructure for automatic snapshots: you only want to use zrepl for replication.
* Run scripts before and after taking snapshots (like locking database tables).
We are working on better integration for this use case: see :issue:`74`.
Note that you will have to trigger replication manually using the ``zrepl wakeup JOB`` subcommand in that case.
::
jobs:
- type: push
filesystems: {
"<": true,
"tmp": false
}
snapshotting:
type: manual
...
.. _job-push:
Job Type ``push``
-----------------
.. list-table::
:widths: 20 80
:header-rows: 1
* - Parameter
- Comment
* - ``type``
- = ``push``
* - ``name``
- unique name of the job
* - ``connect``
- |connect-transport|
* - ``filesystems``
- |filter-spec| for filesystems to be snapshotted and pushed to the sink
* - ``snapshotting``
- |snapshotting-spec|
* - ``pruning``
- |pruning-spec|
Example config: :sampleconf:`/push.yml`
.. _job-sink:
Job Type ``sink``
-----------------
.. list-table::
:widths: 20 80
:header-rows: 1
* - Parameter
- Comment
* - ``type``
- = ``sink``
* - ``name``
- unique name of the job
* - ``serve``
- |serve-transport|
* - ``root_fs``
- ZFS dataset path are received to
``$root_fs/$client_identity``
Example config: :sampleconf:`/sink.yml`
.. _job-pull:
Job Type ``pull``
-----------------
.. list-table::
:widths: 20 80
:header-rows: 1
* - Parameter
- Comment
* - ``type``
- = ``pull``
* - ``name``
- unique name of the job
* - ``connect``
- |connect-transport|
* - ``root_fs``
- ZFS dataset path are received to
``$root_fs/$client_identity``
* - ``interval``
- Interval at which to pull from the source job
* - ``pruning``
- |pruning-spec|
Example config: :sampleconf:`/pull.yml`
.. _job-source:
Source Job
----------
Example: :sampleconf:`pullbackup/productionhost.yml`.
Job Type ``source``
-------------------
.. list-table::
:widths: 20 80
@@ -42,142 +249,19 @@ Example: :sampleconf:`pullbackup/productionhost.yml`.
* - ``serve``
- |serve-transport|
* - ``filesystems``
- |filter| for filesystems to expose to client
* - ``snapshot_prefix``
- prefix for ZFS snapshots taken by this job
* - ``interval``
- snapshotting interval
* - ``prune``
- |prune| for versions of filesytems in ``filesystems``, versions prefixed with ``snapshot_prefix``
- |filter-spec| for filesystems to be snapshotted and exposed to connecting clients
* - ``snapshotting``
- |snapshotting-spec|
Example config: :sampleconf:`/source.yml`
.. _replication-local:
Local replication
-----------------
If you have the need for local replication (most likely between two local storage pools), you can use the :ref:`local transport type <transport-local>` to connect a local push job to a local sink job.
Example config: :sampleconf:`/local.yml`.
- Snapshotting Task (every ``interval``, |patient|)
- A snapshot of filesystems matched by ``filesystems`` is taken every ``interval`` with prefix ``snapshot_prefix``.
- A bookmark of that snapshot is created with the same name.
- The ``prune`` policy is evaluated for versions of filesystems matched by ``filesystems``, versions prefixed with ``snapshot_prefix``.
- Serve Task
- Wait for connections from pull job using ``serve``.
A source job is the counterpart to a :ref:`job-pull`.
Make sure you read the |prune| policy documentation.
.. _job-pull:
Pull Job
--------
Example: :sampleconf:`pullbackup/backuphost.yml`
.. list-table::
:widths: 20 80
:header-rows: 1
* - Parameter
- Comment
* - ``type``
- = ``pull``
* - ``name``
- unqiue name of the job
* - ``connect``
- |connect-transport|
* - ``interval``
- Interval between pull attempts
* - ``mapping``
- |mapping| for remote to local filesystems
* - ``snapshot_prefix``
- prefix snapshots must match to be considered for replication & pruning
* - ``prune``
- |prune| policy for versions of filesystems of local filesystems reachable by ``mapping``, versions prefixed with ``snapshot_prefix``
* Main Task (every ``interval``, |patient|)
#. A connection to the remote source job is established using the strategy in ``connect``
#. ``mapping`` maps filesystems presented by the remote side to local *target filesystems*
#. Those remote filesystems with a local *target filesystem* are replicated
#. Only snapshots with prefix ``snapshot_prefix`` are replicated.
#. If possible, incremental replication takes place.
#. If the local target filesystem does not exist, the most recent snapshot is sent fully (non-incremental).
#. On conflicts, an error is logged but replication of other filesystems with mapping continues.
#. The ``prune`` policy is evaluated for all *target filesystems*
A pull job is the counterpart to a :ref:`job-source`.
Make sure you read the |prune| policy documentation.
.. _job-local:
Local Job
---------
Example: :sampleconf:`localbackup/host1.yml`
.. list-table::
:widths: 20 80
:header-rows: 1
* - Parameter
- Comment
* - ``type``
- = ``local``
* - ``name``
- unqiue name of the job
* - ``mapping``
- |mapping| from source to target filesystem (both local)
* - ``snapshot_prefix``
- prefix for ZFS snapshots taken by this job
* - ``interval``
- snapshotting & replication interval
* - ``prune_lhs``
- pruning policy on left-hand-side (source)
* - ``prune_rhs``
- pruning policy on right-hand-side (target)
* Main Task (every ``interval``, |patient|)
#. Evaluate ``mapping`` for local filesystems, those with a *target filesystem* are called *mapped filesystems*.
#. Snapshot *mapped filesystems* with ``snapshot_prefix``.
#. Bookmark the snapshot created above.
#. Replicate *mapped filesystems* to their respective *target filesystems*:
#. Only snapshots with prefix ``snapshot_prefix`` are replicated.
#. If possible, incremental replication takes place.
#. If the *target filesystem* does not exist, the most recent snapshot is sent fully (non-incremental).
#. On conflicts, an error is logged but replication of other *mapped filesystems* continues.
#. The ``prune_lhs`` policy is triggered for all *mapped filesystems*
#. The ``prune_rhs`` policy is triggered for all *target filesystems*
A local job is combination of source & pull job executed on the same machine.
Terminology
-----------
task
A job consists of one or more tasks and a task consists of one or more steps.
Some tasks may be periodic while others wait for an event to occur.
patient task
.. _job-term-patient:
A patient task is supposed to execute some task every `interval`.
We call the start of the task an *invocation*.
* If the task completes in less than `interval`, the task is restarted at `last_invocation + interval`.
* Otherwise, a patient job
* logs a warning as soon as a task exceeds its configured `interval`
* waits for the last invocation to finish
* logs a warning with the effective task duration
* immediately starts a new invocation of the task
filesystem version
A snapshot or a bookmark.
docs/configuration/logging.rst
+12 -13
View File
@@ -8,18 +8,17 @@ Logging
zrepl uses structured logging to provide users with easily processable log messages.
Logging outlets are configured in the ``global`` section of the |mainconfig|.
Check out :sampleconf:`random/logging_and_monitoring.yml` for an example on how to configure multiple outlets:
::
global:
logging:
- outlet: OUTLET_TYPE
- type: OUTLET_TYPE
level: MINIMUM_LEVEL
format: FORMAT
- outlet: OUTLET_TYPE
- type: OUTLET_TYPE
level: MINIMUM_LEVEL
format: FORMAT
@@ -45,7 +44,7 @@ By default, the following logging configuration is used
global:
logging:
- outlet: "stdout"
- type: "stdout"
level: "warn"
format: "human"
@@ -93,8 +92,8 @@ Formats
* - Format
- Description
* - ``human``
- emphasizes context by putting job, task, step and other context variables into brackets
before the actual message, followed by remaining fields in logfmt style|
- prints job and subsystem into brackets before the actual message,
followed by remaining fields in logfmt style
* - ``logfmt``
- `logfmt <https://brandur.org/logfmt>`_ output. zrepl uses `this Go package <https://github.com/go-logfmt/logfmt>`_.
* - ``json``
@@ -118,7 +117,7 @@ Outlets are the destination for log entries.
* - Parameter
- Comment
* - ``outlet``
* - ``type``
- ``stdout``
* - ``level``
- minimum :ref:`log level <logging-levels>`
@@ -126,9 +125,11 @@ Outlets are the destination for log entries.
- output :ref:`format <logging-formats>`
* - ``time``
- always include time in output (``true`` or ``false``)
* - ``color``
- colorize output according to log level (``true`` or ``false``)
Writes all log entries with minimum level ``level`` formatted by ``format`` to stdout.
If stdout is a tty, interactive usage is assumed and the current time is included in the output.
If stdout is a tty, interactive usage is assumed and both ``time`` and ``color`` are set to ``true``.
Can only be specified once.
@@ -140,7 +141,7 @@ Can only be specified once.
* - Parameter
- Comment
* - ``outlet``
* - ``type``
- ``syslog``
* - ``level``
- minimum :ref:`log level <logging-levels>`
@@ -163,7 +164,7 @@ Can only be specified once.
* - Parameter
- Comment
* - ``outlet``
* - ``type``
- ``tcp``
* - ``level``
- minimum :ref:`log level <logging-levels>`
@@ -179,11 +180,9 @@ Can only be specified once.
- TLS config (see below)
Establishes a TCP connection to ``address`` and sends log messages with minimum level ``level`` formatted by ``format``.
If ``tls`` is not specified, an unencrypted connection is established.
If ``tls`` is specified, the TCP connection is secured with TLS + Client Authentication.
This is particularly useful in combination with log aggregation services that run on an other machine.
The latter is particularly useful in combination with log aggregation services.
.. list-table::
:widths: 10 90
docs/configuration/map_filter_syntax.rst
-108
View File
@@ -1,108 +0,0 @@
.. include:: ../global.rst.inc
Mapping & Filter Syntax
=======================
For various job types, a filesystem ``mapping`` or ``filter`` needs to be
specified.
Both have in common that they take a filesystem path (in the ZFS filesystem hierarchy)as parameters and return something.
Mappings return a *target filesystem* and filters return a *filter result*.
The pattern syntax is the same for mappings and filters and is documented in the following section.
Common Pattern Syntax
---------------------
A mapping / filter is specified as a **YAML dictionary** with patterns as keys and
results as values.
The following rules determine which result is chosen for a given filesystem path:
* More specific path patterns win over less specific ones
* Non-wildcard patterns (full path patterns) win over *subtree wildcards* (`<` at end of pattern)
The **subtree wildcard** ``<`` means "the dataset left of ``<`` and all its children".
Example
~~~~~~~
::
# Rule number and its pattern
1: tank< # tank and all its children
2: tank/foo/bar # full path pattern (no wildcard)
3: tank/foo< # tank/foo and all its children
# Which rule applies to given path?
tank/foo/bar/loo => 3
tank/bar => 1
tank/foo/bar => 2
zroot => NO MATCH
tank/var/log => 1
.. _pattern-mapping:
Mappings
--------
Mappings map a *source filesystem path* to a *target filesystem path*.
Per pattern, either a target filesystem path or ``"!"`` is specified as a result.
* If no pattern matches, there exists no target filesystem (``NO MATCH``).
* If the result is a ``"!"``, there exists no target filesystem (``NO MATCH``).
* If the pattern is a non-wildcard pattern, the source path is mapped to the target path on the right.
* If the pattern ends with a *subtree wildcard* (``<``), the source path is **prefix-trimmed** with the path specified left of ``<``.
* Note: this means that only for *wildcard-only* patterns (pattern= ``<`` ) is the source path simply appended to the target path.
The example is from the :sampleconf:`localbackup/host1.yml` example config.
::
jobs:
- name: mirror_local
type: local
mapping: {
"zroot/var/db<": "storage/backups/local/zroot/var/db",
"zroot/usr/home<": "storage/backups/local/zroot/usr/home",
"zroot/usr/home/paranoid": "!", #don't backup paranoid user
"zroot/poudriere/ports<": "!", #don't backup the ports trees
}
...
::
zroot/var/db => storage/backups/local/zroot/var/db
zroot/var/db/a/child => storage/backups/local/zroot/var/db/a/child
zroot/usr/home => storage/backups/local/zroot/usr/home
zroot/usr/home/paranoid => NOT MAPPED
zroot/usr/home/bob => storage/backups/local/zroot/usr/home/bob
zroot/usr/src => NOT MAPPED
zroot/poudriere/ports/2017Q3 => NOT MAPPED
zroot/poudriere/ports/HEAD => NOT MAPPED
.. TIP::
You can try out patterns for a configured job using the ``zrepl test`` subcommand.
.. _pattern-filter:
Filters
-------
Valid filter results: ``ok`` or ``!``.
The example below show the source job from the :ref:`tutorial <tutorial-configure-app-srv>`:
The corresponding pull job is allowed access to ``zroot/var/db``, ``zroot/usr/home`` + children except ``zroot/usr/home/paranoid``::
jobs:
- name: pull_backup
type: source
...
filesystems: {
"zroot/var/db": "ok",
"zroot/usr/home<": "ok",
"zroot/usr/home/paranoid": "!",
}
...
docs/configuration/monitoring.rst
+1 -2
View File
@@ -6,7 +6,6 @@ Monitoring
==========
Monitoring endpoints are configured in the ``global.monitoring`` section of the |mainconfig|.
Check out :sampleconf:`random/logging_and_monitoring.yml` for examples.
.. _monitoring-prometheus:
@@ -17,7 +16,7 @@ zrepl can expose `Prometheus metrics <https://prometheus.io/docs/instrumenting/e
The ``listen`` attribute is a `net.Listen <https://golang.org/pkg/net/#Listen>`_ string for tcp, e.g. ``:9091`` or ``127.0.0.1:9091``.
The Prometheues monitoring job appears in the ``zrepl control`` job list and may be specified **at most once**.
There is no stability guarantee on the exported metrics.
At the time of writing, there is no stability guarantee on the exported metrics.
::
docs/configuration/preface.rst
+1 -1
View File
@@ -15,7 +15,7 @@ zrepl searches for its main configuration file in the following locations (in th
* ``/etc/zrepl/zrepl.yml``
* ``/usr/local/etc/zrepl/zrepl.yml``
The examples in the :ref:`tutorial` or the ``cmd/sampleconf`` directory should provide a good starting point.
The examples in the :ref:`tutorial` or the :sampleconf:`/` directory should provide a good starting point.
-------------------
Runtime Directories
docs/configuration/prune.rst
+114 -41
View File
@@ -3,46 +3,96 @@
Pruning Policies
================
In zrepl, *pruning* means *destroying filesystem versions by some policy* where filesystem versions are bookmarks and snapshots.
In zrepl, *pruning* means *destroying snapshots*.
Pruning must happen on both sides of a replication or the systems would inevitable run out of disk space at some point.
A *pruning policy* takes a list of filesystem versions and decides for each whether it should be kept or destroyed.
Typically, the requirements to temporal resolution and maximum retention time differ per side.
For example, when using zrepl to back up a busy database server, you will want high temporal resolution (snapshots every 10 min) for the last 24h in case of administrative disasters, but cannot afford to store them for much longer because you might have high turnover volume in the database.
On the receiving side, you may have more disk space available, or need to comply with other backup retention policies.
The job context defines which snapshots are even considered for pruning, for example through the ``snapshot_prefix`` variable.
Check the respective :ref:`job definition <job>` for details.
zrepl uses a set of **keep rules** to determine which snapshots shall be kept per filesystem.
**A snapshot that is not kept by any rule is destroyed.**
The keep rules are **evaluated on the active side** (:ref:`push <job-push>` or :ref:`pull job <job-pull>`) of the replication setup, for both active and passive side, after replication completed or was determined to have failed permanently.
Currently, the :ref:`prune-retention-grid` is the only supported pruning policy.
Example Configuration:
::
jobs:
- type: push
name: ...
connect: ...
filesystems: {
"<": true,
"tmp": false
}
snapshotting:
type: periodic
prefix: zrepl_
interval: 10m
pruning:
keep_sender:
- type: not_replicated
# make sure manually created snapshots by the administrator are kept
- type: regex
regex: "^manual_.*"
- type: grid
grid: 1x1h(keep=all) | 24x1h | 14x1d
regex: "^zrepl_.*"
keep_receiver:
- type: grid
grid: 1x1h(keep=all) | 24x1h | 35x1d | 6x30d
regex: "^zrepl_.*"
# manually created snapshots will be kept forever on receiver
.. TIP::
You can perform a dry-run of a job's pruning policy using the ``zrepl test`` subcommand.
.. _prune-retention-grid:
.. ATTENTION::
Retention Grid
--------------
It is currently not possible to define pruning on a source job.
The source job creates snapshots, which means that extended replication downtime will fill up the source's zpool with snapshots, since pruning is directed by the corresponding active side (pull job).
If this is a potential risk for you, consider using :ref:`push mode <job-push>`.
.. _prune-keep-not-replicated:
Policy ``not_replicated``
-------------------------
::
jobs:
- type: push
pruning:
keep_sender:
- type: not_replicated
...
``not_replicated`` keeps all snapshots that have not been replicated to the receiving side.
It only makes sense to specify this rule on a sender (source or push job).
The state required to evaluate this rule is stored in the :ref:`replication cursor bookmark <replication-cursor-bookmark>` on the sending side.
.. _prune-keep-retention-grid:
Policy ``grid``
---------------
::
jobs:
- name: pull_app-srv
type: pull
...
prune:
policy: grid
grid: 1x1h(keep=all) | 24x1h | 35x1d | 6x30d
│ │
- type: pull
pruning:
keep_receiver:
- type: grid
regex: "^zrepl_.*"
grid: 1x1h(keep=all) | 24x1h | 35x1d | 6x30d
│ │
└─ one hour interval
│
└─ 24 adjacent one-hour intervals
- name: pull_backup
type: source
interval: 10m
prune:
policy: grid
grid: 1x1d(keep=all)
keep_bookmarks: 144
│
└─ 24 adjacent one-hour intervals
...
The retention grid can be thought of as a time-based sieve:
The ``grid`` field specifies a list of adjacent time intervals:
@@ -51,16 +101,13 @@ All intervals to its right describe time intervals further in the past.
Each interval carries a maximum number of snapshots to keep.
It is secified via ``(keep=N)``, where ``N`` is either ``all`` (all snapshots are kept) or a positive integer.
The default value is **1**.
Bookmarks are not affected by the above.
Instead, the ``keep_bookmarks`` field specifies the number of bookmarks to be kept per filesystem.
You only need to specify ``keep_bookmarks`` at the source-side of a replication setup since the destination side does not receive bookmarks.
You can specify ``all`` as a value to keep all bookmarks, but be warned that you should install some other way to prune unneeded ones then (see below).
The default value is **keep=1**.
The following procedure happens during pruning:
#. The list of snapshots eligible for pruning is sorted by ``creation``
#. The list of snapshots is filtered by the regular expression in ``regex``.
Only snapshots names that match the regex are considered for this rule, all others are not affected.
#. The filtered list of snapshots is sorted by ``creation``
#. The left edge of the first interval is aligned to the ``creation`` date of the youngest snapshot
#. A list of buckets is created, one for each interval
#. The list of snapshots is split up into the buckets.
@@ -69,16 +116,42 @@ The following procedure happens during pruning:
#. the contained snapshot list is sorted by creation.
#. snapshots from the list, oldest first, are destroyed until the specified ``keep`` count is reached.
#. all remaining snapshots on the list are kept.
#. The list of bookmarks eligible for pruning is sorted by ``createtxg`` and the most recent ``keep_bookmarks`` bookmarks are kept.
.. _replication-downtime:
.. ATTENTION::
.. _prune-keep-last-n:
Policy ``last_n``
-----------------
::
jobs:
- type: push
pruning:
keep_receiver:
- type: last_n
count: 10
...
``last_n`` keeps the last ``count`` snapshots (last = youngest = most recent creation date).
.. _prune-keep-regex:
Policy ``regex``
----------------
::
jobs:
- type: push
pruning:
keep_receiver:
- type: regex
regex: "^(zrepl|manual)_.*"
...
``regex`` keeps all snapshots whose names are matched by the regular expressionin ``regex``.
Like all other regular expression fields in prune policies, zrepl uses Go's `regexp.Regexp <https://golang.org/pkg/regexp/#Compile>`_ Perl-compatible regular expressions (`Syntax <https://golang.org/pkg/regexp/syntax>`_).
Be aware that ``keep_bookmarks x interval`` (interval of the job level) controls the **maximum allowable replication downtime** between source and destination.
If replication does not work for whatever reason, source and destination will eventually run out of sync because the source will continue pruning snapshots.
The only recovery in that case is full replication, which may not always be viable due to disk space or traffic constraints.
Further note that while bookmarks consume a constant amount of disk space, listing them requires temporary dynamic **kernel memory** proportional to the number of bookmarks.
Thus, do not use ``all`` or an inappropriately high value without good reason.
docs/configuration/transports.rst
+171 -26
View File
@@ -5,42 +5,156 @@
Transports
==========
A transport provides an authenticated `io.ReadWriteCloser <https://golang.org/pkg/io/#ReadWriteCloser>`_ to the RPC layer.
(An ``io.ReadWriteCloser`` is essentially a bidirectional reliable communication channel.)
The zrepl RPC layer uses **transports** to establish a single, bidirectional data stream between an active and passive job.
On the passive (serving) side, the transport also provides the **client identity** to the upper layers:
this string is used for access control and separation of filesystem sub-trees in :ref:`sink jobs <job-sink>`.
Transports are specified in the ``connect`` or ``serve`` section of a job definition.
Currently, only the ``ssh+stdinserver`` transport is supported.
.. ATTENTION::
The **client identities must be valid ZFS dataset path components**
because the :ref:`sink job <job-sink>` uses ``${root_fs}/${client_identity}`` to determine the client's subtree.
.. _transport-tcp:
``tcp`` Transport
-----------------
The ``tcp`` transport uses plain TCP, which means that the data is **not encrypted** on the wire.
Clients are identified by their IPv4 or IPv6 addresses, and the client identity is established through a mapping on the server.
This transport may also be used in conjunction with network-layer encryption and/or VPN tunnels to provide encryption on the wire.
To make the IP-based client authentication effective, such solutions should provide authenticated IP addresses.
Some options to consider:
* `WireGuard <https://www.wireguard.com/>`_: Linux-focussed, in-kernel TLS
* `OpenVPN <https://openvpn.net/>`_: Cross-platform VPN, uses tun on \*nix
* `IPSec <https://en.wikipedia.org/wiki/IPsec>`_: Properly standardized, in-kernel network-layer VPN
* `spiped <http://www.tarsnap.com/spiped.html>`_: think of it as an encrypted pipe between two servers
* SSH
* `sshuttle <https://sshuttle.readthedocs.io/en/stable/overview.html>`_: VPN-like solution, but using SSH
* `SSH port forwarding <https://help.ubuntu.com/community/SSH/OpenSSH/PortForwarding>`_: Systemd user unit & make it start before the zrepl service.
Serve
~~~~~
::
jobs:
- type: sink
serve:
type: tcp
listen: ":8888"
clients: {
"192.168.122.123" : "mysql01"
"192.168.122.123" : "mx01"
}
...
Connect
~~~~~~~
::
jobs:
- type: push
connect:
type: tcp
address: "10.23.42.23:8888"
dial_timeout: # optional, default 10s
...
.. _transport-tcp+tlsclientauth:
``tls`` Transport
-----------------
The ``tls`` transport uses TCP + TLS with client authentication using client certificates.
The client identity is the common name (CN) presented in the client certificate.
It is recommended to set up a dedicated CA infrastructure for this transport, e.g. using OpenVPN's `EasyRSA <https://github.com/OpenVPN/easy-rsa>`_.
The implementation uses `Go's TLS library <https://golang.org/pkg/crypto/tls/>`_.
Since Go binaries are statically linked, you or your distribution need to recompile zrepl when vulnerabilities in that library are disclosed.
All file paths are resolved relative to the zrepl daemon's working directory.
Specify absolute paths if you are unsure what directory that is (or find out from your init system).
Serve
~~~~~
::
jobs:
- type: sink
root_fs: "pool2/backup_laptops"
serve:
type: tls
listen: ":8888"
ca: /etc/zrepl/ca.crt
cert: /etc/zrepl/prod.crt
key: /etc/zrepl/prod.key
client_cns:
- "laptop1"
- "homeserver"
The ``ca`` field specified the certificate authority used to validate client certificates.
The ``client_cns`` list specifies a list of accepted client common names (which are also the client identities for this transport).
Connect
~~~~~~~
::
jobs:
- type: pull
connect:
type: tls
address: "server1.foo.bar:8888"
ca: /etc/zrepl/ca.crt
cert: /etc/zrepl/backupserver.crt
key: /etc/zrepl/backupserver.key
server_cn: "server1"
dial_timeout: # optional, default 10s
The ``ca`` field specifies the CA which signed the server's certificate (``serve.cert``).
The ``server_cn`` specifies the expected common name (CN) of the server's certificate.
It overrides the hostname specified in ``address``.
The connection fails if either do not match.
.. _transport-ssh+stdinserver:
``ssh+stdinserver`` Transport
-----------------------------
The way the ``ssh+stdinserver`` transport works is inspired by `git shell <https://git-scm.com/docs/git-shell>`_ and `Borg Backup <https://borgbackup.readthedocs.io/en/stable/deployment.html>`_.
``ssh+stdinserver`` is inspired by `git shell <https://git-scm.com/docs/git-shell>`_ and `Borg Backup <https://borgbackup.readthedocs.io/en/stable/deployment.html>`_.
It is provided by the Go package ``github.com/problame/go-netssh``.
The config excerpts are taken from the :ref:`tutorial` which you should complete before reading further.
.. _transport-ssh+stdinserver-serve:
Serve Mode
~~~~~~~~~~
Serve
~~~~~
::
jobs:
- name: pull_backup
type: source
- type: source
serve:
type: stdinserver
client_identity: backup-srv.example.com
client_identities:
- "client1"
- "client2"
...
The serving job opens a UNIX socket named after ``client_identity`` in the runtime directory, e.g. ``/var/run/zrepl/stdinserver/backup-srv.example.com``.
First of all, note that ``type=stdinserver`` in this case:
Currently, only ``connect.type=ssh+stdinserver`` can connect to a ``serve.type=stdinserver``, but we want to keep that option open for future extensions.
On the same machine, the ``zrepl stdinserver $client_identity`` command connects to that socket.
For example, ``zrepl stdinserver backup-srv.example.com`` connects to the UNIX socket ``/var/run/zrepl/stdinserver/backup-srv.example.com``.
The serving job opens a UNIX socket named after ``client_identity`` in the runtime directory.
In our example above, that is ``/var/run/zrepl/stdinserver/client1`` and ``/var/run/zrepl/stdinserver/client2``.
On the same machine, the ``zrepl stdinserver $client_identity`` command connects to ``/var/run/zrepl/stdinserver/$client_identity``.
It then passes its stdin and stdout file descriptors to the zrepl daemon via *cmsg(3)*.
zrepl daemon in turn combines them into an ``io.ReadWriteCloser``:
zrepl daemon in turn combines them into an object implementing ``net.Conn``:
a ``Write()`` turns into a write to stdout, a ``Read()`` turns into a read from stdin.
Interactive use of the ``stdinserver`` subcommand does not make much sense.
@@ -54,8 +168,8 @@ This can be achieved with an entry in the ``authorized_keys`` file of the servin
# for older OpenSSH versions
command="zrepl stdinserver CLIENT_IDENTITY",no-port-forwarding,no-X11-forwarding,no-pty,no-agent-forwarding,no-user-rc CLIENT_SSH_KEY
* CLIENT_IDENTITY is substituted with ``backup-srv.example.com`` in our example
* CLIENT_SSH_KEY is substituted with the public part of the SSH keypair specified in the ``connect`` directive on the connecting host.
* CLIENT_IDENTITY is substituted with an entry from ``client_identities`` in our example
* CLIENT_SSH_KEY is substituted with the public part of the SSH keypair specified in the ``connect.identity_file`` directive on the connecting host.
.. NOTE::
@@ -64,24 +178,24 @@ This can be achieved with an entry in the ``authorized_keys`` file of the servin
To recap, this is of how client authentication works with the ``ssh+stdinserver`` transport:
* Connections to the ``client_identity`` UNIX socket are blindly trusted by zrepl daemon.
* Thus, the runtime directory must be private to the zrepl user (checked by zrepl daemon)
* Connections to the ``/var/run/zrepl/stdinserver/${client_identity}`` UNIX socket are blindly trusted by zrepl daemon.
The connection client identity is the name of the socket, i.e. ``${client_identity}``.
* Thus, the runtime directory must be private to the zrepl user (this is checked by zrepl daemon)
* The admin of the host with the serving zrepl daemon controls the ``authorized_keys`` file.
* Thus, the administrator controls the mapping ``PUBKEY -> CLIENT_IDENTITY``.
.. _transport-ssh+stdinserver-connect:
Connect Mode
~~~~~~~~~~~~
Connect
~~~~~~~
::
jobs:
- name: pull_app-srv
type: pull
- type: pull
connect:
type: ssh+stdinserver
host: app-srv.example.com
host: prod.example.com
user: root
port: 22
identity_file: /etc/zrepl/ssh/identity
@@ -102,15 +216,46 @@ The connecting zrepl daemon
#. The remote user, host and port correspond to those configured.
#. Further options can be specified using the ``options`` field, which appends each entry in the list to the command line using ``-o $entry``.
#. Wraps the pipe ends in an ``io.ReadWriteCloser`` and uses it for RPC.
#. Wraps the pipe ends in a ``net.Conn`` and returns it to the RPC layer.
As discussed in the section above, the connecting zrepl daemon expects that ``zrepl stdinserver $client_identity`` is executed automatically via an ``authorized_keys`` file entry.
The ``known_hosts`` file used by the ssh command must contain an entry for the serving host, e.g., ``app-srv.example.com`` in the example above.
The ``known_hosts`` file used by the ssh command must contain an entry for ``connect.host`` prior to starting zrepl.
Thus, run the following on the pulling host's command line (substituting ``connect.host``):
::
ssh -i /etc/zrepl/ssh/identity root@prod.example.com
.. NOTE::
The environment variables of the underlying SSH process are cleared. ``$SSH_AUTH_SOCK`` will not be available.
It is suggested to create a separate, unencrypted SSH key solely for that purpose.
.. _transport-local:
``local`` Transport
-------------------
The local transport can be used to implement :ref:`local replication <replication-local>`, i.e., push replication between a push and sink job defined in the same configuration file.
The ``listener_name`` is analogous to a hostname and must match between ``serve`` and ``connect``.
The ``client_identity`` is used by the sink as documented above.
::
jobs:
- type: sink
serve:
type: local
listener_name: localsink
...
- type: push
connect:
type: local
listener_name: localsink
client_identity: local_backup
...