bookmarking: prune policy for bookmarks

refs #34

docs/configuration/prune.rst

@@ -3,9 +3,9 @@
 Pruning Policies
 ================
 
-In zrepl, *pruning* means *destroying snapshots by some policy*.
+In zrepl, *pruning* means *destroying filesystem versions by some policy* where filesystem versions are bookmarks and snapshots.
 
-A *pruning policy* takes a list of snapshots and -- for each snapshot -- decides whether it should be kept or destroyed.
+A *pruning policy* takes a list of filesystem versions and decides for each whether it should be kept or destroyed.
 
 The job context defines which snapshots are even considered for pruning, for example through the ``snapshot_prefix`` variable.
 Check the respective :ref:`job definition <job>` for details.
@@ -25,6 +25,7 @@ Retention Grid
 
     jobs:
     - name: pull_app-srv
+      type: pull
       ...
       prune:
         policy: grid
@@ -34,6 +35,15 @@ Retention Grid
                                 │
                                 └─ 24 adjacent one-hour intervals
 
+    - name: pull_backup
+      type: source
+      interval: 10m
+      prune:
+        policy: grid
+        grid: 1x1d(keep=all)
+        keep_bookmarks: 144
+
+
 The retention grid can be thought of as a time-based sieve:
 The ``grid`` field specifies a list of adjacent time intervals:
 the left edge of the leftmost (first) interval is the ``creation`` date of the youngest snapshot.
@@ -43,6 +53,11 @@ Each interval carries a maximum number of snapshots to keep.
 It is secified via ``(keep=N)``, where ``N`` is either ``all`` (all snapshots are kept) or a positive integer.
 The default value is **1**.
 
+Bookmarks are not affected by the above.
+Instead, the ``keep_bookmarks`` field specifies the number of bookmarks to be kept per filesystem.
+You only need to specify ``keep_bookmarks`` at the source-side of a replication setup since the destination side does not receive bookmarks.
+You can specify ``all`` as a value to keep all bookmarks, but be warned that you should install some other way to prune unneeded ones then (see below).
+
 The following procedure happens during pruning:
 
 #. The list of snapshots eligible for pruning is sorted by ``creation``
@@ -54,14 +69,16 @@ The following procedure happens during pruning:
    #. the contained snapshot list is sorted by creation.
    #. snapshots from the list, oldest first, are destroyed until the specified ``keep`` count is reached.
    #. all remaining snapshots on the list are kept.
+#. The list of bookmarks eligible for pruning is sorted by ``createtxg`` and the most recent ``keep_bookmarks`` bookmarks are kept.
+
+.. _replication-downtime:
 
 .. ATTENTION::
 
-   The configuration of the first interval (``1x1h(keep=all)`` in the example) determines the **maximum allowable replication lag** because the source and destination pruning policies do not coordinate:
-   if replication does not work for whatever reason, source will continue to execute the prune policy.
-   Eventually, source destroys a snapshot that has never been replicated to destination, degrading the temporal resolution of your backup.
+    Be aware that ``keep_bookmarks x interval`` (interval of the job level) controls the **maximum allowable replication downtime** between source and destination.
+    If replication does not work for whatever reason, source and destination will eventually run out of sync because the source will continue pruning snapshots.
+    The only recovery in that case is full replication, which may not always be viable due to disk space or traffic constraints.
 
-   Thus, **always** configure the first interval to ``1x?(keep=all)``, substituting ``?`` with the maximum time replication may fail due to downtimes, maintenance, connectivity issues, etc.
-
-.. We intentionally do not mention that bookmarks are used to bridge the gap between source and dest that are out of sync snapshot-wise. This is an implementation detail.
+    Further note that while bookmarks consume a constant amount of disk space, listing them requires temporary dynamic **kernel memory** proportional to the number of bookmarks.
+    Thus, do not use ``all`` or an inappropriately high value without good reason.