new features: {resumable,encrypted,hold-protected} send-recv, last-received-hold

- **Resumable Send & Recv Support** No knobs required, automatically used where supported. - **Hold-Protected Send & Recv** Automatic ZFS holds to ensure that we can always resume a replication step. - **Encrypted Send & Recv Support** for OpenZFS native encryption. Configurable at the job level, i.e., for all filesystems a job is responsible for. - **Receive-side hold on last received dataset** The counterpart to the replication cursor bookmark on the send-side. Ensures that incremental replication will always be possible between a sender and receiver. Design Doc ---------- `replication/design.md` doc describes how we use ZFS holds and bookmarks to ensure that a single replication step is always resumable. The replication algorithm described in the design doc introduces the notion of job IDs (please read the details on this design doc). We reuse the job names for job IDs and use `JobID` type to ensure that a job name can be embedded into hold tags, bookmark names, etc. This might BREAK CONFIG on upgrade. Protocol Version Bump --------------------- This commit makes backwards-incompatible changes to the replication/pdu protobufs. Thus, bump the version number used in the protocol handshake. Replication Cursor Format Change -------------------------------- The new replication cursor bookmark format is: `#zrepl_CURSOR_G_${this.GUID}_J_${jobid}` Including the GUID enables transaction-safe moving-forward of the cursor. Including the job id enables that multiple sending jobs can send the same filesystem without interfering. The `zrepl migrate replication-cursor:v1-v2` subcommand can be used to safely destroy old-format cursors once zrepl has created new-format cursors. Changes in This Commit ---------------------- - package zfs - infrastructure for holds - infrastructure for resume token decoding - implement a variant of OpenZFS's `entity_namecheck` and use it for validation in new code - ZFSSendArgs to specify a ZFS send operation - validation code protects against malicious resume tokens by checking that the token encodes the same send parameters that the send-side would use if no resume token were available (i.e. same filesystem, `fromguid`, `toguid`) - RecvOptions support for `recv -s` flag - convert a bunch of ZFS operations to be idempotent - achieved through more differentiated error message scraping / additional pre-/post-checks - package replication/pdu - add field for encryption to send request messages - add fields for resume handling to send & recv request messages - receive requests now contain `FilesystemVersion To` in addition to the filesystem into which the stream should be `recv`d into - can use `zfs recv $root_fs/$client_id/path/to/dataset@${To.Name}`, which enables additional validation after recv (i.e. whether `To.Guid` matched what we received in the stream) - used to set `last-received-hold` - package replication/logic - introduce `PlannerPolicy` struct, currently only used to configure whether encrypted sends should be requested from the sender - integrate encryption and resume token support into `Step` struct - package endpoint - move the concepts that endpoint builds on top of ZFS to a single file `endpoint/endpoint_zfs.go` - step-holds + step-bookmarks - last-received-hold - new replication cursor + old replication cursor compat code - adjust `endpoint/endpoint.go` handlers for - encryption - resumability - new replication cursor - last-received-hold - client subcommand `zrepl holds list`: list all holds and hold-like bookmarks that zrepl thinks belong to it - client subcommand `zrepl migrate replication-cursor:v1-v2`
2019-09-11 17:19:17 +02:00
parent 9a4763ceee
commit 58c08c855f
72 changed files with 5445 additions and 818 deletions
@@ -0,0 +1,342 @@
+The goal of this document is to describe what **logical steps** zrepl takes to replicate ZFS filesystems.
+Note that the actual code executing these steps is spread over the `endpoint.{Sender,Receiver}` RPC endpoint implementations. The code in `replication/{driver,logic}` (mostly `logic.Step.doReplication`) drives the replication and invokes the `endpoint` methods (locally or via RPC).
+Hence, when trying to map algorithm to implementation, use the code in package `replication` as the entrypoint and follow the RPC calls.
+
+
+* The [Single Replication Step](#zrepl-algo-single-step) algorithm is implemented as described
+  * step holds are implemented as described
+  * step bookmarks rely on bookmark cloning, which is WIP in OpenZFS (see respective TODO comments)
+  * the feature to hold receive-side `to` immediately after replication is used to move the `last-received-hold` forward
+    * this is a little more than what we allow in the algorithm (we only allow specifying a hold tag whereas moving the `last-received-hold` is a little more complex)
+  * the algorithm's sender-side callback is used move the `zrepl_CURSOR` bookmark to point to `to`
+* The `zrepl_CURSOR` bookmark and the `last-received-hold` ensure that future replication steps can always be done using incremental replication:
+  * Both reference / hold the last successfully received snapshot `to`.
+  * Thus, `to` or `#zrepl_CURSOR_G_${to.GUID}_J_${jobid}` can always be used for a future incremental replication step `to => future-to`:
+    * incremental send will be possible because `to` doesn't go away because we claim ownership of the `#zrepl_CURSOR` namespace
+    * incremental recv will be possible iff
+      * `to` will still be present on the recv-side because of `last-received-hold`
+      * the recv-side fs doesn't get newer snapshots than `to` in the meantime
+        * guaranteed because the zrepl model of the receiver assumes ownership of the filesystems it receives into
+          * if that assumption is broken, future replication attempts will fail with a conflict
+* The [Algorithm for Planning and Executing Replication of an Filesystems](#zrepl-algo-filesystem) is a design draft and not used
+* We also have [Notes on Planning and Executing Replication of Multiple Filesystems](#zrepl-algo-multiple-filesystems-notes)
+
+---
+
+<a id="zrepl-algo-single-step"></a>
+## Algorithm for a Single Replication Step
+
+The algorithm described below describes how a single replication step must be implemented.
+A *replication step* is a full send of a snapshot `to` for initial replication, or an incremental send (`from => to`) for incremental replication.
+
+The algorithm **ensures resumability** of the replication step in presence of
+
+* uncoordinated or unsynchronized destruction of the filesystem, snapshots, or bookmarks involved in the replication step
+* network failures at any time
+* other instances of this algorithm executing the same step in parallel (e.g. concurrent replication to different destinations)
+
+To accomplish this goal, the algorithm **assumes ownersip of parts of the ZFS hold tag namespace and the bookmark namespace**:
+* holds with prefix `zrepl_STEP` on any snapshot are reserved for zrepl
+* bookmarks with prefix `zrepl_STEP` are reserved for zrepl
+
+Resumability of a step cannot be guaranteed if these sub-namespaces are manipulated through software other than zrepl.
+
+Note that the algorithm **does not ensure** that a replication *plan*, which describes a *set* of replication steps, can be carried out successfully.
+If that is desirable, additional measures outside of this algorithm must be taken.
+
+---
+
+### Definitions:
+
+#### Step Completion & Invariants
+
+The replication step (full `to` send or `from => to` send) is *complete* iff the algorithm ran to completion without errors or a permanent non-network error is reported by sender or receiver.
+
+Specifically, the algorithm may be invoked with the same `from` and `to` arguments, and potentially a `resume_token`, after a temporary (like network-related) failure:
+**Unless permanent errors occur, repeated invocations of the algorithm with updated resume token will converge monotonically (but not strictly monotonically) toward completion.**
+
+Note that the mere existence of `to` on the receiving side does not constitue completion, since there may still be post-recv actions to be performed on sender and receiver.
+
+#### Job and Job ID
+This algorithm supports that *multiple* instance of it run in parallel on the *same* step (full `to` / `from => to` pair).
+An exemplary use case for this feature are concurrent replication jobs that replicate the same dataset to different receivers.
+
+**We require that all parallel invocations of this algorithm provide different and unique `jobid`s.**
+Violation of `jobid` uniqueness across parallel jobs may result in interference between instances of this algorithm, resulting in potential compromise of resumability.
+
+After a step is *completed*, `jobid` is guaranteed to not be encoded in on-disk state.
+Before a step is completed, there is no such guarantee.
+Changing the `jobid` before step completion may compromise resumability and may leak the underlying ZFS holds or step bookmarks (i.e. zrepl won't clean them up)
+Note the definition of *complete* above.
+
+#### Step Bookmark
+
+A step bookmark is our equivalent of ZFS holds, but for bookmarks.<br/>
+A step bookmark is a ZFS bookmark whose name matches the following regex:
+
+```
+STEP_BOOKMARK = #zrepl_STEP_bm_G_([0-9a-f]+)_J_(.+)
+```
+
+- capture group `1` must be the guid of `zfs get guid ${STEP_BOOKMARK}`, encoded hexadecimal (without leading `0x`) and fixed-length (i.e. padded with leading zeroes)
+- capture group `2` must be equal to the `jobid`
+
+### Algorithm
+
+INPUT:
+
+* `jobid`
+* `from`: snapshot or bookmark: may be nil for full send
+* `to`: snapshot, must never be nil
+* `resume_token` (may be nil)
+* (`from` and `to` must be on the same filesystem)
+
+#### Prepare-Phase
+
+Send-side: make sure `to` and `from` don't go away
+
+- hold `to` using `idempotent_hold(to, zrepl_STEP_J_${jobid})`
+- make sure `from` doesn't go away:
+  - if `from` is a snapshot: hold `from` using `idempotent_hold(from, zrepl_STEP_J_${jobid})`
+  - else `idempotent_step_bookmark(from)` (`from` is a bookmark)
+    - PERMANENT ERROR if this fails (e.g. because `from` is a bookmark whose snapshot no longer exists and we don't have bookmark copying yet (ZFS feature is in development) )
+      - Why? we must assume the given bookmark is externally managed (i.e. not in the )
+      - this means the bookmark is externally created bookmark and cannot be trusted to persist until the replication step succeeds
+      - Maybe provide an override-knob for this behavior
+
+Recv-side: no-op
+
+- `from` cannot go away once we received enough data for the step to be resumable:
+    ```text
+    # do a partial incremental recv @a=>@b (i.e. recv with -s, and abort the incremental recv in the middle)
+    # try destroying the incremental source on the receiving side
+    zfs destroy p1/recv@a
+    cannot destroy 'p1/recv@a': snapshot has dependent clones
+    use '-R' to destroy the following datasets:
+    p1/recv/%recv
+    # => doesn't work, because zfs recv is implemented as a `clone` internally, that's exactly what we want
+    ```
+
+- if recv-side `to` exists, goto cleaup-phase (no replication to do)
+  
+- `to` cannot be destroyed while being received, because it isn't visible as a snapshot yet (it isn't yet one after all)
+
+#### Replication Phase
+
+Attempt the replication step:
+start `zfs send` and pipe its output into `zfs recv` until an error occurs or `recv` returns without error.
+
+Let us now think about interferences that could occur during this phase, and ensure that none of them compromise the goals of this algorithm, i.e., monotonic convergence toward step completion using resumable send & recv.
+
+**Safety from External Pruning**
+We are safe from pruning during the replication step because we have guarantees that no external action will destroy send-side `from` and `to`, and recv-side `to` (for both snapshot and bookmark `from`s)<br/>
+
+**Safety In Presence of Network Failures During Replication**
+Network failures during replication can be recovered from using resumable send & recv:
+
+- Network failure before the receive-side could stored data:
+  - Send-side `from` and `to` are guaranteed to be still present due to zfs holds
+  - recv-side `from` may no longer exist because we don't `hold` it explicitly
+    - if that is the case, ERROR OUT, step can never complete
+  - If the step planning algorithm does not include the step, for example because a snapshot filter configuration was changed by the user inbetween which hides `from` or `to` from the second planning attempt: **tough luck, we're leaking all holds**
+- Network failure during the replication
+  - send-side `from` and `to` are still present due to zfs holds
+  - recv-side `from` is still present because the partial receive state prevents its destruction (see prepare-phase)
+  - if recv-side hasa resume token, the resume token will continue to work on the sender because `from`s and `to` are still present
+- Network failure at the end of the replication step stream transmission
+  - Variant A: failure from the sender's perspective, success from the receiver's perspective
+    - receive-side `to` doesn't have a hold and could be destroyed anytime
+    - receive-side `from` doesn't have a hold and could be destroyed anytime
+    - thus, when the step is invoked again, pattern match `(from_exists, to_exists)`
+      - `(true, true)`: prepare-phase will early-exit
+      - `(false,true)`: prepare-phase will error out bc. `from` does not exist
+      - `(true, false)`: entire step will be re-done
+        - FIXME monotonicity requirement does not hold
+      - `(false, false)`: prepare-phase will error out bc. `from` does not exist
+  - Variant B: success from the sender's perspective, failure from the receiver's perspective
+    - No idea how this would happen except for bugs in error reporting in the replication protocol
+    - Misclassification by the sender, most likely broken error handling in the sender or replication protocol
+    - => the sender will release holds and move the replication cursor while the receiver won't => tough luck
+
+If the RPC used for `zfs recv` returned without error, this phase is done.
+(Although the *replication step* (this algorithm) is not yet *complete*, see definition of complete).
+
+
+#### Cleanup-Phase
+
+At the end of this phase, all intermediate state we built up to support the resumable replication of the step is destroyed.
+However, consumers of this algorithm might want to take advantage of the fact that we currently still have holds / step bookmarks.
+
+##### Recv-side: Optionally hold `to` with a caller-defined tag
+
+Users of the algorithm might want to depend on `to` being available on the receiving side for a longer duration than the lifetime of the current step's algorithm invocation.
+For reasons explained in the next paragraph, we cannot guarantee that we have a `hold` when `recv` exists. If that were the case, we could take our time to provide a generalized callback to the user of the algorithm, and have them do whatever they want with `to` while we guarantee that `to` doesn't go away through the hold. But that functionality isn't available, so we only provide a fixed operation right after receive: **take a `hold` with a tag of the algorithm user's choice**. That's what's needed to guarantee that a replication plan, consisting of multiple consecutive steps, can be carried out without a receive-side prune job interfering by destroying `to`. Technically, this step is racy, i.e., it could happen that `to` is destroyed between `recv` and `hold`. But this is a) unlikely and b) not fatal because we can detect that hold fails because the 'dataset does not exist` and re-do the entire transmission since we still hold send-side `from` and `to`, i.e. we just lose a lot of work in rare cases.
+
+So why can't we have a `hold` at the time `recv` exits?
+It seems like [`zfs send --holds`](https://github.com/zfsonlinux/zfs/commit/9c5e88b1ded19cb4b19b9d767d5c71b34c189540) could be used to send the send-side's holds to the receive-side. But that flag always sends all holds, and `--holds` is implemented in user-space libzfs, i.e., doesn't happen in the same txg as the recv completion. Thus, the race window is in fact only smaller (unless we oversaw some synchronization in userland zfs).
+But if we're willing to entertain the idea a little further, we still hit the problem that `--holds` sends _all_ holds, whereas our goal is to _temporarily_ have >= 1 hold that we own until the callback is done, and then release all of the received holds so that no holds created by us exist after this algorithm completes.
+Since there could be concurrent `zfs hold` invocations on the sender and receiver while this algorithm runs, and because `--holds` doesn't provide info about which holds were sent, we cannot correctly destroy _exactly_ those holds that we received.
+
+##### Send-side: callback to consumer, then cleanup
+We can provide the algorithm user with a generic callback because we have holds / step bookmarks for `to` and `from`, respectively.
+
+Example use case for the sender-side callback is the replication cursor, see algorithm for filesystem replication below.
+
+After the callback is done: cleanup holds & step bookmark:
+
+- `idempotent_release(to, zrepl_STEP_J_${jobid})`
+- make sure `from` can now go away:
+  - if `from` is a snapshot: `idempotent_release(from, zrepl_STEP_J_${jobid})`
+  - else `idempotent_step_bookmark_destroy(from)`
+    - if `from` fulfills the properties of a step bookmark: destroy it
+    - otherwise: it's a bookmark not created by this algorithm that happened to be used for replication: leave it alone
+      - that can only happen if user used the override knob
+
+Note that "make sure `from` can now go away" is the inverse of "Send-side: make sure `to` and `from` don't go away". Those are relatively complex operations and should be implemented in the same file next to each other to ease maintainability.
+
+---
+
+### Notes & Pseudo-APIs used in the algorithm
+
+- `idempotent_hold(snapshot s, string tag)` like zfs hold, but doesn't error if hold already exists
+- `idempotent_release(snapshot s, string tag)` like zfs hold, but doesn't error if hold already exists
+- `idempotent_step_bookmark(snapshot_or_bookmark b)` creates a *step bookmark* of b
+  - determine step bookmark name N
+  - if `N` already exists, verify it's a correct step bookmark => DONE or ERROR
+  - if `b` is a snapshot, issue `zfs bookmark`
+  - if `b` is a bookmark:
+    - if bookmark cloning supported, use it to duplicate the bookmark
+    - else ERROR OUT, with an error that upper layers can identify as such, so that they are able to ignore the fact that we couldn't create a step bookmark 
+- `idempotent_destroy(bookmark #b_$GUID, of a snapshot s)` must atomically check that `$GUID == s.guid` before destroying s
+- `idempotent_bookmark(snapshot s, $GUID, name #b_$GUID)` must atomically check that `$GUID == s.guid` at the time of bookmark creation
+- `idempotent_destroy(snapshot s)` must atomically check that zrepl's `s.guid` matches the current `guid` of the snapshot (i.e. destroy by guid)
+
+
+<a id="zrepl-algo-filesystem"></a>
+## Algorithm for Planning and Executing Replication of a Filesystems (planned, not implemented yet)
+
+This algorithm describes how a filesystem or zvol is replicated in zrepl.
+The algorithm is invoked with a `jobid`, `sender`, `receiver`, a sender-side `filesystem path`.
+It builds a diff between the sender and receiver filesystem bookmarks+snapshots and determines whether a replication conflict exists or whether fast-forward replication using full or incremental sends is possible.
+In case of conflict, the algorithm errors out with a conflict description that can be used to manually or automatically resolve the conflict.
+Otherwise, the algorithm builds a list of replication steps that are then worked on sequentially by the "Algorithm for a Single Replication Step".
+
+The algorithm ensures that a plan can be executed exactly as planned by aquiring appropriate zfs holds.
+The algorithm can be configured to retry a plan when encountering non-permanent errors (e.g. network errors).
+However, permanent errors result in the plan being cancelled.
+
+Regardless of whether a plan can be executed to completion or is cancelled, the algorithm guarantees that leftover artifacts (i.e. holds) of its invocation are cleaned up.
+However, since network failure can occur at any point, there might be stale holds on sending or receiving side after a crash or other error.
+These will be cleaned up on a subsequent invocation of this algorithm with the same `jobid`.
+
+The algorithm is fit to be executed in parallel from the same sender to different receivers.
+To be clear: replicating in parallel from the same sender to different receivers is supported.
+But one instance of the algorithm assumes ownership of the `filesystem path` on the receiver.
+
+The algorithm reserves the following sub-namespaces:
+* zfs hold: `zrepl_FS`
+* bookmark names: `zrepl_FS`
+
+### Definitions
+
+#### ZFS Filesystem Path
+We refer to the name of a ZFS filesystem or volume as a *filesystem path*, sometimes just *path*.
+The name includes the pool name.
+
+#### Sender and Receiver
+Sender and Receiver are passed as RPC stubs to the algorithm.
+One of those RPC stubs will typically be local, i.e., call methods on a struct in the same address space.
+The other RPC stub may invoke actual calls over the network (unless local replication is used).
+The algorithm does not make any assumption about which object is local or an actual stub.
+This design decouples the replication logic (described in this algorithm) from the question which side (sender or receiver) initiates the replication.
+
+### Algorithm
+
+**Cleanup Previous Invocations With Same `jobid`** by scanning the filesystem's list of snapshots and step bookmarks, and destroying any which was created by this algorithm when it was invoked with `jobid`.<br/>
+TODO HOW
+
+**Build a fast-forward list of replication steps** `STEPS`.
+`STEPS` may contain an optional initial full send, and subsequent incremental send steps.
+`STEPS[0]` may also have a resume token.
+If fast-forward is not possible, produce a conflict description and ERROR OUT.<br/>
+TODOs:
+- make it configurable what snapshots are included in the list (i.e. every one we see, only most recent, at least one every X hours, ...)
+
+**Ensure that we will be able to carry out all steps** by aquiring holds or fsstep bookmarks on the sending side
+- `idempotent_hold([s.to for s in STEPS], zrepl_FS_J_${jobid})`
+- `if STEPS[0].from != nil: idempotent_FSSTEP_bookmark(STEPS[0].from, zrepl_FSSTEP_bm_G_${STEPS[0].from.guid}_J_${jobid})` 
+  
+**Determine which steps have not been completed (`uncompleted_steps`)** (we might be in an second invocation of this algorithm after a network failure and some steps might already be done):
+- `res_tok := receiver.ResumeToken(fs)`
+- `rmrfsv := receiver.MostRecentFilesystemVersion(fs)`
+- if `res_tok !=  nil`: ensure that `res_tok` has a correspondinng step in `STEPS`, otherwise ERROR OUT
+- if `rmrfsv != nil`: ensure that `res_tok` has a correspondinng step in `STEPS`, otherwise ERROR OUT
+- if `(res_token != nil && rmrfsv != nil)`: ensure that `res_tok` is the subsequent step to the one we found for `rmrfsv`
+- if both are nil, we are at the beginning, `uncompleted_steps = STEPS` and goto next block
+- `rstep := if res_tok != nil { res_tok } else { rmrfsv }`
+- `uncompleted_steps := STEPS[find_step_idx(STEPS, rstep).expect("must exist, checked above"):]`
+  - Note that we do not explicitly check for the completion of prior replication steps.
+    All we care about is what needs to be done from `rstep`.
+  - This is intentional and necessary because we cummutatively release all holds and step bookmarks made for steps that preceed a just-completed step (see next paragraph)
+
+**Execute uncompleted steps**<br/>
+Invoke the "Algorithm for a Single Replication Step" for each step in `uncompleted_steps`.
+Remember to pass the resume token if one exists.
+
+In the wind-down phase of each replication step `from => to`, while the per-step algorithm still holds send-side `from` and `to`, as well as recv-side `to`:
+
+- Sending side: **Idempotently move the replication cursor to `to`.**
+  - Why: replication cursor tracks the sender's last known replication state, outlives the current plan, but is required to guarantee that future invocations of this algorithm find an incremental path.
+  - Impl for 'atomic' move: have two cursors (fixes [#177](https://github.com/zrepl/zrepl/issues/177))
+      - Idempotently cursor-bookmark `to` using `idempotent_bookmark(to, to.guid, #zrepl_replication_cursor_G_${to.guid}_J_${jobid})`
+      - Idempotently destroy old cursor-bookmark of `from` `idempotent_destroy(#zrepl_replication_cursor_G_${from.guid}_J_${jobid}, from)`
+    - If `from` is a snapshot, release the hold on it using `idempotent_release(from,  zrepl_J_${jobid})`
+    - FIXME: resumability if we crash inbetween those operations (or scrap it and wait for channel programs to bring atomicity)
+
+- Receiving side: **`idempotent_hold(to, zrepl_FS_J_${jobid})` because its going to be the next step's `from`**
+  - As discussed in the section on the per-step algorithm, this is a feature provided to us by the per-step algorithm.
+
+- Receiving side + Sending side (order doesn't matter): Make sure all holds & step bookmarks made by this plan on already replicated snapshots are released / destroyed:
+  - `idempotent_release_prior_and_including(from, zrepl_FS_TODO)`
+  - `idempotent_step_bookmark_destroy_prior_and_including(from, zrepl_FS_TODO)`
+
+**Cleanup** receiving and sending side (order doesn't matter)
+
+  - `idempotent_release_prior_and_including(STEPS[-1].to, zrepl_FS_TODO)`
+  - `idempotent_step_bookmark_destroy_prior_and_including(STEPS[-1].from, zrepl_FS_TODO)`
+
+### Notes
+
+- `idempotent_FSSTEP_bookmark` is like `idempotent_STEP_bookmark`, but with prefix `zrepl_FSSTEP` instead of `zrepl_STEP`
+
+
+<a id="zrepl-algo-multiple-filesystems-notes"></a>
+
+## Notes on Planning and Executing Replication of Multiple Filesystems
+
+### The RPC Interfaces Uses Send-side Filesystem Names to Identify a Filesystem
+
+The model of the receiver includes the assumption that it will `Receive` all snapshot streams sent to it into filesystems with paths prefixed with `root_fs`.
+This is useful for separating filesystem path namespaces for multiple clients.
+The receiver hides this prefixing in its RPC interface, i.e., when responding to `ListFilesystems` rpcs, the prefix is removed before sending the response.
+This behavior is useful to achieve symmetry in this algorithm: we do not need to take the prefixing into consideration when computing diffs.
+For the receiver, it has the advantage that `Receive` rpc can enforce the namespacing and need not trust this algorithm to apply it correctly.
+
+The receiver is also allowed (and may need to) do implement other filtering / namespace transformations.
+For example, when only `foo/a/b` has been received, but not `foo` nor `foo/a`, the receiver must not include the latter two in its `ListFilesystems` response.
+The current zrepl receiver endpoint implementation uses the `zrepl:placeholder` property to mark filesystems `foo` and `foo/a` as placeholders, and filters out such filesystems in the `ListFilesystems` response.
+Another approach would be to have a flat list of received filesystems per sender, and have a separate table that associates send-side names and receive-side filesystem paths.
+
+Similarly, the sender is allowed to filter the output of its RPC interface to hide certain filesystems or snapshots.
+
+#### Consequences of the Above Design
+
+Namespace filtering and transformations of filesystem paths on sender and receiver are user-configurable.
+Both ends of the replication setup have their own config, and do not coordinate config changes.
+This results in a number of challenges:
+
+- A sender might change its filter to allow additional filesystems to be replicated.
+  For example, where `foo/a/b` was initially allowed, the new filter might allow `foo` and `foo/a` as well.
+  If the receiver uses the `zrepl:placeholder` approach as sketched out above, this means that the receiver will need to replace the placeholder filesystems `$root_fs/foo` and `$root_fs/foo/a` with the incoming full sends of `foo` and `foo/a`.
+
+- Send-side renames cannot be handled efficiently because send-side rename effectively changes the filesystem identity because we use its name.
@@ -23,6 +23,32 @@ var _ = math.Inf
 // proto package needs to be updated.
 const _ = proto.ProtoPackageIsVersion2 // please upgrade the proto package

+type Tri int32
+
+const (
+	Tri_DontCare Tri = 0
+	Tri_False    Tri = 1
+	Tri_True     Tri = 2
+)
+
+var Tri_name = map[int32]string{
+	0: "DontCare",
+	1: "False",
+	2: "True",
+}
+var Tri_value = map[string]int32{
+	"DontCare": 0,
+	"False":    1,
+	"True":     2,
+}
+
+func (x Tri) String() string {
+	return proto.EnumName(Tri_name, int32(x))
+}
+func (Tri) EnumDescriptor() ([]byte, []int) {
+	return fileDescriptor_pdu_0f43b713cd3bf056, []int{0}
+}
+
 type FilesystemVersion_VersionType int32

 const (
@@ -43,7 +69,7 @@ func (x FilesystemVersion_VersionType) String() string {
 	return proto.EnumName(FilesystemVersion_VersionType_name, int32(x))
 }
 func (FilesystemVersion_VersionType) EnumDescriptor() ([]byte, []int) {
-	return fileDescriptor_pdu_83b7e2a28d820622, []int{5, 0}
+	return fileDescriptor_pdu_0f43b713cd3bf056, []int{5, 0}
 }

 type ListFilesystemReq struct {
@@ -56,7 +82,7 @@ func (m *ListFilesystemReq) Reset()         { *m = ListFilesystemReq{} }
 func (m *ListFilesystemReq) String() string { return proto.CompactTextString(m) }
 func (*ListFilesystemReq) ProtoMessage()    {}
 func (*ListFilesystemReq) Descriptor() ([]byte, []int) {
-	return fileDescriptor_pdu_83b7e2a28d820622, []int{0}
+	return fileDescriptor_pdu_0f43b713cd3bf056, []int{0}
 }
 func (m *ListFilesystemReq) XXX_Unmarshal(b []byte) error {
 	return xxx_messageInfo_ListFilesystemReq.Unmarshal(m, b)
@@ -87,7 +113,7 @@ func (m *ListFilesystemRes) Reset()         { *m = ListFilesystemRes{} }
 func (m *ListFilesystemRes) String() string { return proto.CompactTextString(m) }
 func (*ListFilesystemRes) ProtoMessage()    {}
 func (*ListFilesystemRes) Descriptor() ([]byte, []int) {
-	return fileDescriptor_pdu_83b7e2a28d820622, []int{1}
+	return fileDescriptor_pdu_0f43b713cd3bf056, []int{1}
 }
 func (m *ListFilesystemRes) XXX_Unmarshal(b []byte) error {
 	return xxx_messageInfo_ListFilesystemRes.Unmarshal(m, b)
@@ -118,6 +144,7 @@ type Filesystem struct {
 	Path                 string   `protobuf:"bytes,1,opt,name=Path,proto3" json:"Path,omitempty"`
 	ResumeToken          string   `protobuf:"bytes,2,opt,name=ResumeToken,proto3" json:"ResumeToken,omitempty"`
 	IsPlaceholder        bool     `protobuf:"varint,3,opt,name=IsPlaceholder,proto3" json:"IsPlaceholder,omitempty"`
+	IsEncrypted          bool     `protobuf:"varint,4,opt,name=IsEncrypted,proto3" json:"IsEncrypted,omitempty"`
 	XXX_NoUnkeyedLiteral struct{} `json:"-"`
 	XXX_unrecognized     []byte   `json:"-"`
 	XXX_sizecache        int32    `json:"-"`
@@ -127,7 +154,7 @@ func (m *Filesystem) Reset()         { *m = Filesystem{} }
 func (m *Filesystem) String() string { return proto.CompactTextString(m) }
 func (*Filesystem) ProtoMessage()    {}
 func (*Filesystem) Descriptor() ([]byte, []int) {
-	return fileDescriptor_pdu_83b7e2a28d820622, []int{2}
+	return fileDescriptor_pdu_0f43b713cd3bf056, []int{2}
 }
 func (m *Filesystem) XXX_Unmarshal(b []byte) error {
 	return xxx_messageInfo_Filesystem.Unmarshal(m, b)
@@ -168,6 +195,13 @@ func (m *Filesystem) GetIsPlaceholder() bool {
 	return false
 }

+func (m *Filesystem) GetIsEncrypted() bool {
+	if m != nil {
+		return m.IsEncrypted
+	}
+	return false
+}
+
 type ListFilesystemVersionsReq struct {
 	Filesystem           string   `protobuf:"bytes,1,opt,name=Filesystem,proto3" json:"Filesystem,omitempty"`
 	XXX_NoUnkeyedLiteral struct{} `json:"-"`
@@ -179,7 +213,7 @@ func (m *ListFilesystemVersionsReq) Reset()         { *m = ListFilesystemVersion
 func (m *ListFilesystemVersionsReq) String() string { return proto.CompactTextString(m) }
 func (*ListFilesystemVersionsReq) ProtoMessage()    {}
 func (*ListFilesystemVersionsReq) Descriptor() ([]byte, []int) {
-	return fileDescriptor_pdu_83b7e2a28d820622, []int{3}
+	return fileDescriptor_pdu_0f43b713cd3bf056, []int{3}
 }
 func (m *ListFilesystemVersionsReq) XXX_Unmarshal(b []byte) error {
 	return xxx_messageInfo_ListFilesystemVersionsReq.Unmarshal(m, b)
@@ -217,7 +251,7 @@ func (m *ListFilesystemVersionsRes) Reset()         { *m = ListFilesystemVersion
 func (m *ListFilesystemVersionsRes) String() string { return proto.CompactTextString(m) }
 func (*ListFilesystemVersionsRes) ProtoMessage()    {}
 func (*ListFilesystemVersionsRes) Descriptor() ([]byte, []int) {
-	return fileDescriptor_pdu_83b7e2a28d820622, []int{4}
+	return fileDescriptor_pdu_0f43b713cd3bf056, []int{4}
 }
 func (m *ListFilesystemVersionsRes) XXX_Unmarshal(b []byte) error {
 	return xxx_messageInfo_ListFilesystemVersionsRes.Unmarshal(m, b)
@@ -259,7 +293,7 @@ func (m *FilesystemVersion) Reset()         { *m = FilesystemVersion{} }
 func (m *FilesystemVersion) String() string { return proto.CompactTextString(m) }
 func (*FilesystemVersion) ProtoMessage()    {}
 func (*FilesystemVersion) Descriptor() ([]byte, []int) {
-	return fileDescriptor_pdu_83b7e2a28d820622, []int{5}
+	return fileDescriptor_pdu_0f43b713cd3bf056, []int{5}
 }
 func (m *FilesystemVersion) XXX_Unmarshal(b []byte) error {
 	return xxx_messageInfo_FilesystemVersion.Unmarshal(m, b)
@@ -316,20 +350,18 @@ func (m *FilesystemVersion) GetCreation() string {

 type SendReq struct {
 	Filesystem string `protobuf:"bytes,1,opt,name=Filesystem,proto3" json:"Filesystem,omitempty"`
-	From       string `protobuf:"bytes,2,opt,name=From,proto3" json:"From,omitempty"`
-	// May be empty / null to request a full transfer of From
-	To string `protobuf:"bytes,3,opt,name=To,proto3" json:"To,omitempty"`
-	// If ResumeToken is not empty, the resume token that CAN be tried for 'zfs send' by the sender.
-	// The sender MUST indicate in SendRes.UsedResumeToken
-	// If it does not work, the sender SHOULD clear the resume token on their side
-	// and use From and To instead
-	// If ResumeToken is not empty, the GUIDs of From and To
-	// MUST correspond to those encoded in the ResumeToken.
-	// Otherwise, the Sender MUST return an error.
+	// May be empty / null to request a full transfer of To
+	From *FilesystemVersion `protobuf:"bytes,2,opt,name=From,proto3" json:"From,omitempty"`
+	To   *FilesystemVersion `protobuf:"bytes,3,opt,name=To,proto3" json:"To,omitempty"`
+	// If ResumeToken is not empty, the resume token that CAN be used for 'zfs
+	// send' by the sender. The sender MUST indicate use of ResumeToken in the
+	// reply message SendRes.UsedResumeToken If it does not work, the sender
+	// SHOULD clear the resume token on their side and use From and To instead If
+	// ResumeToken is not empty, the GUIDs of From and To MUST correspond to those
+	// encoded in the ResumeToken. Otherwise, the Sender MUST return an error.
 	ResumeToken          string   `protobuf:"bytes,4,opt,name=ResumeToken,proto3" json:"ResumeToken,omitempty"`
-	Compress             bool     `protobuf:"varint,5,opt,name=Compress,proto3" json:"Compress,omitempty"`
-	Dedup                bool     `protobuf:"varint,6,opt,name=Dedup,proto3" json:"Dedup,omitempty"`
-	DryRun               bool     `protobuf:"varint,7,opt,name=DryRun,proto3" json:"DryRun,omitempty"`
+	Encrypted            Tri      `protobuf:"varint,5,opt,name=Encrypted,proto3,enum=Tri" json:"Encrypted,omitempty"`
+	DryRun               bool     `protobuf:"varint,6,opt,name=DryRun,proto3" json:"DryRun,omitempty"`
 	XXX_NoUnkeyedLiteral struct{} `json:"-"`
 	XXX_unrecognized     []byte   `json:"-"`
 	XXX_sizecache        int32    `json:"-"`
@@ -339,7 +371,7 @@ func (m *SendReq) Reset()         { *m = SendReq{} }
 func (m *SendReq) String() string { return proto.CompactTextString(m) }
 func (*SendReq) ProtoMessage()    {}
 func (*SendReq) Descriptor() ([]byte, []int) {
-	return fileDescriptor_pdu_83b7e2a28d820622, []int{6}
+	return fileDescriptor_pdu_0f43b713cd3bf056, []int{6}
 }
 func (m *SendReq) XXX_Unmarshal(b []byte) error {
 	return xxx_messageInfo_SendReq.Unmarshal(m, b)
@@ -366,18 +398,18 @@ func (m *SendReq) GetFilesystem() string {
 	return ""
 }

-func (m *SendReq) GetFrom() string {
+func (m *SendReq) GetFrom() *FilesystemVersion {
 	if m != nil {
 		return m.From
 	}
-	return ""
+	return nil
 }

-func (m *SendReq) GetTo() string {
+func (m *SendReq) GetTo() *FilesystemVersion {
 	if m != nil {
 		return m.To
 	}
-	return ""
+	return nil
 }

 func (m *SendReq) GetResumeToken() string {
@@ -387,18 +419,11 @@ func (m *SendReq) GetResumeToken() string {
 	return ""
 }

-func (m *SendReq) GetCompress() bool {
+func (m *SendReq) GetEncrypted() Tri {
 	if m != nil {
-		return m.Compress
+		return m.Encrypted
 	}
-	return false
-}
-
-func (m *SendReq) GetDedup() bool {
-	if m != nil {
-		return m.Dedup
-	}
-	return false
+	return Tri_DontCare
 }

 func (m *SendReq) GetDryRun() bool {
@@ -420,7 +445,7 @@ func (m *Property) Reset()         { *m = Property{} }
 func (m *Property) String() string { return proto.CompactTextString(m) }
 func (*Property) ProtoMessage()    {}
 func (*Property) Descriptor() ([]byte, []int) {
-	return fileDescriptor_pdu_83b7e2a28d820622, []int{7}
+	return fileDescriptor_pdu_0f43b713cd3bf056, []int{7}
 }
 func (m *Property) XXX_Unmarshal(b []byte) error {
 	return xxx_messageInfo_Property.Unmarshal(m, b)
@@ -456,6 +481,7 @@ func (m *Property) GetValue() string {

 type SendRes struct {
 	// Whether the resume token provided in the request has been used or not.
+	// If the SendReq.ResumeToken == "", this field has no meaning.
 	UsedResumeToken bool `protobuf:"varint,2,opt,name=UsedResumeToken,proto3" json:"UsedResumeToken,omitempty"`
 	// Expected stream size determined by dry run, not exact.
 	// 0 indicates that for the given SendReq, no size estimate could be made.
@@ -470,7 +496,7 @@ func (m *SendRes) Reset()         { *m = SendRes{} }
 func (m *SendRes) String() string { return proto.CompactTextString(m) }
 func (*SendRes) ProtoMessage()    {}
 func (*SendRes) Descriptor() ([]byte, []int) {
-	return fileDescriptor_pdu_83b7e2a28d820622, []int{8}
+	return fileDescriptor_pdu_0f43b713cd3bf056, []int{8}
 }
 func (m *SendRes) XXX_Unmarshal(b []byte) error {
 	return xxx_messageInfo_SendRes.Unmarshal(m, b)
@@ -511,10 +537,80 @@ func (m *SendRes) GetProperties() []*Property {
 	return nil
 }

+type SendCompletedReq struct {
+	OriginalReq          *SendReq `protobuf:"bytes,2,opt,name=OriginalReq,proto3" json:"OriginalReq,omitempty"`
+	XXX_NoUnkeyedLiteral struct{} `json:"-"`
+	XXX_unrecognized     []byte   `json:"-"`
+	XXX_sizecache        int32    `json:"-"`
+}
+
+func (m *SendCompletedReq) Reset()         { *m = SendCompletedReq{} }
+func (m *SendCompletedReq) String() string { return proto.CompactTextString(m) }
+func (*SendCompletedReq) ProtoMessage()    {}
+func (*SendCompletedReq) Descriptor() ([]byte, []int) {
+	return fileDescriptor_pdu_0f43b713cd3bf056, []int{9}
+}
+func (m *SendCompletedReq) XXX_Unmarshal(b []byte) error {
+	return xxx_messageInfo_SendCompletedReq.Unmarshal(m, b)
+}
+func (m *SendCompletedReq) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
+	return xxx_messageInfo_SendCompletedReq.Marshal(b, m, deterministic)
+}
+func (dst *SendCompletedReq) XXX_Merge(src proto.Message) {
+	xxx_messageInfo_SendCompletedReq.Merge(dst, src)
+}
+func (m *SendCompletedReq) XXX_Size() int {
+	return xxx_messageInfo_SendCompletedReq.Size(m)
+}
+func (m *SendCompletedReq) XXX_DiscardUnknown() {
+	xxx_messageInfo_SendCompletedReq.DiscardUnknown(m)
+}
+
+var xxx_messageInfo_SendCompletedReq proto.InternalMessageInfo
+
+func (m *SendCompletedReq) GetOriginalReq() *SendReq {
+	if m != nil {
+		return m.OriginalReq
+	}
+	return nil
+}
+
+type SendCompletedRes struct {
+	XXX_NoUnkeyedLiteral struct{} `json:"-"`
+	XXX_unrecognized     []byte   `json:"-"`
+	XXX_sizecache        int32    `json:"-"`
+}
+
+func (m *SendCompletedRes) Reset()         { *m = SendCompletedRes{} }
+func (m *SendCompletedRes) String() string { return proto.CompactTextString(m) }
+func (*SendCompletedRes) ProtoMessage()    {}
+func (*SendCompletedRes) Descriptor() ([]byte, []int) {
+	return fileDescriptor_pdu_0f43b713cd3bf056, []int{10}
+}
+func (m *SendCompletedRes) XXX_Unmarshal(b []byte) error {
+	return xxx_messageInfo_SendCompletedRes.Unmarshal(m, b)
+}
+func (m *SendCompletedRes) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
+	return xxx_messageInfo_SendCompletedRes.Marshal(b, m, deterministic)
+}
+func (dst *SendCompletedRes) XXX_Merge(src proto.Message) {
+	xxx_messageInfo_SendCompletedRes.Merge(dst, src)
+}
+func (m *SendCompletedRes) XXX_Size() int {
+	return xxx_messageInfo_SendCompletedRes.Size(m)
+}
+func (m *SendCompletedRes) XXX_DiscardUnknown() {
+	xxx_messageInfo_SendCompletedRes.DiscardUnknown(m)
+}
+
+var xxx_messageInfo_SendCompletedRes proto.InternalMessageInfo
+
 type ReceiveReq struct {
-	Filesystem string `protobuf:"bytes,1,opt,name=Filesystem,proto3" json:"Filesystem,omitempty"`
-	// If true, the receiver should clear the resume token before perfoming the zfs recv of the stream in the request
-	ClearResumeToken     bool     `protobuf:"varint,2,opt,name=ClearResumeToken,proto3" json:"ClearResumeToken,omitempty"`
+	Filesystem string             `protobuf:"bytes,1,opt,name=Filesystem,proto3" json:"Filesystem,omitempty"`
+	To         *FilesystemVersion `protobuf:"bytes,2,opt,name=To,proto3" json:"To,omitempty"`
+	// If true, the receiver should clear the resume token before perfoming the
+	// zfs recv of the stream in the request
+	ClearResumeToken     bool     `protobuf:"varint,3,opt,name=ClearResumeToken,proto3" json:"ClearResumeToken,omitempty"`
 	XXX_NoUnkeyedLiteral struct{} `json:"-"`
 	XXX_unrecognized     []byte   `json:"-"`
 	XXX_sizecache        int32    `json:"-"`
@@ -524,7 +620,7 @@ func (m *ReceiveReq) Reset()         { *m = ReceiveReq{} }
 func (m *ReceiveReq) String() string { return proto.CompactTextString(m) }
 func (*ReceiveReq) ProtoMessage()    {}
 func (*ReceiveReq) Descriptor() ([]byte, []int) {
-	return fileDescriptor_pdu_83b7e2a28d820622, []int{9}
+	return fileDescriptor_pdu_0f43b713cd3bf056, []int{11}
 }
 func (m *ReceiveReq) XXX_Unmarshal(b []byte) error {
 	return xxx_messageInfo_ReceiveReq.Unmarshal(m, b)
@@ -551,6 +647,13 @@ func (m *ReceiveReq) GetFilesystem() string {
 	return ""
 }

+func (m *ReceiveReq) GetTo() *FilesystemVersion {
+	if m != nil {
+		return m.To
+	}
+	return nil
+}
+
 func (m *ReceiveReq) GetClearResumeToken() bool {
 	if m != nil {
 		return m.ClearResumeToken
@@ -568,7 +671,7 @@ func (m *ReceiveRes) Reset()         { *m = ReceiveRes{} }
 func (m *ReceiveRes) String() string { return proto.CompactTextString(m) }
 func (*ReceiveRes) ProtoMessage()    {}
 func (*ReceiveRes) Descriptor() ([]byte, []int) {
-	return fileDescriptor_pdu_83b7e2a28d820622, []int{10}
+	return fileDescriptor_pdu_0f43b713cd3bf056, []int{12}
 }
 func (m *ReceiveRes) XXX_Unmarshal(b []byte) error {
 	return xxx_messageInfo_ReceiveRes.Unmarshal(m, b)
@@ -601,7 +704,7 @@ func (m *DestroySnapshotsReq) Reset()         { *m = DestroySnapshotsReq{} }
 func (m *DestroySnapshotsReq) String() string { return proto.CompactTextString(m) }
 func (*DestroySnapshotsReq) ProtoMessage()    {}
 func (*DestroySnapshotsReq) Descriptor() ([]byte, []int) {
-	return fileDescriptor_pdu_83b7e2a28d820622, []int{11}
+	return fileDescriptor_pdu_0f43b713cd3bf056, []int{13}
 }
 func (m *DestroySnapshotsReq) XXX_Unmarshal(b []byte) error {
 	return xxx_messageInfo_DestroySnapshotsReq.Unmarshal(m, b)
@@ -647,7 +750,7 @@ func (m *DestroySnapshotRes) Reset()         { *m = DestroySnapshotRes{} }
 func (m *DestroySnapshotRes) String() string { return proto.CompactTextString(m) }
 func (*DestroySnapshotRes) ProtoMessage()    {}
 func (*DestroySnapshotRes) Descriptor() ([]byte, []int) {
-	return fileDescriptor_pdu_83b7e2a28d820622, []int{12}
+	return fileDescriptor_pdu_0f43b713cd3bf056, []int{14}
 }
 func (m *DestroySnapshotRes) XXX_Unmarshal(b []byte) error {
 	return xxx_messageInfo_DestroySnapshotRes.Unmarshal(m, b)
@@ -692,7 +795,7 @@ func (m *DestroySnapshotsRes) Reset()         { *m = DestroySnapshotsRes{} }
 func (m *DestroySnapshotsRes) String() string { return proto.CompactTextString(m) }
 func (*DestroySnapshotsRes) ProtoMessage()    {}
 func (*DestroySnapshotsRes) Descriptor() ([]byte, []int) {
-	return fileDescriptor_pdu_83b7e2a28d820622, []int{13}
+	return fileDescriptor_pdu_0f43b713cd3bf056, []int{15}
 }
 func (m *DestroySnapshotsRes) XXX_Unmarshal(b []byte) error {
 	return xxx_messageInfo_DestroySnapshotsRes.Unmarshal(m, b)
@@ -720,21 +823,17 @@ func (m *DestroySnapshotsRes) GetResults() []*DestroySnapshotRes {
 }

 type ReplicationCursorReq struct {
-	Filesystem string `protobuf:"bytes,1,opt,name=Filesystem,proto3" json:"Filesystem,omitempty"`
-	// Types that are valid to be assigned to Op:
-	//	*ReplicationCursorReq_Get
-	//	*ReplicationCursorReq_Set
-	Op                   isReplicationCursorReq_Op `protobuf_oneof:"op"`
-	XXX_NoUnkeyedLiteral struct{}                  `json:"-"`
-	XXX_unrecognized     []byte                    `json:"-"`
-	XXX_sizecache        int32                     `json:"-"`
+	Filesystem           string   `protobuf:"bytes,1,opt,name=Filesystem,proto3" json:"Filesystem,omitempty"`
+	XXX_NoUnkeyedLiteral struct{} `json:"-"`
+	XXX_unrecognized     []byte   `json:"-"`
+	XXX_sizecache        int32    `json:"-"`
 }

 func (m *ReplicationCursorReq) Reset()         { *m = ReplicationCursorReq{} }
 func (m *ReplicationCursorReq) String() string { return proto.CompactTextString(m) }
 func (*ReplicationCursorReq) ProtoMessage()    {}
 func (*ReplicationCursorReq) Descriptor() ([]byte, []int) {
-	return fileDescriptor_pdu_83b7e2a28d820622, []int{14}
+	return fileDescriptor_pdu_0f43b713cd3bf056, []int{16}
 }
 func (m *ReplicationCursorReq) XXX_Unmarshal(b []byte) error {
 	return xxx_messageInfo_ReplicationCursorReq.Unmarshal(m, b)
@@ -761,185 +860,6 @@ func (m *ReplicationCursorReq) GetFilesystem() string {
 	return ""
 }

-type isReplicationCursorReq_Op interface {
-	isReplicationCursorReq_Op()
-}
-
-type ReplicationCursorReq_Get struct {
-	Get *ReplicationCursorReq_GetOp `protobuf:"bytes,2,opt,name=get,proto3,oneof"`
-}
-
-type ReplicationCursorReq_Set struct {
-	Set *ReplicationCursorReq_SetOp `protobuf:"bytes,3,opt,name=set,proto3,oneof"`
-}
-
-func (*ReplicationCursorReq_Get) isReplicationCursorReq_Op() {}
-
-func (*ReplicationCursorReq_Set) isReplicationCursorReq_Op() {}
-
-func (m *ReplicationCursorReq) GetOp() isReplicationCursorReq_Op {
-	if m != nil {
-		return m.Op
-	}
-	return nil
-}
-
-func (m *ReplicationCursorReq) GetGet() *ReplicationCursorReq_GetOp {
-	if x, ok := m.GetOp().(*ReplicationCursorReq_Get); ok {
-		return x.Get
-	}
-	return nil
-}
-
-func (m *ReplicationCursorReq) GetSet() *ReplicationCursorReq_SetOp {
-	if x, ok := m.GetOp().(*ReplicationCursorReq_Set); ok {
-		return x.Set
-	}
-	return nil
-}
-
-// XXX_OneofFuncs is for the internal use of the proto package.
-func (*ReplicationCursorReq) XXX_OneofFuncs() (func(msg proto.Message, b *proto.Buffer) error, func(msg proto.Message, tag, wire int, b *proto.Buffer) (bool, error), func(msg proto.Message) (n int), []interface{}) {
-	return _ReplicationCursorReq_OneofMarshaler, _ReplicationCursorReq_OneofUnmarshaler, _ReplicationCursorReq_OneofSizer, []interface{}{
-		(*ReplicationCursorReq_Get)(nil),
-		(*ReplicationCursorReq_Set)(nil),
-	}
-}
-
-func _ReplicationCursorReq_OneofMarshaler(msg proto.Message, b *proto.Buffer) error {
-	m := msg.(*ReplicationCursorReq)
-	// op
-	switch x := m.Op.(type) {
-	case *ReplicationCursorReq_Get:
-		b.EncodeVarint(2<<3 | proto.WireBytes)
-		if err := b.EncodeMessage(x.Get); err != nil {
-			return err
-		}
-	case *ReplicationCursorReq_Set:
-		b.EncodeVarint(3<<3 | proto.WireBytes)
-		if err := b.EncodeMessage(x.Set); err != nil {
-			return err
-		}
-	case nil:
-	default:
-		return fmt.Errorf("ReplicationCursorReq.Op has unexpected type %T", x)
-	}
-	return nil
-}
-
-func _ReplicationCursorReq_OneofUnmarshaler(msg proto.Message, tag, wire int, b *proto.Buffer) (bool, error) {
-	m := msg.(*ReplicationCursorReq)
-	switch tag {
-	case 2: // op.get
-		if wire != proto.WireBytes {
-			return true, proto.ErrInternalBadWireType
-		}
-		msg := new(ReplicationCursorReq_GetOp)
-		err := b.DecodeMessage(msg)
-		m.Op = &ReplicationCursorReq_Get{msg}
-		return true, err
-	case 3: // op.set
-		if wire != proto.WireBytes {
-			return true, proto.ErrInternalBadWireType
-		}
-		msg := new(ReplicationCursorReq_SetOp)
-		err := b.DecodeMessage(msg)
-		m.Op = &ReplicationCursorReq_Set{msg}
-		return true, err
-	default:
-		return false, nil
-	}
-}
-
-func _ReplicationCursorReq_OneofSizer(msg proto.Message) (n int) {
-	m := msg.(*ReplicationCursorReq)
-	// op
-	switch x := m.Op.(type) {
-	case *ReplicationCursorReq_Get:
-		s := proto.Size(x.Get)
-		n += 1 // tag and wire
-		n += proto.SizeVarint(uint64(s))
-		n += s
-	case *ReplicationCursorReq_Set:
-		s := proto.Size(x.Set)
-		n += 1 // tag and wire
-		n += proto.SizeVarint(uint64(s))
-		n += s
-	case nil:
-	default:
-		panic(fmt.Sprintf("proto: unexpected type %T in oneof", x))
-	}
-	return n
-}
-
-type ReplicationCursorReq_GetOp struct {
-	XXX_NoUnkeyedLiteral struct{} `json:"-"`
-	XXX_unrecognized     []byte   `json:"-"`
-	XXX_sizecache        int32    `json:"-"`
-}
-
-func (m *ReplicationCursorReq_GetOp) Reset()         { *m = ReplicationCursorReq_GetOp{} }
-func (m *ReplicationCursorReq_GetOp) String() string { return proto.CompactTextString(m) }
-func (*ReplicationCursorReq_GetOp) ProtoMessage()    {}
-func (*ReplicationCursorReq_GetOp) Descriptor() ([]byte, []int) {
-	return fileDescriptor_pdu_83b7e2a28d820622, []int{14, 0}
-}
-func (m *ReplicationCursorReq_GetOp) XXX_Unmarshal(b []byte) error {
-	return xxx_messageInfo_ReplicationCursorReq_GetOp.Unmarshal(m, b)
-}
-func (m *ReplicationCursorReq_GetOp) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
-	return xxx_messageInfo_ReplicationCursorReq_GetOp.Marshal(b, m, deterministic)
-}
-func (dst *ReplicationCursorReq_GetOp) XXX_Merge(src proto.Message) {
-	xxx_messageInfo_ReplicationCursorReq_GetOp.Merge(dst, src)
-}
-func (m *ReplicationCursorReq_GetOp) XXX_Size() int {
-	return xxx_messageInfo_ReplicationCursorReq_GetOp.Size(m)
-}
-func (m *ReplicationCursorReq_GetOp) XXX_DiscardUnknown() {
-	xxx_messageInfo_ReplicationCursorReq_GetOp.DiscardUnknown(m)
-}
-
-var xxx_messageInfo_ReplicationCursorReq_GetOp proto.InternalMessageInfo
-
-type ReplicationCursorReq_SetOp struct {
-	Snapshot             string   `protobuf:"bytes,2,opt,name=Snapshot,proto3" json:"Snapshot,omitempty"`
-	XXX_NoUnkeyedLiteral struct{} `json:"-"`
-	XXX_unrecognized     []byte   `json:"-"`
-	XXX_sizecache        int32    `json:"-"`
-}
-
-func (m *ReplicationCursorReq_SetOp) Reset()         { *m = ReplicationCursorReq_SetOp{} }
-func (m *ReplicationCursorReq_SetOp) String() string { return proto.CompactTextString(m) }
-func (*ReplicationCursorReq_SetOp) ProtoMessage()    {}
-func (*ReplicationCursorReq_SetOp) Descriptor() ([]byte, []int) {
-	return fileDescriptor_pdu_83b7e2a28d820622, []int{14, 1}
-}
-func (m *ReplicationCursorReq_SetOp) XXX_Unmarshal(b []byte) error {
-	return xxx_messageInfo_ReplicationCursorReq_SetOp.Unmarshal(m, b)
-}
-func (m *ReplicationCursorReq_SetOp) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
-	return xxx_messageInfo_ReplicationCursorReq_SetOp.Marshal(b, m, deterministic)
-}
-func (dst *ReplicationCursorReq_SetOp) XXX_Merge(src proto.Message) {
-	xxx_messageInfo_ReplicationCursorReq_SetOp.Merge(dst, src)
-}
-func (m *ReplicationCursorReq_SetOp) XXX_Size() int {
-	return xxx_messageInfo_ReplicationCursorReq_SetOp.Size(m)
-}
-func (m *ReplicationCursorReq_SetOp) XXX_DiscardUnknown() {
-	xxx_messageInfo_ReplicationCursorReq_SetOp.DiscardUnknown(m)
-}
-
-var xxx_messageInfo_ReplicationCursorReq_SetOp proto.InternalMessageInfo
-
-func (m *ReplicationCursorReq_SetOp) GetSnapshot() string {
-	if m != nil {
-		return m.Snapshot
-	}
-	return ""
-}
-
 type ReplicationCursorRes struct {
 	// Types that are valid to be assigned to Result:
 	//	*ReplicationCursorRes_Guid
@@ -954,7 +874,7 @@ func (m *ReplicationCursorRes) Reset()         { *m = ReplicationCursorRes{} }
 func (m *ReplicationCursorRes) String() string { return proto.CompactTextString(m) }
 func (*ReplicationCursorRes) ProtoMessage()    {}
 func (*ReplicationCursorRes) Descriptor() ([]byte, []int) {
-	return fileDescriptor_pdu_83b7e2a28d820622, []int{15}
+	return fileDescriptor_pdu_0f43b713cd3bf056, []int{17}
 }
 func (m *ReplicationCursorRes) XXX_Unmarshal(b []byte) error {
 	return xxx_messageInfo_ReplicationCursorRes.Unmarshal(m, b)
@@ -1090,7 +1010,7 @@ func (m *PingReq) Reset()         { *m = PingReq{} }
 func (m *PingReq) String() string { return proto.CompactTextString(m) }
 func (*PingReq) ProtoMessage()    {}
 func (*PingReq) Descriptor() ([]byte, []int) {
-	return fileDescriptor_pdu_83b7e2a28d820622, []int{16}
+	return fileDescriptor_pdu_0f43b713cd3bf056, []int{18}
 }
 func (m *PingReq) XXX_Unmarshal(b []byte) error {
 	return xxx_messageInfo_PingReq.Unmarshal(m, b)
@@ -1129,7 +1049,7 @@ func (m *PingRes) Reset()         { *m = PingRes{} }
 func (m *PingRes) String() string { return proto.CompactTextString(m) }
 func (*PingRes) ProtoMessage()    {}
 func (*PingRes) Descriptor() ([]byte, []int) {
-	return fileDescriptor_pdu_83b7e2a28d820622, []int{17}
+	return fileDescriptor_pdu_0f43b713cd3bf056, []int{19}
 }
 func (m *PingRes) XXX_Unmarshal(b []byte) error {
 	return xxx_messageInfo_PingRes.Unmarshal(m, b)
@@ -1156,6 +1076,82 @@ func (m *PingRes) GetEcho() string {
 	return ""
 }

+type HintMostRecentCommonAncestorReq struct {
+	Filesystem           string             `protobuf:"bytes,1,opt,name=Filesystem,proto3" json:"Filesystem,omitempty"`
+	SenderVersion        *FilesystemVersion `protobuf:"bytes,2,opt,name=SenderVersion,proto3" json:"SenderVersion,omitempty"`
+	XXX_NoUnkeyedLiteral struct{}           `json:"-"`
+	XXX_unrecognized     []byte             `json:"-"`
+	XXX_sizecache        int32              `json:"-"`
+}
+
+func (m *HintMostRecentCommonAncestorReq) Reset()         { *m = HintMostRecentCommonAncestorReq{} }
+func (m *HintMostRecentCommonAncestorReq) String() string { return proto.CompactTextString(m) }
+func (*HintMostRecentCommonAncestorReq) ProtoMessage()    {}
+func (*HintMostRecentCommonAncestorReq) Descriptor() ([]byte, []int) {
+	return fileDescriptor_pdu_0f43b713cd3bf056, []int{20}
+}
+func (m *HintMostRecentCommonAncestorReq) XXX_Unmarshal(b []byte) error {
+	return xxx_messageInfo_HintMostRecentCommonAncestorReq.Unmarshal(m, b)
+}
+func (m *HintMostRecentCommonAncestorReq) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
+	return xxx_messageInfo_HintMostRecentCommonAncestorReq.Marshal(b, m, deterministic)
+}
+func (dst *HintMostRecentCommonAncestorReq) XXX_Merge(src proto.Message) {
+	xxx_messageInfo_HintMostRecentCommonAncestorReq.Merge(dst, src)
+}
+func (m *HintMostRecentCommonAncestorReq) XXX_Size() int {
+	return xxx_messageInfo_HintMostRecentCommonAncestorReq.Size(m)
+}
+func (m *HintMostRecentCommonAncestorReq) XXX_DiscardUnknown() {
+	xxx_messageInfo_HintMostRecentCommonAncestorReq.DiscardUnknown(m)
+}
+
+var xxx_messageInfo_HintMostRecentCommonAncestorReq proto.InternalMessageInfo
+
+func (m *HintMostRecentCommonAncestorReq) GetFilesystem() string {
+	if m != nil {
+		return m.Filesystem
+	}
+	return ""
+}
+
+func (m *HintMostRecentCommonAncestorReq) GetSenderVersion() *FilesystemVersion {
+	if m != nil {
+		return m.SenderVersion
+	}
+	return nil
+}
+
+type HintMostRecentCommonAncestorRes struct {
+	XXX_NoUnkeyedLiteral struct{} `json:"-"`
+	XXX_unrecognized     []byte   `json:"-"`
+	XXX_sizecache        int32    `json:"-"`
+}
+
+func (m *HintMostRecentCommonAncestorRes) Reset()         { *m = HintMostRecentCommonAncestorRes{} }
+func (m *HintMostRecentCommonAncestorRes) String() string { return proto.CompactTextString(m) }
+func (*HintMostRecentCommonAncestorRes) ProtoMessage()    {}
+func (*HintMostRecentCommonAncestorRes) Descriptor() ([]byte, []int) {
+	return fileDescriptor_pdu_0f43b713cd3bf056, []int{21}
+}
+func (m *HintMostRecentCommonAncestorRes) XXX_Unmarshal(b []byte) error {
+	return xxx_messageInfo_HintMostRecentCommonAncestorRes.Unmarshal(m, b)
+}
+func (m *HintMostRecentCommonAncestorRes) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
+	return xxx_messageInfo_HintMostRecentCommonAncestorRes.Marshal(b, m, deterministic)
+}
+func (dst *HintMostRecentCommonAncestorRes) XXX_Merge(src proto.Message) {
+	xxx_messageInfo_HintMostRecentCommonAncestorRes.Merge(dst, src)
+}
+func (m *HintMostRecentCommonAncestorRes) XXX_Size() int {
+	return xxx_messageInfo_HintMostRecentCommonAncestorRes.Size(m)
+}
+func (m *HintMostRecentCommonAncestorRes) XXX_DiscardUnknown() {
+	xxx_messageInfo_HintMostRecentCommonAncestorRes.DiscardUnknown(m)
+}
+
+var xxx_messageInfo_HintMostRecentCommonAncestorRes proto.InternalMessageInfo
+
 func init() {
 	proto.RegisterType((*ListFilesystemReq)(nil), "ListFilesystemReq")
 	proto.RegisterType((*ListFilesystemRes)(nil), "ListFilesystemRes")
@@ -1166,17 +1162,20 @@ func init() {
 	proto.RegisterType((*SendReq)(nil), "SendReq")
 	proto.RegisterType((*Property)(nil), "Property")
 	proto.RegisterType((*SendRes)(nil), "SendRes")
+	proto.RegisterType((*SendCompletedReq)(nil), "SendCompletedReq")
+	proto.RegisterType((*SendCompletedRes)(nil), "SendCompletedRes")
 	proto.RegisterType((*ReceiveReq)(nil), "ReceiveReq")
 	proto.RegisterType((*ReceiveRes)(nil), "ReceiveRes")
 	proto.RegisterType((*DestroySnapshotsReq)(nil), "DestroySnapshotsReq")
 	proto.RegisterType((*DestroySnapshotRes)(nil), "DestroySnapshotRes")
 	proto.RegisterType((*DestroySnapshotsRes)(nil), "DestroySnapshotsRes")
 	proto.RegisterType((*ReplicationCursorReq)(nil), "ReplicationCursorReq")
-	proto.RegisterType((*ReplicationCursorReq_GetOp)(nil), "ReplicationCursorReq.GetOp")
-	proto.RegisterType((*ReplicationCursorReq_SetOp)(nil), "ReplicationCursorReq.SetOp")
 	proto.RegisterType((*ReplicationCursorRes)(nil), "ReplicationCursorRes")
 	proto.RegisterType((*PingReq)(nil), "PingReq")
 	proto.RegisterType((*PingRes)(nil), "PingRes")
+	proto.RegisterType((*HintMostRecentCommonAncestorReq)(nil), "HintMostRecentCommonAncestorReq")
+	proto.RegisterType((*HintMostRecentCommonAncestorRes)(nil), "HintMostRecentCommonAncestorRes")
+	proto.RegisterEnum("Tri", Tri_name, Tri_value)
 	proto.RegisterEnum("FilesystemVersion_VersionType", FilesystemVersion_VersionType_name, FilesystemVersion_VersionType_value)
 }

@@ -1197,6 +1196,8 @@ type ReplicationClient interface {
 	ListFilesystemVersions(ctx context.Context, in *ListFilesystemVersionsReq, opts ...grpc.CallOption) (*ListFilesystemVersionsRes, error)
 	DestroySnapshots(ctx context.Context, in *DestroySnapshotsReq, opts ...grpc.CallOption) (*DestroySnapshotsRes, error)
 	ReplicationCursor(ctx context.Context, in *ReplicationCursorReq, opts ...grpc.CallOption) (*ReplicationCursorRes, error)
+	SendCompleted(ctx context.Context, in *SendCompletedReq, opts ...grpc.CallOption) (*SendCompletedRes, error)
+	HintMostRecentCommonAncestor(ctx context.Context, in *HintMostRecentCommonAncestorReq, opts ...grpc.CallOption) (*HintMostRecentCommonAncestorRes, error)
 }

 type replicationClient struct {
@@ -1252,6 +1253,24 @@ func (c *replicationClient) ReplicationCursor(ctx context.Context, in *Replicati
 	return out, nil
 }

+func (c *replicationClient) SendCompleted(ctx context.Context, in *SendCompletedReq, opts ...grpc.CallOption) (*SendCompletedRes, error) {
+	out := new(SendCompletedRes)
+	err := c.cc.Invoke(ctx, "/Replication/SendCompleted", in, out, opts...)
+	if err != nil {
+		return nil, err
+	}
+	return out, nil
+}
+
+func (c *replicationClient) HintMostRecentCommonAncestor(ctx context.Context, in *HintMostRecentCommonAncestorReq, opts ...grpc.CallOption) (*HintMostRecentCommonAncestorRes, error) {
+	out := new(HintMostRecentCommonAncestorRes)
+	err := c.cc.Invoke(ctx, "/Replication/HintMostRecentCommonAncestor", in, out, opts...)
+	if err != nil {
+		return nil, err
+	}
+	return out, nil
+}
+
 // ReplicationServer is the server API for Replication service.
 type ReplicationServer interface {
 	Ping(context.Context, *PingReq) (*PingRes, error)
@@ -1259,6 +1278,8 @@ type ReplicationServer interface {
 	ListFilesystemVersions(context.Context, *ListFilesystemVersionsReq) (*ListFilesystemVersionsRes, error)
 	DestroySnapshots(context.Context, *DestroySnapshotsReq) (*DestroySnapshotsRes, error)
 	ReplicationCursor(context.Context, *ReplicationCursorReq) (*ReplicationCursorRes, error)
+	SendCompleted(context.Context, *SendCompletedReq) (*SendCompletedRes, error)
+	HintMostRecentCommonAncestor(context.Context, *HintMostRecentCommonAncestorReq) (*HintMostRecentCommonAncestorRes, error)
 }

 func RegisterReplicationServer(s *grpc.Server, srv ReplicationServer) {
@@ -1355,6 +1376,42 @@ func _Replication_ReplicationCursor_Handler(srv interface{}, ctx context.Context
 	return interceptor(ctx, in, info, handler)
 }

+func _Replication_SendCompleted_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
+	in := new(SendCompletedReq)
+	if err := dec(in); err != nil {
+		return nil, err
+	}
+	if interceptor == nil {
+		return srv.(ReplicationServer).SendCompleted(ctx, in)
+	}
+	info := &grpc.UnaryServerInfo{
+		Server:     srv,
+		FullMethod: "/Replication/SendCompleted",
+	}
+	handler := func(ctx context.Context, req interface{}) (interface{}, error) {
+		return srv.(ReplicationServer).SendCompleted(ctx, req.(*SendCompletedReq))
+	}
+	return interceptor(ctx, in, info, handler)
+}
+
+func _Replication_HintMostRecentCommonAncestor_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
+	in := new(HintMostRecentCommonAncestorReq)
+	if err := dec(in); err != nil {
+		return nil, err
+	}
+	if interceptor == nil {
+		return srv.(ReplicationServer).HintMostRecentCommonAncestor(ctx, in)
+	}
+	info := &grpc.UnaryServerInfo{
+		Server:     srv,
+		FullMethod: "/Replication/HintMostRecentCommonAncestor",
+	}
+	handler := func(ctx context.Context, req interface{}) (interface{}, error) {
+		return srv.(ReplicationServer).HintMostRecentCommonAncestor(ctx, req.(*HintMostRecentCommonAncestorReq))
+	}
+	return interceptor(ctx, in, info, handler)
+}
+
 var _Replication_serviceDesc = grpc.ServiceDesc{
 	ServiceName: "Replication",
 	HandlerType: (*ReplicationServer)(nil),
@@ -1379,63 +1436,77 @@ var _Replication_serviceDesc = grpc.ServiceDesc{
 			MethodName: "ReplicationCursor",
 			Handler:    _Replication_ReplicationCursor_Handler,
 		},
+		{
+			MethodName: "SendCompleted",
+			Handler:    _Replication_SendCompleted_Handler,
+		},
+		{
+			MethodName: "HintMostRecentCommonAncestor",
+			Handler:    _Replication_HintMostRecentCommonAncestor_Handler,
+		},
 	},
 	Streams:  []grpc.StreamDesc{},
 	Metadata: "pdu.proto",
 }

-func init() { proto.RegisterFile("pdu.proto", fileDescriptor_pdu_83b7e2a28d820622) }
+func init() { proto.RegisterFile("pdu.proto", fileDescriptor_pdu_0f43b713cd3bf056) }

-var fileDescriptor_pdu_83b7e2a28d820622 = []byte{
-	// 785 bytes of a gzipped FileDescriptorProto
-	0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0x8c, 0x55, 0xd1, 0x8e, 0xe3, 0x34,
-	0x14, 0x9d, 0xb4, 0x69, 0x9b, 0xde, 0x0e, 0xbb, 0x1d, 0x4f, 0x59, 0x85, 0x00, 0xab, 0xca, 0xcb,
-	0x43, 0x17, 0x89, 0x80, 0x0a, 0x2f, 0x08, 0x09, 0x89, 0x4e, 0x67, 0x67, 0x10, 0xb0, 0x54, 0x6e,
-	0x59, 0xad, 0xf6, 0x2d, 0x34, 0x57, 0x6d, 0x34, 0x6d, 0x9d, 0xb5, 0x13, 0xb4, 0xe5, 0x91, 0xbf,
-	0x9a, 0x7f, 0xe0, 0x91, 0x0f, 0x42, 0x76, 0xe3, 0x34, 0x6d, 0x52, 0xa9, 0x4f, 0xf1, 0x39, 0xf7,
-	0xda, 0x3e, 0xf7, 0xd8, 0xd7, 0x81, 0x76, 0x1c, 0xa6, 0x7e, 0x2c, 0x78, 0xc2, 0xe9, 0x35, 0x5c,
-	0xfd, 0x1a, 0xc9, 0xe4, 0x55, 0xb4, 0x42, 0xb9, 0x95, 0x09, 0xae, 0x19, 0xbe, 0xa7, 0xa3, 0x32,
-	0x29, 0xc9, 0x57, 0xd0, 0xd9, 0x13, 0xd2, 0xb5, 0xfa, 0xf5, 0x41, 0x67, 0xd8, 0xf1, 0x0b, 0x49,
-	0xc5, 0x38, 0x5d, 0x02, 0xec, 0x21, 0x21, 0x60, 0x4f, 0x82, 0x64, 0xe9, 0x5a, 0x7d, 0x6b, 0xd0,
-	0x66, 0x7a, 0x4c, 0xfa, 0xd0, 0x61, 0x28, 0xd3, 0x35, 0xce, 0xf8, 0x03, 0x6e, 0xdc, 0x9a, 0x0e,
-	0x15, 0x29, 0xf2, 0x05, 0x7c, 0xf4, 0xb3, 0x9c, 0xac, 0x82, 0x39, 0x2e, 0xf9, 0x2a, 0x44, 0xe1,
-	0xd6, 0xfb, 0xd6, 0xc0, 0x61, 0x87, 0x24, 0xfd, 0x01, 0x3e, 0x39, 0x54, 0xfb, 0x06, 0x85, 0x8c,
-	0xf8, 0x46, 0x32, 0x7c, 0x4f, 0x9e, 0x17, 0x65, 0x64, 0xdb, 0x17, 0x18, 0xfa, 0xcb, 0xe9, 0xc9,
-	0x92, 0xf8, 0xe0, 0x18, 0x98, 0xd5, 0x4b, 0xfc, 0x52, 0x26, 0xcb, 0x73, 0xe8, 0x7f, 0x16, 0x5c,
-	0x95, 0xe2, 0x64, 0x08, 0xf6, 0x6c, 0x1b, 0xa3, 0xde, 0xfc, 0xc9, 0xf0, 0x79, 0x79, 0x05, 0x3f,
-	0xfb, 0xaa, 0x2c, 0xa6, 0x73, 0x95, 0x5f, 0xaf, 0x83, 0x35, 0x66, 0xa6, 0xe8, 0xb1, 0xe2, 0xee,
-	0xd2, 0x28, 0xd4, 0x26, 0xd8, 0x4c, 0x8f, 0xc9, 0x67, 0xd0, 0xbe, 0x11, 0x18, 0x24, 0x38, 0x7b,
-	0x7b, 0xe7, 0xda, 0x3a, 0xb0, 0x27, 0x88, 0x07, 0x8e, 0x06, 0x11, 0xdf, 0xb8, 0x0d, 0xbd, 0x52,
-	0x8e, 0xe9, 0x4b, 0xe8, 0x14, 0xb6, 0x25, 0x97, 0xe0, 0x4c, 0x37, 0x41, 0x2c, 0x97, 0x3c, 0xe9,
-	0x5e, 0x28, 0x34, 0xe2, 0xfc, 0x61, 0x1d, 0x88, 0x87, 0xae, 0x45, 0x1f, 0x2d, 0x68, 0x4d, 0x71,
-	0x13, 0x9e, 0xe1, 0xa7, 0x12, 0xf9, 0x4a, 0xf0, 0xb5, 0x11, 0xae, 0xc6, 0xe4, 0x09, 0xd4, 0x66,
-	0x5c, 0xcb, 0x6e, 0xb3, 0xda, 0x8c, 0x1f, 0x1f, 0xbc, 0x5d, 0x3e, 0x78, 0x25, 0x9c, 0xaf, 0x63,
-	0x81, 0x52, 0x6a, 0xe1, 0x0e, 0xcb, 0x31, 0xe9, 0x41, 0x63, 0x8c, 0x61, 0x1a, 0xbb, 0x4d, 0x1d,
-	0xd8, 0x01, 0xf2, 0x0c, 0x9a, 0x63, 0xb1, 0x65, 0xe9, 0xc6, 0x6d, 0x69, 0x3a, 0x43, 0xf4, 0x3b,
-	0x70, 0x26, 0x82, 0xc7, 0x28, 0x92, 0x6d, 0x6e, 0xaa, 0x55, 0x30, 0xb5, 0x07, 0x8d, 0x37, 0xc1,
-	0x2a, 0x35, 0x4e, 0xef, 0x00, 0xfd, 0x27, 0xaf, 0x58, 0x92, 0x01, 0x3c, 0xfd, 0x43, 0x62, 0x78,
-	0x7c, 0x55, 0x1d, 0x76, 0x4c, 0x13, 0x0a, 0x97, 0xb7, 0x1f, 0x62, 0x9c, 0x27, 0x18, 0x4e, 0xa3,
-	0xbf, 0x51, 0x57, 0x5c, 0x67, 0x07, 0x1c, 0x79, 0x09, 0x90, 0xe9, 0x89, 0x50, 0xba, 0xb6, 0xbe,
-	0x54, 0x6d, 0xdf, 0x48, 0x64, 0x85, 0x20, 0x7d, 0x0b, 0xc0, 0x70, 0x8e, 0xd1, 0x5f, 0x78, 0x8e,
-	0xf1, 0x5f, 0x42, 0xf7, 0x66, 0x85, 0x81, 0x28, 0xeb, 0x2c, 0xf1, 0xf4, 0xb2, 0xb0, 0xb2, 0xa4,
-	0x0b, 0xb8, 0x1e, 0xa3, 0x4c, 0x04, 0xdf, 0x9a, 0x1b, 0x70, 0x4e, 0xe7, 0x90, 0x6f, 0xa0, 0x9d,
-	0xe7, 0xbb, 0xb5, 0x93, 0xdd, 0xb1, 0x4f, 0xa2, 0xef, 0x80, 0x1c, 0x6d, 0x94, 0x35, 0x99, 0x81,
-	0x7a, 0x97, 0x13, 0x4d, 0x66, 0x72, 0xd4, 0x89, 0xdd, 0x0a, 0xc1, 0x85, 0x39, 0x31, 0x0d, 0xe8,
-	0xb8, 0xaa, 0x08, 0xf5, 0x68, 0xb5, 0x54, 0xe1, 0xab, 0xc4, 0x34, 0xf0, 0xb5, 0x5f, 0x96, 0xc0,
-	0x4c, 0x0e, 0xfd, 0xd7, 0x82, 0x1e, 0xc3, 0x78, 0x15, 0xcd, 0x75, 0x93, 0xdc, 0xa4, 0x42, 0x72,
-	0x71, 0x8e, 0x19, 0x5f, 0x43, 0x7d, 0x81, 0x89, 0x96, 0xd4, 0x19, 0x7e, 0xea, 0x57, 0xad, 0xe1,
-	0xdf, 0x61, 0xf2, 0x7b, 0x7c, 0x7f, 0xc1, 0x54, 0xa6, 0x9a, 0x20, 0x31, 0xd1, 0x57, 0xe4, 0xe4,
-	0x84, 0xa9, 0x99, 0x20, 0x31, 0xf1, 0x5a, 0xd0, 0xd0, 0x0b, 0x78, 0x2f, 0xa0, 0xa1, 0x03, 0xaa,
-	0x49, 0x72, 0xe3, 0x76, 0x5e, 0xe4, 0x78, 0x64, 0x43, 0x8d, 0xc7, 0x74, 0x56, 0x59, 0x8d, 0x6a,
-	0xa1, 0xdd, 0x4b, 0xa2, 0xea, 0xb0, 0xef, 0x2f, 0xf2, 0xb7, 0xc4, 0x79, 0xcd, 0x13, 0xfc, 0x10,
-	0xc9, 0xdd, 0x7a, 0xce, 0xfd, 0x05, 0xcb, 0x99, 0x91, 0x03, 0xcd, 0x9d, 0x4b, 0xf4, 0x05, 0xb4,
-	0x26, 0xd1, 0x66, 0xa1, 0x6c, 0x71, 0xa1, 0xf5, 0x1b, 0x4a, 0x19, 0x2c, 0x4c, 0x53, 0x19, 0x48,
-	0x3f, 0x37, 0x49, 0x52, 0xb5, 0xdd, 0xed, 0x7c, 0xc9, 0x4d, 0xdb, 0xa9, 0xf1, 0xf0, 0xb1, 0xa6,
-	0xde, 0x80, 0x5c, 0x1a, 0xf1, 0xc0, 0x56, 0xe9, 0xc4, 0xf1, 0xb3, 0xa5, 0x3d, 0x33, 0x92, 0xe4,
-	0x7b, 0x78, 0x7a, 0xf8, 0x44, 0x4b, 0x42, 0xfc, 0xd2, 0x4f, 0xcb, 0x2b, 0x73, 0x92, 0x4c, 0xe0,
-	0x59, 0xf5, 0xeb, 0x4e, 0x3c, 0xff, 0xe4, 0x3f, 0xc3, 0x3b, 0x1d, 0x93, 0xe4, 0x47, 0xe8, 0x1e,
-	0xdf, 0x33, 0xd2, 0xf3, 0x2b, 0xfa, 0xc7, 0xab, 0x62, 0x25, 0xf9, 0x09, 0xae, 0x4a, 0x47, 0x42,
-	0x3e, 0xae, 0x3c, 0x7f, 0xaf, 0x92, 0x96, 0xa3, 0xc6, 0xbb, 0x7a, 0x1c, 0xa6, 0x7f, 0x36, 0xf5,
-	0x0f, 0xfc, 0xdb, 0xff, 0x03, 0x00, 0x00, 0xff, 0xff, 0x37, 0x0e, 0xf2, 0xe4, 0xcd, 0x07, 0x00,
-	0x00,
+var fileDescriptor_pdu_0f43b713cd3bf056 = []byte{
+	// 892 bytes of a gzipped FileDescriptorProto
+	0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0x8c, 0x56, 0xdf, 0x6f, 0xdb, 0x36,
+	0x10, 0x8e, 0x6c, 0x39, 0x91, 0xcf, 0xe9, 0xea, 0x5c, 0xb2, 0x42, 0x13, 0xba, 0xce, 0xe3, 0x86,
+	0xc1, 0x0d, 0x30, 0x61, 0xc8, 0x7e, 0x60, 0xc3, 0x80, 0x02, 0x8d, 0x93, 0x34, 0xc5, 0xd6, 0xce,
+	0x60, 0xbc, 0x62, 0xe8, 0x9b, 0x6a, 0x1f, 0x12, 0x21, 0xb2, 0xa8, 0x90, 0xf4, 0x50, 0x6f, 0x7b,
+	0xda, 0xe3, 0xfe, 0xbd, 0xe5, 0x0f, 0x2a, 0x44, 0x4b, 0xb6, 0x6c, 0xc9, 0x89, 0x9f, 0xcc, 0xfb,
+	0x78, 0x14, 0xef, 0xbe, 0xfb, 0xee, 0x68, 0x68, 0x26, 0xa3, 0x89, 0x9f, 0x48, 0xa1, 0x05, 0xdb,
+	0x87, 0xbd, 0x5f, 0x43, 0xa5, 0xcf, 0xc2, 0x88, 0xd4, 0x54, 0x69, 0x1a, 0x73, 0xba, 0x61, 0xc7,
+	0x65, 0x50, 0xe1, 0xd7, 0xd0, 0x5a, 0x00, 0xca, 0xb5, 0x3a, 0xf5, 0x6e, 0xeb, 0xa8, 0xe5, 0x17,
+	0x9c, 0x8a, 0xfb, 0xec, 0x3f, 0x0b, 0x60, 0x61, 0x23, 0x82, 0xdd, 0x0f, 0xf4, 0x95, 0x6b, 0x75,
+	0xac, 0x6e, 0x93, 0x9b, 0x35, 0x76, 0xa0, 0xc5, 0x49, 0x4d, 0xc6, 0x34, 0x10, 0xd7, 0x14, 0xbb,
+	0x35, 0xb3, 0x55, 0x84, 0xf0, 0x4b, 0x78, 0xf0, 0x52, 0xf5, 0xa3, 0x60, 0x48, 0x57, 0x22, 0x1a,
+	0x91, 0x74, 0xeb, 0x1d, 0xab, 0xeb, 0xf0, 0x65, 0x30, 0xfd, 0xce, 0x4b, 0x75, 0x1a, 0x0f, 0xe5,
+	0x34, 0xd1, 0x34, 0x72, 0x6d, 0xe3, 0x53, 0x84, 0xd8, 0xcf, 0xf0, 0xc9, 0x72, 0x42, 0x6f, 0x48,
+	0xaa, 0x50, 0xc4, 0x8a, 0xd3, 0x0d, 0x3e, 0x29, 0x06, 0x9a, 0x05, 0x58, 0x40, 0xd8, 0x2f, 0xeb,
+	0x0f, 0x2b, 0xf4, 0xc1, 0xc9, 0xcd, 0x8c, 0x12, 0xf4, 0x4b, 0x9e, 0x7c, 0xee, 0xc3, 0x6e, 0x2d,
+	0xd8, 0x2b, 0xed, 0xe3, 0x11, 0xd8, 0x83, 0x69, 0x42, 0xe6, 0xf2, 0x8f, 0x8e, 0x9e, 0x94, 0xbf,
+	0xe0, 0x67, 0xbf, 0xa9, 0x17, 0x37, 0xbe, 0x29, 0xa3, 0xaf, 0x83, 0x31, 0x65, 0xb4, 0x99, 0x75,
+	0x8a, 0xbd, 0x98, 0x84, 0x23, 0x43, 0x93, 0xcd, 0xcd, 0x1a, 0x1f, 0x43, 0xb3, 0x27, 0x29, 0xd0,
+	0x34, 0xf8, 0xe3, 0x85, 0xe1, 0xc6, 0xe6, 0x0b, 0x00, 0x3d, 0x70, 0x8c, 0x11, 0x8a, 0xd8, 0x6d,
+	0x98, 0x2f, 0xcd, 0x6d, 0xf6, 0x14, 0x5a, 0x85, 0x6b, 0x71, 0x17, 0x9c, 0x8b, 0x38, 0x48, 0xd4,
+	0x95, 0xd0, 0xed, 0xad, 0xd4, 0x3a, 0x16, 0xe2, 0x7a, 0x1c, 0xc8, 0xeb, 0xb6, 0xc5, 0xfe, 0xb7,
+	0x60, 0xe7, 0x82, 0xe2, 0xd1, 0x06, 0x7c, 0xe2, 0x57, 0x60, 0x9f, 0x49, 0x31, 0x36, 0x81, 0x57,
+	0xd3, 0x65, 0xf6, 0x91, 0x41, 0x6d, 0x20, 0x4c, 0x2a, 0xd5, 0x5e, 0xb5, 0x81, 0x58, 0x95, 0x90,
+	0x5d, 0x96, 0x10, 0x83, 0xe6, 0x42, 0x1a, 0x0d, 0xc3, 0xaf, 0xed, 0x0f, 0x64, 0xc8, 0x17, 0x30,
+	0x3e, 0x82, 0xed, 0x13, 0x39, 0xe5, 0x93, 0xd8, 0xdd, 0x36, 0xda, 0xc9, 0x2c, 0xf6, 0x1d, 0x38,
+	0x7d, 0x29, 0x12, 0x92, 0x7a, 0x3a, 0xa7, 0xdb, 0x2a, 0xd0, 0x7d, 0x00, 0x8d, 0x37, 0x41, 0x34,
+	0xc9, 0x6b, 0x30, 0x33, 0xd8, 0xbf, 0x73, 0x2e, 0x14, 0x76, 0xe1, 0xe1, 0xef, 0x8a, 0x46, 0xab,
+	0x32, 0x77, 0xf8, 0x2a, 0x8c, 0x0c, 0x76, 0x4f, 0xdf, 0x27, 0x34, 0xd4, 0x34, 0xba, 0x08, 0xff,
+	0x22, 0x93, 0x77, 0x9d, 0x2f, 0x61, 0xf8, 0x14, 0x20, 0x8b, 0x27, 0x24, 0xe5, 0xda, 0x46, 0x6e,
+	0x4d, 0x3f, 0x0f, 0x91, 0x17, 0x36, 0xd9, 0x33, 0x68, 0xa7, 0x31, 0xf4, 0xc4, 0x38, 0x89, 0x48,
+	0x93, 0x29, 0xcc, 0x21, 0xb4, 0x7e, 0x93, 0xe1, 0x65, 0x18, 0x07, 0x11, 0xa7, 0x9b, 0x8c, 0x7f,
+	0xc7, 0xcf, 0xea, 0xc6, 0x8b, 0x9b, 0x0c, 0x4b, 0xe7, 0x15, 0xfb, 0x07, 0x80, 0xd3, 0x90, 0xc2,
+	0x3f, 0x69, 0x93, 0x32, 0xcf, 0xca, 0x57, 0xbb, 0xb3, 0x7c, 0x87, 0xd0, 0xee, 0x45, 0x14, 0xc8,
+	0x22, 0x3f, 0xb3, 0x16, 0x2f, 0xe1, 0x6c, 0xb7, 0x70, 0xbb, 0x62, 0x97, 0xb0, 0x7f, 0x42, 0x4a,
+	0x4b, 0x31, 0xcd, 0x35, 0xb9, 0x49, 0x2f, 0xe3, 0x37, 0xd0, 0x9c, 0xfb, 0xbb, 0xb5, 0xb5, 0xfd,
+	0xba, 0x70, 0x62, 0x6f, 0x01, 0x57, 0x2e, 0xca, 0xda, 0x3e, 0x37, 0xcd, 0x2d, 0x6b, 0xda, 0x3e,
+	0xf7, 0x49, 0x95, 0x72, 0x2a, 0xa5, 0x90, 0xb9, 0x52, 0x8c, 0xc1, 0x4e, 0xaa, 0x92, 0x48, 0x27,
+	0xed, 0x4e, 0x9a, 0x78, 0xa4, 0xf3, 0x91, 0xb2, 0xef, 0x97, 0x43, 0xe0, 0xb9, 0x0f, 0xfb, 0x01,
+	0x0e, 0x38, 0x25, 0x51, 0x38, 0x34, 0x5d, 0xdb, 0x9b, 0x48, 0x25, 0xe4, 0x26, 0x73, 0x6d, 0x50,
+	0x79, 0x4e, 0xe1, 0x41, 0x36, 0x44, 0xd2, 0x13, 0xf6, 0xf9, 0xd6, 0x7c, 0x8c, 0x38, 0xaf, 0x85,
+	0xa6, 0xf7, 0xa1, 0xd2, 0x33, 0x09, 0x9f, 0x6f, 0xf1, 0x39, 0x72, 0xec, 0xc0, 0xf6, 0x2c, 0x1c,
+	0xf6, 0x05, 0xec, 0xf4, 0xc3, 0xf8, 0x32, 0x0d, 0xc0, 0x85, 0x9d, 0x57, 0xa4, 0x54, 0x70, 0x99,
+	0x77, 0x4d, 0x6e, 0xb2, 0x4f, 0x73, 0x27, 0x95, 0xf6, 0xd5, 0xe9, 0xf0, 0x4a, 0xe4, 0x7d, 0x95,
+	0xae, 0xd9, 0xdf, 0xf0, 0xd9, 0x79, 0x18, 0xeb, 0x57, 0x42, 0xe9, 0xb4, 0xe4, 0xb1, 0xee, 0x89,
+	0xf1, 0x58, 0xc4, 0xcf, 0xe3, 0x21, 0x29, 0xbd, 0x51, 0x72, 0xf8, 0x23, 0x3c, 0x48, 0xf5, 0x4b,
+	0x32, 0xab, 0xc5, 0x1d, 0x42, 0x5c, 0x76, 0x64, 0x9f, 0xdf, 0x77, 0xb9, 0x3a, 0xec, 0x42, 0x7d,
+	0x20, 0xc3, 0x74, 0x04, 0x9e, 0x88, 0x58, 0xf7, 0x02, 0x49, 0xed, 0x2d, 0x6c, 0x42, 0xe3, 0x2c,
+	0x88, 0x14, 0xb5, 0x2d, 0x74, 0xc0, 0x1e, 0xc8, 0x09, 0xb5, 0x6b, 0x47, 0xb7, 0xf5, 0x74, 0x40,
+	0xcd, 0x49, 0x46, 0x0f, 0xec, 0x34, 0x71, 0x74, 0xfc, 0x8c, 0x24, 0x2f, 0x5f, 0x29, 0xfc, 0x09,
+	0x1e, 0x2e, 0xbf, 0x33, 0x0a, 0xd1, 0x2f, 0x3d, 0xce, 0x5e, 0x19, 0x53, 0xd8, 0x87, 0x47, 0xd5,
+	0x4f, 0x14, 0x7a, 0xfe, 0xda, 0x87, 0xcf, 0x5b, 0xbf, 0xa7, 0xf0, 0x19, 0xb4, 0x57, 0xa5, 0x89,
+	0x07, 0x7e, 0x45, 0xcb, 0x79, 0x55, 0xa8, 0xc2, 0xe7, 0xb0, 0x57, 0x12, 0x17, 0x7e, 0xec, 0x57,
+	0x09, 0xd5, 0xab, 0x84, 0x15, 0x7e, 0x3f, 0x2b, 0xe1, 0x7c, 0x04, 0xe1, 0x9e, 0xbf, 0x3a, 0xd2,
+	0xbc, 0x12, 0xa4, 0xf0, 0x1d, 0x3c, 0xbe, 0xab, 0x7e, 0xd8, 0xf1, 0xef, 0xd1, 0x96, 0x77, 0x9f,
+	0x87, 0x3a, 0x6e, 0xbc, 0xad, 0x27, 0xa3, 0xc9, 0xbb, 0x6d, 0xf3, 0x1f, 0xea, 0xdb, 0x0f, 0x01,
+	0x00, 0x00, 0xff, 0xff, 0xad, 0x4e, 0x98, 0x29, 0x50, 0x09, 0x00, 0x00,
 }
@@ -2,131 +2,133 @@ syntax = "proto3";
 option go_package = "pdu";

 service Replication {
-    rpc Ping (PingReq) returns (PingRes);
-    rpc ListFilesystems (ListFilesystemReq) returns (ListFilesystemRes);
-    rpc ListFilesystemVersions (ListFilesystemVersionsReq) returns (ListFilesystemVersionsRes);
-    rpc DestroySnapshots (DestroySnapshotsReq) returns (DestroySnapshotsRes);
-    rpc ReplicationCursor (ReplicationCursorReq) returns (ReplicationCursorRes);
-    // for Send and Recv, see package rpc
+  rpc Ping(PingReq) returns (PingRes);
+  rpc ListFilesystems(ListFilesystemReq) returns (ListFilesystemRes);
+  rpc ListFilesystemVersions(ListFilesystemVersionsReq)
+      returns (ListFilesystemVersionsRes);
+  rpc DestroySnapshots(DestroySnapshotsReq) returns (DestroySnapshotsRes);
+  rpc ReplicationCursor(ReplicationCursorReq) returns (ReplicationCursorRes);
+  rpc SendCompleted(SendCompletedReq) returns (SendCompletedRes);
+  rpc HintMostRecentCommonAncestor(HintMostRecentCommonAncestorReq) returns (HintMostRecentCommonAncestorRes);
+  // for Send and Recv, see package rpc
 }

 message ListFilesystemReq {}

-message ListFilesystemRes {
-    repeated Filesystem Filesystems = 1;
-}
+message ListFilesystemRes { repeated Filesystem Filesystems = 1; }

 message Filesystem {
-    string Path = 1;
-    string ResumeToken = 2;
-    bool IsPlaceholder = 3;
+  string Path = 1;
+  string ResumeToken = 2;
+  bool IsPlaceholder = 3;
+  bool IsEncrypted = 4;
 }

-message ListFilesystemVersionsReq {
-    string Filesystem = 1;
-}
+message ListFilesystemVersionsReq { string Filesystem = 1; }

-message ListFilesystemVersionsRes {
-    repeated FilesystemVersion Versions = 1;
-}
+message ListFilesystemVersionsRes { repeated FilesystemVersion Versions = 1; }

 message FilesystemVersion {
-    enum VersionType {
-        Snapshot = 0;
-        Bookmark = 1;
-    }
-    VersionType Type = 1;
-    string Name = 2;
-    uint64 Guid = 3;
-    uint64 CreateTXG = 4;
-    string Creation = 5; // RFC 3339
+  enum VersionType {
+    Snapshot = 0;
+    Bookmark = 1;
+  }
+  VersionType Type = 1;
+  string Name = 2;
+  uint64 Guid = 3;
+  uint64 CreateTXG = 4;
+  string Creation = 5; // RFC 3339
 }

+enum Tri {
+  DontCare = 0;
+  False = 1;
+  True = 2;
+}

 message SendReq {
-    string Filesystem  = 1;
-    string From = 2;
-    // May be empty / null to request a full transfer of From
-    string To = 3;
+  string Filesystem = 1;
+  // May be empty / null to request a full transfer of To
+  FilesystemVersion From = 2;
+  FilesystemVersion To = 3;

-    // If ResumeToken is not empty, the resume token that CAN be tried for 'zfs send' by the sender.
-    // The sender MUST indicate in SendRes.UsedResumeToken
-    // If it does not work, the sender SHOULD clear the resume token on their side
-    // and use From and To instead
-    // If ResumeToken is not empty, the GUIDs of From and To
-    // MUST correspond to those encoded in the ResumeToken.
-    // Otherwise, the Sender MUST return an error.
-    string ResumeToken  = 4;
-    bool Compress = 5;
-    bool Dedup = 6;
+  // If ResumeToken is not empty, the resume token that CAN be used for 'zfs
+  // send' by the sender. The sender MUST indicate use of ResumeToken in the
+  // reply message SendRes.UsedResumeToken If it does not work, the sender
+  // SHOULD clear the resume token on their side and use From and To instead If
+  // ResumeToken is not empty, the GUIDs of From and To MUST correspond to those
+  // encoded in the ResumeToken. Otherwise, the Sender MUST return an error.
+  string ResumeToken = 4;
+  Tri Encrypted = 5;

-    bool DryRun = 7;
+  bool DryRun = 6;
 }

 message Property {
-    string Name = 1;
-    string Value = 2;
+  string Name = 1;
+  string Value = 2;
 }

 message SendRes {
-    // Whether the resume token provided in the request has been used or not.
-    bool UsedResumeToken = 2;
+  // Whether the resume token provided in the request has been used or not.
+  // If the SendReq.ResumeToken == "", this field has no meaning.
+  bool UsedResumeToken = 2;

-    // Expected stream size determined by dry run, not exact.
-    // 0 indicates that for the given SendReq, no size estimate could be made.
-    int64 ExpectedSize = 3;
+  // Expected stream size determined by dry run, not exact.
+  // 0 indicates that for the given SendReq, no size estimate could be made.
+  int64 ExpectedSize = 3;

-    repeated Property Properties = 4;
+  repeated Property Properties = 4;
 }

-message ReceiveReq {
-    string Filesystem = 1; // FIXME should be snapshot name, we can enforce that on recv
+message SendCompletedReq {
+  SendReq OriginalReq = 2;
+}

-    // If true, the receiver should clear the resume token before perfoming the zfs recv of the stream in the request
-    bool ClearResumeToken = 2;
+message SendCompletedRes {}
+
+message ReceiveReq {
+  string Filesystem = 1;
+  FilesystemVersion To = 2;
+
+  // If true, the receiver should clear the resume token before perfoming the
+  // zfs recv of the stream in the request
+  bool ClearResumeToken = 3;
 }

 message ReceiveRes {}

 message DestroySnapshotsReq {
-    string Filesystem = 1;
-    // Path to filesystem, snapshot or bookmark to be destroyed
-    repeated FilesystemVersion Snapshots = 2;
+  string Filesystem = 1;
+  // Path to filesystem, snapshot or bookmark to be destroyed
+  repeated FilesystemVersion Snapshots = 2;
 }

 message DestroySnapshotRes {
-    FilesystemVersion Snapshot = 1;
-    string Error = 2;
+  FilesystemVersion Snapshot = 1;
+  string Error = 2;
 }

-message DestroySnapshotsRes {
-    repeated DestroySnapshotRes Results = 1;
-}
+message DestroySnapshotsRes { repeated DestroySnapshotRes Results = 1; }

-message ReplicationCursorReq {
-    string Filesystem = 1;
-    message GetOp {}
-    message SetOp {
-        string Snapshot = 2;
-    }
-    oneof op {
-        GetOp get = 2;
-        SetOp set = 3;
-    }
-}
+message ReplicationCursorReq { string Filesystem = 1; }

 message ReplicationCursorRes {
-    oneof Result {
-        uint64 Guid = 1;
-        bool Notexist = 2;
-    }
+  oneof Result {
+    uint64 Guid = 1;
+    bool Notexist = 2;
+  }
 }

-message PingReq {
-    string Message = 1;
-}
+message PingReq { string Message = 1; }

 message PingRes {
-    // Echo must be PingReq.Message
-    string Echo = 1;
-}
+  // Echo must be PingReq.Message
+  string Echo = 1;
+}
+
+message HintMostRecentCommonAncestorReq {
+    string Filesystem = 1;
+    FilesystemVersion SenderVersion = 2;
+}
+message HintMostRecentCommonAncestorRes {}
@@ -7,6 +7,14 @@ import (
 	"github.com/zrepl/zrepl/zfs"
 )

+func (v *FilesystemVersion) GetRelName() string {
+	zv, err := v.ZFSFilesystemVersion()
+	if err != nil {
+		return ""
+	}
+	return zv.String()
+}
+
 func (v *FilesystemVersion) RelName() string {
 	zv, err := v.ZFSFilesystemVersion()
 	if err != nil {
@@ -30,6 +30,7 @@ type Endpoint interface {
 	ListFilesystemVersions(ctx context.Context, req *pdu.ListFilesystemVersionsReq) (*pdu.ListFilesystemVersionsRes, error)
 	DestroySnapshots(ctx context.Context, req *pdu.DestroySnapshotsReq) (*pdu.DestroySnapshotsRes, error)
 	WaitForConnectivity(ctx context.Context) error
+	HintMostRecentCommonAncestor(context.Context, *pdu.HintMostRecentCommonAncestorReq) (*pdu.HintMostRecentCommonAncestorRes, error)
 }

 type Sender interface {
@@ -38,6 +39,7 @@ type Sender interface {
 	// any next call to the parent github.com/zrepl/zrepl/replication.Endpoint.
 	// If the send request is for dry run the io.ReadCloser will be nil
 	Send(ctx context.Context, r *pdu.SendReq) (*pdu.SendRes, zfs.StreamCopier, error)
+	SendCompleted(ctx context.Context, r *pdu.SendCompletedReq) (*pdu.SendCompletedRes, error)
 	ReplicationCursor(ctx context.Context, req *pdu.ReplicationCursorReq) (*pdu.ReplicationCursorRes, error)
 }

@@ -48,9 +50,14 @@ type Receiver interface {
 	Receive(ctx context.Context, req *pdu.ReceiveReq, receive zfs.StreamCopier) (*pdu.ReceiveRes, error)
 }

+type PlannerPolicy struct {
+	EncryptedSend tri // all sends must be encrypted (send -w, and encryption!=off)
+}
+
 type Planner struct {
 	sender   Sender
 	receiver Receiver
+	policy   PlannerPolicy

 	promSecsPerState    *prometheus.HistogramVec // labels: state
 	promBytesReplicated *prometheus.CounterVec   // labels: filesystem
@@ -109,10 +116,11 @@ func (p *Planner) WaitForConnectivity(ctx context.Context) error {
 type Filesystem struct {
 	sender   Sender
 	receiver Receiver
+	policy   PlannerPolicy

-	Path                string // compat
-	receiverFS          *pdu.Filesystem
-	promBytesReplicated prometheus.Counter // compat
+	Path                 string             // compat
+	receiverFS, senderFS *pdu.Filesystem    // receiverFS may be nil, senderFS never nil
+	promBytesReplicated  prometheus.Counter // compat

 	sizeEstimateRequestSem *semaphore.S
 }
@@ -145,8 +153,10 @@ type Step struct {
 	sender   Sender
 	receiver Receiver

-	parent   *Filesystem
-	from, to *pdu.FilesystemVersion // compat
+	parent      *Filesystem
+	from, to    *pdu.FilesystemVersion // from may be nil, indicating full send
+	encrypt     tri
+	resumeToken string // empty means no resume token shall be used

 	expectedSize int64 // 0 means no size estimate present / possible

@@ -186,23 +196,36 @@ func (s *Step) ReportInfo() *report.StepInfo {
 	}
 	s.byteCounterMtx.Unlock()

-	// FIXME stick to zfs convention of from and to
 	from := ""
 	if s.from != nil {
 		from = s.from.RelName()
 	}
+	var encrypted report.EncryptedEnum
+	switch s.encrypt {
+	case DontCare:
+		encrypted = report.EncryptedSenderDependent
+	case True:
+		encrypted = report.EncryptedTrue
+	case False:
+		encrypted = report.EncryptedFalse
+	default:
+		panic(fmt.Sprintf("unknown variant %s", s.encrypt))
+	}
 	return &report.StepInfo{
 		From:            from,
 		To:              s.to.RelName(),
+		Resumed:         s.resumeToken != "",
+		Encrypted:       encrypted,
 		BytesExpected:   s.expectedSize,
 		BytesReplicated: byteCounter,
 	}
 }

-func NewPlanner(secsPerState *prometheus.HistogramVec, bytesReplicated *prometheus.CounterVec, sender Sender, receiver Receiver) *Planner {
+func NewPlanner(secsPerState *prometheus.HistogramVec, bytesReplicated *prometheus.CounterVec, sender Sender, receiver Receiver, policy PlannerPolicy) *Planner {
 	return &Planner{
 		sender:              sender,
 		receiver:            receiver,
+		policy:              policy,
 		promSecsPerState:    secsPerState,
 		promBytesReplicated: bytesReplicated,
 	}
@@ -265,7 +288,9 @@ func (p *Planner) doPlanning(ctx context.Context) ([]*Filesystem, error) {
 		q = append(q, &Filesystem{
 			sender:                 p.sender,
 			receiver:               p.receiver,
+			policy:                 p.policy,
 			Path:                   fs.Path,
+			senderFS:               fs,
 			receiverFS:             receiverFS,
 			promBytesReplicated:    ctr,
 			sizeEstimateRequestSem: sizeEstimateRequestSem,
@@ -281,6 +306,10 @@ func (fs *Filesystem) doPlanning(ctx context.Context) ([]*Step, error) {

 	log.Debug("assessing filesystem")

+	if fs.policy.EncryptedSend == True && !fs.senderFS.GetIsEncrypted() {
+		return nil, fmt.Errorf("sender filesystem is not encrypted but policy mandates encrypted send")
+	}
+
 	sfsvsres, err := fs.sender.ListFilesystemVersions(ctx, &pdu.ListFilesystemVersionsReq{Filesystem: fs.Path})
 	if err != nil {
 		log.WithError(err).Error("cannot get remote filesystem versions")
@@ -306,42 +335,194 @@ func (fs *Filesystem) doPlanning(ctx context.Context) ([]*Step, error) {
 		rfsvs = []*pdu.FilesystemVersion{}
 	}

-	path, conflict := IncrementalPath(rfsvs, sfsvs)
-	if conflict != nil {
-		var msg string
-		path, msg = resolveConflict(conflict) // no shadowing allowed!
-		if path != nil {
-			log.WithField("conflict", conflict).Info("conflict")
-			log.WithField("resolution", msg).Info("automatically resolved")
-		} else {
-			log.WithField("conflict", conflict).Error("conflict")
-			log.WithField("problem", msg).Error("cannot resolve conflict")
+	var resumeToken *zfs.ResumeToken
+	var resumeTokenRaw string
+	if fs.receiverFS != nil && fs.receiverFS.ResumeToken != "" {
+		resumeTokenRaw = fs.receiverFS.ResumeToken // shadow
+		log.WithField("receiverFS.ResumeToken", resumeTokenRaw).Debug("decode receiver fs resume token")
+		resumeToken, err = zfs.ParseResumeToken(ctx, resumeTokenRaw) // shadow
+		if err != nil {
+			// TODO in theory, we could do replication without resume token, but that would mean that
+			// we need to discard the resumable state on the receiver's side.
+			// Would be easy by setting UsedResumeToken=false in the RecvReq ...
+			// FIXME / CHECK semantics UsedResumeToken if SendReq.ResumeToken == ""
+			log.WithError(err).Error("cannot decode resume token, aborting")
+			return nil, err
 		}
-	}
-	if len(path) == 0 {
-		return nil, conflict
+		log.WithField("token", resumeToken).Debug("decode resume token")
 	}

-	steps := make([]*Step, 0, len(path))
-	// FIXME unify struct declarations => initializer?
-	if len(path) == 1 {
-		steps = append(steps, &Step{
+	// give both sides a hint about how far the replication got
+	// This serves as a cummulative variant of SendCompleted and can be useful
+	// for example to release stale holds from an earlier (interrupted) replication.
+	// TODO FIXME: enqueue this as a replication step instead of doing it here during planning
+	//             then again, the step should run regardless of planning success
+	//             so maybe a separate phase before PLANNING, then?
+	path, conflict := IncrementalPath(rfsvs, sfsvs)
+	var sender_mrca *pdu.FilesystemVersion // from sfsvs
+	if conflict == nil && len(path) > 0 {
+		sender_mrca = path[0] // shadow
+	}
+	if sender_mrca != nil {
+		var wg sync.WaitGroup
+		doHint := func(ep Endpoint, name string) {
+			defer wg.Done()
+			log := log.WithField("to_side", name).
+				WithField("sender_mrca", sender_mrca.String())
+			log.Debug("hint most recent common ancestor")
+			hint := &pdu.HintMostRecentCommonAncestorReq{
+				Filesystem:    fs.Path,
+				SenderVersion: sender_mrca,
+			}
+			_, err := ep.HintMostRecentCommonAncestor(ctx, hint)
+			if err != nil {
+				log.WithError(err).Error("error hinting most recent common ancestor")
+			}
+		}
+		wg.Add(2)
+		go doHint(fs.sender, "sender")
+		go doHint(fs.receiver, "receiver")
+		wg.Wait()
+	} else {
+		log.Debug("cannot identify most recent common ancestor, skipping hint")
+	}
+
+	var steps []*Step
+	// build the list of replication steps
+	//
+	// prefer to resume any started replication instead of starting over with a normal IncrementalPath
+	//
+	// look for the step encoded in the resume token in the sender's version
+	// if we find that step:
+	//   1. use it as first step (including resume token)
+	//   2. compute subsequent steps by computing incremental path from the token.To version on
+	//      ...
+	//      that's actually equivalent to simply cutting off earlier versions from rfsvs and sfsvs
+	if resumeToken != nil {
+
+		sfsvs := SortVersionListByCreateTXGThenBookmarkLTSnapshot(sfsvs)
+
+		var fromVersion, toVersion *pdu.FilesystemVersion
+		var toVersionIdx int
+		for idx, sfsv := range sfsvs {
+			if resumeToken.HasFromGUID && sfsv.Guid == resumeToken.FromGUID {
+				if fromVersion != nil && fromVersion.Type == pdu.FilesystemVersion_Snapshot {
+					// prefer snapshots over bookmarks for size estimation
+				} else {
+					fromVersion = sfsv
+				}
+			}
+			if resumeToken.HasToGUID && sfsv.Guid == resumeToken.ToGUID && sfsv.Type == pdu.FilesystemVersion_Snapshot {
+				// `toversion` must always be a snapshot
+				toVersion, toVersionIdx = sfsv, idx
+			}
+		}
+
+		encryptionMatches := false
+		switch fs.policy.EncryptedSend {
+		case True:
+			encryptionMatches = resumeToken.RawOK && resumeToken.CompressOK
+		case False:
+			encryptionMatches = !resumeToken.RawOK && !resumeToken.CompressOK
+		case DontCare:
+			encryptionMatches = true
+		}
+
+		log.WithField("fromVersion", fromVersion).
+			WithField("toVersion", toVersion).
+			WithField("encryptionMatches", encryptionMatches).
+			Debug("result of resume-token-matching to sender's versions")
+
+		if !encryptionMatches {
+			return nil, fmt.Errorf("resume token `rawok`=%v and `compressok`=%v are incompatible with encryption policy=%v", resumeToken.RawOK, resumeToken.CompressOK, fs.policy.EncryptedSend)
+		} else if toVersion == nil {
+			return nil, fmt.Errorf("resume token `toguid` = %v not found on sender (`toname` = %q)", resumeToken.ToGUID, resumeToken.ToName)
+		} else if fromVersion == toVersion {
+			return nil, fmt.Errorf("resume token `fromguid` and `toguid` match same version on sener")
+		}
+		// fromVersion may be nil, toVersion is no nil, encryption matches
+		// good to go this one step!
+		resumeStep := &Step{
 			parent:   fs,
 			sender:   fs.sender,
 			receiver: fs.receiver,
-			from:     nil,
-			to:       path[0],
-		})
-	} else {
-		for i := 0; i < len(path)-1; i++ {
+
+			from:    fromVersion,
+			to:      toVersion,
+			encrypt: fs.policy.EncryptedSend,
+
+			resumeToken: resumeTokenRaw,
+		}
+
+		// by definition, the resume token _must_ be the receiver's most recent version, if they have any
+		// don't bother checking, zfs recv will produce an error if above assumption is wrong
+		//
+		// thus, subsequent steps are just incrementals on the sender's remaining _snapshots_ (not bookmarks)
+
+		var remainingSFSVs []*pdu.FilesystemVersion
+		for _, sfsv := range sfsvs[toVersionIdx:] {
+			if sfsv.Type == pdu.FilesystemVersion_Snapshot {
+				remainingSFSVs = append(remainingSFSVs, sfsv)
+			}
+		}
+
+		steps = make([]*Step, 0, len(remainingSFSVs)) // shadow
+		steps = append(steps, resumeStep)
+		for i := 0; i < len(remainingSFSVs)-1; i++ {
 			steps = append(steps, &Step{
 				parent:   fs,
 				sender:   fs.sender,
 				receiver: fs.receiver,
-				from:     path[i],
-				to:       path[i+1],
+				from:     remainingSFSVs[i],
+				to:       remainingSFSVs[i+1],
+				encrypt:  fs.policy.EncryptedSend,
 			})
 		}
+	} else { // resumeToken == nil
+		path, conflict := IncrementalPath(rfsvs, sfsvs)
+		if conflict != nil {
+			var msg string
+			path, msg = resolveConflict(conflict) // no shadowing allowed!
+			if path != nil {
+				log.WithField("conflict", conflict).Info("conflict")
+				log.WithField("resolution", msg).Info("automatically resolved")
+			} else {
+				log.WithField("conflict", conflict).Error("conflict")
+				log.WithField("problem", msg).Error("cannot resolve conflict")
+			}
+		}
+		if len(path) == 0 {
+			return nil, conflict
+		}
+
+		steps = make([]*Step, 0, len(path)) // shadow
+		if len(path) == 1 {
+			steps = append(steps, &Step{
+				parent:   fs,
+				sender:   fs.sender,
+				receiver: fs.receiver,
+
+				from:    nil,
+				to:      path[0],
+				encrypt: fs.policy.EncryptedSend,
+			})
+		} else {
+			for i := 0; i < len(path)-1; i++ {
+				steps = append(steps, &Step{
+					parent:   fs,
+					sender:   fs.sender,
+					receiver: fs.receiver,
+
+					from:    path[i],
+					to:      path[i+1],
+					encrypt: fs.policy.EncryptedSend,
+				})
+			}
+		}
+	}
+
+	if len(steps) == 0 {
+		log.Info("planning determined that no replication steps are required")
 	}

 	log.Debug("compute send size estimate")
@@ -389,29 +570,6 @@ func (fs *Filesystem) doPlanning(ctx context.Context) ([]*Step, error) {
 	return steps, nil
 }

-// type FilesystemsReplicationFailedError struct {
-// 	FilesystemsWithError []*fsrep.Replication
-// }
-
-// func (e FilesystemsReplicationFailedError) Error() string {
-// 	allSame := true
-// 	lastErr := e.FilesystemsWithError[0].Err().Error()
-// 	for _, fs := range e.FilesystemsWithError {
-// 		fsErr := fs.Err().Error()
-// 		allSame = allSame && lastErr == fsErr
-// 	}
-
-// 	fsstr := "multiple filesystems"
-// 	if len(e.FilesystemsWithError) == 1 {
-// 		fsstr = fmt.Sprintf("filesystem %s", e.FilesystemsWithError[0].FS())
-// 	}
-// 	errorStr := lastErr
-// 	if !allSame {
-// 		errorStr = "multiple different errors"
-// 	}
-// 	return fmt.Sprintf("%s could not be replicated: %s", fsstr, errorStr)
-// }
-
 func (s *Step) updateSizeEstimate(ctx context.Context) error {

 	log := getLogger(ctx)
@@ -430,19 +588,13 @@ func (s *Step) updateSizeEstimate(ctx context.Context) error {

 func (s *Step) buildSendRequest(dryRun bool) (sr *pdu.SendReq) {
 	fs := s.parent.Path
-	if s.from == nil {
-		sr = &pdu.SendReq{
-			Filesystem: fs,
-			To:         s.to.RelName(),
-			DryRun:     dryRun,
-		}
-	} else {
-		sr = &pdu.SendReq{
-			Filesystem: fs,
-			From:       s.from.RelName(),
-			To:         s.to.RelName(),
-			DryRun:     dryRun,
-		}
+	sr = &pdu.SendReq{
+		Filesystem:  fs,
+		From:        s.from, // may be nil
+		To:          s.to,
+		Encrypted:   s.encrypt.ToPDU(),
+		ResumeToken: s.resumeToken,
+		DryRun:      dryRun,
 	}
 	return sr
 }
@@ -478,6 +630,7 @@ func (s *Step) doReplication(ctx context.Context) error {

 	rr := &pdu.ReceiveReq{
 		Filesystem:       fs,
+		To:               sr.GetTo(),
 		ClearResumeToken: !sres.UsedResumeToken,
 	}
 	log.Debug("initiate receive request")
@@ -486,6 +639,7 @@ func (s *Step) doReplication(ctx context.Context) error {
 		log.
 			WithError(err).
 			WithField("errType", fmt.Sprintf("%T", err)).
+			WithField("rr", fmt.Sprintf("%v", rr)).
 			Error("receive request failed (might also be error on sender)")
 		// This failure could be due to
 		// 	- an unexpected exit of ZFS on the sending side
@@ -495,22 +649,12 @@ func (s *Step) doReplication(ctx context.Context) error {
 	}
 	log.Debug("receive finished")

-	log.Debug("advance replication cursor")
-	req := &pdu.ReplicationCursorReq{
-		Filesystem: fs,
-		Op: &pdu.ReplicationCursorReq_Set{
-			Set: &pdu.ReplicationCursorReq_SetOp{
-				Snapshot: s.to.GetName(),
-			},
-		},
-	}
-	_, err = s.sender.ReplicationCursor(ctx, req)
+	log.Debug("tell sender replication completed")
+	_, err = s.sender.SendCompleted(ctx, &pdu.SendCompletedReq{
+		OriginalReq: sr,
+	})
 	if err != nil {
-		log.WithError(err).Error("error advancing replication cursor")
-		// If this fails and replication planning restarts, the diff algorithm will find
-		// that cursor out of place. This is not a problem because then, it would just use another FS
-		// However, we FIXME have no means to just update the cursor in a
-		// second replication attempt right after this one where we don't have new snaps yet
+		log.WithError(err).Error("error telling sender that replication completed successfully")
 		return err
 	}

@@ -0,0 +1,46 @@
+package logic
+
+import (
+	"fmt"
+
+	"github.com/zrepl/zrepl/replication/logic/pdu"
+)
+
+type tri int
+
+const (
+	DontCare = 0x0
+	False    = 0x1
+	True     = 0x2
+)
+
+func (t tri) String() string {
+	switch t {
+	case DontCare:
+		return "dontcare"
+	case False:
+		return "false"
+	case True:
+		return "true"
+	}
+	panic(fmt.Sprintf("unknown variant %v", int(t)))
+}
+
+func (t tri) ToPDU() pdu.Tri {
+	switch t {
+	case DontCare:
+		return pdu.Tri_DontCare
+	case False:
+		return pdu.Tri_False
+	case True:
+		return pdu.Tri_True
+	}
+	panic(fmt.Sprintf("unknown variant %v", int(t)))
+}
+
+func TriFromBool(b bool) tri {
+	if b {
+		return True
+	}
+	return False
+}
@@ -85,8 +85,18 @@ type StepReport struct {
 	Info *StepInfo
 }

+type EncryptedEnum string
+
+const (
+	EncryptedTrue            EncryptedEnum = "yes"
+	EncryptedFalse           EncryptedEnum = "no"
+	EncryptedSenderDependent EncryptedEnum = "sender-dependent"
+)
+
 type StepInfo struct {
 	From, To        string
+	Resumed         bool
+	Encrypted       EncryptedEnum
 	BytesExpected   int64
 	BytesReplicated int64
 }
@@ -150,5 +160,5 @@ func (f *FilesystemReport) NextStep() *StepReport {
 }

 func (f *StepReport) IsIncremental() bool {
-	return f.Info.From != "" // FIXME change to ZFS semantics (To != "")
+	return f.Info.From != ""
 }