adminer/zrepl_patched
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: reflect changes in replication_rewrite branch

Browse Source
This commit is contained in:
Christian Schwarz
2018-10-11 17:46:26 +02:00
parent 125b561df3
commit 1643198713
16 changed files with 823 additions and 554 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/implementation.rst
+12 -13
View File
@@ -5,9 +5,12 @@ Implementation Overview
.. WARNING::
Incomplete / under construction
Incomplete and possibly outdated.
Check out the :ref:`talks about zrepl <pr-talks>` at various conferences for up-to-date material.
Alternatively, have a `look at the source code <http://github.com/zrepl/zrepl>`_ ;)
The following design aspects may convince you that ``zrepl`` is superior to a hacked-together shell script solution.
Also check out the :ref:`talks about zrepl <pr-talks>` at various conferences.
Testability & Performance
-------------------------
@@ -28,7 +31,7 @@ While it is tempting to just issue a few ``ssh remote 'zfs send ...' | zfs recv`
* The snapshot streams need to be compatible.
* Communication is still unidirectional. Thus, you will most likely
* either not take advantage of features such as *compressed send & recv*
* either not take advantage of advanced replication features such as *compressed send & recv*
* or issue additional ``ssh`` commands in advance to figure out what features are supported on the other side.
* Advanced logic in shell scripts is ugly to read, poorly testable and a pain to maintain.
@@ -36,25 +39,21 @@ While it is tempting to just issue a few ``ssh remote 'zfs send ...' | zfs recv`
zrepl takes a different approach:
* Define an RPC protocol.
* Establish an encrypted, authenticated, bidirectional communication channel...
* ... with zrepl running at both ends of it.
* Establish an encrypted, authenticated, bidirectional communication channel.
* Run daemons on both sides of the setup and let them talk to each other.
This has several obvious benefits:
* No blank root shell access is given to the other side.
* Instead, an *authenticated* peer can *request* filesystem lists, snapshot streams, etc.
* Requests are then checked against job-specific ACLs, limiting a client to the filesystems it is actually allowed to replicate.
* The :ref:`transport mechanism <transport>` is decoupled from the remaining logic, keeping it extensible.
* An *authenticated* peer *requests* filesystem lists, snapshot streams, etc.
* The server decides which filesystems it exposes to which peers.
* The :ref:`transport mechanism <transport>` is decoupled from the remaining logic, which allows us to painlessly offer multiple transport mechanisms.
Protocol Implementation
~~~~~~~~~~~~~~~~~~~~~~~
zrepl implements its own RPC protocol.
This is mostly due to the fact that existing solutions do not provide efficient means to transport large amounts of data.
Package `github.com/zrepl/zrepl/rpc <https://github.com/zrepl/zrepl/tree/master/rpc>`_ builds a special-case handling around returning an ``io.Reader`` as part of a unary RPC call.
Measurements show only a single memory-to-memory copy of a snapshot stream is made using ``github.com/zrepl/zrepl/rpc``, and there is still potential for further optimizations.
zrepl uses a custom RPC protol because, at the time of writing, existing solutions like gRPC do not provide efficient means to transport large amounts of data, whose size is unknown at send time (= zfs send streams).
The package used is `github.com/problame/go-streamrpc <https://github.com/problame/go-streamrpc/tree/master>`_.
Logging & Transparency
----------------------