zfs: rewrite SendStream, fix bug in Close() on FreeBSD, add platformtests

This commit was motivated by https://github.com/zrepl/zrepl/issues/495 where, on FreeBSD with OpenZFS 2.0, a SendStream.Close() call might wait indefinitely for `zfs send` to exit. The reason is that, due to the refactoring done for redacted send & recv (https://github.com/openzfs/zfs/commit/30af21b02569ac192f52ce6e6511015f8a8d5729), the `dump_bytes` function, which writes to the pipe, executes in a separate thread (synctask taskq) iff not `HAVE_LARGE_STACKS`. The `zfs send` process/thread waits for that taskq thread using an uninterruptible primitive. So when we SIGKILL `zfs send`, that signal doesn't reach the right thread to interrupt the pipe write. Theoretically this affects both Linux and FreeBSD, but most Linux users `HAVE_LARGE_STACKS` and since https://github.com/penzfs/zfs/pull/12350/files OpenZFS on FreeBSD `HAVE_LARGE_STACKS` as well. However, at least until FreeBSD 13.1, possibly for the entire 13 lifecycle, we're going to have to live with that oddity. Measures taken in this commit: - Report the behavior as an upstream bug https://github.com/openzfs/zfs/issues/12500 - Change SendStream code so that it closes zrepl's read-end of the pipe (see comment in code) - Clean up and make explicit SendStream's state handling - Write extensive platformtests for SendStream - They pass on my Linux install and on FreeBSD 12 - FreeBSD 13 still needs testing. fixes https://github.com/zrepl/zrepl/issues/495
platformtest: fix 'active child tasks' panic for ReceiveForceRollbackWorksUnencrypted
2021-08-23 00:21:34 +02:00 · 2021-08-23 00:21:34 +02:00 · 2021-08-23 00:21:34 +02:00 · 2021-08-23 00:21:34 +02:00
437 changed files with 6227 additions and 10711 deletions
@@ -1,8 +1,5 @@
 version: 2.1
-orbs:
-  # NB: this is not the Go version, but the Orb version
-  # https://circleci.com/developer/orbs/orb/circleci/go#usage-go-modules-cache
-  go: circleci/go@1.11.0
+
 commands:
  setup-home-local-bin:
    steps:
@@ -15,64 +12,132 @@ commands:
              echo "$line" >> $BASH_ENV
            fi

+  invoke-lazy-sh:
+    parameters:
+      subcommand:
+        type: string
+    steps:
+      - run:
+          environment:
+            TERM: xterm
+          command: ./lazy.sh <<parameters.subcommand>>
+
  apt-update-and-install-common-deps:
    steps:
-      - run: sudo apt-get update
-      - run: sudo apt-get install -y gawk make
-      # CircleCI doesn't update its cimg/go images.
-      # So, need to update manually to get up-to-date trust chains.
-      # The need for this was required for cimg/go:1.12, but let's future proof this here and now.
-      - run: sudo apt-get install -y git ca-certificates
+      - run: sudo apt update && sudo apt install gawk make

+  restore-cache-gomod:
+    steps:
+      - restore_cache:
+          key: go-mod-v4-{{ checksum "go.sum" }}
+  save-cache-gomod:
+    steps:
+      - save_cache:
+          key: go-mod-v4-{{ checksum "go.sum" }}
+          paths:
+            - "/go/pkg/mod"
+
+  install-godep:
+    steps:
+      - apt-update-and-install-common-deps
+      - invoke-lazy-sh:
+          subcommand: godep

  install-docdep:
    steps:
      - apt-update-and-install-common-deps
      - run: sudo apt install python3 python3-pip libgirepository1.0-dev
-      - run: pip3 install -r docs/requirements.txt
+      - invoke-lazy-sh:
+          subcommand: docdep

-  docs-publish-sh:
-    parameters:
-      push:
-        type: boolean
+  download-and-install-minio-client:
    steps:
-      - checkout
+      - setup-home-local-bin
+      - restore_cache:
+          key: minio-client-v2
      - run:
+          shell: /bin/bash -eo pipefail
          command: |
-            git config --global user.email "zreplbot@cschwarz.com"
-            git config --global user.name "zrepl-github-io-ci"
+            if which mc; then exit 0; fi
+            sudo curl -sSL https://dl.min.io/client/mc/release/linux-amd64/archive/mc.RELEASE.2020-08-20T00-23-01Z \
+              -o "$HOME/.local/bin/mc"
+            sudo chmod +x "$HOME/.local/bin/mc"
+      - save_cache:
+          key: minio-client-v2
+          paths:
+            - "$HOME/.local/bin/mc"

-      # if we're pushing, we need to add the deploy key
-      # which is stored as "Additional SSH Keys" in the CircleCI project settings.
-      # We can't use the CircleCI-manage deploy key because we're pushing
-      # to a different repo than the one we're building.
-      - when:
-          condition: << parameters.push >>
-          steps:
-            # https://circleci.com/docs/2.0/add-ssh-key/#adding-multiple-keys-with-blank-hostnames
-            - run: ssh-add -D
-            # the default circleci ssh config only additional ssh keys for Host !github.com
-            - run:
-                command: |
-                  cat > ~/.ssh/config \<<EOF
-                  Host *
-                    IdentityFile /home/circleci/.ssh/id_rsa_458e62c517f6c480e40452126ce47421
-                  EOF
-            - add_ssh_keys:
-                fingerprints:
-                  # deploy key for zrepl.github.io
-                  - "45:8e:62:c5:17:f6:c4:80:e4:04:52:12:6c:e4:74:21"
+  upload-minio:
+    parameters:
+      src:
+        type: string
+      dst:
+        type: string
+    steps:
+      - run:
+          shell: /bin/bash -eo pipefail
+          when: always
+          command: |
+            if [ -n "$CIRCLE_PR_NUMBER" ]; then # CIRCLE_PR_NUMBER is guaranteed to be only present in forked PRs (external)
+              echo "Forked PR detected. Sry, can't trust you with credentials to external artifact store, use CircleCI's instead."
+              exit 0
+            fi
+            set -u # from now on

-      # caller must install-docdep
-      - when:
-          condition: << parameters.push >>
-          steps:
-            - run: bash -x docs/publish.sh -c -a -P
-      - when:
-          condition:
-            not: << parameters.push >>
-          steps:
-            - run: bash -x docs/publish.sh -c -a
+            mc config host add --api s3v4 zrepl-minio https://minio.cschwarz.com ${MINIO_ACCESS_KEY} ${MINIO_SECRET_KEY}
+
+            # keep in sync with set-github-minio-status
+            jobprefix=zrepl-ci-artifacts/${CIRCLE_SHA1}-pipeline-<<pipeline.number>>/${CIRCLE_JOB}
+
+            # Upload artifacts
+            mkdir -p ./artifacts
+            mc cp -r <<parameters.src>> "zrepl-minio/$jobprefix/<<parameters.dst>>"
+
+  set-github-minio-status:
+    parameters:
+      context:
+        type: string
+      description:
+        type: string
+      minio-dst:
+        type: string
+    steps:
+      - run:
+          shell: /bin/bash -eo pipefail
+          command: |
+            if [ -n "$CIRCLE_PR_NUMBER" ]; then # CIRCLE_PR_NUMBER is guaranteed to be only present in forked PRs (external)
+              echo "Forked PR detected. Sry, can't trust you with credentials to external artifact store, use CircleCI's instead."
+              exit 0
+            fi
+            set -u # from now on
+
+            # keep in sync with with upload-minio command
+            jobprefix=zrepl-ci-artifacts/${CIRCLE_SHA1}-pipeline-<<pipeline.number>>/${CIRCLE_JOB}
+            # Push Artifact Link to GitHub
+            REPO="zrepl/zrepl"
+            COMMIT="${CIRCLE_SHA1}"
+            JOB_NAME="${CIRCLE_JOB}"
+            CONTEXT="<<parameters.context>>"
+            DESCRIPTION="<<parameters.description>>"
+            TARGETURL=https://minio.cschwarz.com/minio/"$jobprefix"/"<<parameters.minio-dst>>"
+            curl "https://api.github.com/repos/$REPO/statuses/$COMMIT" \
+              -H "Content-Type: application/json" \
+              -H "Authorization: token $GITHUB_COMMIT_STATUS_TOKEN" \
+              -X POST \
+              -d '{"context":"'"$CONTEXT"'", "state": "success", "description":"'"$DESCRIPTION"'", "target_url":"'"$TARGETURL"'"}'
+
+
+  trigger-pipeline:
+    parameters:
+      body_no_shell_subst:
+        type: string
+    steps:
+      - run: |
+          curl -X POST https://circleci.com/api/v2/project/github/zrepl/zrepl/pipeline \
+            -H 'Content-Type: application/json' \
+            -H 'Accept: application/json' \
+            -H "Circle-Token: $ZREPL_BOT_CIRCLE_TOKEN" \
+            --data '<<parameters.body_no_shell_subst>>'

 parameters:
  do_ci:
@@ -83,6 +148,10 @@ parameters:
    type: boolean
    default: false

+  release_docker_baseimage_tag:
+    type: string
+    default: "1.16"
+
 workflows:
  version: 2

@@ -90,20 +159,20 @@ workflows:
    when: << pipeline.parameters.do_ci >>
    jobs:
      - quickcheck-docs
-      - quickcheck-go:
-          name: quickcheck-go-amd64-linux-1.23.1
-          goversion: &latest-go-release "1.23.1"
+      - quickcheck-go: &quickcheck-go-smoketest
+          name: quickcheck-go-amd64-linux-1.16
+          goversion: &latest-go-release "1.16"
          goos: linux
          goarch: amd64
-      - test-go:
+      - test-go-on-latest-go-release:
          goversion: *latest-go-release
      - quickcheck-go:
          requires:
-            - quickcheck-go-amd64-linux-1.23.1 #quickcheck-go-smoketest.name
-          matrix:
+            - quickcheck-go-amd64-linux-1.16 #quickcheck-go-smoketest.name
+          matrix: &quickcheck-go-matrix
            alias: quickcheck-go-matrix
            parameters:
-              goversion: [*latest-go-release, "1.22.7"]
+              goversion: [*latest-go-release, "1.12"]
              goos: ["linux", "freebsd"]
              goarch: ["amd64", "arm64"]
            exclude:
@@ -111,15 +180,10 @@ workflows:
              - goversion: *latest-go-release
                goos: linux
                goarch: amd64
-      - platformtest:
-          matrix:
-            parameters:
-              goversion: [*latest-go-release]
-              goos: ["linux"]
-              goarch: ["amd64"]
-          requires:
-            - test-go
-            - quickcheck-go-<< matrix.goarch >>-<< matrix.goos >>-<< matrix.goversion >>
+              # not supported by Go 1.12
+              - goversion: "1.12"
+                goos: freebsd
+                goarch: arm64

  release:
    when: << pipeline.parameters.do_release >>
@@ -137,7 +201,20 @@ workflows:
            - release-deb
            - release-rpm

-  publish-zrepl.github.io:
+  periodic:
+    triggers:
+      - schedule:
+          cron: "00 17 * * *"
+          filters:
+            branches:
+              only:
+                - master
+                - stable
+                - problame/circleci-build
+    jobs:
+      - periodic-full-pipeline-run
+
+  zrepl.github.io:
    jobs:
      - publish-zrepl-github-io:
          filters:
@@ -148,15 +225,22 @@ workflows:
 jobs:
  quickcheck-docs:
    docker:
-      - image: cimg/base:2024.09
+      - image: cimg/base:2020.08
    steps:
      - checkout
      - install-docdep
-      # do the current docs build
      - run: make docs
-      # does the publish.sh script still work?
-      - docs-publish-sh:
-          push: false
+
+      - store_artifacts:
+          path: artifacts
+      - download-and-install-minio-client
+      - upload-minio:
+          src: artifacts
+          dst: ""
+      - set-github-minio-status:
+          context: artifacts/${CIRCLE_JOB}
+          description: artifacts of CI job ${CIRCLE_JOB}
+          minio-dst: ""

  quickcheck-go:
    parameters:
@@ -167,98 +251,60 @@ jobs:
      goarch:
        type: string
    docker:
-      # any toolchain >= 1.22 will auto-download GOTOOLCHAIN
-      - image: &cimg_with_modern_go cimg/go:1.22
+      - image: circleci/golang:<<parameters.goversion>>
    environment:
      GOOS: <<parameters.goos>>
      GOARCH: <<parameters.goarch>>
-      GOTOOLCHAIN: "go<<parameters.goversion>>"

    steps:
      - checkout

-      - go/load-cache:
-          key: quickcheck-<<parameters.goversion>>
-      - run: make build/install
+      - restore-cache-gomod
      - run: go mod download
      - run: cd build && go mod download
-      - go/save-cache:
-          key: quickcheck-<<parameters.goversion>>
+      - save-cache-gomod

-      # ensure all code has been generated
-      - run: make generate
-      - run: |
-          if output=$(git status --porcelain) && [ -z "$output" ]; then
-            echo "Working directory clean"
-          else
-            echo "Uncommitted changes"
-            echo ""
-            echo "$output"
-            exit 1
-          fi
-
-      # other checks
+      - install-godep
+      - run: make formatcheck
+      - run: make generate-platform-test-list
      - run: make zrepl-bin test-platform-bin
      - run: make vet
      - run: make lint

+      - download-and-install-minio-client
+      - run: rm -f artifacts/generate-platform-test-list
      - store_artifacts:
          path: artifacts
-      - persist_to_workspace:
-          root: .
-          paths: [.]
+      - upload-minio:
+          src: artifacts
+          dst: ""
+      - set-github-minio-status:
+          context: artifacts/${CIRCLE_JOB}
+          description: artifacts of CI job ${CIRCLE_JOB}
+          minio-dst: ""

-  platformtest:
-    parameters:
-      goversion:
-        type: string
-      goos:
-        type: string
-      goarch:
-        type: string
-    machine:
-      image: ubuntu-2204:current
-    resource_class: medium
-    environment:
-      GOOS: <<parameters.goos>>
-      GOARCH: <<parameters.goarch>>
-    steps:
-      - attach_workspace:
-          at: .
-      - run: sudo apt-get update
-      - run: sudo apt-get install -y zfsutils-linux
-      - run: sudo zfs version
-      - run: sudo make test-platform GOOS="$GOOS" GOARCH="$GOARCH"
-
-  test-go:
+  test-go-on-latest-go-release:
    parameters:
      goversion:
        type: string
    docker:
-      - image: *cimg_with_modern_go
-    environment:
-      GOTOOLCHAIN: "go<<parameters.goversion>>"
+      - image: circleci/golang:<<parameters.goversion>>
    steps:
      - checkout
-      - go/load-cache:
-          key: make-test-go
+      - restore-cache-gomod
      - run: make test-go
-      - go/save-cache:
-          key: make-test-go
+      # don't save-cache-gomod here, test-go doesn't pull all the dependencies

  release-build:
-    machine:
-      image: &release-vm-image "ubuntu-2404:current"
-    resource_class: large
+    machine: true
    steps:
      - checkout
-      - run: make release-docker
+      - run: make release-docker RELEASE_DOCKER_BASEIMAGE_TAG=<<pipeline.parameters.release_docker_baseimage_tag>>
      - persist_to_workspace:
          root: .
          paths: [.]
  release-deb:
-    machine:
-      image: *release-vm-image
+    machine: true
    steps:
      - attach_workspace:
          at: .
@@ -269,8 +315,7 @@ jobs:
            - "artifacts/*.deb"

  release-rpm:
-    machine:
-      image: *release-vm-image
+    machine: true
    steps:
      - attach_workspace:
          at: .
@@ -282,19 +327,52 @@ jobs:

  release-upload:
    docker:
-      - image: cimg/base:2024.09
+      - image: cimg/base:2020.08
    steps:
      - attach_workspace:
          at: .
-      - run: make wrapup-and-checksum
      - store_artifacts:
-          path: artifacts/release
+          path: artifacts
+      - download-and-install-minio-client
+      - upload-minio:
+          src: artifacts
+          dst: ""
+      - set-github-minio-status:
+          context: artifacts/release
+          description: CI-generated release artifacts
+          minio-dst: ""
+
+  periodic-full-pipeline-run:
+    docker:
+      - image: cimg/base:2020.08
+    steps:
+      - trigger-pipeline:
+          body_no_shell_subst: '{"branch":"<<pipeline.git.branch>>", "parameters": { "do_ci": true, "do_release": true }}'

  publish-zrepl-github-io:
    docker:
-      - image: cimg/base:2024.09
+      - image: cimg/python:3.7
    steps:
      - checkout
-      - install-docdep
-      - docs-publish-sh:
-          push: true
+      - invoke-lazy-sh:
+          subcommand: docdep
+      - run:
+          command: |
+            git config --global user.email "zreplbot@cschwarz.com"
+            git config --global user.name "zrepl-github-io-ci"
+
+      # https://circleci.com/docs/2.0/add-ssh-key/#adding-multiple-keys-with-blank-hostnames
+      - run: ssh-add -D
+      # the default circleci ssh config only additional ssh keys for Host !github.com
+      - run:
+          command: |
+            cat > ~/.ssh/config \<<EOF
+            Host *
+              IdentityFile /home/circleci/.ssh/id_rsa_458e62c517f6c480e40452126ce47421
+            EOF
+      - add_ssh_keys:
+          fingerprints:
+            # deploy key for zrepl.github.io
+            - "45:8e:62:c5:17:f6:c4:80:e4:04:52:12:6c:e4:74:21"
+
+      - run: bash -x docs/publish.sh -c -a
@@ -1,84 +0,0 @@
-import argparse
-from pathlib import Path
-import re
-
-import requests
-
-import time
-
-import os
-import argparse
-from pathlib import Path
-
-
-circle_token = os.environ.get('CIRCLE_TOKEN')
-if not circle_token:
-    raise ValueError('CIRCLE_TOKEN environment variable not set')
-
-parser = argparse.ArgumentParser(description='Download artifacts from CircleCI')
-parser.add_argument('build_num', type=str, help='Build number')
-parser.add_argument('dst', type=Path, help='Destination directory')
-parser.add_argument('--prefix', type=str, default='', help='Filter for prefix')
-parser.add_argument('--match', type=str, default='.*', help='Only include paths matching the given regex')
-
-args = parser.parse_args()
-
-res = requests.get(
-    f"https://circleci.com/api/v1.1/project/github/zrepl/zrepl/{args.build_num}/artifacts",
-    headers={
-        "Circle-Token": circle_token,
-    },
-)
-res.raise_for_status()
-
-# https://circleci.com/docs/api/v1/index.html#artifacts-of-a-job
-# [ {
-#   "path" : "raw-test-output/go-test-report.xml",
-#   "pretty_path" : "raw-test-output/go-test-report.xml",
-#   "node_index" : 0,
-#   "url" : "https://24-88881093-gh.circle-artifacts.com/0/raw-test-output/go-test-report.xml"
-# }, {
-#   "path" : "raw-test-output/go-test.out",
-#   "pretty_path" : "raw-test-output/go-test.out",
-#   "node_index" : 0,
-#   "url" : "https://24-88881093-gh.circle-artifacts.com/0/raw-test-output/go-test.out"
-# } ]
-res = res.json()
-
-for artifact in res:
-    if not artifact["pretty_path"].startswith(args.prefix):
-        continue
-    if not re.match(args.match, artifact["pretty_path"]):
-        continue
-    stripped = artifact["pretty_path"][len(args.prefix):]
-    print(f"Downloading {artifact['pretty_path']} to {args.dst / stripped}")
-    artifact_rel = Path(stripped)
-    artifact_dst = args.dst / artifact_rel
-    artifact_dst.parent.mkdir(parents=True, exist_ok=True)
-
-    res = requests.get(
-        artifact["url"],
-        headers={
-            "Circle-Token": circle_token,
-        },
-        stream=True,
-    )
-    res.raise_for_status()
-
-    total_size = int(res.headers.get("Content-Length", 0))
-    block_size = 128 * 1024
-    with open(artifact_dst, "wb") as f:
-        progress = 0
-        start_time = time.time()
-        for chunk in res.iter_content(chunk_size=block_size):
-            f.write(chunk)
-            progress += len(chunk)
-            percent = progress / total_size * 100
-            elapsed_time = time.time() - start_time
-            if elapsed_time >= 5:
-                print(f"Downloaded {progress}/{total_size} bytes ({percent:.2f}%)", end="\r")
-                start_time = time.time()
-        print(f"Downloaded {progress}/{total_size} bytes ({percent:.2f}%)")
-    print("Download complete!")
-
-print("All files downloaded")
@@ -0,0 +1,10 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+COMMIT="$1"
+GO_VERSION="$2"
+
+curl -v -X POST https://api.github.com/repos/zrepl/debian-binary-packaging/dispatches \
+    -H 'Accept: application/vnd.github.v3+json' \
+    -H "Authorization: token $GITHUB_ACCESS_TOKEN" \
+    --data '{"event_type": "push", "client_payload": { "zrepl_main_repo_commit": "'"$COMMIT"'", "go_version": "'"$GO_VERSION"'" }}'
@@ -1,9 +0,0 @@
-version: 2
-updates:
-  # Docs use Python only for static site generation.
-  - package-ecosystem: "pip"
-    directory: /docs
-    schedule:
-      interval: "weekly"
-    ignore:
-      - dependency-name: "*"
@@ -32,4 +32,3 @@ _testmain.go
 *.exe
 *.test
 *.prof
-.venv/
@@ -1,24 +1,10 @@
 linters:
  enable:
  - goimports
-  - revive
-
-linters-settings:
-  goimports:
-    local-prefixes: github.com/zrepl/zrepl
-  revive:
-    rules:
-      - name: time-equal

 issues:
  exclude-rules:
    - path: _test\.go
      linters:
        - errcheck
-    # Disable staticcheck 'Empty body in an if or else branch' as it's useful
-    # to put a comment into an empty else-clause that explains why whatever
-    # is done in the if-caluse is not necessary if the condition is false.
-    - linters:
-        - staticcheck
-      text: "SA9003:"

@@ -1,5 +1,5 @@
 .PHONY: generate build test vet cover release docs docs-clean clean format lint platformtest
-.PHONY: release release-noarch
+.PHONY: release bins-all release-noarch
 .DEFAULT_GOAL := zrepl-bin

 ARTIFACTDIR := artifacts
@@ -14,25 +14,22 @@ ifndef _ZREPL_VERSION
    endif
 endif

-ZREPL_PACKAGE_RELEASE := 1
-
 GO := go
 GOOS ?= $(shell bash -c 'source <($(GO) env) && echo "$$GOOS"')
 GOARCH ?= $(shell bash -c 'source <($(GO) env) && echo "$$GOARCH"')
 GOARM ?= $(shell bash -c 'source <($(GO) env) && echo "$$GOARM"')
 GOHOSTOS ?= $(shell bash -c 'source <($(GO) env) && echo "$$GOHOSTOS"')
 GOHOSTARCH ?= $(shell bash -c 'source <($(GO) env) && echo "$$GOHOSTARCH"')
-GO_ENV_VARS := CGO_ENABLED=0
+GO_ENV_VARS := GO111MODULE=on
 GO_LDFLAGS := "-X github.com/zrepl/zrepl/version.zreplVersion=$(_ZREPL_VERSION)"
 GO_MOD_READONLY := -mod=readonly
 GO_EXTRA_BUILDFLAGS :=
 GO_BUILDFLAGS := $(GO_MOD_READONLY) $(GO_EXTRA_BUILDFLAGS)
 GO_BUILD := $(GO_ENV_VARS) $(GO) build $(GO_BUILDFLAGS) -ldflags $(GO_LDFLAGS)
-GOCOVMERGE := $(GO) tool gocovmerge
-RELEASE_GOVERSION ?= go1.24.0
-STRIPPED_GOVERSION := $(subst go,,$(RELEASE_GOVERSION))
-RELEASE_DOCKER_BASEIMAGE ?= golang:$(STRIPPED_GOVERSION)
-RELEASE_DOCKER_CACHEMOUNT :=
+GOLANGCI_LINT := golangci-lint
+GOCOVMERGE := gocovmerge
+RELEASE_DOCKER_BASEIMAGE_TAG ?= 1.16
+RELEASE_DOCKER_BASEIMAGE ?= golang:$(RELEASE_DOCKER_BASEIMAGE_TAG)

 ifneq ($(GOARM),)
 	ZREPL_TARGET_TUPLE := $(GOOS)-$(GOARCH)v$(GOARM)
@@ -40,46 +37,34 @@ else
 	ZREPL_TARGET_TUPLE := $(GOOS)-$(GOARCH)
 endif

+.PHONY: printvars
+printvars:
+	@echo GOOS=$(GOOS)
+	@echo GOARCH=$(GOARCH)
+	@echo GOARM=$(GOARM)

-ifneq ($(RELEASE_DOCKER_CACHEMOUNT),)
-	_RELEASE_DOCKER_CACHEMOUNT := -v $(RELEASE_DOCKER_CACHEMOUNT)/mod:/go/pkg/mod -v $(RELEASE_DOCKER_CACHEMOUNT)/xdg-cache:/root/.cache/go-build
-.PHONY: release-docker-mkcachemount
-release-docker-mkcachemount:
-	mkdir -p $(RELEASE_DOCKER_CACHEMOUNT)
-	mkdir -p $(RELEASE_DOCKER_CACHEMOUNT)/mod
-	mkdir -p $(RELEASE_DOCKER_CACHEMOUNT)/xdg-cache
-else
-	_RELEASE_DOCKER_CACHEMOUNT :=
-.PHONY: release-docker-mkcachemount
-release-docker-mkcachemount:
-	# nothing to do
-endif

 ##################### PRODUCING A RELEASE #############
-.PHONY: release wrapup-and-checksum check-git-clean sign clean ensure-release-toolchain
+.PHONY: release wrapup-and-checksum check-git-clean sign clean

-ensure-release-toolchain:
-	# ensure the toolchain is actually the one we expect
-	test $(RELEASE_GOVERSION) = "$$($(GO_ENV_VARS) $(GO) env GOVERSION)"
-
-release: ensure-release-toolchain
-	$(MAKE) _run_make_foreach_target_tuple RUN_MAKE_FOREACH_TARGET_TUPLE_ARG="vet"
-	$(MAKE) _run_make_foreach_target_tuple RUN_MAKE_FOREACH_TARGET_TUPLE_ARG="lint"
-	$(MAKE) _run_make_foreach_target_tuple RUN_MAKE_FOREACH_TARGET_TUPLE_ARG="zrepl-bin"
-	$(MAKE) _run_make_foreach_target_tuple RUN_MAKE_FOREACH_TARGET_TUPLE_ARG="test-platform-bin"
+release: clean
+	# no cross-platform support for target test
+	$(MAKE) test-go
+	$(MAKE) bins-all
 	$(MAKE) noarch
+	$(MAKE) wrapup-and-checksum
+	$(MAKE) check-git-clean
+ifeq (SIGN, 1)
+	$(make) sign
+endif
+	@echo "ZREPL RELEASE ARTIFACTS AVAILABLE IN artifacts/release"

-release-docker: $(ARTIFACTDIR) release-docker-mkcachemount
-	sed 's/FROM.*!SUBSTITUTED_BY_MAKEFILE/FROM $(RELEASE_DOCKER_BASEIMAGE)/' build/build.Dockerfile > $(ARTIFACTDIR)/build.Dockerfile
-	docker build -t zrepl_release --pull -f $(ARTIFACTDIR)/build.Dockerfile .
-	docker run --rm -i \
-		$(_RELEASE_DOCKER_CACHEMOUNT) \
-		-v $(CURDIR):/src -u $$(id -u):$$(id -g) \
+release-docker: $(ARTIFACTDIR)
+	sed 's/FROM.*!SUBSTITUTED_BY_MAKEFILE/FROM $(RELEASE_DOCKER_BASEIMAGE)/' build.Dockerfile > artifacts/release-docker.Dockerfile
+	docker build -t zrepl_release --pull -f artifacts/release-docker.Dockerfile .
+	docker run --rm -i -v $(CURDIR):/src -u $$(id -u):$$(id -g) \
 		zrepl_release \
-		make release \
-			GOOS=$(GOOS) GOARCH=$(GOARCH) GOARM=$(GOARM) \
-			ZREPL_VERSION=$(ZREPL_VERSION) ZREPL_PACKAGE_RELEASE=$(ZREPL_PACKAGE_RELEASE) \
-			RELEASE_GOVERSION=$(RELEASE_GOVERSION)
+		make release GOOS=$(GOOS) GOARCH=$(GOARCH) GOARM=$(GOARM)

 debs-docker:
 	$(MAKE) _debs_or_rpms_docker _DEB_OR_RPM=deb
@@ -96,14 +81,9 @@ rpm: $(ARTIFACTDIR) # artifacts/_zrepl.zsh_completion artifacts/bash_completion
 	$(eval _ZREPL_RPM_TOPDIR_ABS := $(CURDIR)/$(ARTIFACTDIR)/rpmbuild)
 	rm -rf "$(_ZREPL_RPM_TOPDIR_ABS)"
 	mkdir "$(_ZREPL_RPM_TOPDIR_ABS)"
-	for d in BUILD BUILDROOT RPMS SOURCES SPECS SRPMS; do \
-		mkdir -p "$(_ZREPL_RPM_TOPDIR_ABS)/$$d"; \
-	done
-	sed \
-		-e "s/^Version:.*/Version:          $(_ZREPL_RPM_VERSION)/g" \
-		-e "s/^Release:.*/Release:          $(ZREPL_PACKAGE_RELEASE)/g" \
-	    packaging/rpm/zrepl.spec \
-		>  $(_ZREPL_RPM_TOPDIR_ABS)/SPECS/zrepl.spec
+	mkdir -p "$(_ZREPL_RPM_TOPDIR_ABS)"/{SPECS,RPMS,BUILD,BUILDROOT}
+	sed "s/^Version:.*/Version:          $(_ZREPL_RPM_VERSION)/g" \
+	    packaging/rpm/zrepl.spec >  $(_ZREPL_RPM_TOPDIR_ABS)/SPECS/zrepl.spec

 	# see /usr/lib/rpm/platform
 ifeq ($(GOARCH),amd64)
@@ -130,16 +110,13 @@ rpm-docker:
 	docker build -t zrepl_rpm_pkg --pull -f packaging/rpm/Dockerfile .
 	docker run --rm -i -v $(CURDIR):/build/src -u $$(id -u):$$(id -g) \
 		zrepl_rpm_pkg \
-		make rpm \
-			GOOS=$(GOOS) GOARCH=$(GOARCH) GOARM=$(GOARM) \
-			ZREPL_VERSION=$(ZREPL_VERSION) ZREPL_PACKAGE_RELEASE=$(ZREPL_PACKAGE_RELEASE)
-
+		make rpm GOOS=$(GOOS) GOARCH=$(GOARCH) GOARM=$(GOARM)

 deb: $(ARTIFACTDIR) # artifacts/_zrepl.zsh_completion artifacts/bash_completion docs zrepl-bin

 	cp packaging/deb/debian/changelog.template packaging/deb/debian/changelog
 	sed -i 's/DATE_DASH_R_OUTPUT/$(shell date -R)/' packaging/deb/debian/changelog
-	VERSION="$(subst -,.,$(_ZREPL_VERSION))-$(ZREPL_PACKAGE_RELEASE)"; \
+	VERSION="$(subst -,.,$(_ZREPL_VERSION))"; \
 		export VERSION="$${VERSION#v}"; \
 		sed -i 's/VERSION/'"$$VERSION"'/' packaging/deb/debian/changelog

@@ -157,36 +134,28 @@ endif

 deb-docker:
 	docker build -t zrepl_debian_pkg --pull -f packaging/deb/Dockerfile .
-	# Use a small open file limit to make fakeroot work. If we don't
-	# specify it, docker daemon will use its file limit. I don't know
-	# what changed (Docker, its systemd service, its Go version). But I
-	# observed fakeroot iterating close(i) up to i > 1000000, which costs
-	# a good amount of CPU time and makes the build slow.
 	docker run --rm -i -v $(CURDIR):/build/src -u $$(id -u):$$(id -g) \
-		--ulimit nofile=1024:1024 \
 		zrepl_debian_pkg \
-		make deb \
-			GOOS=$(GOOS) GOARCH=$(GOARCH) GOARM=$(GOARM) \
-			ZREPL_VERSION=$(ZREPL_VERSION) ZREPL_PACKAGE_RELEASE=$(ZREPL_PACKAGE_RELEASE)
+		make deb GOOS=$(GOOS) GOARCH=$(GOARCH) GOARM=$(GOARM)

-# expects `release`, `deb` & `rpm` targets to have run before
+# expects `release` target to have run before
 NOARCH_TARBALL := $(ARTIFACTDIR)/zrepl-noarch.tar
 wrapup-and-checksum:
 	rm -f $(NOARCH_TARBALL)
 	tar --mtime='1970-01-01' --sort=name \
 		--transform 's/$(ARTIFACTDIR)/zrepl-$(_ZREPL_VERSION)-noarch/' \
 		--transform 's#dist#zrepl-$(_ZREPL_VERSION)-noarch/dist#' \
-		--transform 's#internal/config/samples#zrepl-$(_ZREPL_VERSION)-noarch/config#' \
+		--transform 's#config/samples#zrepl-$(_ZREPL_VERSION)-noarch/config#' \
 		-acf $(NOARCH_TARBALL) \
 		$(ARTIFACTDIR)/docs/html \
 		$(ARTIFACTDIR)/bash_completion \
 		$(ARTIFACTDIR)/_zrepl.zsh_completion \
 		$(ARTIFACTDIR)/go_env.txt \
 		dist \
-		internal/config/samples
+		config/samples
 	rm -rf "$(ARTIFACTDIR)/release"
 	mkdir -p "$(ARTIFACTDIR)/release"
-	cp -l $(ARTIFACTDIR)/zrepl* \
+	cp -l $(ARTIFACTDIR)/zrepl-* \
 		$(ARTIFACTDIR)/platformtest-* \
 		"$(ARTIFACTDIR)/release"
 	cd "$(ARTIFACTDIR)/release" && sha512sum $$(ls | sort) > sha512sum.txt
@@ -204,47 +173,38 @@ check-git-clean:
 		fi; \
 	fi;

-tag-release:
-	test -n "$(ZREPL_TAG_VERSION)" || exit 1
-	git tag -u '328A6627FA98061D!' -m "$(ZREPL_TAG_VERSION)" "$(ZREPL_TAG_VERSION)"
-
 sign:
-	gpg -u '328A6627FA98061D!' \
+	gpg -u "89BC 5D89 C845 568B F578  B306 CDBD 8EC8 E27C A5FC" \
 		--armor \
 		--detach-sign $(ARTIFACTDIR)/release/sha512sum.txt

 clean: docs-clean
 	rm -rf "$(ARTIFACTDIR)"

-download-circleci-release:
-	rm -rf "$(ARTIFACTDIR)"
-	mkdir -p "$(ARTIFACTDIR)/release"
-	python3 .circleci/download_artifacts.py --prefix 'artifacts/release/' "$(BUILD_NUM)" "$(ARTIFACTDIR)/release"
+##################### BINARIES #####################
+.PHONY: bins-all lint test-go test-platform cover-merge cover-html vet zrepl-bin test-platform-bin generate-platform-test-list

-##################### MULTI-ARCH HELPERS #####################
-
-_run_make_foreach_target_tuple:
-	if [ "$(RUN_MAKE_FOREACH_TARGET_TUPLE_ARG)" = "" ]; then \
-		echo "RUN_MAKE_FOREACH_TARGET_TUPLE_ARG must be set"; \
-		exit 1; \
-	fi
-	$(MAKE) $(RUN_MAKE_FOREACH_TARGET_TUPLE_ARG) GOOS=freebsd   GOARCH=amd64
-	$(MAKE) $(RUN_MAKE_FOREACH_TARGET_TUPLE_ARG) GOOS=freebsd   GOARCH=386
-	$(MAKE) $(RUN_MAKE_FOREACH_TARGET_TUPLE_ARG) GOOS=freebsd   GOARCH=arm	  GOARM=7
-	$(MAKE) $(RUN_MAKE_FOREACH_TARGET_TUPLE_ARG) GOOS=freebsd   GOARCH=arm64
-	$(MAKE) $(RUN_MAKE_FOREACH_TARGET_TUPLE_ARG) GOOS=linux     GOARCH=amd64
-	$(MAKE) $(RUN_MAKE_FOREACH_TARGET_TUPLE_ARG) GOOS=linux     GOARCH=arm64
-	$(MAKE) $(RUN_MAKE_FOREACH_TARGET_TUPLE_ARG) GOOS=linux     GOARCH=arm     GOARM=7
-	$(MAKE) $(RUN_MAKE_FOREACH_TARGET_TUPLE_ARG) GOOS=linux     GOARCH=386
-	$(MAKE) $(RUN_MAKE_FOREACH_TARGET_TUPLE_ARG) GOOS=darwin    GOARCH=amd64
-	$(MAKE) $(RUN_MAKE_FOREACH_TARGET_TUPLE_ARG) GOOS=solaris   GOARCH=amd64
-	$(MAKE) $(RUN_MAKE_FOREACH_TARGET_TUPLE_ARG) GOOS=illumos   GOARCH=amd64
-
-##################### REGULAR TARGETS #####################
-.PHONY: lint test-go test-platform cover-merge cover-html vet zrepl-bin test-platform-bin
+BINS_ALL_TARGETS := zrepl-bin test-platform-bin vet lint
+GO_SUPPORTS_ILLUMOS := $(shell $(GO) version | gawk -F '.' '/^go version /{split($$0, comps, " "); split(comps[3], v, "."); if (v[1] == "go1" && v[2] >= 13) { print "illumos"; } else { print "noillumos"; }}')
+bins-all:
+	$(MAKE) $(BINS_ALL_TARGETS) GOOS=freebsd   GOARCH=amd64
+	$(MAKE) $(BINS_ALL_TARGETS) GOOS=freebsd   GOARCH=386
+	$(MAKE) $(BINS_ALL_TARGETS) GOOS=linux     GOARCH=amd64
+	$(MAKE) $(BINS_ALL_TARGETS) GOOS=linux     GOARCH=arm64
+	$(MAKE) $(BINS_ALL_TARGETS) GOOS=linux     GOARCH=arm     GOARM=7
+	$(MAKE) $(BINS_ALL_TARGETS) GOOS=linux     GOARCH=386
+	$(MAKE) $(BINS_ALL_TARGETS) GOOS=darwin    GOARCH=amd64
+	$(MAKE) $(BINS_ALL_TARGETS) GOOS=solaris   GOARCH=amd64
+ifeq ($(GO_SUPPORTS_ILLUMOS), illumos)
+	$(MAKE) $(BINS_ALL_TARGETS) GOOS=illumos   GOARCH=amd64
+else ifeq ($(GO_SUPPORTS_ILLUMOS), noillumos)
+	@echo "SKIPPING ILLUMOS BUILD BECAUSE GO VERSION DOESN'T SUPPORT IT"
+else
+	@echo "CANNOT DETERMINE WHETHER GO VERSION SUPPORTS GOOS=illumos"; exit 1
+endif

 lint:
-	$(GO_ENV_VARS) $(GO) tool golangci-lint run ./...
+	$(GO_ENV_VARS) $(GOLANGCI_LINT) run ./...

 vet:
 	$(GO_ENV_VARS) $(GO) vet $(GO_BUILDFLAGS) ./...
@@ -265,6 +225,9 @@ endif
 zrepl-bin:
 	$(GO_BUILD) -o "$(ARTIFACTDIR)/zrepl-$(ZREPL_TARGET_TUPLE)"

+generate-platform-test-list:
+	$(GO_BUILD) -o $(ARTIFACTDIR)/generate-platform-test-list ./platformtest/tests/gen
+
 COVER_PLATFORM_BIN_PATH := $(ARTIFACTDIR)/platformtest-cover-$(ZREPL_TARGET_TUPLE)
 cover-platform-bin:
 	$(GO_ENV_VARS) $(GO) test $(GO_BUILDFLAGS) \
@@ -281,7 +244,7 @@ cover-platform:

 TEST_PLATFORM_BIN_PATH := $(ARTIFACTDIR)/platformtest-$(ZREPL_TARGET_TUPLE)
 test-platform-bin:
-	$(GO_BUILD) -o "$(TEST_PLATFORM_BIN_PATH)" ./internal/platformtest/harness
+	$(GO_BUILD) -o "$(TEST_PLATFORM_BIN_PATH)" ./platformtest/harness
 test-platform:
 	export _TEST_PLATFORM_CMD="\"$(TEST_PLATFORM_BIN_PATH)\""; \
 		$(MAKE) _test-or-cover-platform-impl
@@ -298,7 +261,7 @@ ifndef _TEST_PLATFORM_CMD
 endif
 	rm -f "$(ZREPL_PLATFORMTEST_ZFS_LOG)"
 	rm -f "$(ARTIFACTDIR)/platformtest.cover"
-	internal/platformtest/logmockzfs/logzfsenv "$(ZREPL_PLATFORMTEST_ZFS_LOG)" `which zfs` \
+	platformtest/logmockzfs/logzfsenv "$(ZREPL_PLATFORMTEST_ZFS_LOG)" `which zfs` \
 		$(_TEST_PLATFORM_CMD) \
 		-poolname "$(ZREPL_PLATFORMTEST_POOLNAME)" \
 		-imagepath "$(ZREPL_PLATFORMTEST_IMAGEPATH)" \
@@ -320,38 +283,23 @@ cover-full:

 ##################### DEV TARGETS #####################
 # not part of the build, must do that manually
-.PHONY: generate format
+.PHONY: generate formatcheck format

-build/install:
-	rm -rf build/install.tmp
-	mkdir build/install.tmp
+generate: generate-platform-test-list
+	protoc -I=replication/logic/pdu --go_out=replication/logic/pdu --go-grpc_out=replication/logic/pdu replication/logic/pdu/pdu.proto
+	protoc -I=rpc/grpcclientidentity/example --go_out=rpc/grpcclientidentity/example/pdu --go-grpc_out=rpc/grpcclientidentity/example/pdu rpc/grpcclientidentity/example/grpcauth.proto
+	$(GO_ENV_VARS) $(GO) generate $(GO_BUILDFLAGS) -x ./...

-	-echo "installing protoc"
-	mkdir build/install.tmp/protoc
-	bash -x build/get_protoc.bash build/install.tmp/protoc
+GOIMPORTS := goimports -srcdir . -local 'github.com/zrepl/zrepl'
+FINDSRCFILES := find . -type f -name '*.go' -not -path "./vendor/*" -not -name '*.pb.go' -not -name '*_enumer.go'

-	-echo "installing protoc plugins to GOBIN"
-	$(GO) install tool
-
-	mv build/install.tmp build/install
-
-
-generate: build/install
-	# protoc plugins are installed via 'go install tool' in build/install target
-	build/install/protoc/bin/protoc \
-		-I=internal/replication/logic/pdu \
-		--go_out=internal/replication/logic/pdu --go-grpc_out=internal/replication/logic/pdu \
-		internal/replication/logic/pdu/pdu.proto
-	build/install/protoc/bin/protoc \
-		-I=internal/rpc/grpcclientidentity/example \
-		--go_out=internal/rpc/grpcclientidentity/example/pdu --go-grpc_out=internal/rpc/grpcclientidentity/example/pdu \
-		internal/rpc/grpcclientidentity/example/grpcauth.proto
-	$(GO) generate $(GO_BUILDFLAGS) -x ./...
-
-GOIMPORTS := $(GO) tool goimports -srcdir . -local 'github.com/zrepl/zrepl'
+formatcheck:
+	@# goimports doesn't have a knob to exit with non-zero status code if formatting is needed
+	@# see https://go-review.googlesource.com/c/tools/+/237378
+	@ affectedfiles=$$($(GOIMPORTS) -l $(shell $(FINDSRCFILES)) | tee /dev/stderr | wc -l); test "$$affectedfiles" = 0

 format:
-	@ $(GO) tool goimports -w -local 'github.com/zrepl/zrepl' $$(find . -type f -name '*.go' -not -path "./vendor/*" -not -name '*.pb.go' -not -name '*_enumer.go')
+	@ $(GOIMPORTS) -w -d $(shell  $(FINDSRCFILES))

 ##################### NOARCH #####################
 .PHONY: noarch $(ARTIFACTDIR)/bash_completion $(ARTIFACTDIR)/_zrepl.zsh_completion $(ARTIFACTDIR)/go_env.txt docs docs-clean
@@ -379,12 +327,12 @@ $(ARTIFACTDIR)/go_env.txt:

 docs: $(ARTIFACTDIR)/docs
 	# https://www.sphinx-doc.org/en/master/man/sphinx-build.html
-	$(MAKE) -C docs \
+	make -C docs \
 		html \
 		BUILDDIR=../artifacts/docs \
 		SPHINXOPTS="-W --keep-going -n"

 docs-clean:
-	$(MAKE) -C docs \
+	make -C docs \
 		clean \
 		BUILDDIR=../artifacts/docs
@@ -1,12 +1,11 @@
 [![GitHub license](https://img.shields.io/github/license/zrepl/zrepl.svg)](https://github.com/zrepl/zrepl/blob/master/LICENSE)
-[![Language: Go](https://img.shields.io/badge/lang-Go-6ad7e5.svg)](https://golang.org/)
+[![Language: Go](https://img.shields.io/badge/language-Go-6ad7e5.svg)](https://golang.org/)
 [![User Docs](https://img.shields.io/badge/docs-web-blue.svg)](https://zrepl.github.io)
-[![Support me on Patreon](https://img.shields.io/badge/dynamic/json?color=yellow&label=Patreon&query=data.attributes.patron_count&url=https%3A%2F%2Fwww.patreon.com%2Fapi%2Fcampaigns%2F3095079)](https://patreon.com/zrepl)
+[![Support me on Patreon](https://img.shields.io/badge/dynamic/json?color=yellow&label=Patreon&query=data.attributes.patron_count&suffix=%20patrons&url=https%3A%2F%2Fwww.patreon.com%2Fapi%2Fcampaigns%2F3095079)](https://patreon.com/zrepl)
 [![Donate via GitHub Sponsors](https://img.shields.io/static/v1?label=Sponsor&message=%E2%9D%A4&logo=GitHub&style=flat&color=yellow)](https://github.com/sponsors/problame)
 [![Donate via Liberapay](https://img.shields.io/liberapay/patrons/zrepl.svg?logo=liberapay)](https://liberapay.com/zrepl/donate)
 [![Donate via PayPal](https://img.shields.io/badge/donate-paypal-yellow.svg)](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=R5QSXJVYHGX96)
 [![Twitter](https://img.shields.io/twitter/url/https/github.com/zrepl/zrepl.svg?style=social)](https://twitter.com/intent/tweet?text=Wow:&url=https%3A%2F%2Fgithub.com%2Fzrepl%2Fzrepl)
-[![Chat](https://img.shields.io/badge/chat-matrix-blue.svg)](https://matrix.to/#/#zrepl:matrix.org)

 # zrepl
 zrepl is a one-stop ZFS backup & replication solution.
@@ -26,40 +25,24 @@ zrepl is a one-stop ZFS backup & replication solution.
   If so, think of an expressive configuration example.
 2. Think of at least one use case that generalizes from your concrete application.
 3. Open an issue on GitHub with example conf & use case attached.
-4. **Optional**: [Contact Christian Schwarz](https://cschwarz.com) for contract work.
+4. **Optional**: [Post a bounty](https://www.bountysource.com/teams/zrepl) on the issue, or [contact Christian Schwarz](https://cschwarz.com) for contract work.

 The above does not apply if you already implemented everything.
 Check out the *Coding Workflow* section below for details.

-## Development
+## Building, Releasing, Downstream-Packaging
+
+This section provides an overview of the zrepl build & release process.
+Check out `docs/installation/compile-from-source.rst` for build-from-source instructions.
+
+### Overview

 zrepl is written in [Go](https://golang.org) and uses [Go modules](https://github.com/golang/go/wiki/Modules) to manage dependencies.
 The documentation is written in [ReStructured Text](http://docutils.sourceforge.net/rst.html) using the [Sphinx](https://www.sphinx-doc.org) framework.

-### Building
-
-#### Go Code
-
-Dependencies:
-
-* Go 1.22 or newer
-* GNU Make
-* Git
-* wget (`make generate`)
-* unzip (`make generate`)
-
-Some Go code is **generated**, and generated code is committed to the source tree.
-Therefore, building does not require having code generation tools set up.
-When making changes that require code to be (re-)generated, run `make generate`.
-I downloads and installs pinned versions of the code generation tools into `./build/install`.
-There is a CI check that ensures Git state is clean, i.e., code generation has been done by a PR and is deterministic.
-
-#### Docs
-
-Set up a Python environment that has `docs/requirements.txt` installed via `pip`.
-Use a  [venv](https://docs.python.org/3/library/venv.html) to avoid global state.
-
-### Testing
+Install **build dependencies** using  `./lazy.sh devsetup`.
+`lazy.sh` uses `python3-pip` to fetch the build dependencies for the docs - you might want to use a [venv](https://docs.python.org/3/library/venv.html).
+If you just want to install the Go dependencies, run `./lazy.sh godep`.

 The **test suite** is split into pure **Go tests** (`make test-go`) and **platform tests** that interact with ZFS and thus generally **require root privileges** (`sudo make test-platform`).
 Platform tests run on their own pool with the name `zreplplatformtest`, which is created using the file vdev in `/tmp`.
@@ -67,28 +50,33 @@ Platform tests run on their own pool with the name `zreplplatformtest`, which is
 For a full **code coverage** profile, run `make test-go COVER=1 && sudo make test-platform && make cover-merge`.
 An HTML report can be generated using `make cover-html`.

-### Circle CI
+**Code generation** is triggered by `make generate`. Generated code is committed to the source tree.

-We use CircleCI for automated build & test pre- and post-merge.
+### Build & Release Process

+**The `Makefile` is catering to the needs of developers & CI, not distro packagers**.
+It provides phony targets for
+* local development (building, running tests, etc)
+* building a release in Docker (used by the CI & release management)
+* building .deb and .rpm packages out of the release artifacts.
+
+**Build tooling & dependencies** are documented as code in `lazy.sh`.
+Go dependencies are then fetched by the go command and pip dependencies are pinned through a `requirements.txt`.
+
+**We use CircleCI for continuous integration**.
 There are two workflows:

 * `ci` runs for every commit / branch / tag pushed to GitHub.
  It is supposed to run very fast (<5min and provides quick feedback to developers).
  It runs formatting checks, lints and tests on the most important OSes / architectures.
+  Artifacts are published to minio.cschwarz.com (see GitHub Commit Status).

 * `release` runs
  * on manual triggers through the CircleCI API (in order to produce a release)
  * periodically on `master`
+  Artifacts are published to minio.cschwarz.com (see GitHub Commit Status).

-Artifacts are stored in CircleCI.
-
-### Releasing
-
-All zrepl releases are git-tagged and then published as a GitHub Release.
-There is a git tag for each zrepl release, usually `vMAJOR.MINOR.0`.
-We don't move git tags once the release has been published.
-
+**Releases** are issued via Git tags + GitHub Releases feature.
 The procedure to issue a release is as follows:
 * Issue the source release:
  * Git tag the release on the `master` branch.
@@ -99,35 +87,15 @@ The procedure to issue a release is as follows:
  * Download the artifacts to the release manager's machine.
  * Create a GitHub release, edit the changelog, upload all the release artifacts, including .rpm and .deb files.
  * Issue the GitHub release.
-  * Add the .rpm and .deb files to the official zrepl repos.
-    * Code for management of these repos: https://github.com/zrepl/package-repo-ops (private repo at this time)
+  * Add the .rpm and .deb files to the official zrepl repos, publish those.

-#### Patch releases, Go toolchain updates, APT/RPM Package rebuilds
+**Official binary releases are not re-built when Go receives an update. If the Go update is critical to zrepl (e.g. a Go security update that affects zrepl), we'd issue a new source release**.
+The rationale for this is that whereas distros provide a mechanism for this (`$zrepl_source_release-$distro_package_revision`), GitHub Releases doesn't which means we'd need to update the existing GitHub release's assets, which nobody would notice (no RSS feed updates, etc.).
+Downstream packagers can read the changelog to determine whether they want to push that minor release into their distro or simply skip it.

-`vMAJOR.MINOR.0` is typically a tagged commit on `master`, because development velocity isn't high
-and thus release branches for stabilization aren't necessary.
+### Additional Notes to Distro Package Maintainers

-Occasionally though there is a need for patch changes to a release, e.g.
- security issue in a dependency
- Go toolchain update (e.g. security issue in standard library)
-
-The procedure for this is the following
- create a branch off the release tag we need to patch, named `releases/MAJOR.MINOR.X`
- that branch will never be merged into `master`, it'll be a dead-end for this specific patch
- make changes in that branch
- make the final commit that bumps version numbers
- create the git tag
- follow general procedure for publishing the release (previous sectino
-
-For Go toolchain updates and package rebuilds with no source code changes, leverage the APT/RPM package revision field.
-Control via the `ZREPL_PACKAGE_RELEASE` Makefile variable, see `origin/releases/0.6.1-2` for an example.
-
-
-
-## Notes to Distro Package Maintainers
-
-* The `Makefile` in this project is not suitable for builds in distros.
-* Run the platform tests (Docs -> Usage -> Platform Tests) **on a test system** to validate that zrepl's abstractions on top of ZFS work with the system ZFS.
+* Use `sudo make test-platform-bin && sudo make test-platform` **on a test system** to validate that zrepl's abstractions on top of ZFS work with the system ZFS.
 * Ship a default config that adheres to your distro's `hier` and logging system.
 * Ship a service manager file and _please_ try to upstream it to this repository.
  * `dist/systemd` contains a Systemd unit template.
@@ -136,3 +104,31 @@ Control via the `ZREPL_PACKAGE_RELEASE` Makefile variable, see `origin/releases/
  This is how `zrepl version` knows what version number to show.
  Your build system should set the `ldFlags` flags appropriately and add a prefix or suffix that indicates that the given zrepl binary is a distro build, not an official one.
 * Make sure you are informed about new zrepl versions, e.g. by subscribing to GitHub's release RSS feed.
+
+
+## Contributing Code
+
+* Open an issue when starting to hack on a new feature
+* Commits should reference the issue they are related to
+* Docs improvements not documenting new features do not require an issue.
+
+### Breaking Changes
+
+Backward-incompatible changes must be documented in the git commit message and are listed in `docs/changelog.rst`.
+
+### Glossary & Naming Inconsistencies
+
+In ZFS, *dataset* refers to the objects *filesystem*, *ZVOL* and *snapshot*. <br />
+However, we need a word for *filesystem* & *ZVOL* but not a snapshot, bookmark, etc.
+
+Toward the user, the following terminology is used:
+
+* **filesystem**: a ZFS filesystem or a ZVOL
+* **filesystem version**: a ZFS snapshot or a bookmark
+
+Sadly, the zrepl implementation is inconsistent in its use of these words:
+variables and types are often named *dataset* when they in fact refer to a *filesystem*.
+
+There will not be a big refactoring (an attempt was made, but it's destroying too much history without much gain).
+
+However, new contributions & patches should fix naming without further notice in the commit message.
@@ -0,0 +1,30 @@
+FROM !SUBSTITUTED_BY_MAKEFILE
+
+RUN apt-get update && apt-get install -y \
+    python3-pip \
+    unzip \
+    gawk
+
+ADD build.installprotoc.bash ./
+RUN bash build.installprotoc.bash
+
+ADD lazy.sh /tmp/lazy.sh
+ADD docs/requirements.txt /tmp/requirements.txt
+ENV ZREPL_LAZY_DOCS_REQPATH=/tmp/requirements.txt
+RUN /tmp/lazy.sh docdep
+
+# prepare volume mount of git checkout to /zrepl
+RUN mkdir -p /src/github.com/zrepl/zrepl
+RUN mkdir -p /.cache && chmod -R 0777 /.cache
+
+# $GOPATH is /go
+# Go 1.12 doesn't use modules within GOPATH, but 1.13 and later do
+# => store source outside of GOPATH
+WORKDIR /src
+
+# Install build tools (e.g. protobuf generator, stringer) into $GOPATH/bin
+ADD build/ /tmp/build
+RUN /tmp/lazy.sh godep
+
+RUN chmod -R 0777 /go
+
@@ -0,0 +1,25 @@
+#!/usr/bin/env bash
+set -euo pipefail
+set -x
+
+MACH=$(uname -m)
+MACH="${MACH/aarch64/aarch_64}"
+
+VERSION=3.6.1
+FILENAME=protoc-"$VERSION"-linux-"$MACH".zip
+
+if [ -e "$FILENAME" ]; then
+    echo "$FILENAME" already exists 1>&2
+    exit 1
+fi
+
+wget https://github.com/protocolbuffers/protobuf/releases/download/v"$VERSION"/"$FILENAME"
+
+stat "$FILENAME"
+
+sha256sum -c --ignore-missing <<EOF
+6003de742ea3fcf703cfec1cd4a3380fd143081a2eb0e559065563496af27807  protoc-3.6.1-linux-x86_64.zip
+af8e5aaaf39ddec62ec8dd2be1b8d9602c6da66564883a16393ade5f71170922  protoc-3.6.1-linux-aarch_64.zip
+EOF
+
+unzip -d /usr "$FILENAME"
@@ -1 +0,0 @@
-install/
@@ -1,22 +0,0 @@
-FROM !SUBSTITUTED_BY_MAKEFILE
-
-RUN apt-get update && apt-get install -y \
-    python3-pip \
-    python3-venv \
-    unzip \
-    gawk
-
-# setup venv for docs
-ENV VIRTUAL_ENV=/opt/venv
-RUN python3 -m venv $VIRTUAL_ENV
-ENV PATH="$VIRTUAL_ENV/bin:$PATH"
-ADD docs/requirements.txt /tmp/requirements.txt
-RUN pip3 install -r /tmp/requirements.txt
-
-# Go toolchain uses xdg-cache
-RUN mkdir -p /.cache && chmod -R 0777 /.cache
-
-# Go devtools are managed by Makefile
-
-WORKDIR /src
-
@@ -1,27 +0,0 @@
-#!/usr/bin/env bash
-set -euo pipefail
-set -x
-
-cd "$1"
-
-MACH=$(uname -m)
-MACH="${MACH/aarch64/aarch_64}"
-
-VERSION=28.0
-FILENAME=protoc-"$VERSION"-linux-"$MACH".zip
-
-if [ -e "$FILENAME" ]; then
-    echo "$FILENAME" already exists 1>&2
-    exit 1
-fi
-
-wget --continue https://github.com/protocolbuffers/protobuf/releases/download/v"$VERSION"/"$FILENAME"
-
-stat "$FILENAME"
-
-sha256sum -c --ignore-missing <<EOF
-d622619dcbfb5ecb281cfb92c1a74d6a0f42e752d9a2774b197f475f7ab1c8c4  protoc-28.0-linux-aarch_64.zip
-b2e187c8b9f2d97cd3ecae4926d1bb2cbebe3ab768e7c987cbc86bb17f319358  protoc-28.0-linux-x86_64.zip
-EOF
-
-unzip -d . "$FILENAME"
@@ -0,0 +1,16 @@
+module github.com/zrepl/zrepl/build
+
+go 1.12
+
+require (
+	github.com/alvaroloes/enumer v1.1.1
+	github.com/golangci/golangci-lint v1.35.2
+	github.com/golangci/misspell v0.3.4 // indirect
+	github.com/golangci/revgrep v0.0.0-20180812185044-276a5c0a1039 // indirect
+	github.com/spf13/afero v1.2.2 // indirect
+	github.com/spf13/jwalterweatherman v1.1.0 // indirect
+	github.com/wadey/gocovmerge v0.0.0-20160331181800-b5bfa59ec0ad
+	golang.org/x/tools v0.0.0-20210105210202-9ed45478a130
+	google.golang.org/grpc/cmd/protoc-gen-go-grpc v1.1.0 // indirect
+	google.golang.org/protobuf v1.25.0
+)
@@ -0,0 +1,14 @@
+// +build tools
+
+package main
+
+// the lines are parsed by lazy.sh, do not edit
+import (
+	_ "github.com/alvaroloes/enumer"
+	_ "github.com/golangci/golangci-lint/cmd/golangci-lint"
+	_ "github.com/wadey/gocovmerge"
+	_ "golang.org/x/tools/cmd/goimports"
+	_ "golang.org/x/tools/cmd/stringer"
+	_ "google.golang.org/grpc/cmd/protoc-gen-go-grpc"
+	_ "google.golang.org/protobuf/cmd/protoc-gen-go"
+)
@@ -8,9 +8,9 @@ import (
 	"github.com/spf13/cobra"
 	"github.com/spf13/pflag"

-	"github.com/zrepl/zrepl/internal/daemon/logging/trace"
+	"github.com/zrepl/zrepl/daemon/logging/trace"

-	"github.com/zrepl/zrepl/internal/config"
+	"github.com/zrepl/zrepl/config"
 )

 var rootArgs struct {
@@ -11,17 +11,16 @@ import (
 	"github.com/spf13/pflag"
 	"github.com/zrepl/yaml-config"

-	"github.com/zrepl/zrepl/internal/cli"
-	"github.com/zrepl/zrepl/internal/config"
-	"github.com/zrepl/zrepl/internal/daemon/job"
-	"github.com/zrepl/zrepl/internal/daemon/logging"
-	"github.com/zrepl/zrepl/internal/logger"
+	"github.com/zrepl/zrepl/cli"
+	"github.com/zrepl/zrepl/config"
+	"github.com/zrepl/zrepl/daemon/job"
+	"github.com/zrepl/zrepl/daemon/logging"
+	"github.com/zrepl/zrepl/logger"
 )

 var configcheckArgs struct {
-	format        string
-	what          string
-	skipCertCheck bool
+	format string
+	what   string
 }

 var ConfigcheckCmd = &cli.Subcommand{
@@ -30,7 +29,6 @@ var ConfigcheckCmd = &cli.Subcommand{
 	SetupFlags: func(f *pflag.FlagSet) {
 		f.StringVar(&configcheckArgs.format, "format", "", "dump parsed config object [pretty|yaml|json]")
 		f.StringVar(&configcheckArgs.what, "what", "all", "what to print [all|config|jobs|logging]")
-		f.BoolVar(&configcheckArgs.skipCertCheck, "skip-cert-check", false, "skip checking cert files")
 	},
 	Run: func(ctx context.Context, subcommand *cli.Subcommand, args []string) error {
 		formatMap := map[string]func(interface{}){
@@ -58,16 +56,8 @@ var ConfigcheckCmd = &cli.Subcommand{
 		}

 		var hadErr bool
-
-		parseFlags := config.ParseFlagsNone
-
-		if configcheckArgs.skipCertCheck {
-			parseFlags |= config.ParseFlagsNoCertCheck
-		}
-
 		// further: try to build jobs
-		confJobs, err := job.JobsFromConfig(subcommand.Config(), parseFlags)
-
+		confJobs, err := job.JobsFromConfig(subcommand.Config())
 		if err != nil {
 			err := errors.Wrap(err, "cannot build jobs from config")
 			if configcheckArgs.what == "jobs" {
@@ -9,12 +9,12 @@ import (
 	"github.com/pkg/errors"
 	"github.com/spf13/pflag"

-	"github.com/zrepl/zrepl/internal/daemon/job"
-	"github.com/zrepl/zrepl/internal/endpoint"
-	"github.com/zrepl/zrepl/internal/zfs"
+	"github.com/zrepl/zrepl/daemon/job"
+	"github.com/zrepl/zrepl/endpoint"
+	"github.com/zrepl/zrepl/zfs"

-	"github.com/zrepl/zrepl/internal/cli"
-	"github.com/zrepl/zrepl/internal/config"
+	"github.com/zrepl/zrepl/cli"
+	"github.com/zrepl/zrepl/config"
 )

 var (
@@ -129,7 +129,7 @@ func doMigrateReplicationCursor(ctx context.Context, sc *cli.Subcommand, args []
 	}

 	cfg := sc.Config()
-	jobs, err := job.JobsFromConfig(cfg, config.ParseFlagsNone)
+	jobs, err := job.JobsFromConfig(cfg)
 	if err != nil {
 		fmt.Printf("cannot parse config:\n%s\n\n", err)
 		fmt.Printf("NOTE: this migration was released together with a change in job name requirements.\n")
@@ -1,6 +1,6 @@
 package client

-import "github.com/zrepl/zrepl/internal/cli"
+import "github.com/zrepl/zrepl/cli"

 var PprofCmd = &cli.Subcommand{
 	Use: "pprof",
@@ -8,7 +8,7 @@ import (

 	"golang.org/x/net/websocket"

-	"github.com/zrepl/zrepl/internal/cli"
+	"github.com/zrepl/zrepl/cli"
 )

 var pprofActivityTraceCmd = &cli.Subcommand{
@@ -6,9 +6,9 @@ import (
 	"log"
 	"os"

-	"github.com/zrepl/zrepl/internal/cli"
-	"github.com/zrepl/zrepl/internal/config"
-	"github.com/zrepl/zrepl/internal/daemon"
+	"github.com/zrepl/zrepl/cli"
+	"github.com/zrepl/zrepl/config"
+	"github.com/zrepl/zrepl/daemon"
 )

 var pprofListenCmd struct {
@@ -5,9 +5,9 @@ import (

 	"github.com/pkg/errors"

-	"github.com/zrepl/zrepl/internal/cli"
-	"github.com/zrepl/zrepl/internal/config"
-	"github.com/zrepl/zrepl/internal/daemon"
+	"github.com/zrepl/zrepl/cli"
+	"github.com/zrepl/zrepl/config"
+	"github.com/zrepl/zrepl/daemon"
 )

 var SignalCmd = &cli.Subcommand{
@@ -10,15 +10,15 @@ import (

 	"github.com/pkg/errors"

-	"github.com/zrepl/zrepl/internal/daemon"
+	"github.com/zrepl/zrepl/daemon"
 )

 type Client struct {
-	h *http.Client
+	h http.Client
 }

 func New(network, addr string) (*Client, error) {
-	httpc, err := makeControlHttpClient(func(_ context.Context) (net.Conn, error) { return net.Dial(network, addr) })
+	httpc, err := controlHttpClient(func(_ context.Context) (net.Conn, error) { return net.Dial(network, addr) })
 	if err != nil {
 		return nil, err
 	}
@@ -63,8 +63,8 @@ func (c *Client) SignalReset(job string) error {
 	return c.signal(job, "reset")
 }

-func makeControlHttpClient(dialfunc func(context.Context) (net.Conn, error)) (client *http.Client, err error) {
-	return &http.Client{
+func controlHttpClient(dialfunc func(context.Context) (net.Conn, error)) (client http.Client, err error) {
+	return http.Client{
 		Transport: &http.Transport{
 			DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) {
 				return dialfunc(ctx)
@@ -73,24 +73,14 @@ func makeControlHttpClient(dialfunc func(context.Context) (net.Conn, error)) (cl
 	}, nil
 }

-func jsonRequestResponse(c *http.Client, endpoint string, req interface{}, res interface{}) error {
+func jsonRequestResponse(c http.Client, endpoint string, req interface{}, res interface{}) error {
 	var buf bytes.Buffer
 	encodeErr := json.NewEncoder(&buf).Encode(req)
 	if encodeErr != nil {
 		return encodeErr
 	}

-	hreq, err := http.NewRequest("POST", "http://unix"+endpoint, &buf)
-	if err != nil {
-		return err
-	}
-	hreq.Header.Set("Content-Type", "application/json")
-	// Prevent EOF errors when client request frequency and server keepalive close are at the same time.
-	// Found this by watching http.Server.ConnState changes, then found
-	// https://stackoverflow.com/questions/17714494/golang-http-request-results-in-eof-errors-when-making-multiple-requests-successi
-	// Note: The issue seems even more prounounced with local TCP sockets than unix domain sockets. So, I used that for debugging.
-	hreq.Close = true
-	resp, err := c.Do(hreq)
+	resp, err := c.Post("http://unix"+endpoint, "application/json", &buf)
 	if err != nil {
 		return err
 	}
@@ -9,11 +9,11 @@ import (
 	"github.com/pkg/errors"
 	"github.com/spf13/pflag"

-	"github.com/zrepl/zrepl/internal/cli"
-	"github.com/zrepl/zrepl/internal/client/status/client"
-	"github.com/zrepl/zrepl/internal/config"
-	"github.com/zrepl/zrepl/internal/daemon"
-	"github.com/zrepl/zrepl/internal/util/choices"
+	"github.com/zrepl/zrepl/cli"
+	"github.com/zrepl/zrepl/client/status/client"
+	"github.com/zrepl/zrepl/config"
+	"github.com/zrepl/zrepl/daemon"
+	"github.com/zrepl/zrepl/util/choices"
 )

 type Client interface {
@@ -5,11 +5,11 @@ import (
 	"os"
 	"strings"

-	"github.com/gdamore/tcell/v2"
+	"github.com/gdamore/tcell"
 	"github.com/mattn/go-isatty"
 	"github.com/pkg/errors"

-	"github.com/zrepl/zrepl/internal/client/status/viewmodel"
+	"github.com/zrepl/zrepl/client/status/viewmodel"
 )

 func dump(c Client, job string) error {
@@ -8,9 +8,9 @@ import (
 	"time"

 	"github.com/gdamore/tcell/v2"
-	"github.com/rivo/tview"
+	tview "gitlab.com/tslocum/cview"

-	"github.com/zrepl/zrepl/internal/client/status/viewmodel"
+	"github.com/zrepl/zrepl/client/status/viewmodel"
 )

 func interactive(c Client, flag statusFlags) error {
@@ -29,6 +29,7 @@ func interactive(c Client, flag statusFlags) error {
 	jobMenuRoot.SetSelectable(true)
 	jobMenu.SetRoot(jobMenuRoot)
 	jobMenu.SetCurrentNode(jobMenuRoot)
+	jobMenu.SetSelectedTextColor(tcell.ColorGreen)
 	jobTextDetail := tview.NewTextView()
 	jobTextDetail.SetWrap(false)

@@ -109,8 +110,10 @@ func interactive(c Client, flag statusFlags) error {
 			}
 			app.SetRoot(toolbarSplit, true)
 			app.SetFocus(preModalFocus)
+			app.Draw()
 		})
 		app.SetRoot(m, true)
+		app.Draw()
 	}

 	app.SetRoot(toolbarSplit, true)
@@ -126,7 +129,7 @@ func interactive(c Client, flag statusFlags) error {
 		FSFilter:                func(_ string) bool { return true },
 		DetailViewWidth:         100,
 		DetailViewWrap:          false,
-		ShortKeybindingOverview: "[::b]Q[::-] quit  [::b]<TAB>[::-] switch panes [::b]W[::-] wrap lines  [::b]Shift+M[::-] toggle navbar  [::b]Shift+S[::-] signal job [::b]</>[::-] filter filesystems",
+		ShortKeybindingOverview: "[::b]Q[::-] quit  [::b]<TAB>[::-] switch panes  [::b]Shift+M[::-] toggle navbar  [::b]Shift+S[::-] signal job [::b]</>[::-] filter filesystems",
 	}
 	paramsMtx := &sync.Mutex{}
 	var redraw func()
@@ -167,14 +170,12 @@ func interactive(c Client, flag statusFlags) error {
 			redrawJobsList = true
 		}
 		if redrawJobsList {
-			selectedTextStyle := tcell.StyleDefault.Bold(true)
 			selectedJobN = nil
 			children := make([]*tview.TreeNode, len(jobs))
 			for i := range jobs {
 				jobN := tview.NewTreeNode(jobs[i].JobTreeTitle())
 				jobN.SetReference(jobs[i])
 				jobN.SetSelectable(true)
-				jobN.SetSelectedTextStyle(selectedTextStyle)
 				children[i] = jobN
 				jobN.SetSelectedFunc(func() {
 					viewmodelupdate(func(p *viewmodel.Params) {
@@ -186,7 +187,6 @@ func interactive(c Client, flag statusFlags) error {
 				}
 			}
 			jobMenuRoot.SetChildren(children)
-			jobMenuRoot.SetSelectedTextStyle(selectedTextStyle)
 		}

 		if selectedJobN != nil && jobMenu.GetCurrentNode() != selectedJobN {
@@ -207,6 +207,9 @@ func interactive(c Client, flag statusFlags) error {
 		bottombar.ResizeItem(bottombarDateView, len(bottombardatestring), 0)

 		bottomBarStatus.SetText(m.BottomBarStatus())
+
+		app.Draw()
+
 	}

 	go func() {
@@ -249,7 +252,6 @@ func interactive(c Client, flag statusFlags) error {

 	app.SetInputCapture(func(e *tcell.EventKey) *tcell.EventKey {
 		if e.Key() == tcell.KeyTab {
-			// TODO: only if there's no modal showing (long-time bug in zrepl status)
 			tabbableCycle()
 			return nil
 		}
@@ -282,8 +284,9 @@ func interactive(c Client, flag statusFlags) error {
 			signals := []string{"wakeup", "reset"}
 			clientFuncs := []func(job string) error{c.SignalWakeup, c.SignalReset}
 			sigMod := tview.NewModal()
+			sigMod.SetBackgroundColor(tcell.ColorDefault)
 			sigMod.SetBorder(true)
-			sigMod.SetButtonActivatedStyle(tcell.StyleDefault.Bold(true).Reverse(true))
+			sigMod.GetForm().SetButtonTextColorFocused(tcell.ColorGreen)
 			sigMod.AddButtons(signals)
 			sigMod.SetText(fmt.Sprintf("Send a signal to job %q", job.Name()))
 			showModal(sigMod, func(idx int, _ string) {
@@ -10,9 +10,9 @@ import (
 	"github.com/gdamore/tcell/v2"
 	"github.com/mattn/go-isatty"
 	"github.com/pkg/errors"
-	"github.com/rivo/tview"
+	tview "gitlab.com/tslocum/cview"

-	"github.com/zrepl/zrepl/internal/client/status/viewmodel"
+	"github.com/zrepl/zrepl/client/status/viewmodel"
 )

 func legacy(c Client, flag statusFlags) error {
@@ -28,7 +28,7 @@ func legacy(c Client, flag statusFlags) error {
 	textView := tview.NewTextView()
 	textView.SetWrap(true)
 	textView.SetScrollable(true) // so that it allows us to set scroll position
-	// textView.SetScrollBarVisibility(tview.ScrollBarNever)
+	textView.SetScrollBarVisibility(tview.ScrollBarNever)

 	app.SetRoot(textView, true)

@@ -2,22 +2,14 @@ package viewmodel

 import (
 	"fmt"
-	"math"
 )

-func ByteCountBinaryUint(b uint64) string {
-	if b > math.MaxInt64 {
-		panic(b)
-	}
-	return ByteCountBinary(int64(b))
-}
-
 func ByteCountBinary(b int64) string {
 	const unit = 1024
 	if b < unit {
 		return fmt.Sprintf("%d B", b)
 	}
-	div, exp := unit, 0
+	div, exp := int64(unit), 0
 	for n := b / unit; n >= unit; n /= unit {
 		div *= unit
 		exp++
@@ -4,16 +4,17 @@ import "time"

 type byteProgressMeasurement struct {
 	time time.Time
-	val  uint64
+	val  int64
 }

 type bytesProgressHistory struct {
 	last        *byteProgressMeasurement // pointer as poor man's optional
 	changeCount int
 	lastChange  time.Time
+	bpsAvg      float64
 }

-func (p *bytesProgressHistory) Update(currentVal uint64) (bytesPerSecondAvg int64, changeCount int) {
+func (p *bytesProgressHistory) Update(currentVal int64) (bytesPerSecondAvg int64, changeCount int) {

 	if p.last == nil {
 		p.last = &byteProgressMeasurement{
@@ -33,17 +34,15 @@ func (p *bytesProgressHistory) Update(currentVal uint64) (bytesPerSecondAvg int6
 		return 0, 0
 	}

-	var deltaV int64
-	if currentVal >= p.last.val {
-		deltaV = int64(currentVal - p.last.val)
-	} else {
-		deltaV = -int64(p.last.val - currentVal)
-	}
+	deltaV := currentVal - p.last.val
 	deltaT := time.Since(p.last.time)
 	rate := float64(deltaV) / deltaT.Seconds()

+	factor := 0.3
+	p.bpsAvg = (1-factor)*p.bpsAvg + factor*rate
+
 	p.last.time = time.Now()
 	p.last.val = currentVal

-	return int64(rate), p.changeCount
+	return int64(p.bpsAvg), p.changeCount
 }
@@ -10,12 +10,12 @@ import (
 	"github.com/go-playground/validator/v10"
 	yaml "github.com/zrepl/yaml-config"

-	"github.com/zrepl/zrepl/internal/client/status/viewmodel/stringbuilder"
-	"github.com/zrepl/zrepl/internal/config"
-	"github.com/zrepl/zrepl/internal/daemon/job"
-	"github.com/zrepl/zrepl/internal/daemon/pruner"
-	"github.com/zrepl/zrepl/internal/daemon/snapper"
-	"github.com/zrepl/zrepl/internal/replication/report"
+	"github.com/zrepl/zrepl/client/status/viewmodel/stringbuilder"
+	"github.com/zrepl/zrepl/daemon"
+	"github.com/zrepl/zrepl/daemon/job"
+	"github.com/zrepl/zrepl/daemon/pruner"
+	"github.com/zrepl/zrepl/daemon/snapper"
+	"github.com/zrepl/zrepl/replication/report"
 )

 type M struct {
@@ -85,7 +85,7 @@ func (m *M) Update(p Params) {
 	// filter out internal jobs
 	var jobsList []*Job
 	for _, j := range m.jobsList {
-		if config.IsInternalJobName(j.name) {
+		if daemon.IsInternalJobName(j.name) {
 			continue
 		}
 		jobsList = append(jobsList, j)
@@ -216,7 +216,7 @@ func drawJob(t *stringbuilder.B, name string, v *job.Status, history *bytesProgr
 	}
 }

-func printFilesystemStatus(t *stringbuilder.B, rep *report.FilesystemReport, maxFS int) {
+func printFilesystemStatus(t *stringbuilder.B, rep *report.FilesystemReport, active bool, maxFS int) {

 	expected, replicated, containsInvalidSizeEstimates := rep.BytesSum()
 	sizeEstimationImpreciseNotice := ""
@@ -227,28 +227,15 @@ func printFilesystemStatus(t *stringbuilder.B, rep *report.FilesystemReport, max
 		sizeEstimationImpreciseNotice = " (step lacks size estimation)"
 	}

-	userVisisbleCurrentStep, userVisibleTotalSteps := rep.CurrentStep, len(rep.Steps)
-	// `.CurrentStep` is == len(rep.Steps) if all steps are done.
-	// Until then, it's an index into .Steps that starts at 0.
-	// For the user, we want it to start at 1.
-	if rep.CurrentStep >= len(rep.Steps) {
-		// rep.CurrentStep is what we want to show.
-		// We check for >= and not == for robustness.
-	} else {
-		// We're not done yet, so, make step count start at 1
-		// (The `.State` is included in the output, indicating we're not done yet)
-		userVisisbleCurrentStep = rep.CurrentStep + 1
-	}
 	status := fmt.Sprintf("%s (step %d/%d, %s/%s)%s",
 		strings.ToUpper(string(rep.State)),
-		userVisisbleCurrentStep, userVisibleTotalSteps,
-		ByteCountBinaryUint(replicated), ByteCountBinaryUint(expected),
+		rep.CurrentStep, len(rep.Steps),
+		ByteCountBinary(replicated), ByteCountBinary(expected),
 		sizeEstimationImpreciseNotice,
 	)

 	activeIndicator := " "
-	if rep.BlockedOn == report.FsBlockedOnNothing &&
-		(rep.State == report.FilesystemPlanning || rep.State == report.FilesystemStepping) {
+	if active {
 		activeIndicator = "*"
 	}
 	t.AddIndent(1)
@@ -273,9 +260,9 @@ func printFilesystemStatus(t *stringbuilder.B, rep *report.FilesystemReport, max
 				attribs = append(attribs, "resumed")
 			}

-			if len(attribs) > 0 {
-				next += fmt.Sprintf(" (%s)", strings.Join(attribs, ", "))
-			}
+			attribs = append(attribs, fmt.Sprintf("encrypted=%s", nextStep.Info.Encrypted))
+
+			next += fmt.Sprintf(" (%s)", strings.Join(attribs, ", "))
 		} else {
 			next = "" // individual FSes may still be in planning state
 		}
@@ -371,18 +358,16 @@ func renderReplicationReport(t *stringbuilder.B, rep *report.Report, history *by
 		rate, changeCount := history.Update(replicated)
 		eta := time.Duration(0)
 		if rate > 0 {
-			eta = time.Duration((float64(expected)-float64(replicated))/float64(rate)) * time.Second
+			eta = time.Duration((expected-replicated)/rate) * time.Second
 		}

-		if !latest.State.IsTerminal() {
-			t.Write("Progress: ")
-			t.DrawBar(50, replicated, expected, changeCount)
-			t.Write(fmt.Sprintf(" %s / %s @ %s/s", ByteCountBinaryUint(replicated), ByteCountBinaryUint(expected), ByteCountBinary(rate)))
-			if eta != 0 {
-				t.Write(fmt.Sprintf(" (%s remaining)", humanizeDuration(eta)))
-			}
-			t.Newline()
+		t.Write("Progress: ")
+		t.DrawBar(50, replicated, expected, changeCount)
+		t.Write(fmt.Sprintf(" %s / %s @ %s/s", ByteCountBinary(replicated), ByteCountBinary(expected), ByteCountBinary(rate)))
+		if eta != 0 {
+			t.Write(fmt.Sprintf(" (%s remaining)", humanizeDuration(eta)))
 		}
+		t.Newline()
 		if containsInvalidSizeEstimates {
 			t.Write("NOTE: not all steps could be size-estimated, total estimate is likely imprecise!")
 			t.Newline()
@@ -400,7 +385,7 @@ func renderReplicationReport(t *stringbuilder.B, rep *report.Report, history *by
 			}
 		}
 		for _, fs := range latest.Filesystems {
-			printFilesystemStatus(t, fs, maxFSLen)
+			printFilesystemStatus(t, fs, false, maxFSLen) // FIXME bring 'active' flag back
 		}

 	}
@@ -495,17 +480,15 @@ func renderPrunerReport(t *stringbuilder.B, r *pruner.Report, fsfilter FilterFun
 	}

 	// global progress bar
-	if !state.IsTerminal() {
-		progress := int(math.Round(80 * float64(completedDestroyCount) / float64(totalDestroyCount)))
-		t.Write("Progress: ")
-		t.Write("[")
-		t.Write(stringbuilder.Times("=", progress))
-		t.Write(">")
-		t.Write(stringbuilder.Times("-", 80-progress))
-		t.Write("]")
-		t.Printf(" %d/%d snapshots", completedDestroyCount, totalDestroyCount)
-		t.Newline()
-	}
+	progress := int(math.Round(80 * float64(completedDestroyCount) / float64(totalDestroyCount)))
+	t.Write("Progress: ")
+	t.Write("[")
+	t.Write(stringbuilder.Times("=", progress))
+	t.Write(">")
+	t.Write(stringbuilder.Times("-", 80-progress))
+	t.Write("]")
+	t.Printf(" %d/%d snapshots", completedDestroyCount, totalDestroyCount)
+	t.Newline()

 	sort.SliceStable(all, func(i, j int) bool {
 		return strings.Compare(all[i].Filesystem, all[j].Filesystem) == -1
@@ -551,20 +534,10 @@ func renderPrunerReport(t *stringbuilder.B, r *pruner.Report, fsfilter FilterFun

 func renderSnapperReport(t *stringbuilder.B, r *snapper.Report, fsfilter FilterFunc) {
 	if r == nil {
-		t.Printf("<no snapshotting report available>\n")
+		t.Printf("<snapshot type does not have a report>\n")
 		return
 	}
-	t.Printf("Type: %s\n", r.Type)
-	if r.Periodic != nil {
-		renderSnapperReportPeriodic(t, r.Periodic, fsfilter)
-	} else if r.Cron != nil {
-		renderSnapperReportCron(t, r.Cron, fsfilter)
-	} else {
-		t.Printf("<no details available>")
-	}
-}

-func renderSnapperReportPeriodic(t *stringbuilder.B, r *snapper.PeriodicReport, fsfilter FilterFunc) {
 	t.Printf("Status: %s", r.State)
 	t.Newline()

@@ -575,25 +548,8 @@ func renderSnapperReportPeriodic(t *stringbuilder.B, r *snapper.PeriodicReport,
 		t.Printf("Sleep until: %s\n", r.SleepUntil)
 	}

-	renderSnapperPlanReportFilesystem(t, r.Progress, fsfilter)
-}
-
-func renderSnapperReportCron(t *stringbuilder.B, r *snapper.CronReport, fsfilter FilterFunc) {
-	t.Printf("State: %s\n", r.State)
-
-	now := time.Now()
-	if r.WakeupTime.After(now) {
-		t.Printf("Sleep until: %s (%s remaining)\n", r.WakeupTime, r.WakeupTime.Sub(now).Round(time.Second))
-	} else {
-		t.Printf("Started: %s (lasting %s)\n", r.WakeupTime, now.Sub(r.WakeupTime).Round(time.Second))
-	}
-
-	renderSnapperPlanReportFilesystem(t, r.Progress, fsfilter)
-}
-
-func renderSnapperPlanReportFilesystem(t *stringbuilder.B, fss []*snapper.ReportFilesystem, fsfilter FilterFunc) {
-	sort.Slice(fss, func(i, j int) bool {
-		return strings.Compare(fss[i].Path, fss[j].Path) == -1
+	sort.Slice(r.Progress, func(i, j int) bool {
+		return strings.Compare(r.Progress[i].Path, r.Progress[j].Path) == -1
 	})

 	dur := func(d time.Duration) string {
@@ -606,8 +562,8 @@ func renderSnapperPlanReportFilesystem(t *stringbuilder.B, fss []*snapper.Report
 	var widths struct {
 		path, state, duration int
 	}
-	rows := make([]*row, 0, len(fss))
-	for _, fs := range fss {
+	rows := make([]*row, 0, len(r.Progress))
+	for _, fs := range r.Progress {
 		if !fsfilter(fs.Path) {
 			continue
 		}
@@ -650,11 +606,9 @@ func renderSnapperPlanReportFilesystem(t *stringbuilder.B, fss []*snapper.Report
 		t.Printf("%s %s %s", path, state, duration)
 		t.PrintfDrawIndentedAndWrappedIfMultiline(" %s", r.remainder)
 		if r.hookReport != "" {
-			t.AddIndent(1)
-			t.Newline()
-			t.Printf("%s", r.hookReport)
-			t.AddIndent(-1)
+			t.PrintfDrawIndentedAndWrappedIfMultiline("%s", r.hookReport)
 		}
 		t.Newline()
 	}
+
 }
@@ -99,11 +99,11 @@ func RightPad(str string, length int, pad string) string {
 }

 // changeCount = 0 indicates stall / no progress
-func (w *B) DrawBar(length int, bytes, totalBytes uint64, changeCount int) {
+func (w *B) DrawBar(length int, bytes, totalBytes int64, changeCount int) {
 	const arrowPositions = `>\|/`
 	var completedLength int
 	if totalBytes > 0 {
-		completedLength = int(uint64(length) * bytes / totalBytes)
+		completedLength = int(int64(length) * bytes / totalBytes)
 		if completedLength > length {
 			completedLength = length
 		}
@@ -5,8 +5,8 @@ import (

 	"github.com/problame/go-netssh"

-	"github.com/zrepl/zrepl/internal/cli"
-	"github.com/zrepl/zrepl/internal/config"
+	"github.com/zrepl/zrepl/cli"
+	"github.com/zrepl/zrepl/config"

 	"context"
 	"errors"
@@ -9,10 +9,10 @@ import (
 	"github.com/pkg/errors"
 	"github.com/spf13/pflag"

-	"github.com/zrepl/zrepl/internal/cli"
-	"github.com/zrepl/zrepl/internal/config"
-	"github.com/zrepl/zrepl/internal/daemon/filters"
-	"github.com/zrepl/zrepl/internal/zfs"
+	"github.com/zrepl/zrepl/cli"
+	"github.com/zrepl/zrepl/config"
+	"github.com/zrepl/zrepl/daemon/filters"
+	"github.com/zrepl/zrepl/zfs"
 )

 var TestCmd = &cli.Subcommand{
@@ -139,11 +139,9 @@ var testPlaceholder = &cli.Subcommand{
 func runTestPlaceholder(ctx context.Context, subcommand *cli.Subcommand, args []string) error {

 	var checkDPs []*zfs.DatasetPath
-	var datasetWasExplicitArgument bool

 	// all actions first
 	if testPlaceholderArgs.all {
-		datasetWasExplicitArgument = false
 		out, err := zfs.ZFSList(ctx, []string{"name"})
 		if err != nil {
 			return errors.Wrap(err, "could not list ZFS filesystems")
@@ -156,7 +154,6 @@ func runTestPlaceholder(ctx context.Context, subcommand *cli.Subcommand, args []
 			checkDPs = append(checkDPs, dp)
 		}
 	} else {
-		datasetWasExplicitArgument = true
 		dp, err := zfs.NewDatasetPath(testPlaceholderArgs.ds)
 		if err != nil {
 			return err
@@ -174,12 +171,7 @@ func runTestPlaceholder(ctx context.Context, subcommand *cli.Subcommand, args []
 			return errors.Wrap(err, "cannot get placeholder state")
 		}
 		if !ph.FSExists {
-			if datasetWasExplicitArgument {
-				return errors.Errorf("filesystem %q does not exist", ph.FS)
-			} else {
-				// got deleted between ZFSList and ZFSGetFilesystemPlaceholderState
-				continue
-			}
+			panic("placeholder state inconsistent: filesystem " + ph.FS + " must exist in this context")
 		}
 		is := "yes"
 		if !ph.IsPlaceholder {
@@ -7,10 +7,10 @@ import (

 	"github.com/spf13/pflag"

-	"github.com/zrepl/zrepl/internal/cli"
-	"github.com/zrepl/zrepl/internal/config"
-	"github.com/zrepl/zrepl/internal/daemon"
-	"github.com/zrepl/zrepl/internal/version"
+	"github.com/zrepl/zrepl/cli"
+	"github.com/zrepl/zrepl/config"
+	"github.com/zrepl/zrepl/daemon"
+	"github.com/zrepl/zrepl/version"
 )

 var versionArgs struct {
@@ -7,10 +7,10 @@ import (

 	"github.com/spf13/pflag"

-	"github.com/zrepl/zrepl/internal/cli"
-	"github.com/zrepl/zrepl/internal/daemon/filters"
-	"github.com/zrepl/zrepl/internal/endpoint"
-	"github.com/zrepl/zrepl/internal/zfs"
+	"github.com/zrepl/zrepl/cli"
+	"github.com/zrepl/zrepl/daemon/filters"
+	"github.com/zrepl/zrepl/endpoint"
+	"github.com/zrepl/zrepl/zfs"
 )

 var (
@@ -57,7 +57,7 @@ func (f *zabsFilterFlags) registerZabsFilterFlags(s *pflag.FlagSet, verb string)
 	for v := range endpoint.AbstractionTypesAll {
 		variants = append(variants, string(v))
 	}
-	sort.Strings(variants)
+	variants = sort.StringSlice(variants)
 	variantsJoined := strings.Join(variants, "|")
 	s.Var(&f.Types, "type", fmt.Sprintf("only %s holds of the specified type [default: all] [comma-separated list of %s]", verb, variantsJoined))

@@ -1,6 +1,6 @@
 package client

-import "github.com/zrepl/zrepl/internal/cli"
+import "github.com/zrepl/zrepl/cli"

 var zabsCmdCreate = &cli.Subcommand{
 	Use:             "create",
@@ -7,9 +7,9 @@ import (
 	"github.com/pkg/errors"
 	"github.com/spf13/pflag"

-	"github.com/zrepl/zrepl/internal/cli"
-	"github.com/zrepl/zrepl/internal/endpoint"
-	"github.com/zrepl/zrepl/internal/zfs"
+	"github.com/zrepl/zrepl/cli"
+	"github.com/zrepl/zrepl/endpoint"
+	"github.com/zrepl/zrepl/zfs"
 )

 var zabsCreateStepHoldFlags struct {
@@ -11,9 +11,9 @@ import (
 	"github.com/pkg/errors"
 	"github.com/spf13/pflag"

-	"github.com/zrepl/zrepl/internal/cli"
-	"github.com/zrepl/zrepl/internal/endpoint"
-	"github.com/zrepl/zrepl/internal/util/chainlock"
+	"github.com/zrepl/zrepl/cli"
+	"github.com/zrepl/zrepl/endpoint"
+	"github.com/zrepl/zrepl/util/chainlock"
 )

 var zabsListFlags struct {
@@ -44,11 +44,10 @@ func doZabsList(ctx context.Context, sc *cli.Subcommand, args []string) error {
 		return errors.Wrap(err, "invalid filter specification on command line")
 	}

-	abstractions, errors, drainDone, err := endpoint.ListAbstractionsStreamed(ctx, q)
+	abstractions, errors, err := endpoint.ListAbstractionsStreamed(ctx, q)
 	if err != nil {
 		return err // context clear by invocation of command
 	}
-	defer drainDone()

 	var line chainlock.L
 	var wg sync.WaitGroup
@@ -10,8 +10,8 @@ import (
 	"github.com/pkg/errors"
 	"github.com/spf13/pflag"

-	"github.com/zrepl/zrepl/internal/cli"
-	"github.com/zrepl/zrepl/internal/endpoint"
+	"github.com/zrepl/zrepl/cli"
+	"github.com/zrepl/zrepl/endpoint"
 )

 // shared between release-all and release-step
@@ -2,32 +2,23 @@ package config

 import (
 	"fmt"
+	"io/ioutil"
 	"log/syslog"
 	"os"
-	pathpkg "path"
-	"path/filepath"
-	"strings"
+	"reflect"
+	"regexp"
+	"strconv"
 	"time"

 	"github.com/pkg/errors"
-	"github.com/robfig/cron/v3"
 	"github.com/zrepl/yaml-config"

-	"github.com/zrepl/zrepl/internal/util/datasizeunit"
-	zfsprop "github.com/zrepl/zrepl/internal/zfs/property"
-)
-
-type ParseFlags uint
-
-const (
-	ParseFlagsNone        ParseFlags = 0
-	ParseFlagsNoCertCheck ParseFlags = 1 << iota
+	zfsprop "github.com/zrepl/zrepl/zfs/property"
 )

 type Config struct {
-	Jobs    []JobEnum `yaml:"jobs,optional"`
-	Global  *Global   `yaml:"global,optional,fromdefaults"`
-	Include []string  `yaml:"include,optional"`
+	Jobs   []JobEnum `yaml:"jobs"`
+	Global *Global   `yaml:"global,optional,fromdefaults"`
 }

 func (c *Config) Job(name string) (*JobEnum, error) {
@@ -63,28 +54,26 @@ func (j JobEnum) Name() string {
 }

 type ActiveJob struct {
-	Type               string                `yaml:"type"`
-	Name               string                `yaml:"name"`
-	Connect            ConnectEnum           `yaml:"connect"`
-	Pruning            PruningSenderReceiver `yaml:"pruning"`
-	Replication        *Replication          `yaml:"replication,optional,fromdefaults"`
-	ConflictResolution *ConflictResolution   `yaml:"conflict_resolution,optional,fromdefaults"`
-}
-
-type ConflictResolution struct {
-	InitialReplication string `yaml:"initial_replication,optional,default=most_recent"`
+	Type        string                `yaml:"type"`
+	Name        string                `yaml:"name"`
+	Connect     ConnectEnum           `yaml:"connect"`
+	Pruning     PruningSenderReceiver `yaml:"pruning"`
+	Debug       JobDebugSettings      `yaml:"debug,optional"`
+	Replication *Replication          `yaml:"replication,optional,fromdefaults"`
 }

 type PassiveJob struct {
-	Type  string    `yaml:"type"`
-	Name  string    `yaml:"name"`
-	Serve ServeEnum `yaml:"serve"`
+	Type  string           `yaml:"type"`
+	Name  string           `yaml:"name"`
+	Serve ServeEnum        `yaml:"serve"`
+	Debug JobDebugSettings `yaml:"debug,optional"`
 }

 type SnapJob struct {
 	Type         string            `yaml:"type"`
 	Name         string            `yaml:"name"`
 	Pruning      PruningLocal      `yaml:"pruning"`
+	Debug        JobDebugSettings  `yaml:"debug,optional"`
 	Snapshotting SnapshottingEnum  `yaml:"snapshotting"`
 	Filesystems  FilesystemsFilter `yaml:"filesystems"`
 }
@@ -96,10 +85,8 @@ type SendOptions struct {
 	BackupProperties bool `yaml:"backup_properties,optional,default=false"`
 	LargeBlocks      bool `yaml:"large_blocks,optional,default=false"`
 	Compressed       bool `yaml:"compressed,optional,default=false"`
-	EmbeddedData     bool `yaml:"embedded_data,optional,default=false"`
+	EmbeddedData     bool `yaml:"embbeded_data,optional,default=false"`
 	Saved            bool `yaml:"saved,optional,default=false"`
-
-	BandwidthLimit *BandwidthLimit `yaml:"bandwidth_limit,optional,fromdefaults"`
 }

 type RecvOptions struct {
@@ -109,17 +96,6 @@ type RecvOptions struct {
 	// Reencrypt bool `yaml:"reencrypt"`

 	Properties *PropertyRecvOptions `yaml:"properties,fromdefaults"`
-
-	BandwidthLimit *BandwidthLimit `yaml:"bandwidth_limit,optional,fromdefaults"`
-
-	Placeholder *PlaceholderRecvOptions `yaml:"placeholder,fromdefaults"`
-}
-
-var _ yaml.Unmarshaler = &datasizeunit.Bits{}
-
-type BandwidthLimit struct {
-	Max            datasizeunit.Bits `yaml:"max,default=-1 B"`
-	BucketCapacity datasizeunit.Bits `yaml:"bucket_capacity,default=128 KiB"`
 }

 type Replication struct {
@@ -137,15 +113,15 @@ type ReplicationOptionsConcurrency struct {
 	SizeEstimates int `yaml:"size_estimates,optional,default=4"`
 }

+func (l *RecvOptions) SetDefault() {
+	*l = RecvOptions{Properties: &PropertyRecvOptions{}}
+}
+
 type PropertyRecvOptions struct {
 	Inherit  []zfsprop.Property          `yaml:"inherit,optional"`
 	Override map[zfsprop.Property]string `yaml:"override,optional"`
 }

-type PlaceholderRecvOptions struct {
-	Encryption string `yaml:"encryption,default=unspecified"`
-}
-
 type PushJob struct {
 	ActiveJob    `yaml:",inline"`
 	Snapshotting SnapshottingEnum  `yaml:"snapshotting"`
@@ -187,10 +163,13 @@ func (i *PositiveDurationOrManual) UnmarshalYAML(u func(interface{}, bool) error
 		return fmt.Errorf("value must not be empty")
 	default:
 		i.Manual = false
-		i.Interval, err = parsePositiveDuration(s)
+		i.Interval, err = time.ParseDuration(s)
 		if err != nil {
 			return err
 		}
+		if i.Interval <= 0 {
+			return fmt.Errorf("value must be a positive duration, got %q", s)
+		}
 	}
 	return nil
 }
@@ -222,49 +201,10 @@ type SnapshottingEnum struct {
 }

 type SnapshottingPeriodic struct {
-	Type                    string            `yaml:"type"`
-	Prefix                  string            `yaml:"prefix"`
-	Interval                *PositiveDuration `yaml:"interval"`
-	Hooks                   HookList          `yaml:"hooks,optional"`
-	TimestampFormattingSpec `yaml:",inline"`
-}
-
-type TimestampFormattingSpec struct {
-	TimestampFormat   string `yaml:"timestamp_format,optional,default=dense"`
-	TimestampLocation string `yaml:"timestamp_location,optional,default=UTC"`
-}
-
-type CronSpec struct {
-	Schedule cron.Schedule
-}
-
-var _ yaml.Unmarshaler = &CronSpec{}
-
-func (s *CronSpec) UnmarshalYAML(unmarshal func(v interface{}, not_strict bool) error) error {
-	var specString string
-	if err := unmarshal(&specString, false); err != nil {
-		return err
-	}
-
-	// Use standard cron format.
-	// Disable the various "descriptors" (@daily, etc)
-	// They are just aliases to "top of hour", "midnight", etc.
-	parser := cron.NewParser(cron.Minute | cron.Hour | cron.Dom | cron.Month | cron.Dow | cron.SecondOptional)
-
-	sched, err := parser.Parse(specString)
-	if err != nil {
-		return errors.Wrap(err, "cron syntax invalid")
-	}
-	s.Schedule = sched
-	return nil
-}
-
-type SnapshottingCron struct {
-	Type                    string   `yaml:"type"`
-	Prefix                  string   `yaml:"prefix"`
-	Cron                    CronSpec `yaml:"cron"`
-	Hooks                   HookList `yaml:"hooks,optional"`
-	TimestampFormattingSpec `yaml:",inline"`
+	Type     string        `yaml:"type"`
+	Prefix   string        `yaml:"prefix"`
+	Interval time.Duration `yaml:"interval,positive"`
+	Hooks    HookList      `yaml:"hooks,optional"`
 }

 type SnapshottingManual struct {
@@ -306,6 +246,18 @@ type Global struct {
 	Serve      *GlobalServe           `yaml:"serve,optional,fromdefaults"`
 }

+func Default(i interface{}) {
+	v := reflect.ValueOf(i)
+	if v.Kind() != reflect.Ptr {
+		panic(v)
+	}
+	y := `{}`
+	err := yaml.Unmarshal([]byte(y), v.Interface())
+	if err != nil {
+		panic(err)
+	}
+}
+
 type ConnectEnum struct {
 	Ret interface{}
 }
@@ -472,6 +424,14 @@ type GlobalStdinServer struct {
 	SockDir string `yaml:"sockdir,default=/var/run/zrepl/stdinserver"`
 }

+type JobDebugSettings struct {
+	Conn *struct {
+		ReadDump  string `yaml:"read_dump"`
+		WriteDump string `yaml:"write_dump"`
+	} `yaml:"conn,optional"`
+	RPCLog bool `yaml:"rpc_log,optional,default=false"`
+}
+
 type HookList []HookEnum

 type HookEnum struct {
@@ -570,7 +530,6 @@ func (t *SnapshottingEnum) UnmarshalYAML(u func(interface{}, bool) error) (err e
 	t.Ret, err = enumUnmarshal(u, map[string]interface{}{
 		"periodic": &SnapshottingPeriodic{},
 		"manual":   &SnapshottingManual{},
-		"cron":     &SnapshottingCron{},
 	})
 	return
 }
@@ -659,9 +618,8 @@ var ConfigFileDefaultLocations = []string{
 	"/usr/local/etc/zrepl/zrepl.yml",
 }

-func ParseConfig(path string) (rootConfig *Config, err error) {
+func ParseConfig(path string) (i *Config, err error) {

-	// Parse main configuration file
 	if path == "" {
 		// Try default locations
 		for _, l := range ConfigFileDefaultLocations {
@@ -680,94 +638,11 @@ func ParseConfig(path string) (rootConfig *Config, err error) {

 	var bytes []byte

-	if bytes, err = os.ReadFile(path); err != nil {
+	if bytes, err = ioutil.ReadFile(path); err != nil {
 		return
 	}

-	rootConfig, err = ParseConfigBytes(bytes)
-	if err != nil {
-		return nil, err
-	}
-
-	err = expandConfigInclude(path, rootConfig)
-	if err != nil {
-		return nil, err
-	}
-
-	if err = validateJobNames(rootConfig); err != nil {
-		return nil, err
-	}
-
-	return rootConfig, err
-}
-
-func IsInternalJobName(s string) bool {
-	return strings.HasPrefix(s, "_")
-}
-
-func validateJobNames(config *Config) error {
-	seen := make(map[string]struct{})
-	for _, job := range config.Jobs {
-		name := job.Name()
-		if IsInternalJobName(name) {
-			return errors.Errorf("job name %q is reserved for internal use (starts with _)", name)
-		}
-		if _, ok := seen[name]; ok {
-			return errors.Errorf("duplicate job name %q", name)
-		}
-		seen[name] = struct{}{}
-	}
-	return nil
-}
-
-func expandConfigInclude(configPath string, config *Config) (err error) {
-	var includeConfigPaths []string
-	for _, path := range config.Include {
-		if !pathpkg.IsAbs(configPath) {
-			path = pathpkg.Join(pathpkg.Dir(configPath), path)
-		}
-
-		stat, statErr := os.Stat(path)
-		if statErr != nil {
-			return errors.Wrapf(statErr, "stat path %q", path)
-		}
-
-		if stat.Mode().IsDir() {
-			directoryPaths, err := filepath.Glob(path + "/*.yml")
-			if err != nil {
-				return err
-			}
-
-			includeConfigPaths = append(includeConfigPaths, directoryPaths...)
-		} else if stat.Mode().IsRegular() {
-			if extention := filepath.Ext(path); extention != ".yml" {
-				return fmt.Errorf("include config files must end with `.yml`: %s", path)
-			}
-			includeConfigPaths = append(includeConfigPaths, path)
-		} else {
-			return fmt.Errorf("not a file or directory: %s", path)
-		}
-	}
-
-	for _, path := range includeConfigPaths {
-		var bytes []byte
-		if bytes, err = os.ReadFile(path); err != nil {
-			return errors.Wrapf(err, "read file: %q", path)
-		}
-
-		includedConfig, err := ParseConfigBytes(bytes)
-		if err != nil {
-			return err
-		}
-
-		if len(includedConfig.Include) > 0 {
-			return errors.Errorf("included configuration files must not include other files: %s", path)
-		}
-
-		config.Jobs = append(config.Jobs, includedConfig.Jobs...)
-	}
-
-	return nil
+	return ParseConfigBytes(bytes)
 }

 func ParseConfigBytes(bytes []byte) (*Config, error) {
@@ -775,16 +650,46 @@ func ParseConfigBytes(bytes []byte) (*Config, error) {
 	if err := yaml.UnmarshalStrict(bytes, &c); err != nil {
 		return nil, err
 	}
-	if c != nil {
-		return c, nil
-	}
-	// There was no yaml document in the file, deserialize from default.
-	// => See TestFromdefaultsEmptyDoc in yaml-config package.
-	if err := yaml.UnmarshalStrict([]byte("{}"), &c); err != nil {
-		return nil, err
-	}
 	if c == nil {
-		panic("the fallback to deserialize from `{}` should work")
+		return nil, fmt.Errorf("config is empty or only consists of comments")
 	}
 	return c, nil
 }
+
+var durationStringRegex *regexp.Regexp = regexp.MustCompile(`^\s*(\d+)\s*(s|m|h|d|w)\s*$`)
+
+func parsePositiveDuration(e string) (d time.Duration, err error) {
+	comps := durationStringRegex.FindStringSubmatch(e)
+	if len(comps) != 3 {
+		err = fmt.Errorf("does not match regex: %s %#v", e, comps)
+		return
+	}
+
+	durationFactor, err := strconv.ParseInt(comps[1], 10, 64)
+	if err != nil {
+		return 0, err
+	}
+	if durationFactor <= 0 {
+		return 0, errors.New("duration must be positive integer")
+	}
+
+	var durationUnit time.Duration
+	switch comps[2] {
+	case "s":
+		durationUnit = time.Second
+	case "m":
+		durationUnit = time.Minute
+	case "h":
+		durationUnit = time.Hour
+	case "d":
+		durationUnit = 24 * time.Hour
+	case "w":
+		durationUnit = 24 * 7 * time.Hour
+	default:
+		err = fmt.Errorf("contains unknown time unit '%s'", comps[2])
+		return
+	}
+
+	d = time.Duration(durationFactor) * durationUnit
+	return
+}
@@ -2,8 +2,16 @@ package config

 import (
 	"testing"
+
+	"github.com/stretchr/testify/assert"
 )

+func TestConfigEmptyFails(t *testing.T) {
+	conf, err := testConfig(t, "\n")
+	assert.Nil(t, conf)
+	assert.Error(t, err)
+}
+
 func TestJobsOnlyWorks(t *testing.T) {
 	testValidConfig(t, `
 jobs:
@@ -26,7 +34,7 @@ jobs:
    keep_sender:
    - type: not_replicated
    keep_receiver:
-    - type: last_n
+    - type: last_n 
      count: 1
 `)
 }
@@ -7,7 +7,7 @@ import (
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"

-	zfsprop "github.com/zrepl/zrepl/internal/zfs/property"
+	zfsprop "github.com/zrepl/zrepl/zfs/property"
 )

 func TestRecvOptions(t *testing.T) {
@@ -0,0 +1,90 @@
+package config
+
+import (
+	"fmt"
+	"testing"
+	"time"
+
+	"github.com/stretchr/testify/assert"
+)
+
+func TestSnapshotting(t *testing.T) {
+	tmpl := `
+jobs:
+- name: foo
+  type: push
+  connect:
+    type: local
+    listener_name: foo
+    client_identity: bar
+  filesystems: {"<": true}
+  %s
+  pruning:
+    keep_sender:
+    - type: last_n
+      count: 10
+    keep_receiver:
+    - type: last_n
+      count: 10
+`
+	manual := `
+  snapshotting:
+    type: manual
+`
+	periodic := `
+  snapshotting:
+    type: periodic
+    prefix: zrepl_
+    interval: 10m
+`
+
+	hooks := `
+  snapshotting:
+    type: periodic
+    prefix: zrepl_
+    interval: 10m
+    hooks:
+    - type: command
+      path: /tmp/path/to/command
+    - type: command
+      path: /tmp/path/to/command
+      filesystems: { "zroot<": true, "<": false }
+    - type: postgres-checkpoint
+      dsn: "host=localhost port=5432 user=postgres sslmode=disable"
+      filesystems: {
+          "tank/postgres/data11": true
+      }
+    - type: mysql-lock-tables
+      dsn: "root@tcp(localhost)/"
+      filesystems: {
+        "tank/mysql": true
+      }
+`
+
+	fillSnapshotting := func(s string) string { return fmt.Sprintf(tmpl, s) }
+	var c *Config
+
+	t.Run("manual", func(t *testing.T) {
+		c = testValidConfig(t, fillSnapshotting(manual))
+		snm := c.Jobs[0].Ret.(*PushJob).Snapshotting.Ret.(*SnapshottingManual)
+		assert.Equal(t, "manual", snm.Type)
+	})
+
+	t.Run("periodic", func(t *testing.T) {
+		c = testValidConfig(t, fillSnapshotting(periodic))
+		snp := c.Jobs[0].Ret.(*PushJob).Snapshotting.Ret.(*SnapshottingPeriodic)
+		assert.Equal(t, "periodic", snp.Type)
+		assert.Equal(t, 10*time.Minute, snp.Interval)
+		assert.Equal(t, "zrepl_", snp.Prefix)
+	})
+
+	t.Run("hooks", func(t *testing.T) {
+		c = testValidConfig(t, fillSnapshotting(hooks))
+		hs := c.Jobs[0].Ret.(*PushJob).Snapshotting.Ret.(*SnapshottingPeriodic).Hooks
+		assert.Equal(t, hs[0].Ret.(*HookCommand).Filesystems["<"], true)
+		assert.Equal(t, hs[1].Ret.(*HookCommand).Filesystems["zroot<"], true)
+		assert.Equal(t, hs[2].Ret.(*HookPostgresCheckpoint).Filesystems["tank/postgres/data11"], true)
+		assert.Equal(t, hs[3].Ret.(*HookMySQLLockTables).Filesystems["tank/mysql"], true)
+	})
+
+}
@@ -0,0 +1,88 @@
+package config
+
+import (
+	"bufio"
+	"bytes"
+	"fmt"
+	"path"
+	"path/filepath"
+	"strings"
+	"testing"
+	"text/template"
+
+	"github.com/kr/pretty"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+func TestSampleConfigsAreParsedWithoutErrors(t *testing.T) {
+	paths, err := filepath.Glob("./samples/*")
+	if err != nil {
+		t.Errorf("glob failed: %+v", err)
+	}
+
+	for _, p := range paths {
+
+		if path.Ext(p) != ".yml" {
+			t.Logf("skipping file %s", p)
+			continue
+		}
+
+		t.Run(p, func(t *testing.T) {
+			c, err := ParseConfig(p)
+			if err != nil {
+				t.Errorf("error parsing %s:\n%+v", p, err)
+			}
+
+			t.Logf("file: %s", p)
+			t.Log(pretty.Sprint(c))
+		})
+
+	}
+
+}
+
+// template must be a template/text template with a single '{{ . }}' as placeholder for val
+//nolint[:deadcode,unused]
+func testValidConfigTemplate(t *testing.T, tmpl string, val string) *Config {
+	tmp, err := template.New("master").Parse(tmpl)
+	if err != nil {
+		panic(err)
+	}
+	var buf bytes.Buffer
+	err = tmp.Execute(&buf, val)
+	if err != nil {
+		panic(err)
+	}
+	return testValidConfig(t, buf.String())
+}
+
+func testValidConfig(t *testing.T, input string) *Config {
+	t.Helper()
+	conf, err := testConfig(t, input)
+	require.NoError(t, err)
+	require.NotNil(t, conf)
+	return conf
+}
+
+func testConfig(t *testing.T, input string) (*Config, error) {
+	t.Helper()
+	return ParseConfigBytes([]byte(input))
+}
+
+func trimSpaceEachLineAndPad(s, pad string) string {
+	var out strings.Builder
+	scan := bufio.NewScanner(strings.NewReader(s))
+	for scan.Scan() {
+		fmt.Fprintf(&out, "%s%s\n", pad, bytes.TrimSpace(scan.Bytes()))
+	}
+	return out.String()
+}
+
+func TestTrimSpaceEachLineAndPad(t *testing.T) {
+	foo := `
+	foo
+	bar baz 
+	`
+	assert.Equal(t, "  \n  foo\n  bar baz\n  \n", trimSpaceEachLineAndPad(foo, "  "))
+}
@@ -9,8 +9,8 @@ jobs:
    key:  /etc/zrepl/prod.key
    server_cn: "backups"
  filesystems: {
-    "zroot<": true,
-    "zroot/var/tmp<": false,
+    "zroot/var/db": true,
+    "zroot/usr/home<": true,
    "zroot/usr/home/paranoid": false
  }
  snapshotting:
@@ -8,20 +8,19 @@ import (
 	"io"
 	"net"
 	"net/http"
-	"os"
 	"time"

 	"github.com/pkg/errors"
 	"github.com/prometheus/client_golang/prometheus"

-	"github.com/zrepl/zrepl/internal/daemon/job"
-	"github.com/zrepl/zrepl/internal/daemon/nethelpers"
-	"github.com/zrepl/zrepl/internal/endpoint"
-	"github.com/zrepl/zrepl/internal/logger"
-	"github.com/zrepl/zrepl/internal/util/envconst"
-	"github.com/zrepl/zrepl/internal/version"
-	"github.com/zrepl/zrepl/internal/zfs"
-	"github.com/zrepl/zrepl/internal/zfs/zfscmd"
+	"github.com/zrepl/zrepl/daemon/job"
+	"github.com/zrepl/zrepl/daemon/nethelpers"
+	"github.com/zrepl/zrepl/endpoint"
+	"github.com/zrepl/zrepl/logger"
+	"github.com/zrepl/zrepl/util/envconst"
+	"github.com/zrepl/zrepl/version"
+	"github.com/zrepl/zrepl/zfs"
+	"github.com/zrepl/zrepl/zfs/zfscmd"
 )

 type controlJob struct {
@@ -125,9 +124,8 @@ func (j *controlJob) Run(ctx context.Context) {
 			s := Status{
 				Jobs: jobs,
 				Global: GlobalStatus{
-					ZFSCmds:   globalZFS,
-					Envconst:  envconstReport,
-					OsEnviron: os.Environ(),
+					ZFSCmds:  globalZFS,
+					Envconst: envconstReport,
 				}}
 			return s, nil
 		}})
@@ -158,8 +156,8 @@ func (j *controlJob) Run(ctx context.Context) {
 	server := http.Server{
 		Handler: mux,
 		// control socket is local, 1s timeout should be more than sufficient, even on a loaded system
-		WriteTimeout: envconst.Duration("ZREPL_DAEMON_CONTROL_SERVER_WRITE_TIMEOUT", 1*time.Second),
-		ReadTimeout:  envconst.Duration("ZREPL_DAEMON_CONTROL_SERVER_READ_TIMEOUT", 1*time.Second),
+		WriteTimeout: 1 * time.Second,
+		ReadTimeout:  1 * time.Second,
 	}

 outer:
@@ -3,8 +3,10 @@ package daemon
 import (
 	"context"
 	"fmt"
+	"math/rand"
 	"os"
 	"os/signal"
+	"strings"
 	"sync"
 	"syscall"
 	"time"
@@ -12,18 +14,18 @@ import (
 	"github.com/pkg/errors"
 	"github.com/prometheus/client_golang/prometheus"

-	"github.com/zrepl/zrepl/internal/daemon/logging/trace"
-	"github.com/zrepl/zrepl/internal/endpoint"
-	"github.com/zrepl/zrepl/internal/util/envconst"
+	"github.com/zrepl/zrepl/daemon/logging/trace"
+	"github.com/zrepl/zrepl/endpoint"
+	"github.com/zrepl/zrepl/util/envconst"

-	"github.com/zrepl/zrepl/internal/config"
-	"github.com/zrepl/zrepl/internal/daemon/job"
-	"github.com/zrepl/zrepl/internal/daemon/job/reset"
-	"github.com/zrepl/zrepl/internal/daemon/job/wakeup"
-	"github.com/zrepl/zrepl/internal/daemon/logging"
-	"github.com/zrepl/zrepl/internal/logger"
-	"github.com/zrepl/zrepl/internal/version"
-	"github.com/zrepl/zrepl/internal/zfs/zfscmd"
+	"github.com/zrepl/zrepl/config"
+	"github.com/zrepl/zrepl/daemon/job"
+	"github.com/zrepl/zrepl/daemon/job/reset"
+	"github.com/zrepl/zrepl/daemon/job/wakeup"
+	"github.com/zrepl/zrepl/daemon/logging"
+	"github.com/zrepl/zrepl/logger"
+	"github.com/zrepl/zrepl/version"
+	"github.com/zrepl/zrepl/zfs/zfscmd"
 )

 func Run(ctx context.Context, conf *config.Config) error {
@@ -37,13 +39,19 @@ func Run(ctx context.Context, conf *config.Config) error {
 		cancel()
 	}()

+	// The math/rand package is used presently for generating trace IDs, we
+	// seed it with the current time and pid so that the IDs are mostly
+	// unique.
+	rand.Seed(time.Now().UnixNano())
+	rand.Seed(int64(os.Getpid()))
+
 	outlets, err := logging.OutletsFromConfig(*conf.Global.Logging)
 	if err != nil {
 		return errors.Wrap(err, "cannot build logging from config")
 	}
 	outlets.Add(newPrometheusLogOutlet(), logger.Debug)

-	confJobs, err := job.JobsFromConfig(conf, config.ParseFlagsNone)
+	confJobs, err := job.JobsFromConfig(conf)
 	if err != nil {
 		return errors.Wrap(err, "cannot build jobs from config")
 	}
@@ -63,7 +71,7 @@ func Run(ctx context.Context, conf *config.Config) error {
 	})

 	for _, job := range confJobs {
-		if config.IsInternalJobName(job.Name()) {
+		if IsInternalJobName(job.Name()) {
 			panic(fmt.Sprintf("internal job name used for config job '%s'", job.Name())) //FIXME
 		}
 	}
@@ -152,9 +160,8 @@ type Status struct {
 }

 type GlobalStatus struct {
-	ZFSCmds   *zfscmd.Report
-	Envconst  *envconst.Report
-	OsEnviron []string
+	ZFSCmds  *zfscmd.Report
+	Envconst *envconst.Report
 }

 func (s *jobs) status() map[string]*job.Status {
@@ -210,6 +217,10 @@ const (
 	jobNameControl    = "_control"
 )

+func IsInternalJobName(s string) bool {
+	return strings.HasPrefix(s, "_")
+}
+
 func (s *jobs) start(ctx context.Context, j job.Job, internal bool) {
 	s.m.Lock()
 	defer s.m.Unlock()
@@ -217,12 +228,10 @@ func (s *jobs) start(ctx context.Context, j job.Job, internal bool) {
 	ctx = logging.WithInjectedField(ctx, logging.JobField, j.Name())

 	jobName := j.Name()
-
-	// package `config` enforces these with clean errors, these are just assertions
-	if !internal && config.IsInternalJobName(jobName) {
+	if !internal && IsInternalJobName(jobName) {
 		panic(fmt.Sprintf("internal job name used for non-internal job %s", jobName))
 	}
-	if internal && !config.IsInternalJobName(jobName) {
+	if internal && !IsInternalJobName(jobName) {
 		panic(fmt.Sprintf("internal job does not use internal job name %s", jobName))
 	}
 	if _, ok := s.jobs[jobName]; ok {
@@ -6,8 +6,8 @@ import (

 	"github.com/pkg/errors"

-	"github.com/zrepl/zrepl/internal/endpoint"
-	"github.com/zrepl/zrepl/internal/zfs"
+	"github.com/zrepl/zrepl/endpoint"
+	"github.com/zrepl/zrepl/zfs"
 )

 type DatasetMapFilter struct {
@@ -160,14 +160,6 @@ func (m DatasetMapFilter) Filter(p *zfs.DatasetPath) (pass bool, err error) {
 	return
 }

-func (m DatasetMapFilter) UserSpecifiedDatasets() (datasets zfs.UserSpecifiedDatasetsSet) {
-	datasets = make(zfs.UserSpecifiedDatasetsSet)
-	for i := range m.entries {
-		datasets[m.entries[i].path.ToString()] = true
-	}
-	return
-}
-
 // Construct a new filter-only DatasetMapFilter from a mapping
 // The new filter allows exactly those paths that were not forbidden by the mapping.
 func (m DatasetMapFilter) InvertedFilter() (inv *DatasetMapFilter, err error) {
@@ -3,7 +3,7 @@ package filters
 import (
 	"testing"

-	"github.com/zrepl/zrepl/internal/zfs"
+	"github.com/zrepl/zrepl/zfs"
 )

 func TestDatasetMapFilter(t *testing.T) {
@@ -3,8 +3,8 @@ package hooks
 import (
 	"fmt"

-	"github.com/zrepl/zrepl/internal/config"
-	"github.com/zrepl/zrepl/internal/zfs"
+	"github.com/zrepl/zrepl/config"
+	"github.com/zrepl/zrepl/zfs"
 )

 type List []Hook
@@ -5,7 +5,7 @@
 //
 // This package also provides all supported hook type implementations and abstractions around them.
 //
-// # Use For Other Kinds Of ExpectStepReports
+// Use For Other Kinds Of ExpectStepReports
 //
 // This package REQUIRES REFACTORING before it can be used for other activities than snapshots, e.g. pre- and post-replication:
 //
@@ -15,7 +15,7 @@
 // The hook implementations should move out of this package.
 // However, there is a lot of tight coupling which to untangle isn't worth it ATM.
 //
-// # How This Package Is Used By Package Snapper
+// How This Package Is Used By Package Snapper
 //
 // Deserialize a config.List using ListFromConfig().
 // Then it MUST filter the list to only contain hooks for a particular filesystem using
@@ -30,4 +30,5 @@
 // Command hooks make it available in the environment variable ZREPL_DRYRUN.
 //
 // Plan.Report() can be called while Plan.Run() is executing to give an overview of plan execution progress (future use in "zrepl status").
+//
 package hooks
@@ -7,7 +7,7 @@ import (
 	"sync"
 	"time"

-	"github.com/zrepl/zrepl/internal/zfs"
+	"github.com/zrepl/zrepl/zfs"
 )

 // Re-export type here so that
@@ -6,9 +6,9 @@ import (
 	"context"
 	"sync"

-	"github.com/zrepl/zrepl/internal/daemon/logging"
-	"github.com/zrepl/zrepl/internal/logger"
-	"github.com/zrepl/zrepl/internal/util/envconst"
+	"github.com/zrepl/zrepl/daemon/logging"
+	"github.com/zrepl/zrepl/logger"
+	"github.com/zrepl/zrepl/util/envconst"
 )

 type Logger = logger.Logger
@@ -4,8 +4,8 @@ import (
 	"context"
 	"fmt"

-	"github.com/zrepl/zrepl/internal/daemon/filters"
-	"github.com/zrepl/zrepl/internal/zfs"
+	"github.com/zrepl/zrepl/daemon/filters"
+	"github.com/zrepl/zrepl/zfs"
 )

 type HookJobCallback func(ctx context.Context) error
@@ -12,11 +12,11 @@ import (
 	"sync"
 	"time"

-	"github.com/zrepl/zrepl/internal/config"
-	"github.com/zrepl/zrepl/internal/daemon/filters"
-	"github.com/zrepl/zrepl/internal/logger"
-	"github.com/zrepl/zrepl/internal/util/circlog"
-	"github.com/zrepl/zrepl/internal/util/envconst"
+	"github.com/zrepl/zrepl/config"
+	"github.com/zrepl/zrepl/daemon/filters"
+	"github.com/zrepl/zrepl/logger"
+	"github.com/zrepl/zrepl/util/circlog"
+	"github.com/zrepl/zrepl/util/envconst"
 )

 type HookEnvVar string
@@ -93,14 +93,7 @@ func (r *CommandHookReport) String() string {
 		cmdLine.WriteString(fmt.Sprintf("%s'%s'", sep, a))
 	}

-	var msg string
-	if r.Err == nil {
-		msg = "command hook"
-	} else {
-		msg = fmt.Sprintf("command hook failed with %q", r.Err)
-	}
-
-	return fmt.Sprintf("%s: \"%s\"", msg, cmdLine.String()) // no %q to make copy-pastable
+	return fmt.Sprintf("command hook invocation: \"%s\"", cmdLine.String()) // no %q to make copy-pastable
 }
 func (r *CommandHookReport) Error() string {
 	if r.Err == nil {
@@ -12,27 +12,24 @@ import (

 	"github.com/pkg/errors"

-	"github.com/zrepl/zrepl/internal/config"
-	"github.com/zrepl/zrepl/internal/daemon/filters"
-	"github.com/zrepl/zrepl/internal/zfs"
+	"github.com/zrepl/zrepl/config"
+	"github.com/zrepl/zrepl/daemon/filters"
+	"github.com/zrepl/zrepl/zfs"
 )

-// Hook to implement the following recommmendation from MySQL docs
 // https://dev.mysql.com/doc/mysql-backup-excerpt/5.7/en/backup-methods.html
 //
-//	Making Backups Using a File System Snapshot:
+// Making Backups Using a File System Snapshot:
 //
-//	If you are using a Veritas file system, you can make a backup like this:
+//  If you are using a Veritas file system, you can make a backup like this:
 //
-//	   	From a client program, execute FLUSH TABLES WITH READ LOCK.
-//	   	From another shell, execute mount vxfs snapshot.
-//	   	From the first client, execute UNLOCK TABLES.
-//	   	Copy files from the snapshot.
-//	   	Unmount the snapshot.
+//    	From a client program, execute FLUSH TABLES WITH READ LOCK.
+//    	From another shell, execute mount vxfs snapshot.
+//    	From the first client, execute UNLOCK TABLES.
+//    	Copy files from the snapshot.
+//    	Unmount the snapshot.
 //
-//		Similar snapshot capabilities may be available in other file systems, such as LVM or ZFS.
-//
-
+//	Similar snapshot capabilities may be available in other file systems, such as LVM or ZFS.
 type MySQLLockTables struct {
 	errIsFatal  bool
 	connector   sqldriver.Connector
@@ -10,9 +10,9 @@ import (
 	"github.com/lib/pq"
 	"github.com/pkg/errors"

-	"github.com/zrepl/zrepl/internal/config"
-	"github.com/zrepl/zrepl/internal/daemon/filters"
-	"github.com/zrepl/zrepl/internal/zfs"
+	"github.com/zrepl/zrepl/config"
+	"github.com/zrepl/zrepl/daemon/filters"
+	"github.com/zrepl/zrepl/zfs"
 )

 type PgChkptHook struct {
@@ -11,13 +11,13 @@ import (

 	"github.com/stretchr/testify/require"

-	"github.com/zrepl/zrepl/internal/daemon/logging/trace"
+	"github.com/zrepl/zrepl/daemon/logging/trace"

-	"github.com/zrepl/zrepl/internal/config"
-	"github.com/zrepl/zrepl/internal/daemon/hooks"
-	"github.com/zrepl/zrepl/internal/daemon/logging"
-	"github.com/zrepl/zrepl/internal/logger"
-	"github.com/zrepl/zrepl/internal/zfs"
+	"github.com/zrepl/zrepl/config"
+	"github.com/zrepl/zrepl/daemon/hooks"
+	"github.com/zrepl/zrepl/daemon/logging"
+	"github.com/zrepl/zrepl/logger"
+	"github.com/zrepl/zrepl/zfs"
 )

 type comparisonAssertionFunc func(require.TestingT, interface{}, interface{}, ...interface{})
@@ -161,7 +161,7 @@ jobs:
 					ExpectedEdge: hooks.Pre,
 					ExpectStatus: hooks.StepErr,
 					OutputTest:   containsTest(fmt.Sprintf("TEST ERROR pre_testing %s@%s", testFSName, testSnapshotName)),
-					ErrorTest:    regexpTest("^command hook failed.*exit status 1$"),
+					ErrorTest:    regexpTest("^command hook invocation.*exit status 1$"),
 				},
 				expectStep{ExpectedEdge: hooks.Callback, ExpectStatus: hooks.StepOk},
 				expectStep{
@@ -185,7 +185,7 @@ jobs:
 					ExpectedEdge: hooks.Pre,
 					ExpectStatus: hooks.StepErr,
 					OutputTest:   containsTest(fmt.Sprintf("TEST ERROR pre_testing %s@%s", testFSName, testSnapshotName)),
-					ErrorTest:    regexpTest("^command hook failed.*exit status 1$"),
+					ErrorTest:    regexpTest("^command hook invocation.*exit status 1$"),
 				},
 				expectStep{
 					ExpectedEdge: hooks.Pre,
@@ -234,7 +234,7 @@ jobs:
 					ExpectedEdge: hooks.Post,
 					ExpectStatus: hooks.StepErr,
 					OutputTest:   containsTest(fmt.Sprintf("TEST ERROR post_testing %s@%s", testFSName, testSnapshotName)),
-					ErrorTest:    regexpTest("^command hook failed.*exit status 1$"),
+					ErrorTest:    regexpTest("^command hook invocation.*exit status 1$"),
 				},
 			},
 		},
@@ -267,7 +267,7 @@ jobs:
 					ExpectedEdge: hooks.Pre,
 					ExpectStatus: hooks.StepErr,
 					OutputTest:   containsTest(fmt.Sprintf("TEST ERROR pre_testing %s@%s", testFSName, testSnapshotName)),
-					ErrorTest:    regexpTest("^command hook failed.*exit status 1$"),
+					ErrorTest:    regexpTest("^command hook invocation.*exit status 1$"),
 				},
 				expectStep{ExpectedEdge: hooks.Callback, ExpectStatus: hooks.StepOk},
 				expectStep{
@@ -295,7 +295,7 @@ jobs:
 					ExpectedEdge: hooks.Pre,
 					ExpectStatus: hooks.StepErr,
 					OutputTest:   containsTest(fmt.Sprintf("TEST ERROR pre_testing %s@%s", testFSName, testSnapshotName)),
-					ErrorTest:    regexpTest("^command hook failed.*exit status 1$"),
+					ErrorTest:    regexpTest("^command hook invocation.*exit status 1$"),
 				},
 				expectStep{ExpectedEdge: hooks.Callback, ExpectStatus: hooks.StepOk},
 				expectStep{
@@ -1,5 +1,6 @@
 // Code generated by "enumer -type=StepStatus -trimprefix=Step"; DO NOT EDIT.

+//
 package hooks

 import (
@@ -8,24 +8,25 @@ import (

 	"github.com/pkg/errors"
 	"github.com/prometheus/client_golang/prometheus"
+	"github.com/prometheus/common/log"

-	"github.com/zrepl/zrepl/internal/daemon/logging/trace"
-	"github.com/zrepl/zrepl/internal/util/envconst"
+	"github.com/zrepl/zrepl/daemon/logging/trace"
+	"github.com/zrepl/zrepl/util/envconst"

-	"github.com/zrepl/zrepl/internal/config"
-	"github.com/zrepl/zrepl/internal/daemon/job/reset"
-	"github.com/zrepl/zrepl/internal/daemon/job/wakeup"
-	"github.com/zrepl/zrepl/internal/daemon/pruner"
-	"github.com/zrepl/zrepl/internal/daemon/snapper"
-	"github.com/zrepl/zrepl/internal/endpoint"
-	"github.com/zrepl/zrepl/internal/replication"
-	"github.com/zrepl/zrepl/internal/replication/driver"
-	"github.com/zrepl/zrepl/internal/replication/logic"
-	"github.com/zrepl/zrepl/internal/replication/report"
-	"github.com/zrepl/zrepl/internal/rpc"
-	"github.com/zrepl/zrepl/internal/transport"
-	"github.com/zrepl/zrepl/internal/transport/fromconfig"
-	"github.com/zrepl/zrepl/internal/zfs"
+	"github.com/zrepl/zrepl/config"
+	"github.com/zrepl/zrepl/daemon/job/reset"
+	"github.com/zrepl/zrepl/daemon/job/wakeup"
+	"github.com/zrepl/zrepl/daemon/pruner"
+	"github.com/zrepl/zrepl/daemon/snapper"
+	"github.com/zrepl/zrepl/endpoint"
+	"github.com/zrepl/zrepl/replication"
+	"github.com/zrepl/zrepl/replication/driver"
+	"github.com/zrepl/zrepl/replication/logic"
+	"github.com/zrepl/zrepl/replication/report"
+	"github.com/zrepl/zrepl/rpc"
+	"github.com/zrepl/zrepl/transport"
+	"github.com/zrepl/zrepl/transport/fromconfig"
+	"github.com/zrepl/zrepl/zfs"
 )

 type ActiveSide struct {
@@ -41,7 +42,6 @@ type ActiveSide struct {
 	promPruneSecs         *prometheus.HistogramVec // labels: prune_side
 	promBytesReplicated   *prometheus.CounterVec   // labels: filesystem
 	promReplicationErrors prometheus.Gauge
-	promLastSuccessful    prometheus.Gauge

 	tasksMtx sync.Mutex
 	tasks    activeSideTasks
@@ -100,7 +100,7 @@ type modePush struct {
 	receiver      *rpc.Client
 	senderConfig  *endpoint.SenderConfig
 	plannerPolicy *logic.PlannerPolicy
-	snapper       snapper.Snapper
+	snapper       *snapper.PeriodicOrManual
 }

 func (m *modePush) ConnectEndpoints(ctx context.Context, connecter transport.Connecter) {
@@ -136,8 +136,7 @@ func (m *modePush) RunPeriodic(ctx context.Context, wakeUpCommon chan<- struct{}
 }

 func (m *modePush) SnapperReport() *snapper.Report {
-	r := m.snapper.Report()
-	return &r
+	return m.snapper.Report()
 }

 func (m *modePush) ResetConnectBackoff() {
@@ -162,13 +161,8 @@ func modePushFromConfig(g *config.Global, in *config.PushJob, jobID endpoint.Job
 		return nil, errors.Wrap(err, "field `replication`")
 	}

-	conflictResolution, err := logic.ConflictResolutionFromConfig(in.ConflictResolution)
-	if err != nil {
-		return nil, errors.Wrap(err, "field `conflict_resolution`")
-	}
-
 	m.plannerPolicy = &logic.PlannerPolicy{
-		ConflictResolution:        conflictResolution,
+		EncryptedSend:             logic.TriFromBool(in.Send.Encrypted),
 		ReplicationConfig:         replicationConfig,
 		SizeEstimationConcurrency: in.Replication.Concurrency.SizeEstimates,
 	}
@@ -266,13 +260,8 @@ func modePullFromConfig(g *config.Global, in *config.PullJob, jobID endpoint.Job
 		return nil, errors.Wrap(err, "field `replication`")
 	}

-	conflictResolution, err := logic.ConflictResolutionFromConfig(in.ConflictResolution)
-	if err != nil {
-		return nil, errors.Wrap(err, "field `conflict_resolution`")
-	}
-
 	m.plannerPolicy = &logic.PlannerPolicy{
-		ConflictResolution:        conflictResolution,
+		EncryptedSend:             logic.DontCare,
 		ReplicationConfig:         replicationConfig,
 		SizeEstimationConcurrency: in.Replication.Concurrency.SizeEstimates,
 	}
@@ -298,7 +287,7 @@ func replicationDriverConfigFromConfig(in *config.Replication) (c driver.Config,
 	return c, err
 }

-func activeSide(g *config.Global, in *config.ActiveJob, configJob interface{}, parseFlags config.ParseFlags) (j *ActiveSide, err error) {
+func activeSide(g *config.Global, in *config.ActiveJob, configJob interface{}) (j *ActiveSide, err error) {

 	j = &ActiveSide{}
 	j.name, err = endpoint.MakeJobID(in.Name)
@@ -332,6 +321,7 @@ func activeSide(g *config.Global, in *config.ActiveJob, configJob interface{}, p
 		Help:        "number of bytes replicated from sender to receiver per filesystem",
 		ConstLabels: prometheus.Labels{"zrepl_job": j.name.String()},
 	}, []string{"filesystem"})
+
 	j.promReplicationErrors = prometheus.NewGauge(prometheus.GaugeOpts{
 		Namespace:   "zrepl",
 		Subsystem:   "replication",
@@ -339,15 +329,8 @@ func activeSide(g *config.Global, in *config.ActiveJob, configJob interface{}, p
 		Help:        "number of filesystems that failed replication in the latest replication attempt, or -1 if the job failed before enumerating the filesystems",
 		ConstLabels: prometheus.Labels{"zrepl_job": j.name.String()},
 	})
-	j.promLastSuccessful = prometheus.NewGauge(prometheus.GaugeOpts{
-		Namespace:   "zrepl",
-		Subsystem:   "replication",
-		Name:        "last_successful",
-		Help:        "timestamp of last successful replication",
-		ConstLabels: prometheus.Labels{"zrepl_job": j.name.String()},
-	})

-	j.connecter, err = fromconfig.ConnecterFromConfig(g, in.Connect, parseFlags)
+	j.connecter, err = fromconfig.ConnecterFromConfig(g, in.Connect)
 	if err != nil {
 		return nil, errors.Wrap(err, "cannot build client")
 	}
@@ -377,7 +360,6 @@ func (j *ActiveSide) RegisterMetrics(registerer prometheus.Registerer) {
 	registerer.MustRegister(j.promPruneSecs)
 	registerer.MustRegister(j.promBytesReplicated)
 	registerer.MustRegister(j.promReplicationErrors)
-	registerer.MustRegister(j.promLastSuccessful)
 }

 func (j *ActiveSide) Name() string { return j.name.String() }
@@ -481,7 +463,7 @@ func (j *ActiveSide) do(ctx context.Context) {
 	go func() {
 		select {
 		case <-reset.Wait(ctx):
-			GetLogger(ctx).Info("reset received, cancelling current invocation")
+			log.Info("reset received, cancelling current invocation")
 			cancelThisRun()
 		case <-ctx.Done():
 		}
@@ -512,11 +494,7 @@ func (j *ActiveSide) do(ctx context.Context) {
 		repCancel()   // always cancel to free up context resources

 		replicationReport := j.tasks.replicationReport()
-		var numErrors = replicationReport.GetFailedFilesystemsCountInLatestAttempt()
-		j.promReplicationErrors.Set(float64(numErrors))
-		if numErrors == 0 {
-			j.promLastSuccessful.SetToCurrentTime()
-		}
+		j.promReplicationErrors.Set(float64(replicationReport.GetFailedFilesystemsCountInLatestAttempt()))

 		endSpan()
 	}
@@ -6,8 +6,8 @@ import (
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"

-	"github.com/zrepl/zrepl/internal/endpoint"
-	"github.com/zrepl/zrepl/internal/transport"
+	"github.com/zrepl/zrepl/endpoint"
+	"github.com/zrepl/zrepl/transport"
 )

 func TestFakeActiveSideDirectMethodInvocationClientIdentityDoesNotPassValidityTest(t *testing.T) {
@@ -1,5 +1,6 @@
 // Code generated by "enumer -type=ActiveSideState"; DO NOT EDIT.

+//
 package job

 import (
@@ -7,14 +7,13 @@ import (

 	"github.com/pkg/errors"

-	"github.com/zrepl/zrepl/internal/config"
-	"github.com/zrepl/zrepl/internal/util/bandwidthlimit"
+	"github.com/zrepl/zrepl/config"
 )

-func JobsFromConfig(c *config.Config, parseFlags config.ParseFlags) ([]Job, error) {
+func JobsFromConfig(c *config.Config) ([]Job, error) {
 	js := make([]Job, len(c.Jobs))
 	for i := range c.Jobs {
-		j, err := buildJob(c.Global, c.Jobs[i], parseFlags)
+		j, err := buildJob(c.Global, c.Jobs[i])
 		if err != nil {
 			return nil, err
 		}
@@ -24,22 +23,37 @@ func JobsFromConfig(c *config.Config, parseFlags config.ParseFlags) ([]Job, erro
 		js[i] = j
 	}

+	// receiving-side root filesystems must not overlap
+	{
+		rfss := make([]string, 0, len(js))
+		for _, j := range js {
+			jrfs, ok := j.OwnedDatasetSubtreeRoot()
+			if !ok {
+				continue
+			}
+			rfss = append(rfss, jrfs.ToString())
+		}
+		if err := validateReceivingSidesDoNotOverlap(rfss); err != nil {
+			return nil, err
+		}
+	}
+
 	return js, nil
 }

-func buildJob(c *config.Global, in config.JobEnum, parseFlags config.ParseFlags) (j Job, err error) {
+func buildJob(c *config.Global, in config.JobEnum) (j Job, err error) {
 	cannotBuildJob := func(e error, name string) (Job, error) {
 		return nil, errors.Wrapf(e, "cannot build job %q", name)
 	}
 	// FIXME prettify this
 	switch v := in.Ret.(type) {
 	case *config.SinkJob:
-		j, err = passiveSideFromConfig(c, &v.PassiveJob, v, parseFlags)
+		j, err = passiveSideFromConfig(c, &v.PassiveJob, v)
 		if err != nil {
 			return cannotBuildJob(err, v.Name)
 		}
 	case *config.SourceJob:
-		j, err = passiveSideFromConfig(c, &v.PassiveJob, v, parseFlags)
+		j, err = passiveSideFromConfig(c, &v.PassiveJob, v)
 		if err != nil {
 			return cannotBuildJob(err, v.Name)
 		}
@@ -49,12 +63,12 @@ func buildJob(c *config.Global, in config.JobEnum, parseFlags config.ParseFlags)
 			return cannotBuildJob(err, v.Name)
 		}
 	case *config.PushJob:
-		j, err = activeSide(c, &v.ActiveJob, v, parseFlags)
+		j, err = activeSide(c, &v.ActiveJob, v)
 		if err != nil {
 			return cannotBuildJob(err, v.Name)
 		}
 	case *config.PullJob:
-		j, err = activeSide(c, &v.ActiveJob, v, parseFlags)
+		j, err = activeSide(c, &v.ActiveJob, v)
 		if err != nil {
 			return cannotBuildJob(err, v.Name)
 		}
@@ -93,13 +107,3 @@ func validateReceivingSidesDoNotOverlap(receivingRootFSs []string) error {
 	}
 	return nil
 }
-
-func buildBandwidthLimitConfig(in *config.BandwidthLimit) (c bandwidthlimit.Config, _ error) {
-	if in.Max.ToBytes() > 0 && int64(in.Max.ToBytes()) == 0 {
-		return c, fmt.Errorf("bandwidth limit `max` is too small, must at least specify one byte")
-	}
-	return bandwidthlimit.Config{
-		Max:            int64(in.Max.ToBytes()),
-		BucketCapacity: int64(in.BucketCapacity.ToBytes()),
-	}, nil
-}
@@ -3,11 +3,11 @@ package job
 import (
 	"github.com/pkg/errors"

-	"github.com/zrepl/zrepl/internal/config"
-	"github.com/zrepl/zrepl/internal/daemon/filters"
-	"github.com/zrepl/zrepl/internal/endpoint"
-	"github.com/zrepl/zrepl/internal/util/nodefault"
-	"github.com/zrepl/zrepl/internal/zfs"
+	"github.com/zrepl/zrepl/config"
+	"github.com/zrepl/zrepl/daemon/filters"
+	"github.com/zrepl/zrepl/endpoint"
+	"github.com/zrepl/zrepl/util/nodefault"
+	"github.com/zrepl/zrepl/zfs"
 )

 type SendingJobConfig interface {
@@ -22,12 +22,7 @@ func buildSenderConfig(in SendingJobConfig, jobID endpoint.JobID) (*endpoint.Sen
 		return nil, errors.Wrap(err, "cannot build filesystem filter")
 	}
 	sendOpts := in.GetSendOptions()
-	bwlim, err := buildBandwidthLimitConfig(sendOpts.BandwidthLimit)
-	if err != nil {
-		return nil, errors.Wrap(err, "cannot build bandwith limit config")
-	}
-
-	sc := &endpoint.SenderConfig{
+	return &endpoint.SenderConfig{
 		FSF:   fsf,
 		JobID: jobID,

@@ -39,15 +34,7 @@ func buildSenderConfig(in SendingJobConfig, jobID endpoint.JobID) (*endpoint.Sen
 		SendCompressed:       sendOpts.Compressed,
 		SendEmbeddedData:     sendOpts.EmbeddedData,
 		SendSaved:            sendOpts.Saved,
-
-		BandwidthLimit: bwlim,
-	}
-
-	if err := sc.Validate(); err != nil {
-		return nil, errors.Wrap(err, "cannot build sender config")
-	}
-
-	return sc, nil
+	}, nil
 }

 type ReceivingJobConfig interface {
@@ -66,22 +53,6 @@ func buildReceiverConfig(in ReceivingJobConfig, jobID endpoint.JobID) (rc endpoi
 	}

 	recvOpts := in.GetRecvOptions()
-
-	bwlim, err := buildBandwidthLimitConfig(recvOpts.BandwidthLimit)
-	if err != nil {
-		return rc, errors.Wrap(err, "cannot build bandwith limit config")
-	}
-
-	placeholderEncryption, err := endpoint.PlaceholderCreationEncryptionPropertyString(recvOpts.Placeholder.Encryption)
-	if err != nil {
-		options := []string{}
-		for _, v := range endpoint.PlaceholderCreationEncryptionPropertyValues() {
-			options = append(options, endpoint.PlaceholderCreationEncryptionProperty(v).String())
-		}
-		return rc, errors.Errorf("placeholder encryption value %q is invalid, must be one of %s",
-			recvOpts.Placeholder.Encryption, options)
-	}
-
 	rc = endpoint.ReceiverConfig{
 		JobID:                      jobID,
 		RootWithoutClientComponent: rootFs,
@@ -89,10 +60,6 @@ func buildReceiverConfig(in ReceivingJobConfig, jobID endpoint.JobID) (rc endpoi

 		InheritProperties:  recvOpts.Properties.Inherit,
 		OverrideProperties: recvOpts.Properties.Override,
-
-		BandwidthLimit: bwlim,
-
-		PlaceholderEncryption: placeholderEncryption,
 	}
 	if err := rc.Validate(); err != nil {
 		return rc, errors.Wrap(err, "cannot build receiver config")
@@ -10,7 +10,8 @@ import (
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"

-	"github.com/zrepl/zrepl/internal/config"
+	"github.com/zrepl/zrepl/config"
+	"github.com/zrepl/zrepl/transport/tls"
 )

 func TestValidateReceivingSidesDoNotOverlap(t *testing.T) {
@@ -95,7 +96,7 @@ jobs:
 			conf, err := config.ParseConfigBytes([]byte(fill(c.jobName)))
 			require.NoError(t, err, "not expecting yaml-config to know about job ids")
 			require.NotNil(t, conf)
-			jobs, err := JobsFromConfig(conf, config.ParseFlagsNone)
+			jobs, err := JobsFromConfig(conf)

 			if c.valid {
 				assert.NoError(t, err)
@@ -113,19 +114,11 @@ jobs:
 }

 func TestSampleConfigsAreBuiltWithoutErrors(t *testing.T) {
-	paths, err := filepath.Glob("../../../internal/config/samples/*")
+	paths, err := filepath.Glob("../../config/samples/*")
 	if err != nil {
 		t.Errorf("glob failed: %+v", err)
 	}

-	type additionalCheck struct {
-		state int
-		test  func(t *testing.T, jobs []Job)
-	}
-	additionalChecks := map[string]*additionalCheck{
-		"bandwidth_limit.yml": {test: testSampleConfig_BandwidthLimit},
-	}
-
 	for _, p := range paths {

 		if path.Ext(p) != ".yml" {
@@ -133,78 +126,23 @@ func TestSampleConfigsAreBuiltWithoutErrors(t *testing.T) {
 			continue
 		}

-		filename := path.Base(p)
-		t.Logf("checking for presence additonal checks for file %q", filename)
-		additionalCheck := additionalChecks[filename]
-		if additionalCheck == nil {
-			t.Logf("no additional checks")
-		} else {
-			t.Logf("additional check present")
-			additionalCheck.state = 1
-		}
-
 		t.Run(p, func(t *testing.T) {
 			c, err := config.ParseConfig(p)
 			if err != nil {
-				t.Fatalf("error parsing %s:\n%+v", p, err)
+				t.Errorf("error parsing %s:\n%+v", p, err)
 			}

 			t.Logf("file: %s", p)
 			t.Log(pretty.Sprint(c))

-			jobs, err := JobsFromConfig(c, config.ParseFlagsNoCertCheck)
+			tls.FakeCertificateLoading(t)
+			jobs, err := JobsFromConfig(c)
 			t.Logf("jobs: %#v", jobs)
-			require.NoError(t, err)
-
-			if additionalCheck != nil {
-				additionalCheck.test(t, jobs)
-				additionalCheck.state = 2
-			}
+			assert.NoError(t, err)
 		})

 	}

-	for basename, c := range additionalChecks {
-		if c.state == 0 {
-			panic("univisited additional check " + basename)
-		}
-	}
-
-}
-
-func testSampleConfig_BandwidthLimit(t *testing.T, jobs []Job) {
-	require.Len(t, jobs, 3)
-
-	{
-		limitedSink, ok := jobs[0].(*PassiveSide)
-		require.True(t, ok, "%T", jobs[0])
-		limitedSinkMode, ok := limitedSink.mode.(*modeSink)
-		require.True(t, ok, "%T", limitedSink)
-
-		assert.Equal(t, int64(12345), limitedSinkMode.receiverConfig.BandwidthLimit.Max)
-		assert.Equal(t, int64(1<<17), limitedSinkMode.receiverConfig.BandwidthLimit.BucketCapacity)
-	}
-
-	{
-		limitedPush, ok := jobs[1].(*ActiveSide)
-		require.True(t, ok, "%T", jobs[1])
-		limitedPushMode, ok := limitedPush.mode.(*modePush)
-		require.True(t, ok, "%T", limitedPush)
-
-		assert.Equal(t, int64(54321), limitedPushMode.senderConfig.BandwidthLimit.Max)
-		assert.Equal(t, int64(1024), limitedPushMode.senderConfig.BandwidthLimit.BucketCapacity)
-	}
-
-	{
-		unlimitedSink, ok := jobs[2].(*PassiveSide)
-		require.True(t, ok, "%T", jobs[2])
-		unlimitedSinkMode, ok := unlimitedSink.mode.(*modeSink)
-		require.True(t, ok, "%T", unlimitedSink)
-
-		max := unlimitedSinkMode.receiverConfig.BandwidthLimit.Max
-		assert.Less(t, max, int64(0), max, "unlimited mode <=> negative value for .Max, see bandwidthlimit.Config")
-	}
-
 }

 func TestReplicationOptions(t *testing.T) {
@@ -297,7 +235,7 @@ jobs:
 			t.Logf("testing config:\n%s", cstr)
 			c, err := config.ParseConfigBytes([]byte(cstr))
 			require.NoError(t, err)
-			jobs, err := JobsFromConfig(c, config.ParseFlagsNone)
+			jobs, err := JobsFromConfig(c)
 			if ts.expectOk != nil {
 				require.NoError(t, err)
 				require.NotNil(t, c)
@@ -7,10 +7,10 @@ import (

 	"github.com/prometheus/client_golang/prometheus"

-	"github.com/zrepl/zrepl/internal/daemon/logging"
-	"github.com/zrepl/zrepl/internal/endpoint"
-	"github.com/zrepl/zrepl/internal/logger"
-	"github.com/zrepl/zrepl/internal/zfs"
+	"github.com/zrepl/zrepl/daemon/logging"
+	"github.com/zrepl/zrepl/endpoint"
+	"github.com/zrepl/zrepl/logger"
+	"github.com/zrepl/zrepl/zfs"
 )

 type Logger = logger.Logger
@@ -7,16 +7,16 @@ import (
 	"github.com/pkg/errors"
 	"github.com/prometheus/client_golang/prometheus"

-	"github.com/zrepl/zrepl/internal/daemon/logging/trace"
+	"github.com/zrepl/zrepl/daemon/logging/trace"

-	"github.com/zrepl/zrepl/internal/config"
-	"github.com/zrepl/zrepl/internal/daemon/logging"
-	"github.com/zrepl/zrepl/internal/daemon/snapper"
-	"github.com/zrepl/zrepl/internal/endpoint"
-	"github.com/zrepl/zrepl/internal/rpc"
-	"github.com/zrepl/zrepl/internal/transport"
-	"github.com/zrepl/zrepl/internal/transport/fromconfig"
-	"github.com/zrepl/zrepl/internal/zfs"
+	"github.com/zrepl/zrepl/config"
+	"github.com/zrepl/zrepl/daemon/logging"
+	"github.com/zrepl/zrepl/daemon/snapper"
+	"github.com/zrepl/zrepl/endpoint"
+	"github.com/zrepl/zrepl/rpc"
+	"github.com/zrepl/zrepl/transport"
+	"github.com/zrepl/zrepl/transport/fromconfig"
+	"github.com/zrepl/zrepl/zfs"
 )

 type PassiveSide struct {
@@ -58,7 +58,7 @@ func modeSinkFromConfig(g *config.Global, in *config.SinkJob, jobID endpoint.Job

 type modeSource struct {
 	senderConfig *endpoint.SenderConfig
-	snapper      snapper.Snapper
+	snapper      *snapper.PeriodicOrManual
 }

 func modeSourceFromConfig(g *config.Global, in *config.SourceJob, jobID endpoint.JobID) (m *modeSource, err error) {
@@ -88,11 +88,10 @@ func (m *modeSource) RunPeriodic(ctx context.Context) {
 }

 func (m *modeSource) SnapperReport() *snapper.Report {
-	r := m.snapper.Report()
-	return &r
+	return m.snapper.Report()
 }

-func passiveSideFromConfig(g *config.Global, in *config.PassiveJob, configJob interface{}, parseFlags config.ParseFlags) (s *PassiveSide, err error) {
+func passiveSideFromConfig(g *config.Global, in *config.PassiveJob, configJob interface{}) (s *PassiveSide, err error) {

 	s = &PassiveSide{}

@@ -111,7 +110,7 @@ func passiveSideFromConfig(g *config.Global, in *config.PassiveJob, configJob in
 		return nil, err // no wrapping necessary
 	}

-	if s.listen, err = fromconfig.ListenerFactoryFromConfig(g, in.Serve, parseFlags); err != nil {
+	if s.listen, err = fromconfig.ListenerFactoryFromConfig(g, in.Serve); err != nil {
 		return nil, errors.Wrap(err, "cannot build listener factory")
 	}

@@ -9,24 +9,23 @@ import (
 	"github.com/pkg/errors"
 	"github.com/prometheus/client_golang/prometheus"

-	"github.com/zrepl/zrepl/internal/daemon/logging/trace"
-	"github.com/zrepl/zrepl/internal/util/bandwidthlimit"
-	"github.com/zrepl/zrepl/internal/util/nodefault"
+	"github.com/zrepl/zrepl/daemon/logging/trace"
+	"github.com/zrepl/zrepl/util/nodefault"

-	"github.com/zrepl/zrepl/internal/config"
-	"github.com/zrepl/zrepl/internal/daemon/filters"
-	"github.com/zrepl/zrepl/internal/daemon/job/wakeup"
-	"github.com/zrepl/zrepl/internal/daemon/pruner"
-	"github.com/zrepl/zrepl/internal/daemon/snapper"
-	"github.com/zrepl/zrepl/internal/endpoint"
-	"github.com/zrepl/zrepl/internal/replication/logic/pdu"
-	"github.com/zrepl/zrepl/internal/zfs"
+	"github.com/zrepl/zrepl/config"
+	"github.com/zrepl/zrepl/daemon/filters"
+	"github.com/zrepl/zrepl/daemon/job/wakeup"
+	"github.com/zrepl/zrepl/daemon/pruner"
+	"github.com/zrepl/zrepl/daemon/snapper"
+	"github.com/zrepl/zrepl/endpoint"
+	"github.com/zrepl/zrepl/replication/logic/pdu"
+	"github.com/zrepl/zrepl/zfs"
 )

 type SnapJob struct {
 	name     endpoint.JobID
 	fsfilter zfs.DatasetFilter
-	snapper  snapper.Snapper
+	snapper  *snapper.PeriodicOrManual

 	prunerFactory *pruner.LocalPrunerFactory

@@ -86,8 +85,7 @@ func (j *SnapJob) Status() *Status {
 		s.Pruning = j.pruner.Report()
 	}
 	j.prunerMtx.Unlock()
-	r := j.snapper.Report()
-	s.Snapshotting = &r
+	s.Snapshotting = j.snapper.Report()
 	return &Status{Type: t, JobSpecific: s}
 }

@@ -138,7 +136,7 @@ outer:
 // TODO:
 // This is a work-around for the current package daemon/pruner
 // and package pruning.Snapshot limitation: they require the
-// `Replicated` getter method be present, but obviously,
+//  `Replicated` getter method be present, but obviously,
 // a local job like SnapJob can't deliver on that.
 // But the pruner.Pruner gives up on an FS if no replication
 // cursor is present, which is why this pruner returns the
@@ -148,7 +146,7 @@ type alwaysUpToDateReplicationCursorHistory struct {
 	target pruner.Target
 }

-var _ pruner.Sender = (*alwaysUpToDateReplicationCursorHistory)(nil)
+var _ pruner.History = (*alwaysUpToDateReplicationCursorHistory)(nil)

 func (h alwaysUpToDateReplicationCursorHistory) ReplicationCursor(ctx context.Context, req *pdu.ReplicationCursorReq) (*pdu.ReplicationCursorRes, error) {
 	fsvReq := &pdu.ListFilesystemVersionsReq{
@@ -181,11 +179,8 @@ func (j *SnapJob) doPrune(ctx context.Context) {
 	sender := endpoint.NewSender(endpoint.SenderConfig{
 		JobID: j.name,
 		FSF:   j.fsfilter,
-		// FIXME the following config fields are irrelevant for SnapJob
-		// because the endpoint is only used as pruner.Target.
-		// However, the implementation requires them to be set.
-		Encrypt:        &nodefault.Bool{B: true},
-		BandwidthLimit: bandwidthlimit.NoLimitConfig(),
+		// FIXME encryption setting is irrelevant for SnapJob because the endpoint is only used as pruner.Target
+		Encrypt: &nodefault.Bool{B: true},
 	})
 	j.prunerMtx.Lock()
 	j.pruner = j.prunerFactory.BuildLocalPruner(ctx, sender, alwaysUpToDateReplicationCursorHistory{sender})
@@ -10,10 +10,10 @@ import (
 	"github.com/mattn/go-isatty"
 	"github.com/pkg/errors"

-	"github.com/zrepl/zrepl/internal/config"
-	"github.com/zrepl/zrepl/internal/daemon/logging/trace"
-	"github.com/zrepl/zrepl/internal/logger"
-	"github.com/zrepl/zrepl/internal/tlsconf"
+	"github.com/zrepl/zrepl/config"
+	"github.com/zrepl/zrepl/daemon/logging/trace"
+	"github.com/zrepl/zrepl/logger"
+	"github.com/zrepl/zrepl/tlsconf"
 )

 func OutletsFromConfig(in config.LoggingOutletEnumList) (*logger.Outlets, error) {
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Christian Schwarz	226962b4fb	zfs: rewrite SendStream, fix bug in Close() on FreeBSD, add platformtests This commit was motivated by https://github.com/zrepl/zrepl/issues/495 where, on FreeBSD with OpenZFS 2.0, a SendStream.Close() call might wait indefinitely for `zfs send` to exit. The reason is that, due to the refactoring done for redacted send & recv (https://github.com/openzfs/zfs/commit/30af21b02569ac192f52ce6e6511015f8a8d5729), the `dump_bytes` function, which writes to the pipe, executes in a separate thread (synctask taskq) iff not `HAVE_LARGE_STACKS`. The `zfs send` process/thread waits for that taskq thread using an uninterruptible primitive. So when we SIGKILL `zfs send`, that signal doesn't reach the right thread to interrupt the pipe write. Theoretically this affects both Linux and FreeBSD, but most Linux users `HAVE_LARGE_STACKS` and since https://github.com/penzfs/zfs/pull/12350/files OpenZFS on FreeBSD `HAVE_LARGE_STACKS` as well. However, at least until FreeBSD 13.1, possibly for the entire 13 lifecycle, we're going to have to live with that oddity. Measures taken in this commit: - Report the behavior as an upstream bug https://github.com/openzfs/zfs/issues/12500 - Change SendStream code so that it closes zrepl's read-end of the pipe (see comment in code) - Clean up and make explicit SendStream's state handling - Write extensive platformtests for SendStream - They pass on my Linux install and on FreeBSD 12 - FreeBSD 13 still needs testing. fixes https://github.com/zrepl/zrepl/issues/495	2021-08-23 00:21:34 +02:00
Christian Schwarz	ff6ebe87f3	platformtest: fix 'active child tasks' panic for ReceiveForceRollbackWorksUnencrypted Revealed by rework of SendStream in a prior commit.	2021-08-23 00:21:34 +02:00
Christian Schwarz	126874b423	platformtest: fix replication tests (SizeEstimationConcurrency field in PlannerPolicy was not set) fixup of `0ceea1b792` (parallel replication knobs)	2021-08-23 00:21:34 +02:00
Christian Schwarz	157753d781	platformtest: work around missing feature detection for test 'ReplicationPropertyReplicationWorks'	2021-08-23 00:21:34 +02:00