daemon/active: implement watchdog to handle stuck replication / pruners
ActiveSide.do() can only run sequentially, i.e. we cannot run replication and pruning in parallel. Why? * go-streamrpc only allows one active request at a time (this is bad design and should be fixed at some point) * replication and pruning are implemented independently, but work on the same resources (snapshots) A: pruning might destroy a snapshot that is planned to be replicated B: replication might replicate snapshots that should be pruned We do not have any resource management / locking for A and B, but we have a use case where users don't want their machine fill up with snapshots if replication does not work. That means we _have_ to run the pruners. A further complication is that we cannot just cancel the replication context after a timeout and move on to the pruner: it could be initial replication and we don't know how long it will take. (And we don't have resumable send & recv yet). With the previous commits, we can implement the watchdog using context cancellation. Note that the 'MadeProgress()' calls can only be placed right before non-error state transition. Otherwise, we could end up in a live-lock.
This commit is contained in:
+1
-2
@@ -274,8 +274,7 @@ func (t *tui) renderReplicationReport(rep *replication.Report) {
|
||||
t.printf("Problem: %s", rep.Problem)
|
||||
t.newline()
|
||||
}
|
||||
if rep.SleepUntil.After(time.Now()) &&
|
||||
state & ^(replication.PermanentError|replication.Completed) != 0 {
|
||||
if rep.SleepUntil.After(time.Now()) && !state.IsTerminal() {
|
||||
t.printf("Sleeping until %s (%s left)\n", rep.SleepUntil, rep.SleepUntil.Sub(time.Now()))
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user