daemon/active: implement watchdog to handle stuck replication / pruners
ActiveSide.do() can only run sequentially, i.e. we cannot run replication and pruning in parallel. Why? * go-streamrpc only allows one active request at a time (this is bad design and should be fixed at some point) * replication and pruning are implemented independently, but work on the same resources (snapshots) A: pruning might destroy a snapshot that is planned to be replicated B: replication might replicate snapshots that should be pruned We do not have any resource management / locking for A and B, but we have a use case where users don't want their machine fill up with snapshots if replication does not work. That means we _have_ to run the pruners. A further complication is that we cannot just cancel the replication context after a timeout and move on to the pruner: it could be initial replication and we don't know how long it will take. (And we don't have resumable send & recv yet). With the previous commits, we can implement the watchdog using context cancellation. Note that the 'MadeProgress()' calls can only be placed right before non-error state transition. Otherwise, we could end up in a live-lock.
This commit is contained in:
@@ -0,0 +1,25 @@
|
||||
package envconst
|
||||
|
||||
import (
|
||||
"os"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
var cache sync.Map
|
||||
|
||||
func Duration(varname string, def time.Duration) time.Duration {
|
||||
if v, ok := cache.Load(varname); ok {
|
||||
return v.(time.Duration)
|
||||
}
|
||||
e := os.Getenv(varname)
|
||||
if e == "" {
|
||||
return def
|
||||
}
|
||||
d, err := time.ParseDuration(e)
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
cache.Store(varname, d)
|
||||
return d
|
||||
}
|
||||
Reference in New Issue
Block a user