Lying in bed I came up with an idea that might "solve" the issue with backup
delegates producing forks accidentally.
TL;DR; use the update_active_key feature on failure of the master node
so basically, the delegate signs blocks with its active key and not the owner
key (TODO: check with BM about this).
I assume you have two machines with the delegates owner key imported. both are
in sync regarding the "list of active keys" and the corresponding private keys.
Now, when the master note is not reachable by the backup node anymore (due to
unkown reasons) but you cannot be sure that the master node is maybe still
active and connected to the network, you could end up with two delegates
signing a block which forks the network. This is status quo and the biggest
problems for the backup strategy.
Now I think we can solve this by the following:
1) the backup node thinks the master node is offline
2) the backup node creates a new key active key for the delegate and enables block production
this key can be produced in a way that the master node has no chance of figuring it out.
3) thus, the master node is UNABLE to continue producing blocks. Its simply not
allowed to produce blocks with an obsolete key (don't ask me what the master
node does in this situation, maybe crash)
4) the end result however is a backup machine that solely is allowed to produce
This can be extended to a set of backup machines by scanning all transactions
which change the active key of a particular delegate. Different backup
delegates can jump in on different delays if any node before it "fails to
deliver" a valid block. (reminds me of RAFT)