What is the best practice for not missing any block during an update?
The update_witness command in CLI wallet allows you to change your witness's block signing key. This architecture allows several different downtime-free update procedures according to your specific hosting situation:
(1) Create new witness on new server with new block signing key (you can use suggest_brain_key in CLI wallet, or the get_dev_key binary, to generate a new key). When the new witness is synced, run update_witness to change your key, shutdown old witness and delete old server. Good method if you use a pay-by-the-hour hosting provider that lets you quickly create and destroy servers (e.g. DigitalOcean).
(2) Create new witness on the same server with new block signing key in different datadir. When new witness is synced, update_witness to change your block signing key and shut down old witness. Good method if you have a large server capable of running two witnesses at once (e.g. dedicated host).
(3) Create temporary witness on your own personal machine. When it is synced, update_witness to change your block signing key and cause the temporary witness to start producing. Then shut down the old witness and spin up the new witness on the same server; it is okay if it takes some time, because your temporary witness is still signing blocks. Then when new witness is ready, run update_witness again to change your block signing key back (or create another new key). Good method if you have a decent personal machine you can occasionally put on witness duty for a few blocks when you're doing an upgrade, and don't want to mess with multiple VPS's as in (1) or pay for a large server as in (2).
Also note that the block signing key can be different from the active and owner keys which control account funds. The only key which needs to live unencrypted on a machine with 24/7 internet connectivity is the block signing key; if an attacker compromises the server, the only thing they can do with the block signing key is sign blocks.
I designed this system, and my goal was to give witnesses better options for dealing with the various IT headaches of signing blocks in DPOS.
I just gave update_witness quite a real-world test on this testnet -- I initially ran all of the init witnesses in a single process on the cloud server that I used to create the testnet, then bytemaster and I migrated a bunch of them to different machines within the first day. In prepping them for the hardfork, I've had to shut down and re-create the witnesses, and also migrated them to better balance them between the multiple machines. I used the update_witness command for all of this and achieved it with minimal downtime. In particular for today's hardfork upgrade, I had no downtime on block signing during the upgrade / migration, even though I had to upgrade 8 witnesses on multiple machines, and I also migrated some of those witnesses to better balance them between machines. The update_witness code and its supporting logic in the block production loop are rock solid!
Some of my init witnesses
have been down, but that's mostly due to issues in the p2p layer, the worst of which are resolved in the latest code.
I think I'll write this up in a wiki article sometime this week