avoiding delegate downtime while upgrading client

[wackou]

ah, and another thing: re-indexing takes up a lot of memory which is not necessarily freed afterwards (not sure there's a leak, but memory doesn't seem to get reclaimed by the OS), so I would recommend to exit the client after re-indexing and restart it immediately afterwards, RAM usage will look more "normal" then.

[wackou]

Hmm, I don't fancy re-indexing the DB on one build while the other is producing blocks, surely that's a recipe for disaster?

only if both are supposed to be producing blocks. The way I do it is what people mentioned before, but instead of locking/unlocking the wallet, I just "switch" them using

Code Select Expand

wallet_delegate_set_block_production ALL true or

Code Select Expand

wallet_delegate_set_block_production ALL false which is instantaneous, ie:

Code Select Expand


client 1: producing blocks, this is the client I want to upgrade (usually on a VPS)
run client 2, same version as client 1 (same wallet too, usually on my laptop). By default, it has block_production=false
# when there is time before next block production (use get_info['next_block_production_time'] on client 1 to know)
client 1: wallet_delegate_set_block_production ALL false
client 2: wallet_delegate_set_block_production ALL true
# now client 2 is producing, so basically you can do whatever you want with client 1, in particular
exit client 1
recompile client 1
run client 1, unlock it
# reindexing can indeed take some time, but is not a problem since last time you were running client 1 it had block_production=false,
# so it will still have it, so it can come fully online whenever it wants, you won't produce forks
# wait for another moment before next block production (same as before, but checking on client 2 now), then
client 1: wallet_delegate_set_block_production ALL true
client 2: wallet_delegate_set_block_production ALL false
# at this point, client 1 is upgraded and fully operational again, you can exit client 2 and relax after a smooth upgrade :)


This seems a bit contrived at first, but will feel very natural after a few times ;)

To check on the clients while upgrading, I use some python tools that I wrote specifically for that purpose, in particular I use this view:
https://raw.githubusercontent.com/wackou/bts_tools/master/bts_tools_screenshot2.png

In this screenshot, you can actually see that I run the client on localhost while recompiling on the server (remote host) and that now would be a bad time to switch, as there are only 39 seconds before my next scheduled block production.

You can get the tools here: https://github.com/wackou/bts_tools

[emski]

Hmmm, after building the latest and running the client (with the current delegate still going) I get this:

Code Select Expand

./bitshares_client
Loading blockchain from: /root/.BitShares/chain
Loading config from file: /root/.BitShares/config.json
------------ error --------------
10 assert_exception: Assert Exception
_map->is_open(): Database is not open!


Try using different --data-dir for each delegate instance (you can copy the old datadir).

[monsterer]

Upgrade went smooth enough after shutting down the main client - no reindexing was needed this time.

[monsterer]

Hmmm, after building the latest and running the client (with the current delegate still going) I get this:

Code Select Expand

./bitshares_client
Loading blockchain from: /root/.BitShares/chain
Loading config from file: /root/.BitShares/config.json
------------ error --------------
10 assert_exception: Assert Exception
_map->is_open(): Database is not open!


[xeroc]

btw.. I wrote two python scripts to help out with the switchover to backup/main machines during updates
https://github.com/xeroc/pytshares/blob/master/enable-backup.py
https://github.com/xeroc/pytshares/blob/master/enable-master.py
you just need to added the corresponding data fields into the config.py file and ccan just run the python files accordingly

[svk]

be reminded that sometimes the database has to be reindex when restarting the client after an update .. that reindex might take longer than a round of delegates

Hmm, I don't fancy re-indexing the DB on one build while the other is producing blocks, surely that's a recipe for disaster?

I do that every time, never had any issues with it. Just make sure to not open and unlock the wallet on the second instance while the first one is still running. Another thing to take into account is the time it takes for an instance to shut down as well as a new one to load. At least on my machine that sometimes takes several minutes.

[monsterer]

be reminded that sometimes the database has to be reindex when restarting the client after an update .. that reindex might take longer than a round of delegates

Hmm, I don't fancy re-indexing the DB on one build while the other is producing blocks, surely that's a recipe for disaster?

[xeroc]

be reminded that sometimes the database has to be reindex when restarting the client after an update .. that reindex might take longer than a round of delegates

[monsterer]

Thanks, guys :)

[Xeldal]

I build a second server with the upgrade leaving the old server running while doing this.

typing "info" at command line will give you  "wallet_next_block_production_time": "7 minutes in the future"

showing you when your delegate is up to sign its next block

It only takes a moment to "lock" the old server's wallet(preventing it signing further blocks) and then "unlock" or "wallet_delegate_set_block_production <delegate_name> true"  the new upgraded server's wallet.

Then you should be safe to destroy your old server instance.

There are of course many ways to do this process.  This is just what I do.

[emski]

https://bitsharestalk.org/index.php?topic=7747.0

[monsterer]

What is the best way to avoid delegate downtime while upgrading the bitshares client?

Is there a way to know when you've just produced a block, to give you maximum time before your turn comes around again?