Author Topic: avoiding delegate downtime while upgrading client  (Read 3193 times)

0 Members and 1 Guest are viewing this topic.

Offline wackou

ah, and another thing: re-indexing takes up a lot of memory which is not necessarily freed afterwards (not sure there's a leak, but memory doesn't seem to get reclaimed by the OS), so I would recommend to exit the client after re-indexing and restart it immediately afterwards, RAM usage will look more "normal" then.
Please vote for witness wackou! More info at http://digitalgaia.io

Offline wackou

Hmm, I don't fancy re-indexing the DB on one build while the other is producing blocks, surely that's a recipe for disaster?

only if both are supposed to be producing blocks. The way I do it is what people mentioned before, but instead of locking/unlocking the wallet, I just "switch" them using
Code: [Select]
wallet_delegate_set_block_production ALL true or
Code: [Select]
wallet_delegate_set_block_production ALL false which is instantaneous, ie:

Code: [Select]
client 1: producing blocks, this is the client I want to upgrade (usually on a VPS)
run client 2, same version as client 1 (same wallet too, usually on my laptop). By default, it has block_production=false
# when there is time before next block production (use get_info['next_block_production_time'] on client 1 to know)
client 1: wallet_delegate_set_block_production ALL false
client 2: wallet_delegate_set_block_production ALL true
# now client 2 is producing, so basically you can do whatever you want with client 1, in particular
exit client 1
recompile client 1
run client 1, unlock it
# reindexing can indeed take some time, but is not a problem since last time you were running client 1 it had block_production=false,
# so it will still have it, so it can come fully online whenever it wants, you won't produce forks
# wait for another moment before next block production (same as before, but checking on client 2 now), then
client 1: wallet_delegate_set_block_production ALL true
client 2: wallet_delegate_set_block_production ALL false
# at this point, client 1 is upgraded and fully operational again, you can exit client 2 and relax after a smooth upgrade :)

This seems a bit contrived at first, but will feel very natural after a few times ;)

To check on the clients while upgrading, I use some python tools that I wrote specifically for that purpose, in particular I use this view:
https://raw.githubusercontent.com/wackou/bts_tools/master/bts_tools_screenshot2.png

In this screenshot, you can actually see that I run the client on localhost while recompiling on the server (remote host) and that now would be a bad time to switch, as there are only 39 seconds before my next scheduled block production.

You can get the tools here: https://github.com/wackou/bts_tools
Please vote for witness wackou! More info at http://digitalgaia.io

Offline emski

  • Hero Member
  • *****
  • Posts: 1282
    • View Profile
    • http://lnkd.in/nPbhxG
Hmmm, after building the latest and running the client (with the current delegate still going) I get this:

Code: [Select]
./bitshares_client
Loading blockchain from: /root/.BitShares/chain
Loading config from file: /root/.BitShares/config.json
------------ error --------------
10 assert_exception: Assert Exception
_map->is_open(): Database is not open!

Try using different --data-dir for each delegate instance (you can copy the old datadir).

Offline monsterer

Upgrade went smooth enough after shutting down the main client - no reindexing was needed this time.
My opinions do not represent those of metaexchange unless explicitly stated.
https://metaexchange.info | Bitcoin<->Altcoin exchange | Instant | Safe | Low spreads

Offline monsterer

Hmmm, after building the latest and running the client (with the current delegate still going) I get this:

Code: [Select]
./bitshares_client
Loading blockchain from: /root/.BitShares/chain
Loading config from file: /root/.BitShares/config.json
------------ error --------------
10 assert_exception: Assert Exception
_map->is_open(): Database is not open!
My opinions do not represent those of metaexchange unless explicitly stated.
https://metaexchange.info | Bitcoin<->Altcoin exchange | Instant | Safe | Low spreads

Offline xeroc

  • Board Moderator
  • Hero Member
  • *****
  • Posts: 12922
  • ChainSquad GmbH
    • View Profile
    • ChainSquad GmbH
  • BitShares: xeroc
  • GitHub: xeroc
btw.. I wrote two python scripts to help out with the switchover to backup/main machines during updates
https://github.com/xeroc/pytshares/blob/master/enable-backup.py
https://github.com/xeroc/pytshares/blob/master/enable-master.py
you just need to added the corresponding data fields into the config.py file and ccan just run the python files accordingly

Offline svk

be reminded that sometimes the database has to be reindex when restarting the client after an update .. that reindex might take longer than a round of delegates

Hmm, I don't fancy re-indexing the DB on one build while the other is producing blocks, surely that's a recipe for disaster?

I do that every time, never had any issues with it. Just make sure to not open and unlock the wallet on the second instance while the first one is still running. Another thing to take into account is the time it takes for an instance to shut down as well as a new one to load. At least on my machine that sometimes takes several minutes.
Worker: dev.bitsharesblocks

Offline monsterer

be reminded that sometimes the database has to be reindex when restarting the client after an update .. that reindex might take longer than a round of delegates

Hmm, I don't fancy re-indexing the DB on one build while the other is producing blocks, surely that's a recipe for disaster?
My opinions do not represent those of metaexchange unless explicitly stated.
https://metaexchange.info | Bitcoin<->Altcoin exchange | Instant | Safe | Low spreads

Offline xeroc

  • Board Moderator
  • Hero Member
  • *****
  • Posts: 12922
  • ChainSquad GmbH
    • View Profile
    • ChainSquad GmbH
  • BitShares: xeroc
  • GitHub: xeroc
be reminded that sometimes the database has to be reindex when restarting the client after an update .. that reindex might take longer than a round of delegates

Offline monsterer

My opinions do not represent those of metaexchange unless explicitly stated.
https://metaexchange.info | Bitcoin<->Altcoin exchange | Instant | Safe | Low spreads

Xeldal

  • Guest
I build a second server with the upgrade leaving the old server running while doing this.

typing "info" at command line will give you  "wallet_next_block_production_time": "7 minutes in the future"

showing you when your delegate is up to sign its next block

It only takes a moment to "lock" the old server's wallet(preventing it signing further blocks) and then "unlock" or "wallet_delegate_set_block_production <delegate_name> true"  the new upgraded server's wallet.

Then you should be safe to destroy your old server instance.

There are of course many ways to do this process.  This is just what I do.


Offline monsterer

What is the best way to avoid delegate downtime while upgrading the bitshares client?

Is there a way to know when you've just produced a block, to give you maximum time before your turn comes around again?
My opinions do not represent those of metaexchange unless explicitly stated.
https://metaexchange.info | Bitcoin<->Altcoin exchange | Instant | Safe | Low spreads