BitShares Forum

Main => General Discussion => Topic started by: emski on July 23, 2014, 11:47:56 am

Title: Delegate temporary absence
Post by: emski on July 23, 2014, 11:47:56 am
I think a way for delegate to perform maintenance is needed. This should keep delegate's votes and position but transfer the delegate's slot to another delegate.

For example:
Delegate signs a transaction that marks him inactive for certain period. Nothing changes except his place is taken by 102nd delegate for the time of the maintenance.
This allows delegates to plan maintenance windows without disrupting the network and getting negative statistics. (And without setting up a backup delegate)

EDIT: It could be used if any issue arises that prevents delegate from performing optimally. For example lawsuit, vacation, security issue etc...
Title: Re: Delegate temporary absence
Post by: xeroc on July 23, 2014, 11:57:19 am
As a delegate should have a backup machine IMHO that should not be necessary as you can simply switch over to the backup machine by enabling/disabling block production.
Title: Re: Delegate temporary absence
Post by: Fox on July 23, 2014, 12:33:10 pm
The most important role the delegate serves is the timely and accurate production of blocks. 
Title: Re: Delegate temporary absence
Post by: testz on July 23, 2014, 01:22:58 pm
As a delegate should have a backup machine IMHO that should not be necessary as you can simply switch over to the backup machine by enabling/disabling block production.

 +5% Yes, it's a way how it's should be done. Probably at some point somebody will automate this task but even today it's can be done manually in 10 seconds.
Title: Re: Delegate temporary absence
Post by: wackou on July 23, 2014, 07:25:14 pm
I actually have that planned for my bitshares_delegate_tools (https://github.com/wackou/bitshares_delegate_tools), I'm currently on holidays with intermittent internet but I expect to have it running in 2-3 weeks.

I'd like to have it working with manual intervention, ie: stop 1 on demand for upgrade, maintenance, etc... while the 2nd one takes over, or using automatic detection, for instance if the client crashes, or connections count drop to 0 (happened to me last night while sleeping :( ) then the 2nd one could automatically kick in. Stay tuned!
Title: Re: Delegate temporary absence
Post by: Fox on July 23, 2014, 08:23:32 pm
I actually have that planned for my bitshares_delegate_tools (https://github.com/wackou/bitshares_delegate_tools)

This looks like a great start.  I'll offer my support to your efforts. 
Title: Re: Delegate temporary absence
Post by: maqifrnswa on July 23, 2014, 08:32:57 pm
As a delegate should have a backup machine IMHO that should not be necessary as you can simply switch over to the backup machine by enabling/disabling block production.

 +5% Yes, it's a way how it's should be done. Probably at some point somebody will automate this task but even today it's can be done manually in 10 seconds.

what happens if a delegate has two machines running, each producing blocks as a form of redundancy? It will cause forks and race conditions, but how serious are those? (I know they can be serious, but would redundancy be a good or bad thing?)
Title: Re: Delegate temporary absence
Post by: bytemaster on July 23, 2014, 08:33:27 pm
As a delegate should have a backup machine IMHO that should not be necessary as you can simply switch over to the backup machine by enabling/disabling block production.

 +5% Yes, it's a way how it's should be done. Probably at some point somebody will automate this task but even today it's can be done manually in 10 seconds.

what happens if a delegate has two machines running, each producing blocks as a form of redundancy? It will cause forks and race conditions, but how serious are those?

They will get fired automatically in a future release.
Title: Re: Delegate temporary absence
Post by: liondani on July 23, 2014, 09:16:18 pm
As a delegate should have a backup machine IMHO that should not be necessary as you can simply switch over to the backup machine by enabling/disabling block production.

 +5% Yes, it's a way how it's should be done. Probably at some point somebody will automate this task but even today it's can be done manually in 10 seconds.

what happens if a delegate has two machines running, each producing blocks as a form of redundancy? It will cause forks and race conditions, but how serious are those?

They will get fired automatically in a future release.

ok but what happen with his APPROVAL votes ?
Get he fired for ever? How much time?
Title: Re: Delegate temporary absence
Post by: bytemaster on July 23, 2014, 09:21:55 pm
As a delegate should have a backup machine IMHO that should not be necessary as you can simply switch over to the backup machine by enabling/disabling block production.

 +5% Yes, it's a way how it's should be done. Probably at some point somebody will automate this task but even today it's can be done manually in 10 seconds.

what happens if a delegate has two machines running, each producing blocks as a form of redundancy? It will cause forks and race conditions, but how serious are those?

They will get fired automatically in a future release.

ok but what happen with his APPROVAL votes ?
Get he fired for ever? How much time?

Their approval will be set to a large negative number.  IE: a delegate's job is to make sure they only produce one block at a time.   Approval is hard to get so it is expensive to ask for forgiveness under a new name.

Title: Re: Delegate temporary absence
Post by: wackou on July 23, 2014, 09:42:39 pm
As a delegate should have a backup machine IMHO that should not be necessary as you can simply switch over to the backup machine by enabling/disabling block production.

 +5% Yes, it's a way how it's should be done. Probably at some point somebody will automate this task but even today it's can be done manually in 10 seconds.

what happens if a delegate has two machines running, each producing blocks as a form of redundancy? It will cause forks and race conditions, but how serious are those? (I know they can be serious, but would redundancy be a good or bad thing?)

The idea is to have N machines "ready", meaning properly synced (ie: head block is only a few seconds of age), at least 5 connections, NTP also in sync, wallet open and unlocked. All of them, except for one, should have their block production set to false, so that at a given time you only have a single one producing blocks.

Whenever you want to activate a backup machine, you should first make sure to deactivate your currently producing one (and preferably not too close to your allotted block production time) and then activate the one you want. As other people commented here, you should never produce 2 blocks at once, in case of doubt it is much better to miss a block than produce duplicates.
Title: Re: Delegate temporary absence
Post by: emski on December 22, 2014, 11:29:19 am
As a delegate should have a backup machine IMHO that should not be necessary as you can simply switch over to the backup machine by enabling/disabling block production.

@xeroc Have you changed your mind given the recent events ?

@everyone Is the proposal in OP useful ?
Title: Re: Delegate temporary absence
Post by: cube on December 22, 2014, 12:10:12 pm
I think a way for delegate to perform maintenance is needed. This should keep delegate's votes and position but transfer the delegate's slot to another delegate.

For example:
Delegate signs a transaction that marks him inactive for certain period. Nothing changes except his place is taken by 102nd delegate for the time of the maintenance.
This allows delegates to plan maintenance windows without disrupting the network and getting negative statistics. (And without setting up a backup delegate)

EDIT: It could be used if any issue arises that prevents delegate from performing optimally. For example lawsuit, vacation, security issue etc...

I think this is a neat idea.
Title: Re: Delegate temporary absence
Post by: theoretical on December 22, 2014, 04:26:07 pm
I think a way for delegate to perform maintenance is needed. This should keep delegate's votes and position but transfer the delegate's slot to another delegate.

I agree this is needful.  A delegate may very well decide to resign from their position and stop being a delegate, due to a change in life circumstances, business circumstances, regulatory / legal issues, etc.  They currently have no way to voluntarily do this except by trying to get in contact with voters and convincing them to vote for someone else.  We need a signed message that says "I no longer want to be a delegate, please do not consider me as a candidate in the delegate election."

Temporary resignation for maintenance should be discouraged; this is what manual failover is for (have client running on two nodes, wait for block, then stop block production on the current node and start it on the backup).  Since this is 2014 (soon to be 2015) and you can rent VPS's by the hour from many providers, the costs of having a backup node online for a few hours while the main node undergoes maintenance is small.

They will get fired automatically in a future release.

So how is automatic failover supposed to work then?  Suppose a backup node loses contact with the main node (the main node doesn't respond to any network service, ping, RPC, ssh, etc.).  Then the backup node can't start producing blocks because it can't distinguish between (a) the main node is down and (b) the network path between the backup node and main node is down.  Thus, producing blocks on the backup node might cause a fork.

I'd like to make the following recommendations for delegates then:

- If running automatic failover nodes, DO NOT start producing blocks unless the main node can be CONFIRMED dead (i.e. you can get into the VPS admin function and tell the machine to reboot, get into SSH and see the process is gone, etc.)

- Best practice is to just have one node with automatic failover wrapper script that monitors the connection and kills / restarts the process, re-unlocking the wallet, if RPC dies.

- Put the client in your /etc/init.d, including putting your wallet password on permanent storage, so rebooting the machine should start the delegate automatically.

Of course this means your key is exposed to anyone who has physical access to the storage.  For this reason:

- Use active key instead of owner key and have a script that withdraws pay several times a day

- Use block production key when that functionality is released
Title: Re: Delegate temporary absence
Post by: emski on December 22, 2014, 10:41:34 pm
@theoretical you can consider maintenance as fancy word for any of the following "legal issues", "personal issues", "technical issues" ... "just issues" .
Do you state that the transaction you see useful is something like "I'm not going to be a delegate anymore!" ?
What reasons do you have against "I'm not going to be a delegate for <given amount of time/blocks>" ?