Author Topic: a solution for hot standby(From BitSuperLab)  (Read 6375 times)

0 Members and 1 Guest are viewing this topic.

Offline alt

  • Hero Member
  • *****
  • Posts: 2821
    • View Profile
  • BitShares: baozi
I wonder if the orphan block is harmless or not?
If it's harmless, May I ask to delay the punish rule, until we have confirm there is no issue about missed blocks?
here is the blocks which  should have missed without hot standby.
you can see all these blocks are generated by the slave node, with a latency about 5 seconds
Code: [Select]
20140801T005845.095164       th_a        delegate_loop ] Produced block #106529!                        client.cpp:765
20140807T040155.143352       th_a        delegate_loop ] Produced block #158623!                        client.cpp:765
20140811T150025.185202       th_a        delegate_loop ] Produced block #196508!                        client.cpp:765
20140813T020315.208326       th_a        delegate_loop ] Produced block #208869!                        client.cpp:765
20140816T030035.228557       th_a        delegate_loop ] Produced block #234622!                        client.cpp:765
20140817T100325.246930       th_a        delegate_loop ] Produced block #245657!                        client.cpp:765
20140817T102915.243216       th_a        delegate_loop ] Produced block #245808!                        client.cpp:765
20140817T104105.244196       th_a        delegate_loop ] Produced block #245879!                        client.cpp:765
20140817T110245.247619       th_a        delegate_loop ] Produced block #246006!                        client.cpp:765
20140817T112915.242517       th_a        delegate_loop ] Produced block #246161!                        client.cpp:765
20140817T113155.239689       th_a        delegate_loop ] Produced block #246176!                        client.cpp:765
20140817T125335.259069       th_a        delegate_loop ] Produced block #246656!                        client.cpp:765
20140817T131305.254306       th_a        delegate_loop ] Produced block #246771!                        client.cpp:765
20140817T132925.246843       th_a        delegate_loop ] Produced block #246866!                        client.cpp:765
20140817T133915.336232       th_a        delegate_loop ] Produced block #246924!                        client.cpp:765
And I still don't know why the master node have missed these blocks. here is the log from master node
maybe some is strange  at  date 20140817
Code: [Select]
delegate (unlocked) >>> blockchain_list_blocks 246920 5
HEIGHT  TIMESTAMP           SIGNING DELEGATE                # TXS   SIZE    TOTAL FEES      LATENCY PROCESSING TIME
===================================================================================================================
246920  2014-08-17T13:38:30 eur.bts500                      0       166     0.00000 BTSX    1       0.002601
246921  2014-08-17T13:38:40 delegate.charity                0       166     0.00000 BTSX    1       0.001493
246922  2014-08-17T13:38:50 bitcoiners                      0       166     0.00000 BTSX    32      0.001067
246923  2014-08-17T13:39:00 coolspeed                       1       410     0.10000 BTSX    23      0.021211
246924  2014-08-17T13:39:10 delegate-watchman               0       166     0.00000 BTSX    13      0.002413
delegate (unlocked) >>> blockchain_list_blocks 246862 5
HEIGHT  TIMESTAMP           SIGNING DELEGATE                # TXS   SIZE    TOTAL FEES      LATENCY PROCESSING TIME
===================================================================================================================
246862  2014-08-17T13:28:40 clout-delegate5                 0       166     0.00000 BTSX    1       0.001768
246863  2014-08-17T13:28:50 skyscraperfarms                 0       166     0.00000 BTSX    2       0.001662
246864  2014-08-17T13:29:00 delegate.bitsuperlab            0       166     0.00000 BTSX    0       0.002276
246865  2014-08-17T13:29:10 init73                          0       166     0.00000 BTSX    21      0.000418
246866  2014-08-17T13:29:20 delegate-baozi                  0       166     0.00000 BTSX    11      0.004477
delegate (unlocked) >>> blockchain_list_blocks  246767 5
HEIGHT  TIMESTAMP           SIGNING DELEGATE                # TXS   SIZE    TOTAL FEES      LATENCY PROCESSING TIME
===================================================================================================================
246767  2014-08-17T13:12:20 init85                          0       166     0.00000 BTSX    1       0.013516
246768  2014-08-17T13:12:30 calabiyau                       1       365     0.10000 BTSX    1       0.007375
246769  2014-08-17T13:12:40 init28                          0       166     0.00000 BTSX    61      0.001217
246770  2014-08-17T13:12:50 delegate.taolje                 0       166     0.00000 BTSX    51      0.006725
246771  2014-08-17T13:13:00 delegate.bitsuperlab            0       166     0.00000 BTSX    41      0.009017
delegate (unlocked) >>> blockchain_list_blocks 246652 5
HEIGHT  TIMESTAMP           SIGNING DELEGATE                # TXS   SIZE    TOTAL FEES      LATENCY PROCESSING TIME
===================================================================================================================
246652  2014-08-17T12:52:50 happyshares-2                   0       166     0.00000 BTSX    1       0.001722
246653  2014-08-17T12:53:00 spartako                        0       166     0.00000 BTSX    1       0.001564
246654  2014-08-17T12:53:10 love.ymca                       1       362     0.10000 BTSX    1       0.005774
246655  2014-08-17T12:53:20 delegate.xeldal                 0       166     0.00000 BTSX    15      0.002737
246656  2014-08-17T12:53:30 delegate-watchman               0       166     0.00000 BTSX    6       0.004592
delegate (unlocked) >>> blockchain_list_blocks 246172 5
HEIGHT  TIMESTAMP           SIGNING DELEGATE                # TXS   SIZE    TOTAL FEES      LATENCY PROCESSING TIME
===================================================================================================================
246172  2014-08-17T11:31:10 clout-delegate5                 0       166     0.00000 BTSX    1       0.001608
246173  2014-08-17T11:31:20 delegate.bitder                 0       166     0.00000 BTSX    1       0.001527
246174  2014-08-17T11:31:30 init65                          0       166     0.00000 BTSX    31      0.001175
246175  2014-08-17T11:31:40 init28                          0       166     0.00000 BTSX    21      0.003292
246176  2014-08-17T11:31:50 delegate.bitsuperlab            0       166     0.00000 BTSX    11      0.001498
delegate (unlocked) >>> blockchain_list_blocks 246157 5
HEIGHT  TIMESTAMP           SIGNING DELEGATE                # TXS   SIZE    TOTAL FEES      LATENCY PROCESSING TIME
===================================================================================================================
246157  2014-08-17T11:28:30 clout-delegate1                 0       166     0.00000 BTSX    1       0.001934
246158  2014-08-17T11:28:40 www.minebitshares-com           0       166     0.00000 BTSX    1       0.001773
246159  2014-08-17T11:28:50 riverhead-del-server-1          0       166     0.00000 BTSX    31      0.001019
246160  2014-08-17T11:29:00 cny.bts500                      0       166     0.00000 BTSX    21      0.00554
246161  2014-08-17T11:29:10 delegate-watchman               0       166     0.00000 BTSX    11      0.001573
delegate (unlocked) >>> blockchain_list_blocks 246002 5
HEIGHT  TIMESTAMP           SIGNING DELEGATE                # TXS   SIZE    TOTAL FEES      LATENCY PROCESSING TIME
===================================================================================================================
246002  2014-08-17T11:02:00 liondani-delegate-1             0       166     0.00000 BTSX    1       0.004777
246003  2014-08-17T11:02:10 bitsharesx-delegate             0       166     0.00000 BTSX    3       0.00576
246004  2014-08-17T11:02:20 future.dacwin                   0       166     0.00000 BTSX    27      0.00024
246005  2014-08-17T11:02:30 delegate.charity                0       166     0.00000 BTSX    17      0.007007
246006  2014-08-17T11:02:40 delegate-watchman               0       166     0.00000 BTSX    7       0.001626
delegate (unlocked) >>> blockchain_list_blocks  245875 5
HEIGHT  TIMESTAMP           SIGNING DELEGATE                # TXS   SIZE    TOTAL FEES      LATENCY PROCESSING TIME
===================================================================================================================
245875  2014-08-17T10:40:20 clout-delegate5                 1       410     0.10000 BTSX    1       0.005272
245876  2014-08-17T10:40:30 chinese                         1       602     0.10000 BTSX    0       0.013464
245877  2014-08-17T10:40:40 mr.agsexplorer                  0       166     0.00000 BTSX    0       0.00159
245878  2014-08-17T10:40:50 eur.bts500                      1       410     0.10000 BTSX    16      0.001288
245879  2014-08-17T10:41:00 delegate.bitsuperlab            1       410     0.10000 BTSX    7       0.012646
delegate (unlocked) >>> blockchain_list_blocks 245804 5
HEIGHT  TIMESTAMP           SIGNING DELEGATE                # TXS   SIZE    TOTAL FEES      LATENCY PROCESSING TIME
===================================================================================================================
245804  2014-08-17T10:28:30 google.helloworld               0       166     0.00000 BTSX    1       0.00236
245805  2014-08-17T10:28:40 skyscraperfarms                 0       166     0.00000 BTSX    1       0.00291
245806  2014-08-17T10:28:50 init58                          1       410     0.10000 BTSX    0       0.015231
245807  2014-08-17T10:29:00 now.dacwin                      0       166     0.00000 BTSX    21      0.000282
245808  2014-08-17T10:29:10 delegate-baozi                  0       166     0.00000 BTSX    11      0.006493
delegate (unlocked) >>> blockchain_list_blocks 245653 5
HEIGHT  TIMESTAMP           SIGNING DELEGATE                # TXS   SIZE    TOTAL FEES      LATENCY PROCESSING TIME
===================================================================================================================
245653  2014-08-17T10:02:40 mr.agsexplorer                  0       166     0.00000 BTSX    1       0.00362
245654  2014-08-17T10:02:50 delegate.taolje                 0       166     0.00000 BTSX    1       0.002551
245655  2014-08-17T10:03:00 delegate-baozi                  0       166     0.00000 BTSX    0       0.002826
245656  2014-08-17T10:03:10 maqifrnswa                      0       166     0.00000 BTSX    21      0.000229
245657  2014-08-17T10:03:20 delegate-watchman               0       166     0.00000 BTSX    11      0.016401
delegate (unlocked) >>> blockchain_list_blocks 234618 5
HEIGHT  TIMESTAMP           SIGNING DELEGATE                # TXS   SIZE    TOTAL FEES      LATENCY PROCESSING TIME
===================================================================================================================
234618  2014-08-16T02:59:50 btsnow                          0       166     0.00000 BTSX    0       0.001571
234619  2014-08-16T03:00:00 dele-puppy                      0       166     0.00000 BTSX    0       0.0022
234620  2014-08-16T03:00:10 angel.bitdelegate               0       166     0.00000 BTSX    0       0.001478
234621  2014-08-16T03:00:20 clout-delegate5                 0       166     0.00000 BTSX    0       0.00132
234622  2014-08-16T03:00:30 delegate-watchman               0       166     0.00000 BTSX    5       0.00274
delegate (unlocked) >>> blockchain_list_blocks  208865 5
HEIGHT  TIMESTAMP           SIGNING DELEGATE                # TXS   SIZE    TOTAL FEES      LATENCY PROCESSING TIME
===================================================================================================================
208865  2014-08-13T02:02:30 init58                          0       166     0.00000 BTSX    0       0.003361
208866  2014-08-13T02:02:40 bits                            0       166     0.00000 BTSX    0       0.003819
208867  2014-08-13T02:02:50 delegate-watchman               0       166     0.00000 BTSX    0       0.001733
208868  2014-08-13T02:03:00 bts.coin                        0       166     0.00000 BTSX    0       0.0125
208869  2014-08-13T02:03:10 delegate-watchman               0       166     0.00000 BTSX    6       0.002742

Offline alt

  • Hero Member
  • *****
  • Posts: 2821
    • View Profile
  • BitShares: baozi
I think maybe it's better that people should stop taking this approach, because it increase the chance of producing two blocks which is not for reaching consensus, later versions will have operation to fire the delegate if someone can prove this delegate broadcast two different blocks.
I have think this again.
with this rule, I think the hot standby is less useful.
It can't handle the situation that master  is totally unavailable to  slave. (maybe program crash, maybe network down, maybe lost power ....)
the slave can't replace the master without master's empower.

All we can do is to check the state of the node, like the sync block number, the network connects,  and adjust to a more suitable node.

Offline HackFisher

  • Hero Member
  • *****
  • Posts: 883
    • View Profile
I think maybe it's better that people should stop taking this approach, because it increase the chance of producing two blocks which is not for reaching consensus, later versions will have operation to fire the delegate if someone can prove this delegate broadcast two different blocks.
Anything said on these forums does not constitute an intent to create a legal obligation or contract between myself and anyone else.   These are merely my opinions and I reserve the right to change them at any time.

Offline xeroc

  • Board Moderator
  • Hero Member
  • *****
  • Posts: 12922
  • ChainSquad GmbH
    • View Profile
    • ChainSquad GmbH
  • BitShares: xeroc
  • GitHub: xeroc
ok .. so probably I did sth wrong .. maybe mixed up my machines (currently running 3 VPS nodes :) )
Gonna recheck this afternoon

Offline alt

  • Hero Member
  • *****
  • Posts: 2821
    • View Profile
  • BitShares: baozi
Not sure if I did sth. wrong but by backup delegate produced a fork .. so it seems it does not work as intended... does the backup notice that a valid block is allready there and stops the production?
yes it should stop the production.
I run with version  0.3.1, and it work find.
I'll check it when I upgrade to 0.4.1

Offline xeroc

  • Board Moderator
  • Hero Member
  • *****
  • Posts: 12922
  • ChainSquad GmbH
    • View Profile
    • ChainSquad GmbH
  • BitShares: xeroc
  • GitHub: xeroc
Not sure if I did sth. wrong but by backup delegate produced a fork .. so it seems it does not work as intended... does the backup notice that a valid block is allready there and stops the production?

Offline xeroc

  • Board Moderator
  • Hero Member
  • *****
  • Posts: 12922
  • ChainSquad GmbH
    • View Profile
    • ChainSquad GmbH
  • BitShares: xeroc
  • GitHub: xeroc
Ok .. I am finally setting up a back delegate with this solution ... python polling is shit :)

Offline alt

  • Hero Member
  • *****
  • Posts: 2821
    • View Profile
  • BitShares: baozi
This will likely cause you to sign 2 different blocks.
How you ensure that they will not sign 2 different blocks ?

This!

and if not, why would bytemaster not decrease the block production time  from 10 to 5 sec. on the next BTSX hard fork?

IS IT POSSIBLE ?  ;D


In fact, I run master and slave at the same machine now.
And I add the node to each other to make sure they community with less latency.
for now,  most missed block is because of software issue,  no need to run in two different machine.

And another thing, it's better to  restart slave every 12 hours.
I forgot to set auto restart slave yesterday. my slave run long time, and met the old problem, connection lost much, and sign some wrong blocks...

Offline alt

  • Hero Member
  • *****
  • Posts: 2821
    • View Profile
  • BitShares: baozi
This will likely cause you to sign 2 different blocks.
How you ensure that they will not sign 2 different blocks ?
yes, this can happen.
If the rule forbid do this, we'll change to  another solution, ensure only 1 node active at the same time.

Offline Fox

This will likely cause you to sign 2 different blocks.
How you ensure that they will not sign 2 different blocks ?
Networking the master/slave pair nearest each other will mitigate risks due to propagation delays. However, I do not see an elegant fail safe way with a single master/slave pair to ensure delegate block production.  Introducing additional slaves for consensus building will likely add too much time before the block production window is breached and the next delegate produces on time. 
Witness: fox

Offline Fox

Usage:
wallet_delegate_set_slave <delegate> <master_node> <delay> <enable>

Enable or disable block production for delegate following delayed block production on master node

Parameters:
  delegate (string, required): The delegate name to slave for (may use ALL for multiple delegates)
  master_node (string, required): The IP and port number of the master delegate node.  (Format IP:PORT)
  delay (int, optional) {0-10} : The delay in seconds the slave will wait for receipt of block production from master (default 5)
  enabled (bool, optional): true to enable block production, else false (default: true)

Returns:
  void
« Last Edit: July 30, 2014, 10:16:31 pm by Fox »
Witness: fox

Offline liondani

  • Hero Member
  • *****
  • Posts: 3737
  • Inch by inch, play by play
    • View Profile
    • My detailed info
  • BitShares: liondani
  • GitHub: liondani
This will likely cause you to sign 2 different blocks.
How you ensure that they will not sign 2 different blocks ?

This!

and if not, why would bytemaster not decrease the block production time  from 10 to 5 sec. on the next BTSX hard fork?

IS IT POSSIBLE ?  ;D


Offline emski

  • Hero Member
  • *****
  • Posts: 1282
    • View Profile
    • http://lnkd.in/nPbhxG
This will likely cause you to sign 2 different blocks.
How you ensure that they will not sign 2 different blocks ?

Offline xeroc

  • Board Moderator
  • Hero Member
  • *****
  • Posts: 12922
  • ChainSquad GmbH
    • View Profile
    • ChainSquad GmbH
  • BitShares: xeroc
  • GitHub: xeroc
excellent. can even make a new command "wallet_delegate_set_slave {IP_of_master} {true/false}" which will add the node and add a 5 second delay to production
Oh yhea .. would make things incredibly easy

Offline maqifrnswa

  • Hero Member
  • *****
  • Posts: 661
    • View Profile
excellent. can even make a new command "wallet_delegate_set_slave {IP_of_master} {true/false}" which will add the node and add a 5 second delay to production
maintains an Ubuntu PPA: https://launchpad.net/~showard314/+archive/ubuntu/bitshares [15% delegate] wallet_account_set_approval maqifrnswa true [50% delegate] wallet_account_set_approval delegate1.maqifrnswa true