@puppies where can I find your failover script?

Quote from: puppies on October 15, 2015, 02:31:09 PM
Thanks betax and rng.  Be careful.  I forgot to mention the first thing it will do is kill any running witness_node on your box so that it can start a new one in a screen called witness.  I would not suggest running it on your main production witness until it has been much more thoroughly tested.

Thanks puppies, I'll help testing. It seems that you also improved fork detection  : )

@bytemaster   "bue "  need your vote to back to service . Thanks





get_witness bue
{
    "id": "1.6.25",
  "witness_account": "1.2.18513",
  "last_aslot": 59851,
  "signing_key": "BTS6pDEnP5iUioSfGECyE84LTM4Z1tN6ySpQhuKroixj5zA8igbxt",
  "pay_vb": "1.13.50",
  "vote_id": "1:35",
  "total_votes": "1340207605570",
  "url": "http://bue.com",
  "total_missed": 26,
  "last_confirmed_block_num": 56354
}

Quote from: puppies on October 15, 2015, 02:31:09 PM
I would not suggest running it on your main production witness until it has been much more thoroughly tested.

Thanks for the great work and  +5% for the responsible caution statement.

Thanks betax and rng.  Be careful.  I forgot to mention the first thing it will do is kill any running witness_node on your box so that it can start a new one in a screen called witness.  I would not suggest running it on your main production witness until it has been much more thoroughly tested.

I've switched witness account from  "1.6.33"  "bts-bitshares-argentina" to "1.6.45" "rnglab"

Please downvote bts-bitshares-argentina and vote for rnglab. Thanks

{
  "id": "1.6.45",
  "witness_account": "1.2.31810",
  "last_aslot": 0,
  "signing_key": "BTS6zwC6qHhk19WBkEBT5YWAvWXjALnSTyTRTBXvSAZujckX2JSaZ",
  "vote_id": "1:56",
  "total_votes": 0,
  "url": "",
  "total_missed": 0,
  "last_confirmed_block_num": 0
}

Quote from: puppies on October 15, 2015, 12:35:47 PM

Quote from: rnglab on October 15, 2015, 02:38:57 AM
witness "1.6.33" is back, sorry for the delay

Update: failover script based on update_witness didn't work because witness_node process died on both servers after the fork. Checking logs now

My newest version of failover script should restart witness_node and cli_wallet in case of crash, or witness production falling below 50%.  Unlock the wallet, and switch signing keys if blocks are missed.  The first 3 times it restarts it will attempt a --replay-blockchain to get back on a majority fork.  If that doesn't work it will try a --resync-blockchain.

At least thats what I expect it to do.  I still need to test it a bit more.

great puppies! that's what we need. I can help testing if you want.

Quote from: puppies on October 15, 2015, 12:35:47 PM

Quote from: rnglab on October 15, 2015, 02:38:57 AM
witness "1.6.33" is back, sorry for the delay

Update: failover script based on update_witness didn't work because witness_node process died on both servers after the fork. Checking logs now

My newest version of failover script should restart witness_node and cli_wallet in case of crash, or witness production falling below 50%.  Unlock the wallet, and switch signing keys if blocks are missed.  The first 3 times it restarts it will attempt a --replay-blockchain to get back on a majority fork.  If that doesn't work it will try a --resync-blockchain.

At least thats what I expect it to do.  I still need to test it a bit more.

Great ill test it now as I am not in yet ;)

Quote from: rnglab on October 15, 2015, 02:38:57 AM
witness "1.6.33" is back, sorry for the delay

Update: failover script based on update_witness didn't work because witness_node process died on both servers after the fork. Checking logs now

My newest version of failover script should restart witness_node and cli_wallet in case of crash, or witness production falling below 50%.  Unlock the wallet, and switch signing keys if blocks are missed.  The first 3 times it restarts it will attempt a --replay-blockchain to get back on a majority fork.  If that doesn't work it will try a --resync-blockchain.

At least thats what I expect it to do.  I still need to test it a bit more.

Both witness nodes crashed around 10 minutes after the fork. Seednode also crashed a few minutes later. Each on a different VPS.

{"new_block":{"previous":"000084897bc808eea0f1911cac37901486b5f7c6","timestamp":"2015-10-14T19:36:06","witness":"1.6.19","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"1f162b264a288624d187a02dc21655a3d99add519670b6f094807e873a3c879bca61fb02066f730b8ecaba18ccf01c08e848e5a5f0617c125c6981c31e3c1dfbf3","transactions":[]}}
th_a  db_block.cpp:191 _push_block
2317366ms th_a       fork_database.cpp:51          push_block           ] Pushing block to fork database that failed to link: 0000848b0cb606ed5d3eb45e250db0169efe761e, 33931
2317366ms th_a       fork_database.cpp:52          push_block           ] Head: 33855, 0000843f85f188ee02afcd6bda5e224429c59d5d
2317367ms th_a       application.cpp:477           handle_block         ] Error when pushing block:
3080000 unlinkable_block_exception: unlinkable block
block does not link to known chain
{}
th_a  fork_database.cpp:72 _push_block

{"new_block":{"previous":"0000848a85dd34c82612e01f4544b0ff2a93b00a","timestamp":"2015-10-14T19:36:12","witness":"1.6.1","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"1f74f7f59213b1a5309d9062f5d5885a0f6d0ab78993acdc3db79eafaa5c423e783e5239ebf0ac8f8f99670e2cdd1240d820ec7e85ee1fc779882612810ee1ad3f","transactions":[]}}
th_a  db_block.cpp:191 _push_block
witness_node: /home/failover/bitshares-2/libraries/net/node.cpp:1605: void graphene::net::detail::node_impl::schedule_peer_for_deletion(const peer_connection_ptr&): Assertion `_closing_connections.find(peer_to_delete) == _closing_connections.end()' failed.
Aborted (core dumped)

ulimit set to 0 by default, no core files were saved.

witness "1.6.33" is back, sorry for the delay

Update: failover script based on update_witness didn't work because witness_node process died on both servers after the fork. Checking logs now

bue need some vote  to be back for servicing network



get_witness bue
{
  "id": "1.6.25",
  "witness_account": "1.2.18513",
  "last_aslot": 36341,
  "signing_key": "BTS6pDEnP5iUioSfGECyE84LTM4Z1tN6ySpQhuKroixj5zA8igbxt",
  "pay_vb": "1.13.50",
  "vote_id": "1:35",
  "total_votes": "864423218403",
  "url": "http://bue.com",
  "total_missed": 26,
  "last_confirmed_block_num": 33743
}

Quote from: bytemaster on October 14, 2015, 07:31:11 PM
All witnesses may need to replay the blockchain to get back in sync.  There seems to be an issue that can cause things to get out of sync.

delegate.taolje is syncing nicely now, need some vote for servicing network
get_witness delegate.taolje
{
  "id": "1.6.38",
  "witness_account": "1.2.10091",
  "last_aslot": 0,
  "signing_key": "BTS6oRYTjcbQU8s9TAtEY4qZknY7Xaw3E29ore3L3xX611KxFE5Nr",
  "vote_id": "1:48",
  "total_votes": 0,
  "url": "url-to-proposal",
  "total_missed": 0,
  "last_confirmed_block_num": 0

Quote from: EkremH on October 14, 2015, 10:36:22 PM
No, oct 5 was just an example. I found at the source the genesis.json, but I could not find seed node ?

it should find the seed node and use the genesis with no arguments
./witness_node

Is supposed to just work, no genesis file or seeds are needed. This is different than the testnet where you had to give it seed nodes and genesis.json.

also, witness maqifrnswa needs votes

delegate.baozi in sync, need some vote

Quote from: bytemaster on October 14, 2015, 07:31:11 PM
All witnesses may need to replay the blockchain to get back in sync.  There seems to be an issue that can cause things to get out of sync.