Test Net for Advanced Users

[bytemaster]

I'm getting a whole lot of this on my node running all of the init witnesses

Code: [Select]

599260ms th_a       application.cpp:364           handle_block         ] Error when pushing block:
10 assert_exception: Assert Exception
itr != _index.get<block_id>().end():
    {}
    th_a  fork_database.cpp:51 push_block

    {"new_block":{"previous":"0001d2f41d0e19149625d60d536830b59d41311e","timestamp":"2015-08-18T16:53:10","witness":"1.6.1439","next_secret_hash":"f94d192447e
29642fd594b22a727c5633b23229e","previous_secret":"76f3d6853731d668c0ca1444c69ee2c4105466a8","transaction_merkle_root":"000000000000000000000000000000000000000
0","extensions":[],"witness_signature":"206adacd5a039ab07d1fd430edb4217c87f0ef29cb85ebe9abf597b264cc961a670c0fd4ddff450fc1bd8b0f1123c147e9bc4af9d4d3738fac8017
8ca543747ef1","transactions":[]}}
    th_a  db_block.cpp:173 _push_block
599263ms th_a       application.cpp:364           handle_block         ] Error when pushing block:
10 assert_exception: Assert Exception
itr != _index.get<block_id>().end():
    {}
    th_a  fork_database.cpp:51 push_block

    {"new_block":{"previous":"0001d2f5f40d076e7ed6a39cfd843cf6a239f1c9","timestamp":"2015-08-18T16:53:20","witness":"1.6.1439","next_secret_hash":"8c6ed73e105
dfaad5f6334962c4c0438e0990f2b","previous_secret":"57abab205a18e20ecede92a78aaa06565447d370","transaction_merkle_root":"000000000000000000000000000000000000000
0","extensions":[],"witness_signature":"1f3a5728cc46e0e9d418ace7e1647a28c4a8f346a93d513c23934f87c6e367f5fc778c8d9b8b8a3d790a390d9b47ea9a7ec874b604f1809c619a5f
d9dda3fcc70b","transactions":[]}}
    th_a  db_block.cpp:173 _push_block
599265ms th_a       application.cpp:364           handle_block         ] Error when pushing block:
10 assert_exception: Assert Exception
itr != _index.get<block_id>().end():
    {}
All of it seems to be related to lafonas witness.  I can pull logs, or if it would help I can just PM bytemaster with the login credentials of the machine.

Yes, that is the line I expected would get triggered for "out-of-order" pushing of blocks.    That is very helpful.

[puppies]

I'm getting a whole lot of this on my node running all of the init witnesses

Code: [Select]

599260ms th_a       application.cpp:364           handle_block         ] Error when pushing block:
10 assert_exception: Assert Exception
itr != _index.get<block_id>().end():
    {}
    th_a  fork_database.cpp:51 push_block

    {"new_block":{"previous":"0001d2f41d0e19149625d60d536830b59d41311e","timestamp":"2015-08-18T16:53:10","witness":"1.6.1439","next_secret_hash":"f94d192447e
29642fd594b22a727c5633b23229e","previous_secret":"76f3d6853731d668c0ca1444c69ee2c4105466a8","transaction_merkle_root":"000000000000000000000000000000000000000
0","extensions":[],"witness_signature":"206adacd5a039ab07d1fd430edb4217c87f0ef29cb85ebe9abf597b264cc961a670c0fd4ddff450fc1bd8b0f1123c147e9bc4af9d4d3738fac8017
8ca543747ef1","transactions":[]}}
    th_a  db_block.cpp:173 _push_block
599263ms th_a       application.cpp:364           handle_block         ] Error when pushing block:
10 assert_exception: Assert Exception
itr != _index.get<block_id>().end():
    {}
    th_a  fork_database.cpp:51 push_block

    {"new_block":{"previous":"0001d2f5f40d076e7ed6a39cfd843cf6a239f1c9","timestamp":"2015-08-18T16:53:20","witness":"1.6.1439","next_secret_hash":"8c6ed73e105
dfaad5f6334962c4c0438e0990f2b","previous_secret":"57abab205a18e20ecede92a78aaa06565447d370","transaction_merkle_root":"000000000000000000000000000000000000000
0","extensions":[],"witness_signature":"1f3a5728cc46e0e9d418ace7e1647a28c4a8f346a93d513c23934f87c6e367f5fc778c8d9b8b8a3d790a390d9b47ea9a7ec874b604f1809c619a5f
d9dda3fcc70b","transactions":[]}}
    th_a  db_block.cpp:173 _push_block
599265ms th_a       application.cpp:364           handle_block         ] Error when pushing block:
10 assert_exception: Assert Exception
itr != _index.get<block_id>().end():
    {}
All of it seems to be related to lafonas witness.  I can pull logs, or if it would help I can just PM bytemaster with the login credentials of the machine.   

[puppies]

slightly off topic, but is there a good resource on proper protocol to post these issues directly to github?  I could post directly to github, but I'm such a github noob that I'm not sure I wouldn't be making things worse than just posting here.

[bytemaster]

https://www.dropbox.com/s/q4abwrm8c96wxtg/p2p.log.fork.8.18.15?dl=0

The trouble seems to start around 16:27:33

https://github.com/cryptonomex/graphene/issues/247

[puppies]

https://www.dropbox.com/s/q4abwrm8c96wxtg/p2p.log.fork.8.18.15?dl=0

The trouble seems to start around 16:27:33

[bytemaster]

There are no private keys or anything like that in the p2p.log is there?  I just want to double check before I post.

There shouldn't be. 

[puppies]

There are no private keys or anything like that in the p2p.log is there?  I just want to double check before I post.

[bytemaster]

Those who are reporting that they are stuck on a fork I am very interested in hearing more details.

If my understanding is correct it will fork if the block you produced doesn't get included in the final chain.   I will try to replicate this behavior in a test.

If I am reading the logs correctly in my case it was not a generation issue, but a propogation issues.  It looks like my node received block 118181 before it got 118180.  It tried to push 118181 failed, then got 118180 pushed it successfully, but never went back and tried to push 118181 again so when 118182 came along it also failed to push.

Does that make sense?

That sounds very helpful!  Logs would be great.

[bytemaster]

The network is still up and generating blocks on at least three different systems. 

Code: [Select]

  "head_block_num": 113869,
  "head_block_id": "0001bccd0acdf8b05c58b6cc5e6e36d717de523c",
  "head_block_age": "2 seconds old",
  "next_maintenance_time": "4 minutes in the future",
  "chain_id": "081401ede64c8fe30b23c91d7ab8750103acb1a39548a866fb562f2edf4627d6",
It seems like your node did fork and now wont accept the proper chain.

Code: [Select]

get_block 72042
{
  "previous": "000119696d1d31776e6431586e14fd2e4c0b4f58",
  "timestamp": "2015-08-18T02:07:59",
  "witness": "1.6.1435",
  "next_secret_hash": "72dcf0475fa5d0c145a235f590b5be3597c17753",
  "previous_secret": "2813946aacd0bfda5541bf243f5bc47aee51e314",
  "transaction_merkle_root": "0000000000000000000000000000000000000000",
  "extensions": [],
  "witness_signature": "201506e5034d1df8627196bc97b5293eed3efbb6391a256789ca4a6912f169b45b51045f55f9be468a2db485bfcc4952996e10d1f4be0cc1c0eef2764470649035",
  "transactions": []
}
locked >>> get_block 72043
get_block 72043
{
  "previous": "0001196a1eb9d0e2021fd4f6ac9fe452ea46d4b3",
  "timestamp": "2015-08-18T02:08:00",
  "witness": "1.6.0",
  "next_secret_hash": "031a792684e0d99dab315ce6aea4fa92c85c6707",
  "previous_secret": "cd722b49f39a4b7160e081ae2c26cdf71f8e9194",
  "transaction_merkle_root": "0000000000000000000000000000000000000000",
  "extensions": [],
  "witness_signature": "1f64cf6d9e48683ed2259772c5006c0a1ec195dc280c59b5941250fa1581e8b86c0d32ed49d47d9d57cc87b3088e2f52bc15c7dc500955fa334864a3ed7b08be63",
  "transactions": []
}
locked >>> get_block 72044
get_block 72044
{
  "previous": "0001196bf9cf506fb01511e20c8566694b952a7d",
  "timestamp": "2015-08-18T02:08:01",
  "witness": "1.6.4",
  "next_secret_hash": "42899548ec99624cb2102330c429562510ef09a0",
  "previous_secret": "2777b474efe4774a69d716f61b5cf156deb3a232",
  "transaction_merkle_root": "0000000000000000000000000000000000000000",
  "extensions": [],
  "witness_signature": "1f02312f66e1dcf78335d1c5ee999181cd3a0191d927d48376fa5702a493d418514d01eb530328c2dee956d5824103b8576e7974049e0d414eab142fd3ddbbe991",
  "transactions": []
}
locked >>>
im curious is your block 72042 matches mine.

Looks like my node produced a #72042 block but failed to broadcast it in time (networking issue) then got stuck. It got #72043 and following blocks but failed to push them into chain database, because those blocks are linked to another #72042, but it didn't request for that #72042.
There could be an issue in code. Maybe current code just throw out an exception in this case. Maybe it's better to temporarily save the new block and request for it's previous block if it's not same as the one in current fork.

Just restarted my node and catching up.

The node is in China, network condition will be OK in next 8 hours or so (mid-night), and may go bad again after that. 

//update: my node should have produced a #72042 block but not #72043.

Do you have the p2p logs for this?

[puppies]

Those who are reporting that they are stuck on a fork I am very interested in hearing more details.

If my understanding is correct it will fork if the block you produced doesn't get included in the final chain.   I will try to replicate this behavior in a test.

If I am reading the logs correctly in my case it was not a generation issue, but a propogation issues.  It looks like my node received block 118181 before it got 118180.  It tried to push 118181 failed, then got 118180 pushed it successfully, but never went back and tried to push 118181 again so when 118182 came along it also failed to push.

Does that make sense?

Edit.  NM it did go back and push 118181.  I will dig in the logs more.  I can upload the entire log if it will help

Edit #
Just for the record it did push 118181 then 118182, but by that time it had already recieved 118183 and 118184.  It failed pushing 118184, and I don't think it ever went back and tried 118183 again.  Seems the problem was from being two blocks behind

[betax]

and put it in the same folder as the ./witness_node

[puppies]

Code: [Select]

#!/bin/bash
cd ~/bts2.0/graphene/programs/witness_node;
./witness_node --rpc-endpoint "127.0.0.1:8090" --resync-blockchain --witness-id '"1.6.1530"' --private-key '["MY_SIGNING_KEY","PRIVATE_KEY_OF_MY_SIGNING_KEY"]' -d test_net -s 45.55.6.216:1776


try adding

Code: [Select]

--genesis-json aug-14-test-genesis.jsonso the final line in you script will be

Code: [Select]

./witness_node --rpc-endpoint "127.0.0.1:8090" --resync-blockchain --witness-id '"1.6.1530"' --private-key '["MY_SIGNING_KEY","PRIVATE_KEY_OF_MY_SIGNING_KEY"]' -d test_net -s 45.55.6.216:1776 --genesis-json aug-14-test-genesis.json

[Thom]

I've got to step out briefly, but I hope someone will have addressed these questions when I get back. Puppies check out the PM I sent you as well.

Thanks.

[bytemaster]

Those who are reporting that they are stuck on a fork I am very interested in hearing more details.

If my understanding is correct it will fork if the block you produced doesn't get included in the final chain.   I will try to replicate this behavior in a test. 

[Thom]

I use -w test_wallet (see below), but there is no such folder created in the same folder as the cli_wallet binary. It doesn't seem to create one. Do I need to do that manually before launching the cli_wallet?

About the aug-genesis file, I downloaded it twice, checked it with md5 / sha256 and all copies are identical. The hashes I have for the file are:

md5: 743fef728e800f82bf381f10fb7f4b04
sha256: 83a757d971cfd7f6fe6793f98ff97e22f5637f077a8a2ffbeaaef0d48faac3ea

So this issue is probably where the file lives. If you recall I started with it in some arbitrary Downloads folder. I then moved it into the programs/witness_nodes folder. I use the -d test_net which also lives in the same folder as the binary executable, and uses has your config.ini that specifies the genesis file in it. Got any other ideas?

It seems folder / invocation issues have been common in this thread, that's what bit me yesterday in fact. Sounds like there are a few more to expose.

I start the witness and cli_wallet with shell scripts so it is the same every time. They are brain-dead simple, but it allows for consistency.

Code: [Select]

#!/bin/bash
cd ~/bts2.0/graphene/programs/witness_node;
./witness_node --rpc-endpoint "127.0.0.1:8090" --resync-blockchain --witness-id '"1.6.1530"' --private-key '["MY_SIGNING_KEY","PRIVATE_KEY_OF_MY_SIGNING_KEY"]' -d test_net -s 45.55.6.216:1776


Code: [Select]

#!/bin/bash
cd ~/bts2.0/graphene/programs/cli_wallet;
./cli_wallet -w test_wallet  --chain-id 081401ede64c8fe30b23c91d7ab8750103acb1a39548a866fb562f2edf4627d6