happen at block 19245

Code: [Select]

2971563ms th_a       application.cpp:356           handle_block         ] Got block #19244 from network
2971564ms th_a       api.cpp:867                   on_objects_changed   ] ids: ["2.8.19244","1.6.1","1.13.63","2.1.0"]
2971564ms th_a       api.cpp:867                   on_objects_changed   ] ids: ["2.8.19244","1.6.1","1.13.63","2.1.0"]
2975337ms th_a       application.cpp:356           handle_block         ] Got block #19246 from network
2975337ms th_a       fork_database.cpp:57          push_block           ] Pushing block to fork database that failed to link.
2976894ms th_a       application.cpp:356           handle_block         ] Got block #19247 from network
2976895ms th_a       fork_database.cpp:57          push_block           ] Pushing block to fork database that failed to link.
2977641ms th_a       application.cpp:356           handle_block         ] Got block #19248 from network
2977641ms th_a       fork_database.cpp:57          push_block           ] Pushing block to fork database that failed to link.
2982830ms th_a       application.cpp:356           handle_block         ] Got block #19249 from network
2982830ms th_a       fork_database.cpp:57          push_block           ] Pushing block to fork database that failed to link.
2985413ms th_a       application.cpp:356           handle_block         ] Got block #19250 from network
2985413ms th_a       fork_database.cpp:57          push_block           ] Pushing block to fork database that failed to link.
2991159ms th_a       application.cpp:356           handle_block         ] Got block #19251 from network
2991159ms th_a       fork_database.cpp:57          push_block           ] Pushing block to fork database that failed to link.
.......
75407ms th_a       application.cpp:356           handle_block         ] Got block #19379 from network
75407ms th_a       fork_database.cpp:57          push_block           ] Pushing block to fork database that failed to link.
80415ms th_a       application.cpp:356           handle_block         ] Got block #19380 from network
80415ms th_a       fork_database.cpp:57          push_block           ] Pushing block to fork database that failed to link.
1550122ms th_a       application.cpp:500           get_blockchain_synop ] reference_point: 0000000000000000000000000000000000000000 number_of_blocks_after_reference_point: 0 result: ["00000b2c27d70d418df26d4c6f7ea9c3bf278881","00002b2c4baf627959890ae6fcb7fa58a0a1fd89","00003b2c9362d330e2c2fc81702d2e365fac966f","0000432c4c5baebfdd5189f12edf44c1dd7c2d93","0000472c1bef0cf45f0e0be7fe2a60d34a83711c","0000492c197c3deeb618c982d64acdb26aa5427e","00004a2cfc255375ac702afb59a3c0516e77675f","00004aac0824351772902693a572330a2f5b5562","00004aec708a8a7265569807cc0aaf2907bb01f9","00004b0c8a04e3876fc3487ebef34d96b1e946d6","00004b1c50342d1c658d156a835e808f67a336fc","00004b247ee7327a4b4088921aa8317db8d009e5","00004b28f364f49e3e1c942a6368c8d190eba2a9","00004b2a5a9540578d4a3a03eac584f3acf88383","00004b2bad1237b1cf7c26dee14009b8812f4400","00004b2c721c0a7d372e05e78a846636be3c56e6"]
1551317ms th_a       application.cpp:378           handle_block         ] Error when pushing block:
10 assert_exception: Assert Exception
second_branch_itr != _index.get<block_id>().end():
    {}
    th_a  fork_database.cpp:194 fetch_branch_from

    {"first":"00004bb44c5ea6f09c4523f59c1ee2d4d4c46795","second":"00004b2c721c0a7d372e05e78a846636be3c56e6"}
    th_a  fork_database.cpp:225 fetch_branch_from

    {"new_block":{"previous":"00004b2c721c0a7d372e05e78a846636be3c56e6","timestamp":"2015-09-02T00:49:20","witness":"1.6.30","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"1f06a8c6497711b502229c690492345f8df531e3b88916acaed6cc69dba4ef31a0234bac5808e0c550f9b3c9f0627791c663fe07aee3d60d4f5777fb2f83fbf2d0","transactions":[]}}
    th_a  db_block.cpp:176 _push_block
1551320ms th_a       application.cpp:378           handle_block         ] Error when pushing block:
10 assert_exception: Assert Exception
item->num > std::max<int64_t>( 0, int64_t(_head->num) - (_max_size) ): attempting to push a block that is too old
    {"item->num":19246,"head":19380,"max_size":10}
    th_a  fork_database.cpp:69 _push_block
    {"new_block":{"previous":"00004b2d2c2ca421c5c1dea971536d794fb39909","timestamp":"2015-09-02T00:49:25","witness":"1.6.71","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"2000f6e8c6e35dfe28afdd202990416a3575852211b7d0b2872f5b69127da88bb9232179133fbaae82fb9c7af4d115552703ac668f9cdce4f103733099994e6ff6","transactions":[]}}
    th_a  db_block.cpp:176 _push_block
1551320ms th_a       application.cpp:378           handle_block         ] Error when pushing block:
10 assert_exception: Assert Exception
item->num > std::max<int64_t>( 0, int64_t(_head->num) - (_max_size) ): attempting to push a block that is too old
    {"item->num":19247,"head":19380,"max_size":10}
    th_a  fork_database.cpp:69 _push_block

    {"new_block":{"previous":"00004b2e2a410dbcbf7d98869f49d3965c4ff628","timestamp":"2015-09-02T00:49:30","witness":"1.6.92","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"2063834aa931f46e4650b465f130c88c489666e89f6635a5226f059185e6b1eeca4c12d552b3da728c58e4dc41aec3b425e0636e9ff408782371c17346e24203d1","transactions":[]}}
    th_a  db_block.cpp:176 _push_block
1551320ms th_a       application.cpp:378           handle_block         ] Error when pushing block:
10 assert_exception: Assert Exception
item->num > std::max<int64_t>( 0, int64_t(_head->num) - (_max_size) ): attempting to push a block that is too old
    {"item->num":19248,"head":19380,"max_size":10}
    th_a  fork_database.cpp:69 _push_block

    {"new_block":{"previous":"00004b2fbf3e595635934a9d1e0bc71c361650b8","timestamp":"2015-09-02T00:49:35","witness":"1.6.21","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"2050531102e02418d6369765e238355619d025960629043d2bf86af1b95a2c18336053cd369119a3b253fdca0c700d686187e7befaccfc4377f464c7f41987c64c","transactions":[]}}
    th_a  db_block.cpp:176 _push_block
1551321ms th_a       application.cpp:378           handle_block         ] Error when pushing block:
10 assert_exception: Assert Exception
item->num > std::max<int64_t>( 0, int64_t(_head->num) - (_max_size) ): attempting to push a block that is too old
    {"item->num":19249,"head":19380,"max_size":10}
    th_a  fork_database.cpp:69 _push_block

    {"new_block":{"previous":"00004b30452602f87ae249ca2b484965ff9bce20","timestamp":"2015-09-02T00:49:40","witness":"1.6.13","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"2007094c94e1f988276184214c83c584994974914d5b052e5c4060f2ad1b4baed634b6249c604ef8ffac4af06f8e92524aa4d399abe617cd14195175f947e67a4b","transactions":[]}}
    th_a  db_block.cpp:176 _push_block
1551321ms th_a       application.cpp:378           handle_block         ] Error when pushing block:
10 assert_exception: Assert Exception
item->num > std::max<int64_t>( 0, int64_t(_head->num) - (_max_size) ): attempting to push a block that is too old
    {"item->num":19250,"head":19380,"max_size":10}
    th_a  fork_database.cpp:69 _push_block

    {"new_block":{"previous":"00004b31be06fe671e4b289061961220d83b45d3","timestamp":"2015-09-02T00:49:45","witness":"1.6.54","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"20708f798868d17b3237e6e79a19e19397ba716c97cc92c881deb03190879904b04ffe7b3e8b50fab1ba20bd75b815bfd3ead8c5f407c480b7a2dd378faa369a74","transactions":[]}}
    th_a  db_block.cpp:176 _push_block
...................
tem->num > std::max<int64_t>( 0, int64_t(_head->num) - (_max_size) ): attempting to push a block that is too old
    {"item->num":19254,"head":19380,"max_size":10}
    th_a  fork_database.cpp:69 _push_block

    {"new_block":{"previous":"00004b356ca14f441e66d190ea1316a710fbc7c6","timestamp":"2015-09-02T00:50:20","witness":"1.6.19","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"206ce785ed9cac4a4eff9d2e3c6b5c7a1eefbbd6da7a00e9ad08c4b23aa16c130940e2de4421de0a7ff654cd7b75946aaa07de0bbc34a8ffe14634746b5e8b6d78","transactions":[]}}
    th_a  db_block.cpp:176 _push_block
1817912ms ntp        ntp.cpp:77                    request_now          ] resolving... ["pool.ntp.org",123]
1817919ms ntp        ntp.cpp:81                    request_now          ] sending request to 202.118.1.130:123
1817962ms ntp        ntp.cpp:147                   read_loop            ] received ntp reply from 202.118.1.130:123
1817962ms ntp        ntp.cpp:161                   read_loop            ] ntp offset: 618, round_trip_delay 43201
1817962ms ntp        ntp.cpp:177                   read_loop            ] ntp_delta_time updated to 618
1860115ms th_a       application.cpp:500           get_blockchain_synop ] reference_point: 0000000000000000000000000000000000000000 number_of_blocks_after_reference_point: 0 result: ["00000b2c27d70d418df26d4c6f7ea9c3bf278881","00002b2c4baf627959890ae6fcb7fa58a0a1fd89","00003b2c9362d330e2c2fc81702d2e365fac966f","0000432c4c5baebfdd5189f12edf44c1dd7c2d93","0000472c1bef0cf45f0e0be7fe2a60d34a83711c","0000492c197c3deeb618c982d64acdb26aa5427e","00004a2cfc255375ac702afb59a3c0516e77675f","00004aac0824351772902693a572330a2f5b5562","00004aec708a8a7265569807cc0aaf2907bb01f9","00004b0c8a04e3876fc3487ebef34d96b1e946d6","00004b1c50342d1c658d156a835e808f67a336fc","00004b247ee7327a4b4088921aa8317db8d009e5","00004b28f364f49e3e1c942a6368c8d190eba2a9","00004b2a5a9540578d4a3a03eac584f3acf88383","00004b2bad1237b1cf7c26dee14009b8812f4400","00004b2c721c0a7d372e05e78a846636be3c56e6"]
1861225ms th_a       application.cpp:378           handle_block         ] Error when pushing block:
10 assert_exception: Assert Exception
item->num > std::max<int64_t>( 0, int64_t(_head->num) - (_max_size) ): attempting to push a block that is too old
    {"item->num":19245,"head":19380,"max_size":10}
    th_a  fork_database.cpp:69 _push_block

    {"new_block":{"previous":"00004b2c721c0a7d372e05e78a846636be3c56e6","timestamp":"2015-09-02T00:49:20","witness":"1.6.30","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"1f06a8c6497711b502229c690492345f8df531e3b88916acaed6cc69dba4ef31a0234bac5808e0c550f9b3c9f0627791c663fe07aee3d60d4f5777fb2f83fbf2d0","transactions":[]}}
    th_a  db_block.cpp:176 _push_block
1861226ms th_a       application.cpp:378           handle_block         ] Error when pushing block:
10 assert_exception: Assert Exception
item->num > std::max<int64_t>( 0, int64_t(_head->num) - (_max_size) ): attempting to push a block that is too old
    {"item->num":19246,"head":19380,"max_size":10}
    th_a  fork_database.cpp:69 _push_block


Quote from: bytemaster on September 01, 2015, 01:01:29 pm

Quote from: abit on September 01, 2015, 11:58:20 am

Quote from: bytemaster on August 26, 2015, 01:15:44 pm

Quote from: alt on August 26, 2015, 09:13:00 am

several times, my witeness node blocked at some block
I watch the p2p log, it received block 419543, then 419544, then missed 419545, then received 419546, .......
and it will never receive block 419545, I have to  restart witness node, then it can request the block from other node.
I think we need a logic, when the witness node have missed a block more than 10 blocks, it should request from other node.

Eric checked in some fixes that may address this particular problem.    We do have logic to request from another node, but the timeouts were still tuned to BTS 1 timeframes so we reduced the timeout for BTS 2.

Still have same problem with test2 tag.
Please check this issue https://github.com/cryptonomex/graphene/issues/264

In the test2b tag I have increase the allowance for out-of-order blocks.  If it still happens then something else is going on.

Same here at block #20188 with tag test2b.

2015-09-02T02:12:15 th_a:invoke handle_block         handle_block ] Got block #20187 from network                       application.cpp:356
2015-09-02T02:12:15 th_a:invoke handle_block   on_objects_changed ] ids: ["2.8.20187","1.6.64","1.13.72","2.1.0"]
        api.cpp:867
2015-09-02T02:12:15 th_a:invoke handle_block   on_objects_changed ] ids: ["2.8.20187","1.6.64","1.13.72","2.1.0"]
        api.cpp:867
2015-09-02T02:12:25 th_a:invoke handle_block         handle_block ] Got block #20189 from network                       application.cpp:356
2015-09-02T02:12:25 th_a:invoke handle_block           push_block ] Pushing block to fork database that failed to link.
        fork_database.cpp:57
2015-09-02T02:12:30 th_a:invoke handle_block         handle_block ] Got block #20190 from network                       application.cpp:356
2015-09-02T02:12:30 th_a:invoke handle_block           push_block ] Pushing block to fork database that failed to link.
        fork_database.cpp:57
...
...
...
2015-09-02T02:20:22 th_a:invoke handle_block         handle_block ] Error when pushing block:
10 assert_exception: Assert Exception
item->num > std::max<int64_t>( 0, int64_t(_head->num) - (_max_size) ): attempting to push a block that is too old
    {"item->num":20188,"head":20278,"max_size":10}
    th_a  fork_database.cpp:69 _push_block

    {"new_block":{"previous":"00004edb437dfbc741e818a8a9754a653da79111","timestamp":"2015-09-02T02:12:20","witness":"1.6.3706","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"1f0f14d642abd9b0d26e0d2e5960ad5b4e1d340282317fd1948f942e1f857804393bbeca54636b9f3ad68e44f9843648aa403fb5dc13024649d259558db3c119f1","transactions":[]}}
    th_a  db_block.cpp:176 _push_block                  application.cpp:378

See https://github.com/cryptonomex/graphene/issues/264

Quote from: lafona on September 01, 2015, 12:08:05 am

Anyone else getting this when trying to update the submodule

Code: [Select]

fatal: reference is not a tree: 80d967a70d21d26d27ef3a1544a177925b2a7bbe
Unable to checkout '80d967a70d21d26d27ef3a1544a177925b2a7bbe' in submodule path 'libraries/fc'

Sorry about that, I goofed -- I forgot to push a commit to fc.  Try again, it should be good now.

Yes, the test2b tag works fine.
I'm running.

Quote from: alt on August 26, 2015, 09:13:00 am

several times, my witeness node blocked at some block
I watch the p2p log, it received block 419543, then 419544, then missed 419545, then received 419546, .......
and it will never receive block 419545, I have to  restart witness node, then it can request the block from other node.
I think we need a logic, when the witness node have missed a block more than 10 blocks, it should request from other node.

Eric checked in some fixes that may address this particular problem.    We do have logic to request from another node, but the timeouts were still tuned to BTS 1 timeframes so we reduced the timeout for BTS 2.

Still have same problem with test2 tag.
Please check this issue https://github.com/cryptonomex/graphene/issues/264

Quote from: puppies on September 01, 2015, 12:57:11 am

build failed for me.  Please let me know if you find a way around it.

Code: [Select]

/home/user/src/graphene/libraries/plugins/witness/witness.cpp:229:10: note: in expansion of macro ‘elog’
          elog("Not producing block because node appears to be on a minority fork with only ${pct}% witness participation", (capture) );
          ^
make[2]: *** [libraries/plugins/witness/CMakeFiles/graphene_witness.dir/witness.cpp.o] Error 1
make[1]: *** [libraries/plugins/witness/CMakeFiles/graphene_witness.dir/all] Error 2
make: *** [all] Error 2
user@user-desktop:~/src/graphene$



Same error occurred. Hopefully devs will fix it ASAP.

Fixed by editing this file:
libraries/plugins/witness/witness.cpp
line 214:
         ilog("Generated block #${n} with timestamp ${t} at time ${c}", ("n",capture)("t",capture)("c",capture));
line 226:
         ilog("Not producing block because I don't have the private key for ${scheduled_key}", ("scheduled_key",capture) );
line 229:
         elog("Not producing block because node appears to be on a minority fork with only ${pct}% witness participation", ("pct",capture) );

Pull request created: https://github.com/cryptonomex/graphene/pull/285

Hmm, the latest build seems much more stable than before. BTW, is flood_network command disabled? I got a segmentation fault error.

Not so stable for me.. I still encounter segment fault error frequently. Like https://github.com/cryptonomex/graphene/issues/261. Perhaps because of my unstable network connection?

Anyway re-indexing is much faster than before, that's good.

That has always allowed me to restart, but now I can't without this crashing. Here's the output:

Code: [Select]

./witness: line 7:  32471 Segmentation fault      ./witness_node --resync-blockchain -d test_net

I also encountered segment fault problem, most time is due to producing a block while out of sync. See https://github.com/cryptonomex/graphene/issues/261

If the program is crashing, try run it inside gdb:

$ gdb
...
(gdb) file ./witness_node
...
(gdb) set args --resync-blockchain -d test_net
...
(gdb) run
...

Anytime you can end the program by pressing ctrl+C, then typing signal SIGINT:
(gdb) signal SIGINT

After crash, dump the back trace and post it here or to Github.

I'm going to restart after clearing both the blockchain & p2p folders. My cmd line invocation is: ./witness_node --resync-blockchain -d test_net --enable-stale-production, all other parameters are in the config.ini.  After restarting the blockchain age reported by info started at 4 days and is now down to 57 hours. The witness output is much different than before I wiped the blk chn & p2p folders. It looks like I may finally be on the verge of producing blocks! I now see my total_votes is no longer zero.

Don't use '--enable-stale-production' parameter. It's for the init node.

Is there a checkpoint I can use to resync faster, should I need to restart?

Try start with more seed nodes:
-s "104.236.51.238:1776" -s "176.221.43.130:33323" -s "45.55.6.216:1776" -s "114.92.254.159:62015" -s 104.156.226.183:60715 -s 104.156.226.183:40479 -s 104.236.255.53:52995 -s 176.9.234.167:34858 -s 176.9.234.167:57727 -s 178.62.88.151:59148 -s 178.62.88.151:41574 -s 188.226.252.109:58843 -s 45.115.36.171:57281 -s 45.55.6.216:37308


Quote from: betax on August 24, 2015, 11:15:36 am

Quote from: abit on August 24, 2015, 11:00:15 am

Code: [Select]

3508896ms th_a       application.cpp:265           startup              ] Detected unclean shutdown. Replaying blockchain...
It takes too much time to replay blockchain.. How to "cleanly" shutdown the witness node?

Question bump.

I use this strategy:
When I am synced, I shutdown with C-c (it should shutdown in a clean way) and I copy the blockchain folder as backup (if it shutdown without errors).
Everytime my blockchain is corrupted, I remove the blockchain folder and I copy with the backup one and restart the witness.
Finally I backup the blockchain folder every day.

Thanks, it works. 

If witness_node was launched in gdb, we need to press ctrl+c, then from the (gdb) prompt, type 'signal SIGINT'. This will send SIGINT to the program being debugged. I always type 'kill' before I know this, so the program always shutdown uncleanly.

I have another problem, when I restart witness_node, usually it takes a long time to get connected to network, and sometimes won't get connected at all.