October 5 Test Network

[clayop]

I can replay it on my machine in:

Code: [Select]

456696ms th_a       db_management.cpp:93          reindex              ] Done reindexing, elapsed time: 6.88446099999999994 sec

I built with -DCMAKE_BUILD_TYPE=Debug, that would be the reason.

 

[abit]

I can replay it on my machine in:

Code: [Select]

456696ms th_a       db_management.cpp:93          reindex              ] Done reindexing, elapsed time: 6.88446099999999994 sec

I built with -DCMAKE_BUILD_TYPE=Debug, that would be the reason.

[bytemaster]

I can replay it on my machine in:

Code: [Select]

456696ms th_a       db_management.cpp:93          reindex              ] Done reindexing, elapsed time: 6.88446099999999994 sec

[abit]

Every day so many witnesses are not online, what is the reason?

We've identified an issue where the assignment of new object ID's is inconsistent (we use a hashed index with an undefined iteration order).  Fortunately the testnet's main fork has them in increasing order (I suspect it has to do with the memory allocator tending to put things allocated later at higher addresses), in the next hardfork this condition will be enforced (by replacing most of our hashed indexes with ordered indexes).  We suspect some witnesses are getting their state wrong when they assign an ID differently from the main network, if someone publishes a transaction using that ID then the wrong state turns into a desync (they'll no longer sign or pay attention to blocks on the main fork because they think the main fork has an illegal transaction.)

If you get stuck and the first error message mentions a vesting balance object, this is almost certainly what's happening to you.

The next hardfork will fix this issue, and in addition, I'm working on a tool to help us find if there are any similar issues.

The transactions being executed by the community testers have been incredibly helpful to us.

Running with latest commit (e68e99ed3ae11ddac607e983e7549e8278fdecc4), but it can't go over block 70872

Code: [Select]

2015-10-08T21:37:17 p2p:send_sync_block_to_node_delegate send_sync_block_to_n ] Failed to push sync block 70872 (id:000114d879ab3c305edeaaae837c2b68b7da64d2): client rejected sync block sent by peer: {"code":10,"name":"assert_exception","message":"Assert Exception","stack":[{"context":{"level":"error","file":"vesting_balance_evaluator.cpp","line":103,"method":"do_evaluate","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T21:37:17"},"format":"vbo.is_withdraw_allowed( now, op.amount ): ","data":{"now":"2015-10-08T05:43:09","op":{"fee":{"amount":100000,"asset_id":"1.3.0"},"vesting_balance":"1.13.60","owner":"1.2.8517","amount":{"amount":45300000,"asset_id":"1.3.0"}},"vbo":{"id":"1.13.60","owner":"1.2.8517","balance":{"amount":197500000,"asset_id":"1.3.0"},"policy":[1,{"vesting_seconds":86400,"start_claim":"1970-01-01T00:00:00","coin_seconds_earned":"3341610000000","coin_seconds_earned_last_update":"2015-10-08T05:43:06"}]}}},{"context":{"level":"warn","file":"vesting_balance_evaluator.cpp","line":109,"method":"do_evaluate","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T21:37:17"},"format":"","data":{"op":{"fee":{"amount":100000,"asset_id":"1.3.0"},"vesting_balance":"1.13.60","owner":"1.2.8517","amount":{"amount":45300000,"asset_id":"1.3.0"}}}},{"context":{"level":"warn","file":"evaluator.cpp","line":42,"method":"start_evaluate","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T21:37:17"},"format":"","data":{}},{"context":{"level":"warn","file":"db_block.cpp","line":628,"method":"apply_operation","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T21:37:17"},"format":"","data":{}},{"context":{"level":"warn","file":"db_block.cpp","line":611,"method":"_apply_transaction","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T21:37:17"},"format":"","data":{"trx":{"ref_block_num":5335,"ref_block_prefix":4159425851,"expiration":"2015-10-08T05:43:39","operations":[[33,{"fee":{"amount":100000,"asset_id":"1.3.0"},"vesting_balance":"1.13.60","owner":"1.2.8517","amount":{"amount":45300000,"asset_id":"1.3.0"}}]],"extensions":[],"signatures":["1f6f7ea6a4188c8fc36aaf457b5fe4433fb74af11b250507262a00d909af025ad065c03ef0f61925070489d1630a3fbe459795b265200544de9ba1694414fefbcf"]}}},{"context":{"level":"warn","file":"db_block.cpp","line":514,"method":"_apply_block","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T21:37:17"},"format":"","data":{"next_block.block_num()":70872}},{"context":{"level":"warn","file":"db_block.cpp","line":197,"method":"_push_block","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T21:37:17"},"format":"","data":{"new_block":{"previous":"000114d73bcdebf717450d1a45e7ea119844b22e","timestamp":"2015-10-08T05:43:15","witness":"1.6.10","transaction_merkle_root":"05b459df123ff6baf5dbc44890e6ae3e8a8caaf6","extensions":[],"witness_signature":"20232fc2bb2ead0cbc644d78e4462881dc812787ad28f1517229cc346c252acb2904cb13139a8e957c8087d8b03790f12f5c2ef0dbb2cfceead7a47521f75f0950","transactions":[{"ref_block_num":5335,"ref_block_prefix":4159425851,"expiration":"2015-10-08T05:43:39","operations":[[33,{"fee":{"amount":100000,"asset_id":"1.3.0"},"vesting_balance":"1.13.60","owner":"1.2.8517","amount":{"amount":45300000,"asset_id":"1.3.0"}}]],"extensions":[],"signatures":["1f6f7ea6a4188c8fc36aaf457b5fe4433fb74af11b250507262a00d909af025ad065c03ef0f61925070489d1630a3fbe459795b265200544de9ba1694414fefbcf"],"operation_results":[[0,{}]]}]}}},{"context":{"level":"warn","file":"application.cpp","line":441,"method":"handle_block","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T21:37:17"},"format":"","data":{"blk_msg":{"block":{"previous":"000114d73bcdebf717450d1a45e7ea119844b22e","timestamp":"2015-10-08T05:43:15","witness":"1.6.10","transaction_merkle_root":"05b459df123ff6baf5dbc44890e6ae3e8a8caaf6","extensions":[],"witness_signature":"20232fc2bb2ead0cbc644d78e4462881dc812787ad28f1517229cc346c252acb2904cb13139a8e957c8087d8b03790f12f5c2ef0dbb2cfceead7a47521f75f0950","transactions":[{"ref_block_num":5335,"ref_block_prefix":4159425851,"expiration":"2015-10-08T05:43:39","operations":[[33,{"fee":{"amount":100000,"asset_id":"1.3.0"},"vesting_balance":"1.13.60","owner":"1.2.8517","amount":{"amount":45300000,"asset_id":"1.3.0"}}]],"extensions":[],"signatures":["1f6f7ea6a4188c8fc36aaf457b5fe4433fb74af11b250507262a00d909af025ad065c03ef0f61925070489d1630a3fbe459795b265200544de9ba1694414fefbcf"],"operation_results":[[0,{}]]}]},"block_id":"000114d879ab3c305edeaaae837c2b68b7da64d2"},"sync_mode":true}}]}                    node.cpp:3043
2015-10-08T21:37:17 p2p:send_sync_block_to_node_delegate send_sync_block_to_n ] disconnecting client 104.236.51.238:2005 because it offered us the rejected block                       node.cpp:3157



Did you do --replay-blockchain?    What OS?

I was running with a backup chain folder which is created about 16 hours ago, so I don't think it's need to replay.
Ubuntu 14.04 LTS 64 bit.

Code: [Select]

Linux 3.16.0-44-generic #59~14.04.1-Ubuntu SMP Tue Jul 7 15:07:27 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux


It absolutely needs to replay because prior to the last hour or two the master branch was producing non-deterministic ID allocation and your old backup probably imported that.

The backup is at block 65916. Replaying.

//Update:
in sync now.
It took 67 seconds to replay a 87261-block chain.

[bytemaster]

Every day so many witnesses are not online, what is the reason?

We've identified an issue where the assignment of new object ID's is inconsistent (we use a hashed index with an undefined iteration order).  Fortunately the testnet's main fork has them in increasing order (I suspect it has to do with the memory allocator tending to put things allocated later at higher addresses), in the next hardfork this condition will be enforced (by replacing most of our hashed indexes with ordered indexes).  We suspect some witnesses are getting their state wrong when they assign an ID differently from the main network, if someone publishes a transaction using that ID then the wrong state turns into a desync (they'll no longer sign or pay attention to blocks on the main fork because they think the main fork has an illegal transaction.)

If you get stuck and the first error message mentions a vesting balance object, this is almost certainly what's happening to you.

The next hardfork will fix this issue, and in addition, I'm working on a tool to help us find if there are any similar issues.

The transactions being executed by the community testers have been incredibly helpful to us.

Running with latest commit (e68e99ed3ae11ddac607e983e7549e8278fdecc4), but it can't go over block 70872

Code: [Select]

2015-10-08T21:37:17 p2p:send_sync_block_to_node_delegate send_sync_block_to_n ] Failed to push sync block 70872 (id:000114d879ab3c305edeaaae837c2b68b7da64d2): client rejected sync block sent by peer: {"code":10,"name":"assert_exception","message":"Assert Exception","stack":[{"context":{"level":"error","file":"vesting_balance_evaluator.cpp","line":103,"method":"do_evaluate","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T21:37:17"},"format":"vbo.is_withdraw_allowed( now, op.amount ): ","data":{"now":"2015-10-08T05:43:09","op":{"fee":{"amount":100000,"asset_id":"1.3.0"},"vesting_balance":"1.13.60","owner":"1.2.8517","amount":{"amount":45300000,"asset_id":"1.3.0"}},"vbo":{"id":"1.13.60","owner":"1.2.8517","balance":{"amount":197500000,"asset_id":"1.3.0"},"policy":[1,{"vesting_seconds":86400,"start_claim":"1970-01-01T00:00:00","coin_seconds_earned":"3341610000000","coin_seconds_earned_last_update":"2015-10-08T05:43:06"}]}}},{"context":{"level":"warn","file":"vesting_balance_evaluator.cpp","line":109,"method":"do_evaluate","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T21:37:17"},"format":"","data":{"op":{"fee":{"amount":100000,"asset_id":"1.3.0"},"vesting_balance":"1.13.60","owner":"1.2.8517","amount":{"amount":45300000,"asset_id":"1.3.0"}}}},{"context":{"level":"warn","file":"evaluator.cpp","line":42,"method":"start_evaluate","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T21:37:17"},"format":"","data":{}},{"context":{"level":"warn","file":"db_block.cpp","line":628,"method":"apply_operation","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T21:37:17"},"format":"","data":{}},{"context":{"level":"warn","file":"db_block.cpp","line":611,"method":"_apply_transaction","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T21:37:17"},"format":"","data":{"trx":{"ref_block_num":5335,"ref_block_prefix":4159425851,"expiration":"2015-10-08T05:43:39","operations":[[33,{"fee":{"amount":100000,"asset_id":"1.3.0"},"vesting_balance":"1.13.60","owner":"1.2.8517","amount":{"amount":45300000,"asset_id":"1.3.0"}}]],"extensions":[],"signatures":["1f6f7ea6a4188c8fc36aaf457b5fe4433fb74af11b250507262a00d909af025ad065c03ef0f61925070489d1630a3fbe459795b265200544de9ba1694414fefbcf"]}}},{"context":{"level":"warn","file":"db_block.cpp","line":514,"method":"_apply_block","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T21:37:17"},"format":"","data":{"next_block.block_num()":70872}},{"context":{"level":"warn","file":"db_block.cpp","line":197,"method":"_push_block","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T21:37:17"},"format":"","data":{"new_block":{"previous":"000114d73bcdebf717450d1a45e7ea119844b22e","timestamp":"2015-10-08T05:43:15","witness":"1.6.10","transaction_merkle_root":"05b459df123ff6baf5dbc44890e6ae3e8a8caaf6","extensions":[],"witness_signature":"20232fc2bb2ead0cbc644d78e4462881dc812787ad28f1517229cc346c252acb2904cb13139a8e957c8087d8b03790f12f5c2ef0dbb2cfceead7a47521f75f0950","transactions":[{"ref_block_num":5335,"ref_block_prefix":4159425851,"expiration":"2015-10-08T05:43:39","operations":[[33,{"fee":{"amount":100000,"asset_id":"1.3.0"},"vesting_balance":"1.13.60","owner":"1.2.8517","amount":{"amount":45300000,"asset_id":"1.3.0"}}]],"extensions":[],"signatures":["1f6f7ea6a4188c8fc36aaf457b5fe4433fb74af11b250507262a00d909af025ad065c03ef0f61925070489d1630a3fbe459795b265200544de9ba1694414fefbcf"],"operation_results":[[0,{}]]}]}}},{"context":{"level":"warn","file":"application.cpp","line":441,"method":"handle_block","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T21:37:17"},"format":"","data":{"blk_msg":{"block":{"previous":"000114d73bcdebf717450d1a45e7ea119844b22e","timestamp":"2015-10-08T05:43:15","witness":"1.6.10","transaction_merkle_root":"05b459df123ff6baf5dbc44890e6ae3e8a8caaf6","extensions":[],"witness_signature":"20232fc2bb2ead0cbc644d78e4462881dc812787ad28f1517229cc346c252acb2904cb13139a8e957c8087d8b03790f12f5c2ef0dbb2cfceead7a47521f75f0950","transactions":[{"ref_block_num":5335,"ref_block_prefix":4159425851,"expiration":"2015-10-08T05:43:39","operations":[[33,{"fee":{"amount":100000,"asset_id":"1.3.0"},"vesting_balance":"1.13.60","owner":"1.2.8517","amount":{"amount":45300000,"asset_id":"1.3.0"}}]],"extensions":[],"signatures":["1f6f7ea6a4188c8fc36aaf457b5fe4433fb74af11b250507262a00d909af025ad065c03ef0f61925070489d1630a3fbe459795b265200544de9ba1694414fefbcf"],"operation_results":[[0,{}]]}]},"block_id":"000114d879ab3c305edeaaae837c2b68b7da64d2"},"sync_mode":true}}]}                    node.cpp:3043
2015-10-08T21:37:17 p2p:send_sync_block_to_node_delegate send_sync_block_to_n ] disconnecting client 104.236.51.238:2005 because it offered us the rejected block                       node.cpp:3157



Did you do --replay-blockchain?    What OS?

I was running with a backup chain folder which is created about 16 hours ago, so I don't think it's need to replay.
Ubuntu 14.04 LTS 64 bit.

Code: [Select]

Linux 3.16.0-44-generic #59~14.04.1-Ubuntu SMP Tue Jul 7 15:07:27 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux


It absolutely needs to replay because prior to the last hour or two the master branch was producing non-deterministic ID allocation and your old backup probably imported that.

[abit]

Every day so many witnesses are not online, what is the reason?

We've identified an issue where the assignment of new object ID's is inconsistent (we use a hashed index with an undefined iteration order).  Fortunately the testnet's main fork has them in increasing order (I suspect it has to do with the memory allocator tending to put things allocated later at higher addresses), in the next hardfork this condition will be enforced (by replacing most of our hashed indexes with ordered indexes).  We suspect some witnesses are getting their state wrong when they assign an ID differently from the main network, if someone publishes a transaction using that ID then the wrong state turns into a desync (they'll no longer sign or pay attention to blocks on the main fork because they think the main fork has an illegal transaction.)

If you get stuck and the first error message mentions a vesting balance object, this is almost certainly what's happening to you.

The next hardfork will fix this issue, and in addition, I'm working on a tool to help us find if there are any similar issues.

The transactions being executed by the community testers have been incredibly helpful to us.

Running with latest commit (e68e99ed3ae11ddac607e983e7549e8278fdecc4), but it can't go over block 70872

Code: [Select]

2015-10-08T21:37:17 p2p:send_sync_block_to_node_delegate send_sync_block_to_n ] Failed to push sync block 70872 (id:000114d879ab3c305edeaaae837c2b68b7da64d2): client rejected sync block sent by peer: {"code":10,"name":"assert_exception","message":"Assert Exception","stack":[{"context":{"level":"error","file":"vesting_balance_evaluator.cpp","line":103,"method":"do_evaluate","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T21:37:17"},"format":"vbo.is_withdraw_allowed( now, op.amount ): ","data":{"now":"2015-10-08T05:43:09","op":{"fee":{"amount":100000,"asset_id":"1.3.0"},"vesting_balance":"1.13.60","owner":"1.2.8517","amount":{"amount":45300000,"asset_id":"1.3.0"}},"vbo":{"id":"1.13.60","owner":"1.2.8517","balance":{"amount":197500000,"asset_id":"1.3.0"},"policy":[1,{"vesting_seconds":86400,"start_claim":"1970-01-01T00:00:00","coin_seconds_earned":"3341610000000","coin_seconds_earned_last_update":"2015-10-08T05:43:06"}]}}},{"context":{"level":"warn","file":"vesting_balance_evaluator.cpp","line":109,"method":"do_evaluate","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T21:37:17"},"format":"","data":{"op":{"fee":{"amount":100000,"asset_id":"1.3.0"},"vesting_balance":"1.13.60","owner":"1.2.8517","amount":{"amount":45300000,"asset_id":"1.3.0"}}}},{"context":{"level":"warn","file":"evaluator.cpp","line":42,"method":"start_evaluate","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T21:37:17"},"format":"","data":{}},{"context":{"level":"warn","file":"db_block.cpp","line":628,"method":"apply_operation","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T21:37:17"},"format":"","data":{}},{"context":{"level":"warn","file":"db_block.cpp","line":611,"method":"_apply_transaction","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T21:37:17"},"format":"","data":{"trx":{"ref_block_num":5335,"ref_block_prefix":4159425851,"expiration":"2015-10-08T05:43:39","operations":[[33,{"fee":{"amount":100000,"asset_id":"1.3.0"},"vesting_balance":"1.13.60","owner":"1.2.8517","amount":{"amount":45300000,"asset_id":"1.3.0"}}]],"extensions":[],"signatures":["1f6f7ea6a4188c8fc36aaf457b5fe4433fb74af11b250507262a00d909af025ad065c03ef0f61925070489d1630a3fbe459795b265200544de9ba1694414fefbcf"]}}},{"context":{"level":"warn","file":"db_block.cpp","line":514,"method":"_apply_block","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T21:37:17"},"format":"","data":{"next_block.block_num()":70872}},{"context":{"level":"warn","file":"db_block.cpp","line":197,"method":"_push_block","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T21:37:17"},"format":"","data":{"new_block":{"previous":"000114d73bcdebf717450d1a45e7ea119844b22e","timestamp":"2015-10-08T05:43:15","witness":"1.6.10","transaction_merkle_root":"05b459df123ff6baf5dbc44890e6ae3e8a8caaf6","extensions":[],"witness_signature":"20232fc2bb2ead0cbc644d78e4462881dc812787ad28f1517229cc346c252acb2904cb13139a8e957c8087d8b03790f12f5c2ef0dbb2cfceead7a47521f75f0950","transactions":[{"ref_block_num":5335,"ref_block_prefix":4159425851,"expiration":"2015-10-08T05:43:39","operations":[[33,{"fee":{"amount":100000,"asset_id":"1.3.0"},"vesting_balance":"1.13.60","owner":"1.2.8517","amount":{"amount":45300000,"asset_id":"1.3.0"}}]],"extensions":[],"signatures":["1f6f7ea6a4188c8fc36aaf457b5fe4433fb74af11b250507262a00d909af025ad065c03ef0f61925070489d1630a3fbe459795b265200544de9ba1694414fefbcf"],"operation_results":[[0,{}]]}]}}},{"context":{"level":"warn","file":"application.cpp","line":441,"method":"handle_block","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T21:37:17"},"format":"","data":{"blk_msg":{"block":{"previous":"000114d73bcdebf717450d1a45e7ea119844b22e","timestamp":"2015-10-08T05:43:15","witness":"1.6.10","transaction_merkle_root":"05b459df123ff6baf5dbc44890e6ae3e8a8caaf6","extensions":[],"witness_signature":"20232fc2bb2ead0cbc644d78e4462881dc812787ad28f1517229cc346c252acb2904cb13139a8e957c8087d8b03790f12f5c2ef0dbb2cfceead7a47521f75f0950","transactions":[{"ref_block_num":5335,"ref_block_prefix":4159425851,"expiration":"2015-10-08T05:43:39","operations":[[33,{"fee":{"amount":100000,"asset_id":"1.3.0"},"vesting_balance":"1.13.60","owner":"1.2.8517","amount":{"amount":45300000,"asset_id":"1.3.0"}}]],"extensions":[],"signatures":["1f6f7ea6a4188c8fc36aaf457b5fe4433fb74af11b250507262a00d909af025ad065c03ef0f61925070489d1630a3fbe459795b265200544de9ba1694414fefbcf"],"operation_results":[[0,{}]]}]},"block_id":"000114d879ab3c305edeaaae837c2b68b7da64d2"},"sync_mode":true}}]}                    node.cpp:3043
2015-10-08T21:37:17 p2p:send_sync_block_to_node_delegate send_sync_block_to_n ] disconnecting client 104.236.51.238:2005 because it offered us the rejected block                       node.cpp:3157



Did you do --replay-blockchain?    What OS?

I was running with a backup chain folder which is created about 16 hours ago, so I don't think it's need to replay.
Ubuntu 14.04 LTS 64 bit.

Code: [Select]

Linux 3.16.0-44-generic #59~14.04.1-Ubuntu SMP Tue Jul 7 15:07:27 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
Anyway I'll try replay.

[bytemaster]

Every day so many witnesses are not online, what is the reason?

We've identified an issue where the assignment of new object ID's is inconsistent (we use a hashed index with an undefined iteration order).  Fortunately the testnet's main fork has them in increasing order (I suspect it has to do with the memory allocator tending to put things allocated later at higher addresses), in the next hardfork this condition will be enforced (by replacing most of our hashed indexes with ordered indexes).  We suspect some witnesses are getting their state wrong when they assign an ID differently from the main network, if someone publishes a transaction using that ID then the wrong state turns into a desync (they'll no longer sign or pay attention to blocks on the main fork because they think the main fork has an illegal transaction.)

If you get stuck and the first error message mentions a vesting balance object, this is almost certainly what's happening to you.

The next hardfork will fix this issue, and in addition, I'm working on a tool to help us find if there are any similar issues.

The transactions being executed by the community testers have been incredibly helpful to us.

Running with latest commit (e68e99ed3ae11ddac607e983e7549e8278fdecc4), but it can't go over block 70872

Code: [Select]

2015-10-08T21:37:17 p2p:send_sync_block_to_node_delegate send_sync_block_to_n ] Failed to push sync block 70872 (id:000114d879ab3c305edeaaae837c2b68b7da64d2): client rejected sync block sent by peer: {"code":10,"name":"assert_exception","message":"Assert Exception","stack":[{"context":{"level":"error","file":"vesting_balance_evaluator.cpp","line":103,"method":"do_evaluate","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T21:37:17"},"format":"vbo.is_withdraw_allowed( now, op.amount ): ","data":{"now":"2015-10-08T05:43:09","op":{"fee":{"amount":100000,"asset_id":"1.3.0"},"vesting_balance":"1.13.60","owner":"1.2.8517","amount":{"amount":45300000,"asset_id":"1.3.0"}},"vbo":{"id":"1.13.60","owner":"1.2.8517","balance":{"amount":197500000,"asset_id":"1.3.0"},"policy":[1,{"vesting_seconds":86400,"start_claim":"1970-01-01T00:00:00","coin_seconds_earned":"3341610000000","coin_seconds_earned_last_update":"2015-10-08T05:43:06"}]}}},{"context":{"level":"warn","file":"vesting_balance_evaluator.cpp","line":109,"method":"do_evaluate","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T21:37:17"},"format":"","data":{"op":{"fee":{"amount":100000,"asset_id":"1.3.0"},"vesting_balance":"1.13.60","owner":"1.2.8517","amount":{"amount":45300000,"asset_id":"1.3.0"}}}},{"context":{"level":"warn","file":"evaluator.cpp","line":42,"method":"start_evaluate","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T21:37:17"},"format":"","data":{}},{"context":{"level":"warn","file":"db_block.cpp","line":628,"method":"apply_operation","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T21:37:17"},"format":"","data":{}},{"context":{"level":"warn","file":"db_block.cpp","line":611,"method":"_apply_transaction","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T21:37:17"},"format":"","data":{"trx":{"ref_block_num":5335,"ref_block_prefix":4159425851,"expiration":"2015-10-08T05:43:39","operations":[[33,{"fee":{"amount":100000,"asset_id":"1.3.0"},"vesting_balance":"1.13.60","owner":"1.2.8517","amount":{"amount":45300000,"asset_id":"1.3.0"}}]],"extensions":[],"signatures":["1f6f7ea6a4188c8fc36aaf457b5fe4433fb74af11b250507262a00d909af025ad065c03ef0f61925070489d1630a3fbe459795b265200544de9ba1694414fefbcf"]}}},{"context":{"level":"warn","file":"db_block.cpp","line":514,"method":"_apply_block","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T21:37:17"},"format":"","data":{"next_block.block_num()":70872}},{"context":{"level":"warn","file":"db_block.cpp","line":197,"method":"_push_block","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T21:37:17"},"format":"","data":{"new_block":{"previous":"000114d73bcdebf717450d1a45e7ea119844b22e","timestamp":"2015-10-08T05:43:15","witness":"1.6.10","transaction_merkle_root":"05b459df123ff6baf5dbc44890e6ae3e8a8caaf6","extensions":[],"witness_signature":"20232fc2bb2ead0cbc644d78e4462881dc812787ad28f1517229cc346c252acb2904cb13139a8e957c8087d8b03790f12f5c2ef0dbb2cfceead7a47521f75f0950","transactions":[{"ref_block_num":5335,"ref_block_prefix":4159425851,"expiration":"2015-10-08T05:43:39","operations":[[33,{"fee":{"amount":100000,"asset_id":"1.3.0"},"vesting_balance":"1.13.60","owner":"1.2.8517","amount":{"amount":45300000,"asset_id":"1.3.0"}}]],"extensions":[],"signatures":["1f6f7ea6a4188c8fc36aaf457b5fe4433fb74af11b250507262a00d909af025ad065c03ef0f61925070489d1630a3fbe459795b265200544de9ba1694414fefbcf"],"operation_results":[[0,{}]]}]}}},{"context":{"level":"warn","file":"application.cpp","line":441,"method":"handle_block","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T21:37:17"},"format":"","data":{"blk_msg":{"block":{"previous":"000114d73bcdebf717450d1a45e7ea119844b22e","timestamp":"2015-10-08T05:43:15","witness":"1.6.10","transaction_merkle_root":"05b459df123ff6baf5dbc44890e6ae3e8a8caaf6","extensions":[],"witness_signature":"20232fc2bb2ead0cbc644d78e4462881dc812787ad28f1517229cc346c252acb2904cb13139a8e957c8087d8b03790f12f5c2ef0dbb2cfceead7a47521f75f0950","transactions":[{"ref_block_num":5335,"ref_block_prefix":4159425851,"expiration":"2015-10-08T05:43:39","operations":[[33,{"fee":{"amount":100000,"asset_id":"1.3.0"},"vesting_balance":"1.13.60","owner":"1.2.8517","amount":{"amount":45300000,"asset_id":"1.3.0"}}]],"extensions":[],"signatures":["1f6f7ea6a4188c8fc36aaf457b5fe4433fb74af11b250507262a00d909af025ad065c03ef0f61925070489d1630a3fbe459795b265200544de9ba1694414fefbcf"],"operation_results":[[0,{}]]}]},"block_id":"000114d879ab3c305edeaaae837c2b68b7da64d2"},"sync_mode":true}}]}                    node.cpp:3043
2015-10-08T21:37:17 p2p:send_sync_block_to_node_delegate send_sync_block_to_n ] disconnecting client 104.236.51.238:2005 because it offered us the rejected block                       node.cpp:3157



Did you do --replay-blockchain?    What OS?

[abit]

Every day so many witnesses are not online, what is the reason?

We've identified an issue where the assignment of new object ID's is inconsistent (we use a hashed index with an undefined iteration order).  Fortunately the testnet's main fork has them in increasing order (I suspect it has to do with the memory allocator tending to put things allocated later at higher addresses), in the next hardfork this condition will be enforced (by replacing most of our hashed indexes with ordered indexes).  We suspect some witnesses are getting their state wrong when they assign an ID differently from the main network, if someone publishes a transaction using that ID then the wrong state turns into a desync (they'll no longer sign or pay attention to blocks on the main fork because they think the main fork has an illegal transaction.)

If you get stuck and the first error message mentions a vesting balance object, this is almost certainly what's happening to you.

The next hardfork will fix this issue, and in addition, I'm working on a tool to help us find if there are any similar issues.

The transactions being executed by the community testers have been incredibly helpful to us.

Running with latest commit (e68e99ed3ae11ddac607e983e7549e8278fdecc4), but it can't go over block 70872

Code: [Select]

2015-10-08T21:37:17 p2p:send_sync_block_to_node_delegate send_sync_block_to_n ] Failed to push sync block 70872 (id:000114d879ab3c305edeaaae837c2b68b7da64d2): client rejected sync block sent by peer: {"code":10,"name":"assert_exception","message":"Assert Exception","stack":[{"context":{"level":"error","file":"vesting_balance_evaluator.cpp","line":103,"method":"do_evaluate","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T21:37:17"},"format":"vbo.is_withdraw_allowed( now, op.amount ): ","data":{"now":"2015-10-08T05:43:09","op":{"fee":{"amount":100000,"asset_id":"1.3.0"},"vesting_balance":"1.13.60","owner":"1.2.8517","amount":{"amount":45300000,"asset_id":"1.3.0"}},"vbo":{"id":"1.13.60","owner":"1.2.8517","balance":{"amount":197500000,"asset_id":"1.3.0"},"policy":[1,{"vesting_seconds":86400,"start_claim":"1970-01-01T00:00:00","coin_seconds_earned":"3341610000000","coin_seconds_earned_last_update":"2015-10-08T05:43:06"}]}}},{"context":{"level":"warn","file":"vesting_balance_evaluator.cpp","line":109,"method":"do_evaluate","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T21:37:17"},"format":"","data":{"op":{"fee":{"amount":100000,"asset_id":"1.3.0"},"vesting_balance":"1.13.60","owner":"1.2.8517","amount":{"amount":45300000,"asset_id":"1.3.0"}}}},{"context":{"level":"warn","file":"evaluator.cpp","line":42,"method":"start_evaluate","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T21:37:17"},"format":"","data":{}},{"context":{"level":"warn","file":"db_block.cpp","line":628,"method":"apply_operation","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T21:37:17"},"format":"","data":{}},{"context":{"level":"warn","file":"db_block.cpp","line":611,"method":"_apply_transaction","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T21:37:17"},"format":"","data":{"trx":{"ref_block_num":5335,"ref_block_prefix":4159425851,"expiration":"2015-10-08T05:43:39","operations":[[33,{"fee":{"amount":100000,"asset_id":"1.3.0"},"vesting_balance":"1.13.60","owner":"1.2.8517","amount":{"amount":45300000,"asset_id":"1.3.0"}}]],"extensions":[],"signatures":["1f6f7ea6a4188c8fc36aaf457b5fe4433fb74af11b250507262a00d909af025ad065c03ef0f61925070489d1630a3fbe459795b265200544de9ba1694414fefbcf"]}}},{"context":{"level":"warn","file":"db_block.cpp","line":514,"method":"_apply_block","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T21:37:17"},"format":"","data":{"next_block.block_num()":70872}},{"context":{"level":"warn","file":"db_block.cpp","line":197,"method":"_push_block","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T21:37:17"},"format":"","data":{"new_block":{"previous":"000114d73bcdebf717450d1a45e7ea119844b22e","timestamp":"2015-10-08T05:43:15","witness":"1.6.10","transaction_merkle_root":"05b459df123ff6baf5dbc44890e6ae3e8a8caaf6","extensions":[],"witness_signature":"20232fc2bb2ead0cbc644d78e4462881dc812787ad28f1517229cc346c252acb2904cb13139a8e957c8087d8b03790f12f5c2ef0dbb2cfceead7a47521f75f0950","transactions":[{"ref_block_num":5335,"ref_block_prefix":4159425851,"expiration":"2015-10-08T05:43:39","operations":[[33,{"fee":{"amount":100000,"asset_id":"1.3.0"},"vesting_balance":"1.13.60","owner":"1.2.8517","amount":{"amount":45300000,"asset_id":"1.3.0"}}]],"extensions":[],"signatures":["1f6f7ea6a4188c8fc36aaf457b5fe4433fb74af11b250507262a00d909af025ad065c03ef0f61925070489d1630a3fbe459795b265200544de9ba1694414fefbcf"],"operation_results":[[0,{}]]}]}}},{"context":{"level":"warn","file":"application.cpp","line":441,"method":"handle_block","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T21:37:17"},"format":"","data":{"blk_msg":{"block":{"previous":"000114d73bcdebf717450d1a45e7ea119844b22e","timestamp":"2015-10-08T05:43:15","witness":"1.6.10","transaction_merkle_root":"05b459df123ff6baf5dbc44890e6ae3e8a8caaf6","extensions":[],"witness_signature":"20232fc2bb2ead0cbc644d78e4462881dc812787ad28f1517229cc346c252acb2904cb13139a8e957c8087d8b03790f12f5c2ef0dbb2cfceead7a47521f75f0950","transactions":[{"ref_block_num":5335,"ref_block_prefix":4159425851,"expiration":"2015-10-08T05:43:39","operations":[[33,{"fee":{"amount":100000,"asset_id":"1.3.0"},"vesting_balance":"1.13.60","owner":"1.2.8517","amount":{"amount":45300000,"asset_id":"1.3.0"}}]],"extensions":[],"signatures":["1f6f7ea6a4188c8fc36aaf457b5fe4433fb74af11b250507262a00d909af025ad065c03ef0f61925070489d1630a3fbe459795b265200544de9ba1694414fefbcf"],"operation_results":[[0,{}]]}]},"block_id":"000114d879ab3c305edeaaae837c2b68b7da64d2"},"sync_mode":true}}]}                    node.cpp:3043
2015-10-08T21:37:17 p2p:send_sync_block_to_node_delegate send_sync_block_to_n ] disconnecting client 104.236.51.238:2005 because it offered us the rejected block                       node.cpp:3157



[bytemaster]

Building...

When is the next hardfork?

About 20 hours from now.

[clayop]

Building...

When is the next hardfork?

[abit]

can't switch to correct chain, even restart witness_node(build from bfef440968e98219612a9bbd24e715e8835ad975),
get block 70872, but failed to push sync block
here is the error message from log file:

Code: [Select]

2015-10-08T06:31:23 p2p:send_sync_block_to_node_delegate send_sync_block_to_n ] Failed to push sync block 70872 (id:000114d879ab3c305edeaaae837c2b68b7da64d2): client   rejected sync block sent by peer: {"code":10,"name":"assert_exception","message":"Assert Exception","stack":[{"context":{"level":"error","file":"vesting_balance_evalu  ator.cpp","line":103,"method":"do_evaluate","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T06:31:23"},"format":"vbo.is_withdraw_allowed( now, op.amount ):   ","data":{"now":"2015-10-08T05:43:09","op":{"fee":{"amount":100000,"asset_id":"1.3.0"},"vesting_balance":"1.13.60","owner":"1.2.8517","amount":{"amount":45300000,"as  set_id":"1.3.0"}},"vbo":{"id":"1.13.60","owner":"1.2.8517","balance":{"amount":202500000,"asset_id":"1.3.0"},"policy":[1,{"vesting_seconds":86400,"start_claim":"1970-  01-01T00:00:00","coin_seconds_earned":"3290619000000","coin_seconds_earned_last_update":"2015-10-08T05:43:06"}]}}},{"context":{"level":"warn","file":"vesting_balance_  evaluator.cpp","line":109,"method":"do_evaluate","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T06:31:23"},"format":"","data":{"op":{"fee":{"amount":10000  0,"asset_id":"1.3.0"},"vesting_balance":"1.13.60","owner":"1.2.8517","amount":{"amount":45300000,"asset_id":"1.3.0"}}}},{"context":{"level":"warn","file":"evaluator.c  pp","line":42,"method":"start_evaluate","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T06:31:23"},"format":"","data":{}},{"context":{"level":"warn","file"  :"db_block.cpp","line":625,"method":"apply_operation","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T06:31:23"},"format":"","data":{}},{"context":{"level"  :"warn","file":"db_block.cpp","line":608,"method":"_apply_transaction","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T06:31:23"},"format":"","data":{"trx"  :{"ref_block_num":5335,"ref_block_prefix":4159425851,"expiration":"2015-10-08T05:43:39","operations":[[33,{"fee":{"amount":100000,"asset_id":"1.3.0"},"vesting_balance  ":"1.13.60","owner":"1.2.8517","amount":{"amount":45300000,"asset_id":"1.3.0"}}]],"extensions":[],"signatures":["1f6f7ea6a4188c8fc36aaf457b5fe4433fb74af11b250507262a0  0d909af025ad065c03ef0f61925070489d1630a3fbe459795b265200544de9ba1694414fefbcf"]}}},{"context":{"level":"warn","file":"db_block.cpp","line":511,"method":"_apply_block"  ,"hostname":"","thread_name":"th_a","timestamp":"2015-10-08T06:31:23"},"format":"","data":{"next_block.block_num()":70872}},{"context":{"level":"warn","file":"db_bloc  k.cpp","line":197,"method":"_push_block","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T06:31:23"},"format":"","data":{"new_block":{"previous":"000114d73b  cdebf717450d1a45e7ea119844b22e","timestamp":"2015-10-08T05:43:15","witness":"1.6.10","transaction_merkle_root":"05b459df123ff6baf5dbc44890e6ae3e8a8caaf6","extensions"  :[],"witness_signature":"20232fc2bb2ead0cbc644d78e4462881dc812787ad28f1517229cc346c252acb2904cb13139a8e957c8087d8b03790f12f5c2ef0dbb2cfceead7a47521f75f0950","transact  ions":[{"ref_block_num":5335,"ref_block_prefix":4159425851,"expiration":"2015-10-08T05:43:39","operations":[[33,{"fee":{"amount":100000,"asset_id":"1.3.0"},"vesting_b  alance":"1.13.60","owner":"1.2.8517","amount":{"amount":45300000,"asset_id":"1.3.0"}}]],"extensions":[],"signatures":["1f6f7ea6a4188c8fc36aaf457b5fe4433fb74af11b25050  7262a00d909af025ad065c03ef0f61925070489d1630a3fbe459795b265200544de9ba1694414fefbcf"],"operation_results":[[0,{}]]}]}}},{"context":{"level":"warn","file":"application  .cpp","line":434,"method":"handle_block","hostname":"","thread_name":"th_a","timestamp":"2015-10-08T06:31:23"},"format":"","data":{"blk_msg":{"block":{"previous":"000  114d73bcdebf717450d1a45e7ea119844b22e","timestamp":"2015-10-08T05:43:15","witness":"1.6.10","transaction_merkle_root":"05b459df123ff6baf5dbc44890e6ae3e8a8caaf6","exte  nsions":[],"witness_signature":"20232fc2bb2ead0cbc644d78e4462881dc812787ad28f1517229cc346c252acb2904cb13139a8e957c8087d8b03790f12f5c2ef0dbb2cfceead7a47521f75f0950","t  ransactions":[{"ref_block_num":5335,"ref_block_prefix":4159425851,"expiration":"2015-10-08T05:43:39","operations":[[33,{"fee":{"amount":100000,"asset_id":"1.3.0"},"ve  sting_balance":"1.13.60","owner":"1.2.8517","amount":{"amount":45300000,"asset_id":"1.3.0"}}]],"extensions":[],"signatures":["1f6f7ea6a4188c8fc36aaf457b5fe4433fb74af1  1b250507262a00d909af025ad065c03ef0f61925070489d1630a3fbe459795b265200544de9ba1694414fefbcf"],"operation_results":[[0,{}]]}]},"block_id":"000114d879ab3c305edeaaae837c2  b68b7da64d2"},"sync_mode":true}}]}      node.cpp:3017                                                                                                                 

output from console

Code: [Select]

1883626ms th_a       db_block.cpp:191              _push_block          ] Failed to push new block:
10 assert_exception: Assert Exception
vbo.is_withdraw_allowed( now, op.amount ):
    {"now":"2015-10-08T05:43:09","op":{"fee":{"amount":100000,"asset_id":"1.3.0"},"vesting_balance":"1.13.60","owner":"1.2.8517","amount":{"amount":45300000,"asset_id":
"1.3.0"}},"vbo":{"id":"1.13.60","owner":"1.2.8517","balance":{"amount":202500000,"asset_id":"1.3.0"},"policy":[1,{"vesting_seconds":86400,"start_claim":"1970-01-01T00:0
0:00","coin_seconds_earned":"3290619000000","coin_seconds_earned_last_update":"2015-10-08T05:43:06"}]}}
    th_a  vesting_balance_evaluator.cpp:103 do_evaluate

    {"op":{"fee":{"amount":100000,"asset_id":"1.3.0"},"vesting_balance":"1.13.60","owner":"1.2.8517","amount":{"amount":45300000,"asset_id":"1.3.0"}}}
    th_a  vesting_balance_evaluator.cpp:109 do_evaluate

    {}
    th_a  evaluator.cpp:42 start_evaluate

    {}
    th_a  db_block.cpp:625 apply_operation

    {"trx":{"ref_block_num":5335,"ref_block_prefix":4159425851,"expiration":"2015-10-08T05:43:39","operations":[[33,{"fee":{"amount":100000,"asset_id":"1.3.0"},"vesting
_balance":"1.13.60","owner":"1.2.8517","amount":{"amount":45300000,"asset_id":"1.3.0"}}]],"extensions":[],"signatures":["1f6f7ea6a4188c8fc36aaf457b5fe4433fb74af11b25050
7262a00d909af025ad065c03ef0f61925070489d1630a3fbe459795b265200544de9ba1694414fefbcf"]}}
    th_a  db_block.cpp:608 _apply_transaction

    {"next_block.block_num()":70872}
    th_a  db_block.cpp:511 _apply_block


Same issue here. My build was bfef440968e98219612a9bbd24e715e8835ad975 "fork_database.cpp:  Fix overflow".
delegate.baozi, in.abit and delegate-clayop were on the same fork.
Anyway I'm rebuilding with latest commit now. It should be fixed by this:

Every day so many witnesses are not online, what is the reason?

We've identified an issue where the assignment of new object ID's is inconsistent (we use a hashed index with an undefined iteration order).  Fortunately the testnet's main fork has them in increasing order (I suspect it has to do with the memory allocator tending to put things allocated later at higher addresses), in the next hardfork this condition will be enforced (by replacing most of our hashed indexes with ordered indexes).  We suspect some witnesses are getting their state wrong when they assign an ID differently from the main network, if someone publishes a transaction using that ID then the wrong state turns into a desync (they'll no longer sign or pay attention to blocks on the main fork because they think the main fork has an illegal transaction.)

If you get stuck and the first error message mentions a vesting balance object, this is almost certainly what's happening to you.

The next hardfork will fix this issue, and in addition, I'm working on a tool to help us find if there are any similar issues.

The transactions being executed by the community testers have been incredibly helpful to us.

[puppies]

here is my automatic failover script powered by xerocs python-graphenelib.  I am a noob at teh programming, and if I did anything wrong I would love the feedback.

The wallet this script is running through needs to be unlocked, and must contain the owner key (I think its the owner key, rather than the active key) of the witness.

Awesome!  This gives me an idea for an "industrial strength" version.

The active key should be sufficient.

Thanks theoretical.  Its nice to be able to contribute.

[abit]

We have successfully managed to resolve a number of issues without requiring the test network to be reset.   Please upgrade to the latest master branch and verify that you can still sync.

Issues Resolved:

1. Replaced Hash Indexes with Ordered Indexes because there are places where we iterate over the Hash Index and the iteration order was not deterministic
2. Fix reference to deleted memory during margin call evaluation
3. Cover some corner cases for rational math overflows
4. Fix for publishing invalid price feeds

Wow.. lots of updates.
Rebuilding.

[abit]

My observer node got out of sync as well.

Looks like it's stuck at the fork produced by delegate.baozi 

Caused by:

Code: [Select]

2015-10-08T05:43:09 th_a:invoke handle_block         handle_block ] Got block: #70871 time: 2015-10-08T05:43:09 latency: 573 ms from:
elmato  irreversible: 70846 (-25)                 application.cpp:394
2015-10-08T05:43:13 th_a:invoke handle_transaction   handle_transaction ] Got transaction from network                  application.cp
p:438
2015-10-08T05:43:15 th_a:invoke handle_block         handle_block ] Got block: #70872 time: 2015-10-08T05:43:15 latency: 398 ms from: init9  irreversible: 70847 (-25)                  application.cpp:394
2015-10-08T05:43:15 th_a:invoke handle_block          _push_block ] Failed to push new block:
10 assert_exception: Assert Exception
vbo.is_withdraw_allowed( now, op.amount ):
    {"now":"2015-10-08T05:43:09","op":{"fee":{"amount":100000,"asset_id":"1.3.0"},"vesting_balance":"1.13.60","owner":"1.2.8517","amount":{"amount":45300000,"asset_id":"1.3.0"}},"vbo":{"id":"1.13.60","owner":"1.2.8517","balance":{"amount":197500000,"asset_id":"1.3.0"},"policy":[1,{"vesting_seconds":86400,"start_claim":"1970-01-01T00:00:00","coin_seconds_earned":"3341610000000","coin_seconds_earned_last_update":"2015-10-08T05:43:06"}]}}
    th_a  vesting_balance_evaluator.cpp:103 do_evaluate

    {"op":{"fee":{"amount":100000,"asset_id":"1.3.0"},"vesting_balance":"1.13.60","owner":"1.2.8517","amount":{"amount":45300000,"asset_id":"1.3.0"}}}
    th_a  vesting_balance_evaluator.cpp:109 do_evaluate

    {}
    th_a  evaluator.cpp:42 start_evaluate

    {}
    th_a  db_block.cpp:625 apply_operation

    {"trx":{"ref_block_num":5335,"ref_block_prefix":4159425851,"expiration":"2015-10-08T05:43:39","operations":[[33,{"fee":{"amount":100000,"asset_id":"1.3.0"},"vesting_balance":"1.13.60","owner":"1.2.8517","amount":{"amount":45300000,"asset_id":"1.3.0"}}]],"extensions":[],"signatures":["1f6f7ea6a4188c8fc36aaf457b5fe4433fb74af11b250507262a00d909af025ad065c03ef0f61925070489d1630a3fbe459795b265200544de9ba1694414fefbcf"]}}
    th_a  db_block.cpp:608 _apply_transaction
    {"next_block.block_num()":70872}
    th_a  db_block.cpp:511 _apply_block                 db_block.cpp:191



[bytemaster]

We have successfully managed to resolve a number of issues without requiring the test network to be reset.   Please upgrade to the latest master branch and verify that you can still sync.

Issues Resolved:

1. Replaced Hash Indexes with Ordered Indexes because there are places where we iterate over the Hash Index and the iteration order was not deterministic
2. Fix reference to deleted memory during margin call evaluation
3. Cover some corner cases for rational math overflows
4. Fix for publishing invalid price feeds