Test Net for Advanced Users

[bytemaster]

It took 75 seconds to replay the 280k-blocks chain.

Maybe on a SLOW vps, but on my machine it takes a mere 11 8.79 seconds.

In Debug build it took me 59 seconds to reindex.
In Release build it took me 8.79 seconds to reindex.

There were a total of 444758 operations in the blockchain which means we are averaging over 50K TPS.
The average maintenance calculation time is:  7 ms 
1.5 seconds were spent doing maintenance calculations (on the hour, instead of every day) leaving us at 62K TPS average
The vast majority of blocks have been empty which means our TPS average is being diluted by block processing overhead.

It looks like in Debug mode we process closer to 10K TPS.

[bytemaster]

It took 75 seconds to replay the 280k-blocks chain.

Maybe on a SLOW vps, but on my machine it takes a mere 11 seconds.

[mindphlux]

I'm thinking of writing a functional test for the UI frontend. One that makes sure that every function still works as designed after modyfing/adding a new feature.

For example, also market order creation could be automated, including a stress test of market orders on multiple instances of the functional test.

Can you please give me examples of other items that are worth testing, UI-wise and network wise? It will automatically report when something doesn't work as expected, and it can be also be integrated into jenkins if that is used yet.

I tried using phantomjs, but it does not support IndexedDB, so I'll use selenium and chrome as a real browser.

[alt]

I think it's more reasonable to copy those blocks which is not fork from the database directoly than resync from the network.

forked, have to resync.
can we add the block number for parameter --resync-blockchain?
so we only need to  resync from the given block number.

Make a backup of the blockchain directory when you're in sync.
After forked, restore with the backup.

[clayop]

I will flood the testnet at 9/29 0:10 UTC with 101 VPSs. Tips for VPS costs are more than welcome. 

BTS ID: clayop

Due to the limit of numbers per region, I can only manage about 60 VPSs.

Sent you 500BTS for to help with your costs 

Matching funds sent. 

I will match up to 50,000 BTS thru 23:59 UTC 29 SEP 2015 to support Clayop with his VPS needs.  Please do not spam this thread.

Please PM me your Transaction ID and I will match it.

Thank you fox! Fund received.

[Fox]

I will flood the testnet at 9/29 0:10 UTC with 101 VPSs. Tips for VPS costs are more than welcome. 

BTS ID: clayop

Due to the limit of numbers per region, I can only manage about 60 VPSs.

Sent you 500BTS for to help with your costs 

Matching funds sent. 

I will match up to 50,000 BTS thru 23:59 UTC 29 SEP 2015 to support Clayop with his VPS needs.  Please do not spam this thread.

Please PM me your Transaction ID and I will match it. 

[cube]

My witness node passed through the spam with 20 missed blocks.  So far it has managed to survive all the spam attempts with about 20 missed blocks.

[abit]

It took 75 seconds to replay the 280k-blocks chain.

[clayop]

[abit]

forked, have to resync.
can we add the block number for parameter --resync-blockchain?
so we only need to  resync from the given block number.

Make a backup of the blockchain directory when you're in sync.
After forked, restore with the backup.

[abit]

During the spam, I can observe a high latency

Code: [Select]

1410962ms th_a       application.cpp:388           handle_block         ] Got block #281678
with time 2015-09-29T01:23:27 from network with latency of 3975 ms from spartako

My block producing node is dead.
Observer node's CPU is 100% but alive.

[abit]

Here are some of the tons of errors found in p2p.logs while spamming.

At first:

Code: [Select]

2015-09-28T07:02:09 p2p:message read_loop process_ordinary_mes ] client rejected message sent by peer 127.0.0.1:62015, {"code":10,"name":"assert_exception","message":"Assert Exception","stack":[{"context":{"level":"error","file":"db_block.cpp","line":534,"method":"_apply_transaction","hostname":"","thread_name":"th_a","timestamp":"2015-09-28T07:02:09"},
"format":"(skip & skip_transaction_dupe_check) || trx_idx.indices().get<by_trx_id>().find(trx_id) == trx_idx.indices().get<by_trx_id>().end(): ","data":{}},{"context":{"level":"warn","file":"db_block.cpp","line":592,"method":"_apply_transaction","hostname":"","thread_name":"th_a","timestamp":"2015-09-28T07:02:09"},"format":"","data":{"trx":{"ref_block_num":65467,"ref_block_prefix":2922805723,"expiration":"2015-09-28T07:02:48","operations":[[1,{"fee":{"amount":250000,"asset_id":"1.3.0"},"seller":"1.2.17357","amount_to_sell":{"amount":1,"asset_id":"1.3.664"},"min_to_receive":{"amount":100000,"asset_id":"1.3.0"},"expiration":"2015-09-28T07:03:43","fill_or_kill":false,"extensions":[]}]],"extensions":[],"signatures":["1f17bdb09411ca7eeaf6c7e28b81222ae240c68dddd71d4ce6a68049c38b9429420e5ab73e01ca705fa52ced97a1b723069eac4c8fb3683df09ae6e168d8ce5a87"]}}},{"context":{"level":"warn","file":"db_block.cpp","line":214,"method":"push_transaction","hostname":"","thread_name":"th_a","timestamp":"2015-09-28T07:02:09"},"format":"","data":{"trx":{"ref_block_num":65467,"ref_block_prefix":2922805723,"expiration":"2015-09-28T07:02:48","operations":[[1,{"fee":{"amount":250000,"asset_id":"1.3.0"},"seller":"1.2.17357","amount_to_sell":{"amount":1,"asset_id":"1.3.664"},"min_to_receive":{"amount":100000,"asset_id":"1.3.0"},"expiration":"2015-09-28T07:03:43","fill_or_kill":false,"extensions":[]}]],"extensions":[],"signatures":["1f17bdb09411ca7eeaf6c7e28b81222ae240c68dddd71d4ce6a68049c38b9429420e5ab73e01ca705fa52ced97a1b723069eac4c8fb3683df09ae6e168d8ce5a87"]}}},{"context":{"level":"warn","file":"application.cpp","line":434,"method":"handle_transaction","hostname":"","thread_name":"th_a","timestamp":"2015-09-28T07:02:09"},"format":"","data":{"transaction_message":{"trx":{"ref_block_num":65467,"ref_block_prefix":2922805723,"expiration":"2015-09-28T07:02:48","operations":[[1,{"fee":{"amount":250000,"asset_id":"1.3.0"},"seller":"1.2.17357","amount_to_sell":{"amount":1,"asset_id":"1.3.664"},"min_to_receive":{"amount":100000,"asset_id":"1.3.0"},"expiration":"2015-09-28T07:03:43","fill_or_kill":false,"extensions":[]}]],"extensions":[],"signatures":["1f17bdb09411ca7eeaf6c7e28b81222ae240c68dddd71d4ce6a68049c38b9429420e5ab73e01ca705fa52ced97a1b723069eac4c8fb3683df09ae6e168d8ce5a87"]}}}}]}                  node.cpp:3759


Code: [Select]

2015-09-28T07:03:13 p2p:message read_loop process_ordinary_mes ] client rejected message sent by peer 104.236.11.171:48991, {"code":10,"name":"assert_exception","message":"Assert Exception","stack":[{"context":{"level":"error","file":"db_block.cpp","line":562,"method":"_apply_transaction","hostname":"","thread_name":"th_a","timestamp":"2015-09-28T07:03:13"},
"format":"now <= trx.expiration: ","data":{"now":"2015-09-28T07:03:09","trx.exp":"2015-09-28T07:03:00"}},{"context":{"level":"warn","file":"db_block.cpp","line":592,"method":"_apply_transaction","hostname":"","thread_name":"th_a","timestamp":"2015-09-28T07:03:13"},"format":"","data":{"trx":{"ref_block_num":65475,"ref_block_prefix":3967048843,"expiration":"2015-09-28T07:03:00","operations":[[1,{"fee":{"amount":250000,"asset_id":"1.3.0"},"seller":"1.2.17357","amount_to_sell":{"amount":1,"asset_id":"1.3.664"},"min_to_receive":{"amount":100000,"asset_id":"1.3.0"},"expiration":"2015-09-28T07:04:19","fill_or_kill":false,"extensions":[]}]],"extensions":[],"signatures":["1f51cc04c73f2236caa7da4ce80eb8134b5065c632fae74f8f16e839a819f511822a5d375a3457be4fe8028686b6766bcd7c3d77fce19ef35c609c0a15f80fb76f"]}}},{"context":{"level":"warn","file":"db_block.cpp","line":214,"method":"push_transaction","hostname":"","thread_name":"th_a","timestamp":"2015-09-28T07:03:13"},"format":"","data":{"trx":{"ref_block_num":65475,"ref_block_prefix":3967048843,"expiration":"2015-09-28T07:03:00","operations":[[1,{"fee":{"amount":250000,"asset_id":"1.3.0"},"seller":"1.2.17357","amount_to_sell":{"amount":1,"asset_id":"1.3.664"},"min_to_receive":{"amount":100000,"asset_id":"1.3.0"},"expiration":"2015-09-28T07:04:19","fill_or_kill":false,"extensions":[]}]],"extensions":[],"signatures":["1f51cc04c73f2236caa7da4ce80eb8134b5065c632fae74f8f16e839a819f511822a5d375a3457be4fe8028686b6766bcd7c3d77fce19ef35c609c0a15f80fb76f"]}}},{"context":{"level":"warn","file":"application.cpp","line":434,"method":"handle_transaction","hostname":"","thread_name":"th_a","timestamp":"2015-09-28T07:03:13"},"format":"","data":{"transaction_message":{"trx":{"ref_block_num":65475,"ref_block_prefix":3967048843,"expiration":"2015-09-28T07:03:00","operations":[[1,{"fee":{"amount":250000,"asset_id":"1.3.0"},"seller":"1.2.17357","amount_to_sell":{"amount":1,"asset_id":"1.3.664"},"min_to_receive":{"amount":100000,"asset_id":"1.3.0"},"expiration":"2015-09-28T07:04:19","fill_or_kill":false,"extensions":[]}]],"extensions":[],"signatures":["1f51cc04c73f2236caa7da4ce80eb8134b5065c632fae74f8f16e839a819f511822a5d375a3457be4fe8028686b6766bcd7c3d77fce19ef35c609c0a15f80fb76f"]}}}}]}                        node.cpp:3759


Code: [Select]

2015-09-28T07:04:23 p2p:message read_loop process_ordinary_mes ] client rejected message sent by peer 127.0.0.1:62015, {"code":10,"name":"assert_exception","message":"Assert Exception","stack":[{"context":{"level":"error","file":"db_block.cpp","line":555,"method":"_apply_transaction","hostname":"","thread_name":"th_a","timestamp":"2015-09-28T07:04:23"},
"format":"trx.ref_block_prefix == tapos_block_summary.block_id._hash[1]: ","data":{}},{"context":{"level":"warn","file":"db_block.cpp","line":592,"method":"_apply_transaction","hostname":"","thread_name":"th_a","timestamp":"2015-09-28T07:04:23"},"format":"","data":{"trx":{"ref_block_num":65498,"ref_block_prefix":3583879970,"expiration":"2015-09-28T07:04:43","operations":[[1,{"fee":{"amount":250000,"asset_id":"1.3.0"},"seller":"1.2.17357","amount_to_sell":{"amount":1,"asset_id":"1.3.664"},"min_to_receive":{"amount":100000,"asset_id":"1.3.0"},"expiration":"2015-09-28T07:05:58","fill_or_kill":false,"extensions":[]}]],"extensions":[],"signatures":["20332b3cbc6e337b00ae7b5ebc2e4cf2903fdbd4046558fb51496aee33e45ece6032386c5b3979c85ff1035e6561bb823198fb47f52e2dec224ba48452dd572c53"]}}},{"context":{"level":"warn","file":"db_block.cpp","line":214,"method":"push_transaction","hostname":"","thread_name":"th_a","timestamp":"2015-09-28T07:04:23"},"format":"","data":{"trx":{"ref_block_num":65498,"ref_block_prefix":3583879970,"expiration":"2015-09-28T07:04:43","operations":[[1,{"fee":{"amount":250000,"asset_id":"1.3.0"},"seller":"1.2.17357","amount_to_sell":{"amount":1,"asset_id":"1.3.664"},"min_to_receive":{"amount":100000,"asset_id":"1.3.0"},"expiration":"2015-09-28T07:05:58","fill_or_kill":false,"extensions":[]}]],"extensions":[],"signatures":["20332b3cbc6e337b00ae7b5ebc2e4cf2903fdbd4046558fb51496aee33e45ece6032386c5b3979c85ff1035e6561bb823198fb47f52e2dec224ba48452dd572c53"]}}},{"context":{"level":"warn","file":"application.cpp","line":434,"method":"handle_transaction","hostname":"","thread_name":"th_a","timestamp":"2015-09-28T07:04:23"},"format":"","data":{"transaction_message":{"trx":{"ref_block_num":65498,"ref_block_prefix":3583879970,"expiration":"2015-09-28T07:04:43","operations":[[1,{"fee":{"amount":250000,"asset_id":"1.3.0"},"seller":"1.2.17357","amount_to_sell":{"amount":1,"asset_id":"1.3.664"},"min_to_receive":{"amount":100000,"asset_id":"1.3.0"},"expiration":"2015-09-28T07:05:58","fill_or_kill":false,"extensions":[]}]],"extensions":[],"signatures":["20332b3cbc6e337b00ae7b5ebc2e4cf2903fdbd4046558fb51496aee33e45ece6032386c5b3979c85ff1035e6561bb823198fb47f52e2dec224ba48452dd572c53"]}}}}]}                        node.cpp:3759

My witness node got out of sync here after some seconds.

Later:

Code: [Select]

2015-09-28T07:22:20 p2p:send_sync_block_to_node_delegate send_sync_block_to_n ] Failed to push sync block 262132 (id:0003fff4f020495812404736ce0e866d4801c01b): client rejected sync block sent by peer: {"code":10,"name":"assert_exception","message":"Assert Exception","stack":[{"context":{"level":"error","file":"db_block.cpp","line":613,"method":"validate_block_header","hostname":"","thread_name":"th_a","timestamp":"2015-09-28T07:22:20"},
"format":"head_block_id() == next_block.previous: ","data":{"head_block_id":"0003fff3004e3e0a5eed50770023bf74e4c246f1","next.prev":"0003fff36a9b0c61dd3922f19fe1f4a040ff1ae2"}},{"context":{"level":"warn","file":"db_block.cpp","line":495,"method":"_apply_block","hostname":"","thread_name":"th_a","timestamp":"2015-09-28T07:22:20"},"format":"","data":{"next_block.block_num()":262132}},{"context":{"level":"warn","file":"db_block.cpp","line":195,"method":"_push_block","hostname":"","thread_name":"th_a","timestamp":"2015-09-28T07:22:20"},"format":"","data":{"new_block":{"previous":"0003fff36a9b0c61dd3922f19fe1f4a040ff1ae2","timestamp":"2015-09-28T07:06:12","witness":"1.6.2","transaction_merkle_root":"93a42a8d8c1db93d55ce90defea956544834b5e0","extensions":[],"witness_signature":"20045fae9169b63f0d2f764cfbacf9be2ab7d7044192158b6cf9eca676079b3e302c0f04fa6dd99913c3ef53f7f8423d9809932d3457747dd96fcf920cc920ebbf","transactions":[.....]}


[clayop]

During the spam, I can observe a high latency

Code: [Select]

1410962ms th_a       application.cpp:388           handle_block         ] Got block #281678
with time 2015-09-29T01:23:27 from network with latency of 3975 ms from spartako

[clayop]

I will flood the testnet at 9/29 0:10 UTC with 101 VPSs.

I changed some parameter and will do it again at 0:40 UTC

[alt]

maybe need to notice, before block 275997, already  met  a fork switch at block 275996, and it  switch to the correct chain.

here is a piece from the log when it forked:

Code: [Select]

  2015-09-28T20:08:30 p2p:message read_loop process_block_during ] received a block from peer 216.252.204.69:54183, passing it to client      node.cpp:3232
  2015-09-28T20:08:30 p2p:message read_loop process_block_during ] Successfully pushed block 275994 (id:0004361a7b4321512a38bc112c8c8ecd84d3e59c)     node.cpp:3254
....................
      2015-09-28T20:08:36 p2p:message read_loop process_block_during ] received a block from peer 114.92.254.159:62015, passing it to client      node.cpp:3232
.........................
  2015-09-28T20:08:36 p2p:message read_loop process_block_during ] Failed to push block 275996 (id:0004361c004527fceb33c4dfe9062a92a6421508), client rejected block sent   by peer      node.cpp:3346
  2015-09-28T20:08:36 p2p:message read_loop process_block_during ] disconnecting client 114.92.254.159:62015 because it offered us the rejected block     node.cpp:3368
...............
2015-09-28T20:08:37 p2p:message read_loop process_block_during ] received a block from peer 185.42.242.124:40060, passing it to client      node.cpp:3232
  2015-09-28T20:08:37 p2p:message read_loop process_block_during ] Failed to push block 275996 (id:0004361c004527fceb33c4dfe9062a92a6421508), client rejected block sent   by peer      node.cpp:3346
  2015-09-28T20:08:37 p2p:message read_loop process_block_during ] disconnecting client 185.42.242.124:40060 because it offered us the rejected block     node.cpp:3368
 ..............
 2015-09-28T20:08:39 p2p:message read_loop process_block_during ] received a block from peer 178.62.88.151:46944, passing it to client     node.cpp:3232
  2015-09-28T20:08:39 p2p:message read_loop process_block_during ] Successfully pushed block 275996 (id:0004361c4c4b8410260663693fd59b7ca1977b86)     node.cpp:3254
  2015-09-28T20:08:39 p2p:message read_loop process_block_during ] client validated the block, advertising it to other peers      node.cpp:3278
..............
  2015-09-28T20:08:42 p2p:message read_loop process_block_during ] received a block from peer 178.62.88.151:46944, passing it to client     node.cpp:3232
  2015-09-28T20:08:42 p2p:message read_loop process_block_during ] Peer 178.62.88.151:46944 sent me a block that didn't link to our blockchain.  Restarting sync mode wi  th them to get the missing block. Error pushing block was: {"code":90006,"name":"unlinkable_block_exception","message":"unlinkable block","stack":[{"context":{"level"  :"error","file":"application.cpp","line":417,"method":"handle_block","hostname":"","thread_name":"th_a","timestamp":"2015-09-28T20:08:42"},"format":"Error when pushin  g block:\n${e}","data":{"e":"3080000 unlinkable_block_exception: unlinkable block\nblock does not link to known chain\n    {}\n    th_a  fork_database.cpp:79 _push_bl  ock\n\n    {\"new_block\":{\"previous\":\"0004361c004527fceb33c4dfe9062a92a6421508\",\"timestamp\":\"2015-09-28T20:08:42\",\"witness\":\"1.6.5248\",\"transaction_merk  le_root\":\"0000000000000000000000000000000000000000\",\"extensions\":[],\"witness_signature\":\"1f051e73526e48fd46a4359daaede6e5e5a8b6f451194c24caad9a1287506ef0cd026  c0ecdadb5ea35b9843dc0bd68faee59053278a7ce6e571f838a2d870bebc9\",\"transactions\":[]}}\n    th_a  db_block.cpp:195 _push_block"}},{"context":{"level":"warn","file":"ap  plication.cpp","line":428,"method":"handle_block","hostname":"","thread_name":"th_a","timestamp":"2015-09-28T20:08:42"},"format":"","data":{"blk_msg":{"block":{"previ  ous":"0004361c004527fceb33c4dfe9062a92a6421508","timestamp":"2015-09-28T20:08:42","witness":"1.6.5248","transaction_merkle_root":"000000000000000000000000000000000000  0000","extensions":[],"witness_signature":"1f051e73526e48fd46a4359daaede6e5e5a8b6f451194c24caad9a1287506ef0cd026c0ecdadb5ea35b9843dc0bd68faee59053278a7ce6e571f838a2d8  70bebc9","transactions":[]},"block_id":"0004361d1c4d414512a8ae8e99ecf2026627d08e"},"sync_mode":false}}]}      node.cpp:3362
d>2015-09-28T20:08:42 p2p:message read_loop fetch_next_batch_of_ ] sync: sending a request for the next items after 0004361c4c4b8410260663693fd59b7ca1977b86 to peer 178  .62.88.151:46944, (full request is ["0004361158e0898268252af7afa90b1d59ffa354","0004361707141c5119a53184c26912df003d6ac2","0004361a7b4321512a38bc112c8c8ecd84d3e59c","  0004361c4c4b8410260663693fd59b7ca1977b86"])     node.cpp:2354       
............................
2015-09-28T20:08:42 p2p:message read_loop on_blockchain_item_i ] popping item because delegate has already seen it.  peer 178.62.88.151:46944's last block the delegat  e has seen is now 0004361a7b4321512a38bc112c8c8ecd84d3e59c (actual block #275994)     node.cpp:2511
  2015-09-28T20:08:42 p2p:message read_loop on_blockchain_item_i ] popping item because delegate has already seen it.  peer 178.62.88.151:46944's last block the delegat  e has seen is now 0004361bd4f6fdf618a74290b09a8a018ab7867b (actual block #275995)     node.cpp:2511
  2015-09-28T20:08:42 p2p:message read_loop on_blockchain_item_i ] after removing all items we have already seen, item_hashes_received.size() = 2     node.cpp:2515
  2015-09-28T20:08:42 p2p:message read_loop trigger_fetch_sync_i ] Triggering fetch sync items loop now     node.cpp:1083
  2015-09-28T20:08:42 p2p:fetch_sync_items_loop fetch_sync_items_loo ] beginning another iteration of the sync items loop     node.cpp:1022
  2015-09-28T20:08:42 p2p:fetch_sync_items_loop request_sync_items_f ] requesting 2 item(s) ["0004361c004527fceb33c4dfe9062a92a6421508","0004361d1c4d414512a8ae8e99ecf20  26627d08e"] from peer 178.62.88.151:46944     node.cpp:1006
  2015-09-28T20:08:42 p2p:fetch_sync_items_loop fetch_sync_items_loo ] no sync items to fetch right now, going to sleep     node.cpp:1072
  2015-09-28T20:08:42 p2p:message read_loop           on_message ] handling message fetch_blockchain_item_ids_message_type f9797929aee8529c183c2a43787da7a9b3d24e14 size   85 from peer 23.102.65.247:1984      node.cpp:1684
  2015-09-28T20:08:42 p2p:message read_loop on_fetch_blockchain_ ] sync: received a request for item ids after 0004361c4c4b8410260663693fd59b7ca1977b86 from peer 23.102  .65.247:1984 (full request: ["0004361158e0898268252af7afa90b1d59ffa354","0004361707141c5119a53184c26912df003d6ac2","0004361a7b4321512a38bc112c8c8ecd84d3e59c","0004361  c4c4b8410260663693fd59b7ca1977b86"])      node.cpp:2171
  2015-09-28T20:08:42 p2p:message read_loop on_fetch_blockchain_ ] reply_message: {"total_remaining_item_count":0,"item_type":1001,"item_hashes_available":["0004361c4c4  b8410260663693fd59b7ca1977b86"]} fetch_blockchain_item_ids_message_received.blockchain_synopsis: ["0004361158e0898268252af7afa90b1d59ffa354","0004361707141c5119a53184  c26912df003d6ac2","0004361a7b4321512a38bc112c8c8ecd84d3e59c","0004361c4c4b8410260663693fd59b7ca1977b86"]      node.cpp:2194
  2015-09-28T20:08:42 p2p:message read_loop on_fetch_blockchain_ ] sync: peer is already in sync with us      node.cpp:2213
  2015-09-28T20:08:43 p2p:message read_loop           on_message ] handling message fetch_blockchain_item_ids_message_type f9797929aee8529c183c2a43787da7a9b3d24e14 size   85 from peer 207.46.141.218:1344     node.cpp:1684
  2015-09-28T20:08:43 p2p:message read_loop on_fetch_blockchain_ ] sync: received a request for item ids after 0004361c4c4b8410260663693fd59b7ca1977b86 from peer 207.46  .141.218:1344 (full request: ["0004361158e0898268252af7afa90b1d59ffa354","0004361707141c5119a53184c26912df003d6ac2","0004361a7b4321512a38bc112c8c8ecd84d3e59c","000436  1c4c4b8410260663693fd59b7ca1977b86"])     node.cpp:2171
  2015-09-28T20:08:43 p2p:message read_loop on_fetch_blockchain_ ] reply_message: {"total_remaining_item_count":0,"item_type":1001,"item_hashes_available":["0004361c4c4  b8410260663693fd59b7ca1977b86"]} fetch_blockchain_item_ids_message_received.blockchain_synopsis: ["0004361158e0898268252af7afa90b1d59ffa354","0004361707141c5119a53184  c26912df003d6ac2","0004361a7b4321512a38bc112c8c8ecd84d3e59c","0004361c4c4b8410260663693fd59b7ca1977b86"]      node.cpp:2194
  2015-09-28T20:08:43 p2p:message read_loop on_fetch_blockchain_ ] sync: peer is already in sync with us      node.cpp:2213
2015-09-28T20:08:43 p2p:message read_loop           on_message ] handling message block_message_type 1003c4eca3a31fd0e4933c919f37b072876911dc size 133 from peer 71.19  7.2.119:1776      node.cpp:1684
  2015-09-28T20:08:43 p2p:message read_loop process_block_during ] received a block from peer 71.197.2.119:1776, passing it to client     node.cpp:3232
  2015-09-28T20:08:43 p2p:message read_loop process_block_during ] Peer 71.197.2.119:1776 sent me a block that didn't link to our blockchain.  Restarting sync mode with   them to get the missing block. Error pushing block was: {"code":90006,"name":"unlinkable_block_exception","message":"unlinkable block","stack":[{"context":{"level":"  error","file":"application.cpp","line":417,"method":"handle_block","hostname":"","thread_name":"th_a","timestamp":"2015-09-28T20:08:43"},"format":"Error when pushing   block:\n${e}","data":{"e":"3080000 unlinkable_block_exception: unlinkable block\nblock does not link to known chain\n    {}\n    th_a  fork_database.cpp:79 _push_bloc  k\n\n    {\"new_block\":{\"previous\":\"0004361c004527fceb33c4dfe9062a92a6421508\",\"timestamp\":\"2015-09-28T20:08:42\",\"witness\":\"1.6.5248\",\"transaction_merkle  _root\":\"0000000000000000000000000000000000000000\",\"extensions\":[],\"witness_signature\":\"1f051e73526e48fd46a4359daaede6e5e5a8b6f451194c24caad9a1287506ef0cd026c0  ecdadb5ea35b9843dc0bd68faee59053278a7ce6e571f838a2d870bebc9\",\"transactions\":[]}}\n    th_a  db_block.cpp:195 _push_block"}},{"context":{"level":"warn","file":"appl  ication.cpp","line":428,"method":"handle_block","hostname":"","thread_name":"th_a","timestamp":"2015-09-28T20:08:43"},"format":"","data":{"blk_msg":{"block":{"previou  s":"0004361c004527fceb33c4dfe9062a92a6421508","timestamp":"2015-09-28T20:08:42","witness":"1.6.5248","transaction_merkle_root":"00000000000000000000000000000000000000  00","extensions":[],"witness_signature":"1f051e73526e48fd46a4359daaede6e5e5a8b6f451194c24caad9a1287506ef0cd026c0ecdadb5ea35b9843dc0bd68faee59053278a7ce6e571f838a2d870  bebc9","transactions":[]},"block_id":"0004361d1c4d414512a8ae8e99ecf2026627d08e"},"sync_mode":false}}]}      node.cpp:3362
  2015-09-28T20:08:43 p2p:message read_loop fetch_next_batch_of_ ] sync: sending a request for the next items after 0004361c4c4b8410260663693fd59b7ca1977b86 to peer 71.  197.2.119:1776, (full request is ["0004361158e0898268252af7afa90b1d59ffa354","0004361707141c5119a53184c26912df003d6ac2","0004361a7b4321512a38bc112c8c8ecd84d3e59c","00  04361c4c4b8410260663693fd59b7ca1977b86"])     node.cpp:2354
..............
  2015-09-28T20:08:43 p2p:send_sync_block_to_node_delegate send_sync_block_to_n ] Successfully pushed sync block 275996 (id:0004361c004527fceb33c4dfe9062a92a6421508)       node.cpp:2935
..............
 2015-09-28T20:08:43 p2p:send_sync_block_to_node_delegate fetch_next_batch_of_ ] sync: sending a request for the next items after 0004361c4c4b8410260663693fd59b7ca1977  b86 to peer 71.197.2.119:61371, (full request is ["0004361158e0898268252af7afa90b1d59ffa354","0004361707141c5119a53184c26912df003d6ac2","0004361a7b4321512a38bc112c8c8  ecd84d3e59c","0004361c4c4b8410260663693fd59b7ca1977b86"])     node.cpp:2354
...............
 2015-09-28T20:08:43 p2p:message read_loop on_blockchain_item_i ] popping item because delegate has already seen it.  peer 109.73.172.144:43494's last block the delega  te has seen is now 0004361a7b4321512a38bc112c8c8ecd84d3e59c (actual block #275994)      node.cpp:2511
  2015-09-28T20:08:43 p2p:message read_loop on_blockchain_item_i ] popping item because delegate has already seen it.  peer 109.73.172.144:43494's last block the delega  te has seen is now 0004361bd4f6fdf618a74290b09a8a018ab7867b (actual block #275995)      node.cpp:2511
  2015-09-28T20:08:43 p2p:message read_loop on_blockchain_item_i ] popping item because delegate has already seen it.  peer 109.73.172.144:43494's last block the delega  te has seen is now 0004361c004527fceb33c4dfe9062a92a6421508 (actual block #275996)      node.cpp:2511
...................
2015-09-28T20:08:43 p2p:message read_loop on_blockchain_item_i ] popping item because delegate has already seen it.  peer 188.138.122.20:60328's last block the delega  te has seen is now 0004361a7b4321512a38bc112c8c8ecd84d3e59c (actual block #275994)      node.cpp:2511
  2015-09-28T20:08:43 p2p:message read_loop on_blockchain_item_i ] popping item because delegate has already seen it.  peer 188.165.233.53:1776's last block the delegat  e has seen is now 0004361a7b4321512a38bc112c8c8ecd84d3e59c (actual block #275994)     node.cpp:2511
  2015-09-28T20:08:43 p2p:message read_loop on_blockchain_item_i ] popping item because delegate has already seen it.  peer 188.138.122.20:60328's last block the delega  te has seen is now 0004361bd4f6fdf618a74290b09a8a018ab7867b (actual block #275995)      node.cpp:2511
  2015-09-28T20:08:43 p2p:message read_loop on_blockchain_item_i ] popping item because delegate has already seen it.  peer 188.165.233.53:1776's last block the delegat  e has seen is now 0004361bd4f6fdf618a74290b09a8a018ab7867b (actual block #275995)     node.cpp:2511
  2015-09-28T20:08:43 p2p:message read_loop on_blockchain_item_i ] popping item because delegate has already seen it.  peer 188.138.122.20:60328's last block the delega  te has seen is now 0004361c004527fceb33c4dfe9062a92a6421508 (actual block #275996)      node.cpp:2511
  2015-09-28T20:08:43 p2p:message read_loop on_blockchain_item_i ] after removing all items we have already seen, item_hashes_received.size() = 1     node.cpp:2515
  2015-09-28T20:08:43 p2p:message read_loop trigger_fetch_sync_i ] Triggering fetch sync items loop now     node.cpp:1083
  2015-09-28T20:08:43 p2p:fetch_sync_items_loop fetch_sync_items_loo ] beginning another iteration of the sync items loop     node.cpp:1022
  2015-09-28T20:08:43 p2p:fetch_sync_items_loop fetch_sync_items_loo ] no sync items to fetch right now, going to sleep     node.cpp:1072
  2015-09-28T20:08:43 p2p:message read_loop on_blockchain_item_i ] popping item because delegate has already seen it.  peer 188.165.233.53:1776's last block the delegat  e has seen is now 0004361c004527fceb33c4dfe9062a92a6421508 (actual block #275996)     node.cpp:2511
.......................
  2015-09-28T20:08:43 p2p:fetch_sync_items_loop fetch_sync_items_loo ] beginning another iteration of the sync items loop     node.cpp:1022
  2015-09-28T20:08:43 p2p:fetch_sync_items_loop fetch_sync_items_loo ] no sync items to fetch right now, going to sleep     node.cpp:1072
f>2015-09-28T20:08:43 p2p:send_sync_block_to_node_delegate send_sync_block_to_n ] Failed to push sync block 275997 (id:0004361d1c4d414512a8ae8e99ecf2026627d08e): client   rejected sync block sent by peer: {"code":10,"name":"assert_exception","message":"Assert Exception","stack":[{"context":{"level":"error","file":"vesting_balance_eval  uator.cpp","line":103,"method":"do_evaluate","hostname":"","thread_name":"th_a","timestamp":"2015-09-28T20:08:43"},"format":"vbo.is_withdraw_allowed( now, op.amount )  : ","data":{"now":"2015-09-28T20:08:33","op":{"fee":{"amount":50000,"asset_id":"1.3.0"},"vesting_balance":"1.13.30","owner":"1.2.22404","amount":{"amount":"6400000000  ","asset_id":"1.3.0"}},"vbo":{"id":"1.13.30","owner":"1.2.22404","balance":{"amount":"6392481409","asset_id":"1.3.0"},"policy":[1,{"vesting_seconds":86400,"start_clai  m":"1970-01-01T00:00:00","coin_seconds_earned":"552223993737600","coin_seconds_earned_last_update":"2015-09-28T20:07:51"}]}}},{"context":{"level":"warn","file":"vesti  ng_balance_evaluator.cpp","line":109,"method":"do_evaluate","hostname":"","thread_name":"th_a","timestamp":"2015-09-28T20:08:43"},"format":"","data":{"op":{"fee":{"am  ount":50000,"asset_id":"1.3.0"},"vesting_balance":"1.13.30","owner":"1.2.22404","amount":{"amount":"6400000000","asset_id":"1.3.0"}}}},{"context":{"level":"warn","fil  e":"evaluator.cpp","line":42,"method":"start_evaluate","hostname":"","thread_name":"th_a","timestamp":"2015-09-28T20:08:43"},"format":"","data":{}},{"context":{"level  ":"warn","file":"db_block.cpp","line":609,"method":"apply_operation","hostname":"","thread_name":"th_a","timestamp":"2015-09-28T20:08:43"},"format":"","data":{}},{"co  ntext":{"level":"warn","file":"db_block.cpp","line":592,"method":"_apply_transaction","hostname":"","thread_name":"th_a","timestamp":"2015-09-28T20:08:43"},"format":"  ","data":{"trx":{"ref_block_num":13848,"ref_block_prefix":996090894,"expiration":"2015-09-28T20:08:54","operations":[[33,{"fee":{"amount":50000,"asset_id":"1.3.0"},"v  esting_balance":"1.13.30","owner":"1.2.22404","amount":{"amount":"6400000000","asset_id":"1.3.0"}}]],"extensions":[],"signatures":["202c842047ea693db88068f8a5cb2e289d  372a8ab1226655695e7db2a2a427c7d41ff6887c12e5769aebc2e4e70953c17b95fd423b44ac03696cd413fd67c55aef"]}}},{"context":{"level":"warn","file":"db_block.cpp","line":495,"met  hod":"_apply_block","hostname":"","thread_name":"th_a","timestamp":"2015-09-28T20:08:43"},"format":"","data":{"next_block.block_num()":275996}},{"context":{"level":"w  arn","file":"db_block.cpp","line":195,"method":"_push_block","hostname":"","thread_name":"th_a","timestamp":"2015-09-28T20:08:43"},"format":"","data":{"new_block":{"p  revious":"0004361c004527fceb33c4dfe9062a92a6421508","timestamp":"2015-09-28T20:08:42","witness":"1.6.5248","transaction_merkle_root":"00000000000000000000000000000000  00000000","extensions":[],"witness_signature":"1f051e73526e48fd46a4359daaede6e5e5a8b6f451194c24caad9a1287506ef0cd026c0ecdadb5ea35b9843dc0bd68faee59053278a7ce6e571f838  a2d870bebc9","transactions":[]}}},{"context":{"level":"warn","file":"application.cpp","line":428,"method":"handle_block","hostname":"","thread_name":"th_a","timestamp  ":"2015-09-28T20:08:43"},"format":"","data":{"blk_msg":{"block":{"previous":"0004361c004527fceb33c4dfe9062a92a6421508","timestamp":"2015-09-28T20:08:42","witness":"1.  6.5248","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"1f051e73526e48fd46a4359daaede6e5e5a8b6f451194c24caad  9a1287506ef0cd026c0ecdadb5ea35b9843dc0bd68faee59053278a7ce6e571f838a2d870bebc9","transactions":[]},"block_id":"0004361d1c4d414512a8ae8e99ecf2026627d08e"},"sync_mode":  true}}]}      node.cpp:2959                                                                                                                                           
  2015-09-28T20:08:43 p2p:send_sync_block_to_node_delegate send_sync_block_to_n ] disconnecting client 104.155.223.175:32832 because it offered us the rejected block       node.cpp:3073
....................
  2015-09-28T20:08:56 p2p:send_sync_block_to_node_delegate send_sync_block_to_n ] Successfully pushed sync block 275997 (id:0004361d7bf5aecfd85eac9e45b1d1e84287a640)       node.cpp:2935
...................
  2015-09-28T20:09:09 p2p:message read_loop process_block_during ] Successfully pushed block 275998 (id:0004361e9fceb5c9825c54df7adb585bc4976602)     node.cpp:3254
....................
  2015-09-28T20:09:15 p2p:message read_loop process_block_during ] Successfully pushed block 275999 (id:0004361f189606417518259ba750f34aff126187)     node.cpp:3254
.....................
.................
...............
b>2015-09-28T20:26:24 p2p:message read_loop process_block_during ] Successfully pushed block 276099 (id:000436831df20b58af1fda3d1439dd6c1423ec24)     node.cpp:3254     
...............
forked at block 275997.
from the log, it have received the correct block 275997:0004361d1c4d414512a8ae8e99ecf2026627d08e
but it denied it.
then switch to the wrong block: 0004361d7bf5aecfd85eac9e45b1d1e84287a640