One of my node got out of sync when you're flooding the network. An assertion failure which is never seen before.
Here is some related log:
2015-09-03T00:30:20 th_a:invoke handle_block handle_block ] Got block #34498 from network application.cpp:356
...
...
2015-09-03T00:30:30 th_a:invoke handle_transaction handle_transaction ] Got transaction from network application.cpp:391
2015-09-03T00:30:30 th_a:invoke handle_transaction handle_transaction ] Got transaction from network application.cpp:391
2015-09-03T00:30:30 th_a:invoke handle_block handle_block ] Got block #34500 from network application.cpp:356
2015-09-03T00:30:30 th_a:invoke handle_block push_block ] Pushing block to fork database that failed to link. fork_database.cpp:57
2015-09-03T00:30:35 th_a:invoke handle_transaction handle_transaction ] Got transaction from network application.cpp:391
2015-09-03T00:30:35 th_a:invoke handle_block handle_block ] Got block #34501 from network application.cpp:356
2015-09-03T00:30:35 th_a:invoke handle_block push_block ] Pushing block to fork database that failed to link. fork_database.cpp:57
2015-09-03T00:30:37 th_a:invoke handle_block handle_block ] Got block #34499 from network application.cpp:3
...
...
2015-09-03T00:30:37 th_a:invoke handle_block handle_block ] Error when pushing block:
10 assert_exception: Assert Exception
(skip & skip_transaction_dupe_check) || trx_idx.indices().get<by_trx_id>().find(trx_id) == trx_idx.indices().get<by_trx_id>().end():
{}
th_a db_block.cpp:487 _apply_transaction
{"trx":{"ref_block_num":34497,"ref_block_prefix":2564640669,"expiration":"2015-09-03T00:31:49","operations":[[0,{"fee":{"amount":2000000,"asset_id":"1.3.0"},"from":"1.2.63354","to":"1.2.63355","amount":{"amount":100000,"asset_id":"1.3.0"},"extensions":[]}]],"extensions":[],"signatures":["1f5f41592441c4f40af47b118e80a0c4786a1d768cf6a8d3204d7c30d79d33e971463c4dc78f1c5fc8f2ae7637ec21e8b523b089baef754422fe99108576b343c8"]}}
th_a db_block.cpp:543 _apply_transaction
{"next_block.block_num()":34499}
th_a db_block.cpp:448 _apply_block
...
...
2015-09-03T00:30:37 th_a:invoke handle_transaction handle_transaction ] Got transaction from network application.cpp:391
2015-09-03T00:30:37 th_a:invoke handle_transaction on_objects_changed ] ids: ["2.5.40","2.5.18","2.6.63354","2.7.4698"] api.cpp:867
2015-09-03T00:30:37 th_a:invoke handle_transaction on_objects_changed ] ids: ["2.5.40","2.5.18","2.6.63354","2.7.4698"] api.cpp:867
2015-09-03T00:30:40 th_a:invoke handle_block handle_block ] Got block #34502 from network application.cpp:356
2015-09-03T00:30:40 th_a:invoke handle_block push_block ] Pushing block to fork database that failed to link. fork_database.cpp:57
Full log files are here:
https://drive.google.com/open?id=0B3xrm70jSHn4U3h3V3liOUlOakk
https://drive.google.com/open?id=0B3xrm70jSHn4TmtNYWNaVm1yalU
https://drive.google.com/open?id=0B3xrm70jSHn4SURQZjBZaE1rVGc
The other node works fine.
Strange. Restarting doesn't help. Will try resync.
//Edit: resync worked. Will post here if I found something interesting in the log files.
//Edit2: here is related logs which possibly explains why unable to sync after restarted.
* The head_block of the chain was block 34498, id = 000086c202f51da427ebf0507404bc9b6d192f96
* Witness failed to push block 34499 last time, id = 000086c3bf20d2e5eeb1039f38c221b46880f2f0
* After restarted, the witness ignores block 34499 (id 000086c3bf20d2e5eeb1039f38c221b46880f2f0) because it's considered "seen"
* Following blocks are unable to push to chain because they're not linked (because 34499 is not on chain)
default.log:
2015-09-03T09:46:28 th_a:?unnamed? main ] Started witness node on a chain with 34498 blocks. main.cpp:165
...
2015-09-03T09:48:39 th_a:invoke handle_block push_block ] Pushing block to fork database that failed to link. fork_database.cpp:57
p2p.log:
2015-09-03T09:46:30 p2p:message read_loop fetch_next_batch_of_ ] sync: sending a request for the next items after 000086c202f51da427ebf0507404bc9b6d192f96 to peer 176.9.234.167:40936, (full request is ["000006c280cdf25ca122561981716c74045f16fe","000046c2482fc56dda591b0bea7a47ccbd64d6b2","000066c27fc3abd468f7dd5ad1ce8939614d9d64","000076c2a8ad357d929c0967d98ba955521c11b4","00007ec23da7bcf6188fa3e6e57f0216e78cba1a","000082c22a40c3cbc1152bbd18239e9ffca051dc","000084c22add8beb2d06e3adb13c3573a56ae0b4","000085c29de6e5d439166100ab9d809c730e006c","00008642bd45a1a5711a74bacf4e73cfd5ae69cd","00008682ea2e75d79000e7a4c3411368d7a0a74a","000086a273f0cd7e751e91f6b0795486303009a2","000086b28863612b25b49a1c0a95a3525e42529c","000086ba6b31127d85ba6e1a8b5ea1145a0766e0","000086be74b88f7adb24796112e15c72b04a4e59","000086c0a1c19fb2c9b2d8d92f1caf812fda4bde","000086c19d4fdd989e8e7a62830c6f58e73f33be","000086c202f51da427ebf0507404bc9b6d192f96"]) node.cpp:2294
...
...
2015-09-03T09:46:30 p2p:message read_loop on_blockchain_item_i ] sync: received a list of 669 available items from 192.241.198.6:39809 node.cpp:2310
2015-09-03T09:46:30 p2p:message read_loop on_blockchain_item_i ] is_first_item_for_other_peer: false. item_hashes_received.size() = 669 node.cpp:2356
2015-09-03T09:46:30 p2p:message read_loop on_blockchain_item_i ] popping item because delegate has already seen it. peer's last block the delegate has seen is now 000086c202f51da427ebf0507404bc9b6d192f96 (1) node.cpp:2368
2015-09-03T09:46:30 p2p:message read_loop on_blockchain_item_i ] popping item because delegate has already seen it. peer's last block the delegate has seen is now 000086c3bf20d2e5eeb1039f38c221b46880f2f0 (2) node.cpp:2368
2015-09-03T09:46:30 p2p:message read_loop on_blockchain_item_i ] after removing all items we have already seen, item_hashes_received.size() = 667 node.cpp:2371
2015-09-03T09:46:30 p2p:message read_loop trigger_fetch_sync_i ] Triggering fetch sync items loop now node.cpp:1084
2015-09-03T09:46:30 p2p:fetch_sync_items_loop fetch_sync_items_loo ] beginning another iteration of the sync items loop node.cpp:1023
2015-09-03T09:46:30 p2p:fetch_sync_items_loop request_sync_items_f ] requesting 100 item(s) ["000086c4e3da8babd770344cb8db18ee89a9d09a","000086c50ebdc7f4b5ff303fee02b53f3aec5773",...,"00008727dc9a24b61d0d51454278f2282e89672d"] from peer 192.241.198.6:39809 node.cpp:1007
...
...
2015-09-03T09:48:39 p2p:send_sync_block_to_node_delegate send_sync_block_to_n ] Successfully pushed sync block 34500 (id:000086c4e3da8babd770344cb8db18ee89a9d09a) node.cpp:2793
2015-09-03T09:48:39 p2p:send_sync_block_to_node_delegate send_sync_block_to_n ] sync: client accpted the block, we now have only 5937 items left to fetch before we're in sync node.cpp:2831
Full default.log:
https://drive.google.com/open?id=0B3xrm70jSHn4UWJ2N2QzdkVOX2MFull p2p.log:
https://drive.google.com/open?id=0B3xrm70jSHn4QVNoanZfMS13UXM