If it was only replaying it never caught up. It just ended in a firey crash:
237745ms th_a application.cpp:712 get_blockchain_synop ] synopsis: ["0000dd20f0f688d88c6ecafa73e9396da128575e","0000dd2ddbe5e2f349056489807735ee69a72f64","0000dd3361101334b8cd482d158f26075aa19cda","0000dd36f4282b71818c080ed38b031694afacc9","0000dd388f8db8edf41e928637f20e2720b34082"]
237755ms th_a application.cpp:712 get_blockchain_synop ] synopsis: ["0000dd20f0f688d88c6ecafa73e9396da128575e","0000dd2ddbe5e2f349056489807735ee69a72f64","0000dd3361101334b8cd482d158f26075aa19cda","0000dd36f4282b71818c080ed38b031694afacc9","0000dd388f8db8edf41e928637f20e2720b34082"]
237789ms th_a application.cpp:712 get_blockchain_synop ] synopsis: ["0000dd20f0f688d88c6ecafa73e9396da128575e","0000dd2ddbe5e2f349056489807735ee69a72f64","0000dd3361101334b8cd482d158f26075aa19cda","0000dd36f4282b71818c080ed38b031694afacc9","0000dd388f8db8edf41e928637f20e2720b34082"]
237792ms th_a application.cpp:712 get_blockchain_synop ] synopsis: ["0000dd20f0f688d88c6ecafa73e9396da128575e","0000dd2ddbe5e2f349056489807735ee69a72f64","0000dd3361101334b8cd482d158f26075aa19cda","0000dd36f4282b71818c080ed38b031694afacc9","0000dd388f8db8edf41e928637f20e2720b34082"]
237800ms th_a application.cpp:712 get_blockchain_synop ] synopsis: ["0000dd20f0f688d88c6ecafa73e9396da128575e","0000dd2ddbe5e2f349056489807735ee69a72f64","0000dd3361101334b8cd482d158f26075aa19cda","0000dd36f4282b71818c080ed38b031694afacc9","0000dd388f8db8edf41e928637f20e2720b34082"]
237856ms th_a application.cpp:712 get_blockchain_synop ] synopsis: ["0000dd20f0f688d88c6ecafa73e9396da128575e"]
witness_node: /home/deletech/BitShares2_build/libraries/fc/src/thread/thread_d.hpp:370: bool fc::thread_d::start_next_fiber(bool): Assertion `std::current_exception() == std::exception_ptr()' failed.
Aborted
This was an attempt to start from scratch. Wiped object_database, blockchain, p2p, logs. No --replay-blockchain or --resync-blockchain. No cmd line params except --data-dir, everything is in the config.ini file. All essentials from config noted in my previous posts.
Here's the start of another attempt. This time left everything intact from the run that crashed except I removed the object_database folder AND I used puppies seed instead of the :2005 seed.
973661ms th_a witness.cpp:111 plugin_initialize ] witness plugin: plugin_initialize() end
973662ms th_a application.cpp:302 startup ] Detected unclean shutdown. Replaying blockchain...
973662ms th_a application.cpp:243 operator() ] Initializing database...
1073034ms th_a db_management.cpp:45 reindex ] reindexing blockchain
1073047ms th_a db_management.cpp:98 wipe ] Wiping database
1073179ms th_a object_database.cpp:81 wipe ] Wiping object_database.
1133390ms th_a db_debug.cpp:79 debug_dump ] total_balances[asset_id_type()].value: 14239376866944 core_asset_data.current_supply.value: 271289987694689
1133406ms th_a db_debug.cpp:82 debug_dump ] core_in_orders: 14239376866944 reported_core_in_orders: 14239376866944
1133477ms th_a db_management.cpp:147 open ] last_block->id(): 0000dd388f8db8edf41e928637f20e2720b34082 last_block->block_num(): 56632
1133477ms th_a db_management.cpp:59 reindex ] Replaying blocks...
3.53157% 2000 of 56632
7.06314% 4000 of 56632
10.5947% 6000 of 56632
14.1263% 8000 of 56632
17.6579% 10000 of 56632
21.1894% 12000 of 56632
24.721% 14000 of 56632
28.2526% 16000 of 56632
31.7842% 18000 of 56632
35.3157% 20000 of 56632
38.8473% 22000 of 56632
42.3789% 24000 of 56632
45.9104% 26000 of 56632
49.442% 28000 of 56632
52.9736% 30000 of 56632
56.5052% 32000 of 56632
60.0367% 34000 of 56632
1272503ms th_a asset_evaluator.cpp:553 do_evaluate ] USD feed pub with wrong ass
et by wackou in block 34129
1272511ms th_a asset_evaluator.cpp:618 do_apply ] Ignoring bad feed
1272603ms th_a asset_evaluator.cpp:553 do_evaluate ] SEK feed pub with wrong asset by wackou in block 34215
1272616ms th_a asset_evaluator.cpp:618 do_apply ] Ignoring bad feed
63.5683% 36000 of 56632
1279203ms th_a asset_evaluator.cpp:553 do_evaluate ] SILVER feed pub with wrong asset by wackou in block 36010
1279220ms th_a asset_evaluator.cpp:618 do_apply ] Ignoring bad feed
67.0999% 38000 of 56632
70.6314% 40000 of 56632
74.163% 42000 of 56632
77.6946% 44000 of 56632
81.2262% 46000 of 56632
84.7577% 48000 of 56632
1425106ms th_a asset_evaluator.cpp:553 do_evaluate ] EUR feed pub with wrong ass
et by wackou in block 49903
1425118ms th_a asset_evaluator.cpp:618 do_apply ] Ignoring bad feed
88.2893% 50000 of 56632
1427034ms th_a asset_evaluator.cpp:553 do_evaluate ] RUB feed pub with wrong ass
et by wackou in block 50353
1427045ms th_a asset_evaluator.cpp:618 do_apply ] Ignoring bad feed
91.8209% 52000 of 56632
95.3525% 54000 of 56632
98.884% 56000 of 56632
1459060ms th_a db_management.cpp:93 reindex ] Done reindexing, elapsed ti
me: 325.58274599999998600 sec
1459300ms th_a thread.cpp:95 thread ] name:ntp tid:14049966996044
8
1459327ms th_a thread.cpp:95 thread ] name:p2p tid:14049965107788
8
1459390ms th_a application.cpp:123 reset_p2p_node ] Adding seed node 104.236.14
4.84:1776
1459394ms th_a application.cpp:135 reset_p2p_node ] Configured p2p node to list
en on 0.0.0.0:1776
1459419ms th_a application.cpp:185 reset_websocket_serv ] Configured websocket rpc to
listen on 127.0.0.1:8090
1459435ms th_a witness.cpp:116 plugin_startup ] witness plugin: plugin_sta
rtup() begin
1459435ms th_a witness.cpp:123 plugin_startup ] Launching block production
for 1 witnesses.
1459435ms th_a witness.cpp:134 plugin_startup ] witness plugin: plugin_sta
rtup() end
1459435ms th_a main.cpp:167 main ] Started witness node on a c
hain with 56632 blocks.
1459435ms th_a main.cpp:168 main ] Chain ID is 60e21871125ea99
95fe498b7f68a87a85c6583725ea5448f6fd969c59a37df83
1459450ms ntp ntp.cpp:177 read_loop ] ntp_delta_time updated to -
23481 us
1459561ms th_a application.cpp:712 get_blockchain_synop ] synopsis: ["0000dd388f8db8e
df41e928637f20e2720b34082"]
1459627ms th_a application.cpp:712 get_blockchain_synop ] synopsis: ["0000dd388f8db8e
df41e928637f20e2720b34082"]
1459678ms th_a application.cpp:712 get_blockchain_synop ] synopsis: ["0000dd388f8db8e
df41e928637f20e2720b34082"]
1459724ms th_a application.cpp:712 get_blockchain_synop ] synopsis: ["0000dd388f8db8e
df41e928637f20e2720b34082"]
1459774ms th_a application.cpp:712 get_blockchain_synop ] synopsis: ["0000dd388f8db8e
df41e928637f20e2720b34082"]
1459823ms th_a application.cpp:712 get_blockchain_synop ] synopsis: ["0000dd388f8db8e
df41e928637f20e2720b34082"]
Note that it never gets to 100%, but I see now that these percentages aren't for the replay but rather for the reindexing. If the replay is still in progress and the blockchain is yet to be synced, why does it consistently fail and crash? I notice these "get_blockchain_synop" lines also appear quite frequently in the witness node that is working properly and signing blocks.
At least I think it's working properly, I don't see my missed blocks count increasing and I do see blocks from other witnesses and these lines:
2748001ms th_a witness.cpp:176 block_production_loo ] Generated block #111989 with timestamp 2015-10-09T20:45:48 at time 2015-10-09T20:45:48
Which tell me all is well.