witness_node completely froze overnight

[pc]

I think the hangs only happen when you restart the node without rescanning. At least my nodes never had problems after rescanning.

[abit]

Why only some witnesses have this issue

[alt]

I'm sure this frozen happend because of settlement. my 3 node all frozen at block 1849845

Code: [Select]

2478528ms th_a       application.cpp:496           handle_block         ] Got block: #1849845 time: 2015-12-17T05:41:18 latency: 577 ms from: delegate.taolje  irreversi
ble: 1849827 (-18)

and  the settlement happend at 05:41:18

Code: [Select]

PRICE CNY/BTS      AMOUNT(BTS)      VALUE(CNY)      DATE
0.02404            0.0042           0.000           13:42:42
0.02400            207,186.7375     4,972.482       13:41:18


[alt]

this happend very often,
everytime this happened, it's run in the infinite loop: clear_expired_orders()
maybe it's caused by the expired propose?

Code: [Select]

285       // At each iteration, we either consume the current order and remove it, or we move to the next asset
286       for( auto itr = settlement_index.lower_bound(current_asset);
287            itr != settlement_index.end();
288            itr = settlement_index.lower_bound(current_asset) )                                                                                                     
289       {
290          const force_settlement_object& order = *itr;
291          auto order_id = order.id;
292          current_asset = order.settlement_asset_id();
293          const asset_object& mia_object = get(current_asset);
294          const asset_bitasset_data_object mia = mia_object.bitasset_data(*this);
295
.......................
......................


[merivercap]

I was just trying to start up a full node with some help, but one time it froze at block 587,634.  The rest of the time it wouldn't get past the first block so I'm stuck right now.   Does my problem have anything to do with this issue or is it probably something else?

Sounds like a separate issue... mine froze well after resync, and yours froze during.

Which block are you stuck on now?

What command are you using to launch the witness?

How much free space (disk/mem) do you have?

Ahhh... You know I think I might not have that much disk space on this laptop so that might be the problem.  I'll have to try another machine.    Thanks! 

[roadscape]

I was just trying to start up a full node with some help, but one time it froze at block 587,634.  The rest of the time it wouldn't get past the first block so I'm stuck right now.   Does my problem have anything to do with this issue or is it probably something else?

Sounds like a separate issue... mine froze well after resync, and yours froze during.

Which block are you stuck on now?

What command are you using to launch the witness?

How much free space (disk/mem) do you have?

[alt]

I have met this froze several times, cpu usage is  100%, here is some info from gdb

Code: [Select]

(gdb) bt
#0  0x0000000000dcecf5 in void fc::divide<fc::uint128>(fc::uint128 const&, fc::uint128 const&, fc::uint128&, fc::uint128&) ()
#1  0x0000000000dcedf4 in fc::uint128::operator/=(fc::uint128 const&) ()
#2  0x0000000000be6d9a in graphene::chain::database::clear_expired_orders() ()
#3  0x0000000000beda0f in graphene::chain::database::_apply_block(graphene::chain::signed_block const&) ()
#4  0x0000000000bee835 in graphene::chain::database::apply_block(graphene::chain::signed_block const&, unsigned int) ()
#5  0x0000000000beed34 in graphene::chain::database::_push_block(graphene::chain::signed_block const&) ()
#6  0x0000000000bf0af3 in graphene::chain::database::push_block(graphene::chain::signed_block const&, unsigned int) ()
#7  0x0000000000a3783d in graphene::app::detail::application_impl::handle_block(graphene::net::block_message const&, bool, std::vector<fc::ripemd160, std::allocator<fc::ripemd160> >&) ()
#8  0x0000000000f3f9d7 in fc::detail::functor_run<graphene::net::detail::statistics_gathering_node_delegate_wrapper::handle_block(graphene::net::block_message const&, bool, std::vector<fc::ripemd160, std::allocator<fc::ripemd160> >&)::{lambda()#1}>::run(void*, fc::detail::functor_run<graphene::net::detail::statistics_gathering_node_delegate_wrapper::handle_block(graphene::net::block_message const&, bool, std::vector<fc::ripemd160, std::allocator<fc::ripemd160> >&)::{lambda()#1}>) ()
#9  0x0000000000dedfa4 in fc::task_base::run_impl() ()
#10 0x0000000000debbaf in fc::thread_d::process_tasks() ()
#11 0x0000000000debdf1 in fc::thread_d::start_process_tasks(long) ()
#12 0x0000000001061621 in make_fcontext ()
#13 0x0000000000000000 in ?? ()
seems froze at here

Code: [Select]

285       // At each iteration, we either consume the current order and remove it, or we move to the next asset
286       for( auto itr = settlement_index.lower_bound(current_asset);
287            itr != settlement_index.end();
288            itr = settlement_index.lower_bound(current_asset) )                                                                                                     
289       {
290          const force_settlement_object& order = *itr;
291          auto order_id = order.id;
292          current_asset = order.settlement_asset_id();
293          const asset_object& mia_object = get(current_asset);
294          const asset_bitasset_data_object mia = mia_object.bitasset_data(*this);
295
.......................
......................


[merivercap]

I was just trying to start up a full node with some help, but one time it froze at block 587,634.  The rest of the time it wouldn't get past the first block so I'm stuck right now.   Does my problem have anything to do with this issue or is it probably something else?

[roadscape]

This has happened to two other witnesses and a bug report was submitted.

It is unclear what is causing it so far. The same thing has caused cryptofresh to stop loading twice in the last few days.

Correction: only once due to this mysterious hang

The other time was a HD full of log files..

[monsterer]

It's something in the latest version which is causing it. Unclear what, though.

[BunkerChainLabs-DataSecurityNode]

This has happened to two other witnesses and a bug report was submitted.

It is unclear what is causing it so far. The same thing has caused cryptofresh to stop loading twice in the last few days.

[monsterer]

Unfortunately, no really log information available, but the latest witness_node just froze overnight; this is the first time this has happened.