Test Net for Advanced Users

[bytemaster]

All of my updated nodes also crashed.  I am now unable to --resync-blockchain.  I only have a single seed node listed in my startup script, but it is not responding. 

Please post your peers so that I may sync.

The network is effectively dead at this point.  It will require a checkpoint to revive.   

Could everyone post their head block number and hash?

[bytemaster]

All of my nodes were hung when I came in this morning.  Looks like a major issue was discovered and we are looking into the cause. 

Based upon the crash reports it looks like someone attempted a force settle. 

[Fox]

All of my updated nodes also crashed.  I am now unable to --resync-blockchain.  I only have a single seed node listed in my startup script, but it is not responding. 

Please post your peers so that I may sync.

[bytemaster]

I wake up this morning and my witness for all night was not able to sync the correct fork.
I have a while [ True ] in my bash script so if it crash, it start again but every times gives me this error and started again to try to resync (without success):

Code: [Select]

   93.3428%   284000 of 304255
1818515ms th_a       undo_database.hpp:62          ~session             ] 10 assert_exception: Assert Exception
!_disabled:
    {}
    th_a  undo_database.cpp:88 undo

    {}
    th_a  undo_database.cpp:116 undo
terminate called after throwing an instance of 'fc::assert_exception'
Aborted (core dumped)

Now the  witness if updated to the new master.

This issue was fixed yesterday. 

[iHashFury]

Crashed on witness and one of my nodes at the same block

# witness crash

Code: [Select]

1244000ms th_a       witness.cpp:182               block_production_loo ] Not producing block because it isn't my turn
witness_node: /home/ihashfury/tmp/graphene/libraries/chain/db_market.cpp:290: bool graphene::chain::database::fill_order(const graphene:
:chain::call_order_object&, const graphene::chain::asset&, const graphene::chain::asset&): Assertion `order.get_collateral() >= pays' fa
iled.
Aborted (core dumped)
# node crash

Code: [Select]

1272262ms th_a       application.cpp:388           handle_block         ] Got block #314815 with time 2015-09-30T08:21:12 from network with latency of 263 ms from init0
witness_node: /home/ihashfury/tmp/graphene/libraries/chain/db_market.cpp:290: bool graphene::chain::database::fill_order(const graphene::chain::call_order_object&, const graphene::chain::asset&, const graphene::chain::asset&): Assertion `order.get_collateral() >= pays' failed.
Aborted

[wackou]

not sure if that's helpful, but I tried to get some more info out of it, and I could retrieve this:

the block in push_block (in stack frame #16) has a header that starts like this:

Code: [Select]

{<graphene::chain::block_header> = {previous = {_hash = {3201106944, 1397224070, 591270816, 3178160838, 2663318844}}, timestamp = {utc_seconds = 1443601296} [...]


[wackou]

had this crash too:

Code: [Select]

1294000ms th_a       witness.cpp:182               block_production_loo ] Not producing block because it isn't my turn
1295000ms th_a       witness.cpp:182               block_production_loo ] Not producing block because it isn't my turn
witness_node: /home/admin/.BitShares2_build/libraries/chain/db_market.cpp:290: bool graphene::chain::database::fill_order(const graphene::chain::call_order_object&, const graphene::chain::asset&, const graphene::chain::asset&): Assertion `order.get_collateral() >= pays' failed.

Program received signal SIGABRT, Aborted.
0x00007ffff6516107 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0  0x00007ffff6516107 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007ffff65174e8 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007ffff650f226 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007ffff650f2d2 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#4  0x00000000024ec8a3 in graphene::chain::database::fill_order (this=0x360f160, order=..., pays=..., receives=...)
    at /home/admin/.BitShares2_build/libraries/chain/db_market.cpp:290
#5  0x00000000024ec173 in graphene::chain::database::match (this=0x360f160, call=..., settle=..., match_price=..., max_settlement=...)
    at /home/admin/.BitShares2_build/libraries/chain/db_market.cpp:239
#6  0x00000000024f1f82 in graphene::chain::database::clear_expired_orders (this=0x360f160) at /home/admin/.BitShares2_build/libraries/chain/db_update.cpp:237
#7  0x00000000024d4b90 in graphene::chain::database::_apply_block (this=0x360f160, next_block=...)
    at /home/admin/.BitShares2_build/libraries/chain/db_block.cpp:478
#8  0x00000000024d4088 in graphene::chain::database::<lambda()>::operator()(void) const (__closure=0x7fffe7df1330)
    at /home/admin/.BitShares2_build/libraries/chain/db_block.cpp:436
#9  0x00000000024f2f55 in graphene::chain::detail::with_skip_flags<graphene::chain::database::apply_block(const graphene::chain::signed_block&, uint32_t)::<lambda()> >(graphene::chain::database &, uint32_t, graphene::chain::database::<lambda()>) (db=..., skip_flags=0, callback=...)
    at /home/admin/.BitShares2_build/libraries/chain/include/graphene/chain/db_with.hpp:123
#10 0x00000000024d44ad in graphene::chain::database::apply_block (this=0x360f160, next_block=..., skip=0)
    at /home/admin/.BitShares2_build/libraries/chain/db_block.cpp:437
#11 0x00000000024ce648 in graphene::chain::database::_push_block (this=0x360f160, new_block=...)
    at /home/admin/.BitShares2_build/libraries/chain/db_block.cpp:185
#12 0x00000000024cdc21 in graphene::chain::database::<lambda()>::<lambda()>::operator()(void) const (__closure=0x7fffe7df1e20)
    at /home/admin/.BitShares2_build/libraries/chain/db_block.cpp:112
#13 0x00000000024f2d5e in graphene::chain::detail::without_pending_transactions<graphene::chain::database::push_block(const graphene::chain::signed_block&, uint32_t)::<lambda()>::<lambda()> >(graphene::chain::database &, <unknown type in /home/admin/.BitShares2_bin/witness_node_2015-09-24_test3c-1-gd01fc0a, CU 0xfa3feb, DIE 0x11f4221>, graphene::chain::database::<lambda()>::<lambda()>) (db=...,
    pending_transactions=<unknown type in /home/admin/.BitShares2_bin/witness_node_2015-09-24_test3c-1-gd01fc0a, CU 0xfa3feb, DIE 0x11f4221>, callback=...)
    at /home/admin/.BitShares2_build/libraries/chain/include/graphene/chain/db_with.hpp:140
#14 0x00000000024cdc92 in graphene::chain::database::<lambda()>::operator()(void) const (__closure=0x7fffe7df1ed0)
    at /home/admin/.BitShares2_build/libraries/chain/db_block.cpp:113
#15 0x00000000024f2dd9 in graphene::chain::detail::with_skip_flags<graphene::chain::database::push_block(const graphene::chain::signed_block&, uint32_t)::<lambda()> >(graphene::chain::database &, uint32_t, graphene::chain::database::<lambda()>) (db=..., skip_flags=0, callback=...)
    at /home/admin/.BitShares2_build/libraries/chain/include/graphene/chain/db_with.hpp:123
#16 0x00000000024cdce3 in graphene::chain::database::push_block (this=0x360f160, new_block=..., skip=0)
    at /home/admin/.BitShares2_build/libraries/chain/db_block.cpp:114
---Type <return> to continue, or q <return> to quit---
#17 0x00000000024d158b in graphene::chain::database::_generate_block (this=0x360f160, when=..., witness_id=..., block_signing_private_key=..., retry_on_failure=true) at /home/admin/.BitShares2_build/libraries/chain/db_block.cpp:368
#18 0x00000000024d07d3 in graphene::chain::database::<lambda()>::operator()(void) const (__closure=0x7fffe7df2c40) at /home/admin/.BitShares2_build/libraries/chain/db_block.cpp:276
#19 0x00000000024f2ecf in graphene::chain::detail::with_skip_flags<graphene::chain::database::generate_block(fc::time_point_sec, graphene::chain::witness_id_type, const fc::ecc::private_key&, uint32_t)::<lambda()> >(graphene::chain::database &, uint32_t, graphene::chain::database::<lambda()>) (db=..., skip_flags=0, callback=...) at /home/admin/.BitShares2_build/libraries/chain/include/graphene/chain/db_with.hpp:123
#20 0x00000000024d089f in graphene::chain::database::generate_block (this=0x360f160, when=..., witness_id=..., block_signing_private_key=..., skip=0) at /home/admin/.BitShares2_build/libraries/chain/db_block.cpp:277
#21 0x00000000024c218f in graphene::witness_plugin::witness_plugin::maybe_produce_block (this=0x3618410, capture=...) at /home/admin/.BitShares2_build/libraries/plugins/witness/witness.cpp:276
#22 0x00000000024bff51 in graphene::witness_plugin::witness_plugin::block_production_loop (this=0x3618410) at /home/admin/.BitShares2_build/libraries/plugins/witness/witness.cpp:160
#23 0x00000000024bfe19 in graphene::witness_plugin::witness_plugin::<lambda()>::operator()(void) const (__closure=0x1639d188) at /home/admin/.BitShares2_build/libraries/plugins/witness/witness.cpp:150
#24 0x00000000024c3288 in fc::detail::void_functor_run<graphene::witness_plugin::witness_plugin::schedule_production_loop()::<lambda()> >::run(void *, void *) (functor=0x1639d188, prom=0x1639d180)
    at /home/admin/.BitShares2_build/libraries/fc/include/fc/thread/task.hpp:83
#25 0x00000000027e209d in fc::task_base::run_impl (this=0x1639d190) at /home/admin/.BitShares2_build/libraries/fc/src/thread/task.cpp:43
#26 0x00000000027e202c in fc::task_base::run (this=0x1639d190) at /home/admin/.BitShares2_build/libraries/fc/src/thread/task.cpp:32
#27 0x00000000027d608a in fc::thread_d::run_next_task (this=0x364d2f0) at /home/admin/.BitShares2_build/libraries/fc/src/thread/thread_d.hpp:498
#28 0x00000000027d6554 in fc::thread_d::process_tasks (this=0x364d2f0) at /home/admin/.BitShares2_build/libraries/fc/src/thread/thread_d.hpp:547
#29 0x00000000027d5b7e in fc::thread_d::start_process_tasks (my=56939248) at /home/admin/.BitShares2_build/libraries/fc/src/thread/thread_d.hpp:475
#30 0x0000000002b50711 in make_fcontext ()
#31 0x00010102464c457f in ?? ()
#32 0x0000000000000000 in ?? ()
(gdb)
Note that I hadn't upgraded to the latest master yet, will do so now.

[Riverhead]

Ditto. I'll restart in gdb.

Code: [Select]

witness_node: /home/riverhead/github/graphene/libraries/chain/db_market.cpp:290: bool graphene::chain::database::fill_order(const graphene::chain::call_order_object&, const graphene::chain::asset&, const graphene::chain::asset&): Assertion `order.get_collateral() >= pays' failed.
Aborted


[CalabiYau]

calabiyau updated

Code: [Select]

witness_node: /home/calabiyau/graphene/libraries/chain/db_market.cpp:290: bool graphene::chain::database::fill_order(const graphene::chain::call_order_object&, const graphene::chain::asset&, const graphene::chain::asset&): Assertion `order.get_collateral() >= pays' failed.
Aborted (core dumped)

seems to sync now after rm blockchain again

uumm - no:

Code: [Select]

witness_node: /home/calabiyau/graphene/libraries/chain/db_market.cpp:290: bool graphene::chain::database::fill_order(const graphene::chain::call_order_object&, const graphene::chain::asset&, const graphene::chain::asset&): Assertion `order.get_collateral() >= pays' failed.
Aborted (core dumped)


[clayop]

It seems that the network has some serious problem. My cli_client says that the head block age is 30 min old.

[spartako]

I am no more able to run my witness, every time I get this error:

Code: [Select]

witness_node: /home/spartako/graphene/libraries/chain/db_market.cpp:290: bool graphene::chain::database::fill_order(const graphene::chain::call_order_object&, const graphene::chain::asset&, const graphene::chain::asset&): Assertion `order.get_collateral() >= pays' failed.
Aborted (core dumped)


Also https://graphene.bitshares.org/ seems down

[spartako]

I wake up this morning and my witness for all night was not able to sync the correct fork.
I have a while [ True ] in my bash script so if it crash, it start again but every times gives me this error and started again to try to resync (without success):

Code: [Select]

   93.3428%   284000 of 304255
1818515ms th_a       undo_database.hpp:62          ~session             ] 10 assert_exception: Assert Exception
!_disabled:
    {}
    th_a  undo_database.cpp:88 undo

    {}
    th_a  undo_database.cpp:116 undo
terminate called after throwing an instance of 'fc::assert_exception'
Aborted (core dumped)

Now the  witness if updated to the new master.

[clayop]

I tested new network code with preliminary spam transactions, say 1500 txs. The network processed all the transactions very successfully.

I will run the third stress test tomorrow at around 0:00 UTC (the schedule is subject to be changed with further announcement)

All testnet witnesses, please update your witness node before then.


Add: a drill will take place in 20 min (23:45 PST) for 3 min.

[clayop]

Special thanks to the five (5) community members taking part in the fund matching offer to support Clayop's efforts with spamming the network from a global VPS entourage.  These individuals combined to contribute 9500 BTS.  I kicked in an extra 500 BTS extra, making it a 5-digit match.

Again, thanks to the community and Clayop. 

Best,
Fox

Many thanks to all community members with invaluable support for BitShares.

[roadscape]

roadscape (1.6.5249) is up to date but I need more votes!