Test Net for Advanced Users

[spartako]

It seems the stress test has genereted a big block (> 100tx/sec):

blockNum IdWitness  nameWitness    nTx
196454     '1.6.3356'   'mr.agsexplorer' 507

[Akado]

Glad the testing thread has more pages than the price discussion one, it's a good sign, keep it up guys! 

[puppies]

My witness producing node is no longer losing sync during stress tests, but it does stop producing blocks.  I can't find anything in logs that sheds any light on it.

How are you determining that it stopped producing blocks?  Perhaps the absence of logs is a clue

I am just watching the witness vesting pay 1.13.163.  The witness is still voted in.  Info shows a recent block time.  At this point, I am rebuilding, and ill verify the issue remains.  Then I was thinking I would run it on a more powerful machine and see if that helped.  I'll also keep digging in the logs to see if I can find anything.

Is there a way to see the scheduled witness order?  Some way to tell which witness was scheduled when a block isn't produced?

Does restarting the witness fix the production issue?

Yes. 
In fact it just started producing again after I stopped stress testing.  Without a restart

[bytemaster]

My witness producing node is no longer losing sync during stress tests, but it does stop producing blocks.  I can't find anything in logs that sheds any light on it.

How are you determining that it stopped producing blocks?  Perhaps the absence of logs is a clue

I am just watching the witness vesting pay 1.13.163.  The witness is still voted in.  Info shows a recent block time.  At this point, I am rebuilding, and ill verify the issue remains.  Then I was thinking I would run it on a more powerful machine and see if that helped.  I'll also keep digging in the logs to see if I can find anything.

Is there a way to see the scheduled witness order?  Some way to tell which witness was scheduled when a block isn't produced?

Does restarting the witness fix the production issue?

[puppies]

My witness producing node is no longer losing sync during stress tests, but it does stop producing blocks.  I can't find anything in logs that sheds any light on it.

How are you determining that it stopped producing blocks?  Perhaps the absence of logs is a clue

I am just watching the witness vesting pay 1.13.163.  The witness is still voted in.  Info shows a recent block time.  At this point, I am rebuilding, and ill verify the issue remains.  Then I was thinking I would run it on a more powerful machine and see if that helped.  I'll also keep digging in the logs to see if I can find anything.

Is there a way to see the scheduled witness order?  Some way to tell which witness was scheduled when a block isn't produced?

[svk]

Also got this when I re-cloned graphene-ui. If there's another thread for the UI I'll move this post there.

Code: [Select]

> webgui@0.0.1 start /home/james/github/graphene-ui/web
> node server.js

Using DEV options

70.00% 1/1 build modules
events.js:72
        throw er; // Unhandled 'error' event
              ^
Error: listen EADDRINUSE
    at errnoException (net.js:901:11)
    at Server._listen2 (net.js:1039:14)
    at listen (net.js:1061:10)
    at net.js:1143:9
    at dns.js:72:18
    at process._tickCallback (node.js:415:13)
    at Function.Module.runMain (module.js:499:11)
    at startup (node.js:119:16)
    at node.js:902:3

npm ERR! webgui@0.0.1 start: `node server.js`
npm ERR! Exit status 8
npm ERR!
npm ERR! Failed at the webgui@0.0.1 start script.
npm ERR! This is most likely a problem with the webgui package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node server.js
npm ERR! You can get their info via:
npm ERR!     npm owner ls webgui
npm ERR! There is likely additional logging output above.
npm ERR! System Linux 3.19.0-28-generic
npm ERR! command "/usr/bin/nodejs" "/usr/bin/npm" "start"
npm ERR! cwd /home/james/github/graphene-ui/web
npm ERR! node -v v0.10.25
npm ERR! npm -v 1.4.21
npm ERR! code ELIFECYCLE
npm WARN This failure might be due to the use of legacy binary "node"
npm WARN For further explanations, please read
/usr/share/doc/nodejs/README.Debian

npm ERR!
npm ERR! Additional logging details can be found in:
npm ERR!     /home/james/github/graphene-ui/web/npm-debug.log
npm ERR! not ok code 0
james@james-desktop:~/github/graphene-ui$


You already have a server running on port 8080, either something else or another copy of the gui.

[bytemaster]

My witness producing node is no longer losing sync during stress tests, but it does stop producing blocks.  I can't find anything in logs that sheds any light on it.

How are you determining that it stopped producing blocks?  Perhaps the absence of logs is a clue

[bytemaster]

My stats.bitshares.eu node died with

Code: [Select]

3498879ms th_a       application.cpp:422           handle_transaction   ] Got transaction from network
3498880ms th_a       application.cpp:422           handle_transaction   ] Got transaction from network
witness_node: /home/delegate/graphene/libraries/fc/src/thread/thread_d.hpp:370: bool fc::thread_d::start_next_fiber(bool): Assertion `std::current_exception() == std::exception_ptr()' failed.
Aborted


The latest push to *fc* should fix it.   Fetch the latest graphene and be sure to update submodules.

[puppies]

My witness producing node is no longer losing sync during stress tests, but it does stop producing blocks.  I can't find anything in logs that sheds any light on it. 

[xeroc]

My stats.bitshares.eu node died with

Code: [Select]

3498879ms th_a       application.cpp:422           handle_transaction   ] Got transaction from network
3498880ms th_a       application.cpp:422           handle_transaction   ] Got transaction from network
witness_node: /home/delegate/graphene/libraries/fc/src/thread/thread_d.hpp:370: bool fc::thread_d::start_next_fiber(bool): Assertion `std::current_exception() == std::exception_ptr()' failed.
Aborted


[bytemaster]

The compilation from Saturday was running smoothly over the weekend. This morning I found the witness crashed:

Code: [Select]

973361ms th_a       db_with.hpp:82                ~pending_transaction ] The invalid pending transaction caused exception {"code":10,"name":"assert_exception","message":"Assert Exception","stack":[{"context":{"level":"error","file":"db_block.cpp","line":507,"method":"_apply_transaction","hostname":"","thread_name":"th_a","timestamp":"2015-09-13T23:16:13"},"format":"(skip & skip_transaction_dupe_check) || trx_idx.indices().get<by_trx_id>().find(trx_id) == trx_idx.indices().get<by_trx_id>().end(): ","data":{}},{"context":{"level":"warn","file":"db_block.cpp","line":563,"method":"_apply_transaction","hostname":"","thread_name":"th_a","timestamp":"2015-09-13T23:16:13"},"format":"","data":{"trx":{"ref_block_num":57237,"ref_block_prefix":141629248,"expiration":"2015-09-13T23:16:57","operations":[[0,{"fee":{"amount":2000000,"asset_id":"1.3.0"},"from":"1.2.63354","to":"1.2.17263","amount":{"amount":1,"asset_id":"1.3.0"},"extensions":[]}]],"extensions":[],"signatures":["206f76beb28d9a139bbb8f061d787a83aa3d0565040be653ae2330a802eafbfe0f780f4ec86c269c278eb81d6458d53ea8c0d7bfe78eadae56e124a529d6ffa6bc"]}}},{"context":{"level":"warn","file":"db_block.cpp","line":205,"method":"push_transaction","hostname":"","thread_name":"th_a","timestamp":"2015-09-13T23:16:13"},"format":"","data":{"trx":{"ref_block_num":57237,"ref_block_prefix":141629248,"expiration":"2015-09-13T23:16:57","operations":[[0,{"fee":{"amount":2000000,"asset_id":"1.3.0"},"from":"1.2.63354","to":"1.2.17263","amount":{"amount":1,"asset_id":"1.3.0"},"extensions":[]}]],"extensions":[],"signatures":["206f76beb28d9a139bbb8f061d787a83aa3d0565040be653ae2330a802eafbfe0f780f4ec86c269c278eb81d6458d53ea8c0d7bfe78eadae56e124a529d6ffa6bc"]}}}]}
973391ms th_a       application.cpp:421           handle_transaction   ] Got transaction from network
973392ms th_a       application.cpp:421           handle_transaction   ] Got transaction from network
973493ms th_a       application.cpp:421           handle_transaction   ] Got transaction from network
973558ms th_a       application.cpp:421           handle_transaction   ] Got transaction from network
973682ms th_a       application.cpp:421           handle_transaction   ] Got transaction from network
973756ms th_a       application.cpp:421           handle_transaction   ] Got transaction from network
973823ms th_a       application.cpp:421           handle_transaction   ] Got transaction from network
witness_node: /home/calabiyau/graphene/libraries/fc/src/thread/thread_d.hpp:370: bool fc::thread_d::start_next_fiber(bool): Assertion `std::current_exception() == std::exception_ptr()' failed.
Aborted (core dumped)


Woke up to this too

Code: [Select]

3169758ms th_a       db_with.hpp:82                ~pending_transaction ] The invalid pending transaction caused exception {"code":10,"name":"assert_exception","message":"Assert Exception","stack":[{"context":{"level":"error","file":"db_block.cpp","line":507,"method":"_apply_transaction","hostname":"","thread_name":"th_a","timestamp":"2015-09-14T05:52:49"},"format":"(skip & skip_transaction_dupe_check) || trx_idx.indices().get<by_trx_id>().find(trx_id) == trx_idx.indices().get<by_trx_id>().end(): ","data":{}},{"context":{"level":"warn","file":"db_block.cpp","line":563,"method":"_apply_transaction","hostname":"","thread_name":"th_a","timestamp":"2015-09-14T05:52:49"},"format":"","data":{"trx":{"ref_block_num":60494,"ref_block_prefix":2734476150,"expiration":"2015-09-14T05:56:32","operations":[[0,{"fee":{"amount":2000000,"asset_id":"1.3.0"},"from":"1.2.63354","to":"1.2.17263","amount":{"amount":1,"asset_id":"1.3.0"},"extensions":[]}]],"extensions":[],"signatures":["205c4049776969e7b3fa9ad3ee401a46b757a6b0821dcf2c0c2f4e79c9beac8bf60c37d51f490092708be2348d11228ea6ad225db2d884a96b26c5b990e6746ada"]}}},{"context":{"level":"warn","file":"db_block.cpp","line":205,"method":"push_transaction","hostname":"","thread_name":"th_a","timestamp":"2015-09-14T05:52:49"},"format":"","data":{"trx":{"ref_block_num":60494,"ref_block_prefix":2734476150,"expiration":"2015-09-14T05:56:32","operations":[[0,{"fee":{"amount":2000000,"asset_id":"1.3.0"},"from":"1.2.63354","to":"1.2.17263","amount":{"amount":1,"asset_id":"1.3.0"},"extensions":[]}]],"extensions":[],"signatures":["205c4049776969e7b3fa9ad3ee401a46b757a6b0821dcf2c0c2f4e79c9beac8bf60c37d51f490092708be2348d11228ea6ad225db2d884a96b26c5b990e6746ada"]}}}]}


This problem (if not just a warning) seems to have disappeared now.

I think I have identified the cause of this problem.  Can you confirm that it occurred while you were using the RPC interface?

[puppies]

graphene ui is working for me on commit 25b7c4db6816684ae11f731aab522630cbeda66a.  The Dashboard page wont load, and the blockexplorer eats up massive resources (enough to where it locks up firefox) while spamming though. 

[Riverhead]

Also got this when I re-cloned graphene-ui. If there's another thread for the UI I'll move this post there.

Code: [Select]

> webgui@0.0.1 start /home/james/github/graphene-ui/web
> node server.js

Using DEV options

70.00% 1/1 build modules
events.js:72
        throw er; // Unhandled 'error' event
              ^
Error: listen EADDRINUSE
    at errnoException (net.js:901:11)
    at Server._listen2 (net.js:1039:14)
    at listen (net.js:1061:10)
    at net.js:1143:9
    at dns.js:72:18
    at process._tickCallback (node.js:415:13)
    at Function.Module.runMain (module.js:499:11)
    at startup (node.js:119:16)
    at node.js:902:3

npm ERR! webgui@0.0.1 start: `node server.js`
npm ERR! Exit status 8
npm ERR!
npm ERR! Failed at the webgui@0.0.1 start script.
npm ERR! This is most likely a problem with the webgui package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node server.js
npm ERR! You can get their info via:
npm ERR!     npm owner ls webgui
npm ERR! There is likely additional logging output above.
npm ERR! System Linux 3.19.0-28-generic
npm ERR! command "/usr/bin/nodejs" "/usr/bin/npm" "start"
npm ERR! cwd /home/james/github/graphene-ui/web
npm ERR! node -v v0.10.25
npm ERR! npm -v 1.4.21
npm ERR! code ELIFECYCLE
npm WARN This failure might be due to the use of legacy binary "node"
npm WARN For further explanations, please read
/usr/share/doc/nodejs/README.Debian

npm ERR!
npm ERR! Additional logging details can be found in:
npm ERR!     /home/james/github/graphene-ui/web/npm-debug.log
npm ERR! not ok code 0
james@james-desktop:~/github/graphene-ui$


[puppies]

I woke up to this on my home machine

Code: [Select]

770ms th_a       application.cpp:422           handle_transaction   ] Got transaction from network
2492315ms th_a       application.cpp:383           handle_block         ] Got block #191949 from network
2492315ms th_a       db_block.cpp:97               push_block           ] new_block.block_num(): 191949 new_block.id(): 0002edcda9a6ead041565776fc44dabfbe2e1047
2493260ms th_a       application.cpp:683           get_blockchain_synop ] synopsis: ["0002edb0f9f9f46a5131e683d7cdf03630d0b105","0002edbffba5cc2aee85c854c9690d8640766f70","0002edc70bac5a6448cb68f234642c829cac4352","0002edcb696c768b569ade47a988f9ad8bb34344","0002edcd10fc9e1db43fd6ffbc4895169c245915"]
2493714ms th_a       application.cpp:683           get_blockchain_synop ] synopsis: ["0002edb0f9f9f46a5131e683d7cdf03630d0b105","0002edbffba5cc2aee85c854c9690d8640766f70","0002edc70bac5a6448cb68f234642c829cac4352","0002edcb696c768b569ade47a988f9ad8bb34344","0002edcd10fc9e1db43fd6ffbc4895169c245915"]
witness_node: /home/user/src/graphene/libraries/chain/db_block.cpp:82: std::vector<fc::ripemd160> graphene::chain::database::get_block_ids_on_fork(graphene::chain::block_id_type) const: Assertion `branches.first.back()->id == branches.second.back()->id' failed.

Program received signal SIGABRT, Aborted.
0x00007ffff6c01cc9 in __GI_raise (sig=sig@entry=6)
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  0x00007ffff6c01cc9 in __GI_raise (sig=sig@entry=6)
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007ffff6c050d8 in __GI_abort () at abort.c:89
#2  0x00007ffff6bfab86 in __assert_fail_base (
    fmt=0x7ffff6d4b830 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
    assertion=assertion@entry=0x2ac0c40 "branches.first.back()->id == branches.second.back()->id",
    file=file@entry=0x2ac0be0 "/home/user/src/graphene/libraries/chain/db_block.cpp", line=line@entry=82,
    function=function@entry=0x2acb220 <graphene::chain::database::get_block_ids_on_fork(fc::ripemd160) const::__PRETTY_FUNCTION__> "std::vector<fc::ripemd160> graphene::chain::database::get_block_ids_on_fork(graphene::chain::block_id_type) const") at assert.c:92
#3  0x00007ffff6bfac32 in __GI___assert_fail (
    assertion=0x2ac0c40 "branches.first.back()->id == branches.second.back()->id", file=0x2ac0be0 "/home/user/src/graphene/libraries/chain/db_block.cpp",
    line=82,
    function=0x2acb220 <graphene::chain::database::get_block_ids_on_fork(fc::ripemd160) const::__PRETTY_FUNCTION__> "std::vector<fc::ripemd160> graphene::chain::database::get_block_ids_on_fork(graphene::chain::block_id_type) const")
    at assert.c:101
#4  0x0000000002302824 in graphene::chain::database::get_block_ids_on_fork(fc::ripemd160) const ()
---Type <return> to continue, or q <return> to quit---
#5  0x0000000001f99e28 in graphene::app::detail::application_impl::get_blockchain_synopsis(fc::ripemd160 const&, unsigned int) ()
#6  0x000000000281686f in graphene::net::detail::statistics_gathering_node_delegate_wrapper::get_blockchain_synopsis(fc::ripemd160 const&, unsigned int)::{lambda()#1}::operator()() const ()
#7  0x0000000002826910 in fc::detail::functor_run<graphene::net::detail::statistics_gathering_node_delegate_wrapper::get_blockchain_synopsis(fc::ripemd160 const&, unsigned int)::{lambda()#1}>::run(void*, fc::detail::functor_run<graphene::net::detail::statistics_gathering_node_delegate_wrapper::get_blockchain_synopsis(fc::ripemd160 const&, unsigned int)::{lambda()#1}>) ()
#8  0x00000000025f2011 in fc::task_base::run_impl() ()
#9  0x00000000025f1fa2 in fc::task_base::run() ()
#10 0x00000000025e69c2 in fc::thread_d::run_next_task() ()
#11 0x00000000025e6e66 in fc::thread_d::process_tasks() ()
#12 0x00000000025e64d9 in fc::thread_d::start_process_tasks(long) ()
#13 0x0000000002943a81 in make_fcontext ()
#14 0x0000000000000000 in ?? ()
(gdb)
this is running on commit 30ae8e4f3433d4ee500f0b98e30f05e1ebe806ea. 

[Riverhead]

I started getting this about six hours ago. I switched back to "Master" which had a pile of updates and am rebuilding my witness binary. We'll see if that fixes it.

Code: [Select]

2842000ms th_a       witness.cpp:220               block_production_loo ] Not producing block because it isn't my turn
2842001ms th_a       witness.cpp:187               schedule_production_ ] now.time_since_epoch().count(): 1442231242001571 next_second: 2015-09-14T11:47:23
2842014ms th_a       application.cpp:356           handle_block         ] Got block #194679 from network
2842015ms th_a       application.cpp:378           handle_block         ] Error when pushing block:
10 assert_exception: Assert Exception
item->num < _head->num + MAX_BLOCK_REORDERING:
    {}
    th_a  fork_database.cpp:70 _push_block

    {"new_block":{"previous":"0002f876121cf55147d14b4c30a8aaf0877a38eb","timestamp":"2015-09-14T11:47:20","witness":"1.6.48","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"1f5b899e8a4e1a406629a6a98d1dc2f463bae6b60517435ffe68825184d0680f240a7ed917a916c5e1475f05191d67536f74f78bf76e5323537f0990eebd0adc21","transactions":[]}}
    th_a  db_block.cpp:176 _push_block
2843000ms th_a       witness.cpp:220               block_production_loo ] Not producing block because it isn't my turn
2843001ms th_a       witness.cpp:187               schedule_production_ ] now.time_since_epoch().count(): 1442231243001656 next_second: 2015-09-14T11:47:24
2843787ms th_a       main.cpp:169                  main                 ] Exiting from signal 2
2843798ms th_a       thread.cpp:115                ~thread              ] calling quit() on ntp
2843799ms th_a       thread.cpp:160                quit                 ] destroying boost thread 139689216190208
2843802ms ntp        thread.cpp:246                exec                 ] thread canceled: 9 canceled_exception: Canceled
cancellation reason: [none given]
    {"reason":"[none given]"}
    ntp  thread_d.hpp:463 start_next_fiber
2843836ms th_a       thread.cpp:115                ~thread              ] calling quit() on p2p
2843836ms th_a       thread.cpp:160                quit                 ] destroying boost thread 139689180436224
2843840ms p2p        thread.cpp:246                exec                 ] thread canceled: 9 canceled_exception: Canceled
cancellation reason: [none given]
    {"reason":"[none given]"}