cli_wallet keeps crashing

[roadscape]

Updated to the 2.0.160316b release, been running it with --enable-permessage-deflate for 2 days, still can't get it to crash.

So whatever the issue was, it seems to have been resolved.

[roadscape]

I couldn't reproduce the issue on the old revision with a fresh checkout/make. Too many variables changed anyway so I'm just going to test using head on bitshares branch.

I finally caught a cli_wallet crash!

Code: [Select]

new >>> Server has disconnected us.
9 canceled_exception: Canceled

    {}
    th_a  thread_d.hpp:461 start_next_fiber
192877ms th_a       wallet.cpp:787                save_wallet_file     ] saving wallet to file wallet.json
193528ms th_a       http_api.cpp:118              on_request           ] e.to_detail_string(): 9 canceled_exception: Canceled

    {}
    th_a  thread_d.hpp:461 start_next_fiber
193619ms th_a       http_api.cpp:118              on_request           ] e.to_detail_string(): 9 canceled_exception: Canceled

    {}
    th_a  thread_d.hpp:461 start_next_fiber
pure virtual method called
terminate called without an active exception

Program received signal SIGABRT, Aborted.
0x00007ffff67a2cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56      ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  0x00007ffff67a2cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007ffff67a60d8 in __GI_abort () at abort.c:89
#2  0x00007ffff70ad535 in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x00007ffff70ab6d6 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007ffff70ab703 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007ffff70ac1bf in __cxa_pure_virtual () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x0000000000b27f87 in fc::rpc::websocket_api_connection::send_call(unsigned int, std::string, std::vector<fc::variant, std::allocator<fc::variant> >) ()
#7  0x00000000008d0a51 in void fc::api_connection::api_visitor::operator()<>(char const*, std::function<void ()>&) const::{lambda()#1}::operator()() const ()
#8  0x0000000000a38671 in graphene::wallet::detail::wallet_api_impl::~wallet_api_impl() ()
#9  0x0000000000a387a9 in graphene::wallet::detail::wallet_api_impl::~wallet_api_impl() ()
#10 0x00000000008af3b9 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() ()
#11 0x00000000009dc199 in graphene::wallet::wallet_api::~wallet_api() ()
#12 0x00000000008af3b9 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() ()
#13 0x00000000008af4f9 in boost::any::holder<std::shared_ptr<graphene::wallet::wallet_api> >::~holder() ()
#14 0x00000000008af3b9 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() ()
#15 0x00000000008af442 in fc::api<graphene::wallet::wallet_api, fc::identity_member>::~api() ()
#16 0x00000000008af494 in boost::any::holder<fc::api<graphene::wallet::wallet_api, fc::identity_member> >::~holder() ()
#17 0x00000000008977de in std::default_delete<fc::generic_api>::operator()(fc::generic_api*) const [clone .isra.726] ()
#18 0x00000000008c0745 in std::vector<std::unique_ptr<fc::generic_api, std::default_delete<fc::generic_api> >, std::allocator<std::unique_ptr<fc::generic_api, std::default_delete<fc::generic_api> > > >::~vector() ()
#19 0x00000000008c07aa in fc::api_connection::~api_connection() ()
#20 0x0000000000b20a36 in fc::rpc::cli::~cli() ()
#21 0x00000000008af3b9 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() ()
#22 0x00000000008b0e4a in boost::function0<void>::clear() ()
#23 0x00000000008b5a3d in boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (), boost::function<void ()> >, boost::signals2::mutex>::~connection_body() ()
#24 0x00000000008b5a69 in boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (), boost::function<void ()> >, boost::signals2::mutex>::~connection_body() ()
#25 0x00000000008ad5ee in boost::detail::sp_counted_base::release() ()
#26 0x00000000008b6e7c in std::_List_base<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (), boost::function<void ()> >, boost::signals2::mutex> >, std::allocator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (), boost::function<void ()> >, boost::signals2::mutex> > > >::_M_clear() ()
#27 0x00000000008b6f1f in boost::detail::sp_counted_impl_p<boost::signals2::detail::grouped_list<int, std::less<int>, boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (), boost::function<void ()> >, boost::signals2::mutex> > > >::dispose() ()
#28 0x00000000008ad5ee in boost::detail::sp_counted_base::release() ()
#29 0x00000000008ad716 in boost::detail::sp_counted_impl_p<boost::signals2::detail::signal_impl<void (), boost::signals2::optional_last_value<void>, int, std::less<int>, boost::function<void ()>, boost::function<void (boost::signals2::connection const&)>, boost::signals2::mutex>::invocation_state>::dispose() ()
#30 0x00000000008ad5ee in boost::detail::sp_counted_base::release() ()
#31 0x00000000008ad691 in boost::detail::sp_counted_impl_p<boost::signals2::detail::signal_impl<void (), boost::signals2::optional_last_value<void>, int, std::less<int>, boost::function<void ()>, boost::function<void (boost::signals2::connection const&)>, boost::signals2::mutex> >::dispose() ()
#32 0x00000000008ad5ee in boost::detail::sp_counted_base::release() ()
---Type <return> to continue, or q <return> to quit---
#33 0x0000000000b6465e in fc::http::websocket_connection::~websocket_connection() ()
#34 0x00000000008af3b9 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() ()
#35 0x0000000000892352 in main ()
witness_node still running fine. Is it worth pursuing this further or should I switch to testing with the latest?

[roadscape]

Also the websocketpp deflate code is very new, so it might be a novel bug in websocketpp.  A quick run-through of the websocketpp issue tracker shows some known race condition crash bugs in websocketpp [1] [2] [3] and it is not out of the realm of possibility that one of these leads to a use-after-free.  Some of these bugs are TLS only, so it is possible that disabling TLS fixes the issue.

[1] https://github.com/zaphoyd/websocketpp/issues/459
[2] https://github.com/zaphoyd/websocketpp/issues/451
[3] https://github.com/zaphoyd/websocketpp/issues/336

I am using --rpc-endpoint and not --rpc-tls-endpoint, does this still apply?

If it's helpful I can try to recreate this setup on a separate server today or tomorrow.

We've pushed a changed to bitshares branch of bitshares2 that allows you to disable the compression via command line, if you want to test if it is related to the problem. The option to disable compression is:
option is:  --disable-permessage-deflate

Excellent, thanks, I will test this when I get the new environment up. First I will try to reproduce the error using the same revision I had issues with, then I will update and check the result.

[dannotestein]

Also the websocketpp deflate code is very new, so it might be a novel bug in websocketpp.  A quick run-through of the websocketpp issue tracker shows some known race condition crash bugs in websocketpp [1] [2] [3] and it is not out of the realm of possibility that one of these leads to a use-after-free.  Some of these bugs are TLS only, so it is possible that disabling TLS fixes the issue.

[1] https://github.com/zaphoyd/websocketpp/issues/459
[2] https://github.com/zaphoyd/websocketpp/issues/451
[3] https://github.com/zaphoyd/websocketpp/issues/336

I am using --rpc-endpoint and not --rpc-tls-endpoint, does this still apply?

If it's helpful I can try to recreate this setup on a separate server today or tomorrow.

We've pushed a changed to bitshares branch of bitshares2 that allows you to disable the compression via command line, if you want to test if it is related to the problem. The option to disable compression is:
option is:  --disable-permessage-deflate

Can this be disabled by default?

naturally, but it would only make sense to do that if we discover there is a potential problem.

[abit]

Also the websocketpp deflate code is very new, so it might be a novel bug in websocketpp.  A quick run-through of the websocketpp issue tracker shows some known race condition crash bugs in websocketpp [1] [2] [3] and it is not out of the realm of possibility that one of these leads to a use-after-free.  Some of these bugs are TLS only, so it is possible that disabling TLS fixes the issue.

[1] https://github.com/zaphoyd/websocketpp/issues/459
[2] https://github.com/zaphoyd/websocketpp/issues/451
[3] https://github.com/zaphoyd/websocketpp/issues/336

I am using --rpc-endpoint and not --rpc-tls-endpoint, does this still apply?

If it's helpful I can try to recreate this setup on a separate server today or tomorrow.

We've pushed a changed to bitshares branch of bitshares2 that allows you to disable the compression via command line, if you want to test if it is related to the problem. The option to disable compression is:
option is:  --disable-permessage-deflate

Can this be disabled by default?

[dannotestein]

Also the websocketpp deflate code is very new, so it might be a novel bug in websocketpp.  A quick run-through of the websocketpp issue tracker shows some known race condition crash bugs in websocketpp [1] [2] [3] and it is not out of the realm of possibility that one of these leads to a use-after-free.  Some of these bugs are TLS only, so it is possible that disabling TLS fixes the issue.

[1] https://github.com/zaphoyd/websocketpp/issues/459
[2] https://github.com/zaphoyd/websocketpp/issues/451
[3] https://github.com/zaphoyd/websocketpp/issues/336

I am using --rpc-endpoint and not --rpc-tls-endpoint, does this still apply?

If it's helpful I can try to recreate this setup on a separate server today or tomorrow.

We've pushed a changed to bitshares branch of bitshares2 that allows you to disable the compression via command line, if you want to test if it is related to the problem. The option to disable compression is:
option is:  --disable-permessage-deflate

[roadscape]

Also the websocketpp deflate code is very new, so it might be a novel bug in websocketpp.  A quick run-through of the websocketpp issue tracker shows some known race condition crash bugs in websocketpp [1] [2] [3] and it is not out of the realm of possibility that one of these leads to a use-after-free.  Some of these bugs are TLS only, so it is possible that disabling TLS fixes the issue.

[1] https://github.com/zaphoyd/websocketpp/issues/459
[2] https://github.com/zaphoyd/websocketpp/issues/451
[3] https://github.com/zaphoyd/websocketpp/issues/336

I am using --rpc-endpoint and not --rpc-tls-endpoint, does this still apply?

If it's helpful I can try to recreate this setup on a separate server today or tomorrow.

[roadscape]

Possible memory corruption?

If you're running head instead of the last release, there are three changes in FC which might be the cause of the problem:

• websockets now compress data
• websocketpp dependency has been updated
• Use-after-free bug was fixed in future.cpp

If you could run this command:

Code: [Select]

(cd libraries/fc/vendor/websocketpp && git log -1)

it would be useful.  I'm thinking it sounds like you upgraded an existing source checkout, so it might be possible you updated submodules non-recursively, meaning you'd get the new FC code but older websocketpp code, and maybe the new FC code requires more recent websocketpp (I'm just guessing here).

 - I was running head last night, but now I'm on 2.0.160223 (seems to be running much better)
 - My build script always calls git submodule update --init --recursive
 - I'm still using the "existing source checkout" -- I haven't cloned a fresh repo in months
 - Can it be memory corruption even though I restarted both node+wallet many times? (imho it seems more likely the mangled messages are due to compression somehow)

I'm guessing this is not useful now that I've already checked out an older tag, but here's the output:

Code: [Select]

cd libraries/fc/vendor/websocketpp && git log -1
commit c5510d6de04917812b910a8dd44735c1f17061d9
Merge: 13f6da6 f931734
Author: Peter Thorson <git@zaphoyd.com>
Date:   Tue Jun 2 08:55:24 2015 -0400
Could it be a problem with the websocket client I'm using? Perhaps its compression implementation is buggy or something?

[theoretical]

Also the websocketpp deflate code is very new, so it might be a novel bug in websocketpp.  A quick run-through of the websocketpp issue tracker shows some known race condition crash bugs in websocketpp [1] [2] [3] and it is not out of the realm of possibility that one of these leads to a use-after-free.  Some of these bugs are TLS only, so it is possible that disabling TLS fixes the issue.

[1] https://github.com/zaphoyd/websocketpp/issues/459
[2] https://github.com/zaphoyd/websocketpp/issues/451
[3] https://github.com/zaphoyd/websocketpp/issues/336

[cube]

Possible memory corruption?

it would be useful.  I'm thinking it sounds like you upgraded an existing source checkout, so it might be possible you updated submodules non-recursively, meaning you'd get the new FC code but older websocketpp code, and maybe the new FC code requires more recent websocketpp (I'm just guessing here).

Can cmake check for the right modules' version for the dependencies?

[theoretical]

Possible memory corruption?

If you're running head instead of the last release, there are three changes in FC which might be the cause of the problem:

• websockets now compress data
• websocketpp dependency has been updated
• Use-after-free bug was fixed in future.cpp

If you could run this command:

Code: [Select]

(cd libraries/fc/vendor/websocketpp && git log -1)

it would be useful.  I'm thinking it sounds like you upgraded an existing source checkout, so it might be possible you updated submodules non-recursively, meaning you'd get the new FC code but older websocketpp code, and maybe the new FC code requires more recent websocketpp (I'm just guessing here).

[roadscape]

I performed a clean build using 2.0.160223 tag.

Wallet is now running in gdb and I'll keep an eye on it..

[roadscape]

Building @ 36%...

When it's done I'll try running the cli_wallet in gdb and report back with any findings

[bytemaster]

interesting errors.  If we can get a backtrace of the crash that would be helpful.

[roadscape]

xeroc suggested 'try with a plain and empty new wallet by providing the --wallet-file newwallet.json parameter'

I got more stange errors and an unresponsive cli_wallet:

Code: [Select]

    {"str":"{\"methoen_bnotice018params_q[0,[\"0040e76895dbba1c116982218286d077c82062c5\"]]}"}
    th_a  json.cpp:478 from_string
1446545ms th_a       websocket_api.cpp:120         on_message           ] e.to_detail_string(): 4 parse_error_exception: Parse Error
Expected ':' after key "methoen_bnotice018params_q[0,["
    {"key":"methoen_bnotice018params_q[0,["}
    th_a  json.cpp:196 objectFromStream
Error parsing object
    {}
    th_a  json.cpp:218 objectFromStream

    {"str":"{\"methoen_bnotice018params_q[0,[\"0040e769851a9f6ec4457a6a6ea69492772f400b\"]]}"}
    th_a  json.cpp:478 from_string
1449231ms th_a       websocket_api.cpp:120         on_message           ] e.to_detail_string(): 4 parse_error_exception: Parse Error
Expected ':' after key "methoen_bnotice018params_q[0,["
    {"key":"methoen_bnotice018params_q[0,["}
    th_a  json.cpp:196 objectFromStream
Error parsing object
    {}
    th_a  json.cpp:218 objectFromStream

    {"str":"{\"methoen_bnotice018params_q[0,[\"0040e76a38b276e387222c44cf7cc5fd3e213042\"]]}"}
    th_a  json.cpp:478 from_string
1452511ms th_a       websocket_api.cpp:120         on_message           ] e.to_detail_string(): 4 parse_error_exception: Parse Error
Expected ':' after key "methoen_bnotice018params_q[0,["
    {"key":"methoen_bnotice018params_q[0,["}
    th_a  json.cpp:196 objectFromStream
Error parsing object
    {}
    th_a  json.cpp:218 objectFromStream
This looks horribly malformed.. but what would cause this to happen intermittently?

Next I'm trying:
 - make clean and clearing cmake files, rebuilding using tag 2.0.160223
 - connecting to a different ws server (we still can't tell if this is a node or wallet issue)