0 Members and 1 Guest are viewing this topic.
I have job interview in a bit, I just saw we need to upgrade. I will do so ASAP when I get home.Just FYI in case I can't make it.
0.4.20 seems to hold up fine. Great job on the quick fix, thanks! I will be able to sleep tight tonight
Quote from: bytemaster on October 02, 2014, 02:19:00 pmIt is not a memory issue, we have fixed the crash and notified DAC Sun.Is it wise to upgrade already to master or should we wait for an announcement from DAC Sun?
It is not a memory issue, we have fixed the crash and notified DAC Sun.
Quote from: emski on October 02, 2014, 02:35:48 pmDo we expect increased CPU time consumption?I see some higher values.We are auditing all resource usage right now.
Do we expect increased CPU time consumption?I see some higher values.
Quote from: crazybit on October 02, 2014, 02:19:39 pmmy delegate server is under real time monitoring, the RAM did reached to 80% yet before the client crashed,it may not due to the RAM size.what software do you use for this? Couldn't get started monitoring my delegate machine(s) in more details
my delegate server is under real time monitoring, the RAM did reached to 80% yet before the client crashed,it may not due to the RAM size.
UPDATE: Increased memory consumption might be related to DB reindexing. If you restart the client after reindexing is completed memory consumption goes back to normal
Because downgrading is so unpleasant and we want the blockchain updates asap Dan & Eric are looking into the crash, but those that are experiencing it could you please report the specs of the machine you are running on?Lets update to 0.4.19...
Quote from: bytemaster on October 02, 2014, 02:07:15 pm" ...and delegates can run machines with 128 GB of memory." 128 GB ?Sent from my ALCATEL ONE TOUCH 997D
" ...and delegates can run machines with 128 GB of memory."
# free total used free shared buffers cachedMem: 4048312 3068232 980080 332 152304 2587972-/+ buffers/cache: 327956 3720356
experienced segfault4GB RAM
Lets update to 0.4.19...
hmmm .. 2GB is already low memory .. oha
Quote from: emski on October 02, 2014, 01:28:09 pmUpdating to v0.4.19 seems reasonable to me. Segfault by itself isn't that much of a deal.However bytemaster recommended not to update. Maybe he has other reasons?There is still plenty of time for an informed decision. I suggest waiting for bytemaster's input and update to v0.4.19 as default option (if he is silent about this).I agree. everyone downgrading would be very unpleasant.
Updating to v0.4.19 seems reasonable to me. Segfault by itself isn't that much of a deal.However bytemaster recommended not to update. Maybe he has other reasons?There is still plenty of time for an informed decision. I suggest waiting for bytemaster's input and update to v0.4.19 as default option (if he is silent about this).
$ python3 timeofblock.py 640000 block 640000 to appear in <= 6:02:20UTC time: 2014-10-02 19:36:10
#!/usr/bin/expect -fset timeout -1set default_port 1776set port $default_port### change wallet_name hereset wallet_name "delegate"send_user "wallet name is: $wallet_name\n"send_user "wallet passphrase: "stty -echoexpect_user -re "(.*)\n" stty echoset wallet_pass $expect_out(1,string) proc run_wallet {} { global wallet_name wallet_pass default_port port ### change command line here spawn ./bitshares_client --data-dir=delegate --p2p-port $port --server --httpport 9989 --rpcuser user --rpcpassword pass expect -exact "(wallet closed) >>> " send -- "about\r" expect -exact "(wallet closed) >>> " send -- "wallet_open $wallet_name\r" expect -exact "$wallet_name (locked) >>> " send -- "wallet_unlock 99999999\r" expect -exact "passphrase: " send -- "$wallet_pass\r" expect -exact "$wallet_name (unlocked) >>> " send -- "wallet_delegate_set_block_production ALL true\r" expect -exact "$wallet_name (unlocked) >>> " send -- "info\r" expect -exact "$wallet_name (unlocked) >>> " send -- "wallet_list_my_accounts\r" interact wait if { $port == $default_port } { set port [expr $port+1] } else { set port [expr $port-1] } }while true { run_wallet}
Quote from: bytemaster on October 02, 2014, 12:44:03 pmDon't upgrade to .19. Clearly it has issues. any advice on how to downgrade smoothly? Should we just relaunch 0.4.18 on an empty DB and reimport keys? I imagine that as everything is already re-indexed for 0.4.19 there is no way to downgrade this to 0.4.18 directly without wiping some indices first, right?
Don't upgrade to .19. Clearly it has issues.
Quote from: GaltReport on October 02, 2014, 01:07:32 pmQuote from: emski on October 02, 2014, 01:06:20 pmQuote from: GaltReport on October 02, 2014, 01:02:27 pmQuote from: wackou on October 02, 2014, 12:59:50 pmQuote from: bytemaster on October 02, 2014, 12:44:03 pmDon't upgrade to .19. Clearly it has issues. any advice on how to downgrade smoothly? Should we just relaunch 0.4.18 on an empty DB and reimport keys? I imagine that as everything is already re-indexed for 0.4.19 there is no way to downgrade this to 0.4.18 directly without wiping some indices first, right?I would suggest that if it is working for you, you should not have to downgrade but hopefully BM will weigh in.All delegates should be on the same version. Unless we want forks.Can't they delay planned forks ? Not sure if that is what you are referring to or not. I only notice that when we do upgrades their are days when people are on different versions.Forks are hardcoded in the code for block number. Current v0.4.19 fork is scheduled for block 640000. All v0.4.19 will fork at that block. Any previous versions might not accept blocks produced by v0.4.19. That is why either all (most) of the delegates upgrade by block 640000 or none (very few) of them upgrade. Note that v0.4.19-RC1 is different version -> it should not accept blocks by both v0.4.19 and v0.4.18.
Quote from: emski on October 02, 2014, 01:06:20 pmQuote from: GaltReport on October 02, 2014, 01:02:27 pmQuote from: wackou on October 02, 2014, 12:59:50 pmQuote from: bytemaster on October 02, 2014, 12:44:03 pmDon't upgrade to .19. Clearly it has issues. any advice on how to downgrade smoothly? Should we just relaunch 0.4.18 on an empty DB and reimport keys? I imagine that as everything is already re-indexed for 0.4.19 there is no way to downgrade this to 0.4.18 directly without wiping some indices first, right?I would suggest that if it is working for you, you should not have to downgrade but hopefully BM will weigh in.All delegates should be on the same version. Unless we want forks.Can't they delay planned forks ? Not sure if that is what you are referring to or not. I only notice that when we do upgrades their are days when people are on different versions.
Quote from: GaltReport on October 02, 2014, 01:02:27 pmQuote from: wackou on October 02, 2014, 12:59:50 pmQuote from: bytemaster on October 02, 2014, 12:44:03 pmDon't upgrade to .19. Clearly it has issues. any advice on how to downgrade smoothly? Should we just relaunch 0.4.18 on an empty DB and reimport keys? I imagine that as everything is already re-indexed for 0.4.19 there is no way to downgrade this to 0.4.18 directly without wiping some indices first, right?I would suggest that if it is working for you, you should not have to downgrade but hopefully BM will weigh in.All delegates should be on the same version. Unless we want forks.
Quote from: wackou on October 02, 2014, 12:59:50 pmQuote from: bytemaster on October 02, 2014, 12:44:03 pmDon't upgrade to .19. Clearly it has issues. any advice on how to downgrade smoothly? Should we just relaunch 0.4.18 on an empty DB and reimport keys? I imagine that as everything is already re-indexed for 0.4.19 there is no way to downgrade this to 0.4.18 directly without wiping some indices first, right?I would suggest that if it is working for you, you should not have to downgrade but hopefully BM will weigh in.
I would suggest that if it is working for you, you should not have to downgrade but hopefully BM will weigh in.
Quote from: bytemaster on October 02, 2014, 12:44:03 pmDon't upgrade to .19. Clearly it has issues. What if you already did and it's working?
network_set_advanced_node_parameters { "peer_connection_retry_timeout": 10, "desired_number_of_connections": 10, "maximum_number_of_connections": 10 }
(wallet closed) >>>Program received signal SIGSEGV, Segmentation fault.[Switching to Thread 0x7fffb3fff700 (LWP 41682)]0x0000000000a55777 in std::_Hashtable<bts::net::item_id, bts::net::item_id, std::allocator<bts::net::item_id>, std::__detail::_Identity, std::equal_to<bts::net::item_id>, std::hash<bts::net::item_id>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, true, true> >::_M_find_before_node(unsigned long, bts::net::item_id const&, unsigned long) const () at /usr/include/c++/4.8/bits/hashtable.h:11591159 __node_base* __prev_p = _M_buckets[__n];(gdb) bt#0 0x0000000000a55777 in std::_Hashtable<bts::net::item_id, bts::net::item_id, std::allocator<bts::net::item_id>, std::__detail::_Identity, std::equal_to<bts::net::item_id>, std::hash<bts::net::item_id>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, true, true> >::_M_find_before_node(unsigned long, bts::net::item_id const&, unsigned long) const () at /usr/include/c++/4.8/bits/hashtable.h:1159#1 0x0000000000a55840 in std::_Hashtable<bts::net::item_id, bts::net::item_id, std::allocator<bts::net::item_id>, std::__detail::_Identity, std::equal_to<bts::net::item_id>, std::hash<bts::net::item_id>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, true, true> >::find(bts::net::item_id const&) () at /usr/include/c++/4.8/bits/hashtable.h:604#2 0x0000000000a3dbc3 in bts::net::detail::node_impl::process_block_during_normal_operation(bts::net::peer_connection*, bts::client::block_message const&, fc::ripemd160 const&) () at /usr/include/c++/4.8/bits/unordered_set.h:517#3 0x0000000000a3f48b in bts::net::detail::node_impl::process_block_message(bts::net::peer_connection*, bts::net::message const&, fc::ripemd160 const&) () at /home/fabiux/ssd/bitsharesx/libraries/net/node.cpp:2880#4 0x0000000000a40bd3 in bts::net::detail::node_impl::on_message(bts::net::peer_connection*, bts::net::message const&) () at /home/fabiux/ssd/bitsharesx/libraries/net/node.cpp:1598#5 0x0000000000aaef3a in bts::net::detail::message_oriented_connection_impl::read_loop() () at /home/fabiux/ssd/bitsharesx/libraries/net/message_oriented_connection.cpp:157#6 0x0000000000ab130c in fc::detail::void_functor_run<bts::net::detail::message_oriented_connection_impl::accept()::{lambda()#1}>::run(void*, fc::detail::void_functor_run<bts::net::detail::message_oriented_connection_impl::accept()::{lambda()#1}>) () at /home/fabiux/ssd/bitsharesx/libraries/net/message_oriented_connection.cpp:100#7 0x00000000006adc53 in fc::task_base::run_impl() () at /home/fabiux/ssd/bitsharesx/libraries/fc/src/thread/task.cpp:43#8 0x00000000006ac47b in fc::thread_d::process_tasks() () at /home/fabiux/ssd/bitsharesx/libraries/fc/src/thread/thread_d.hpp:415#9 0x00000000006ac716 in fc::thread_d::start_process_tasks(long) () at /home/fabiux/ssd/bitsharesx/libraries/fc/src/thread/thread_d.hpp:395#10 0x0000000000f1f9ce in make_fcontext ()#11 0x00007fffa80008c0 in ?? ()#12 0x00007fff8d02a530 in ?? ()#13 0x0000000000000000 in ?? ()(gdb)
Quote from: svk on October 02, 2014, 10:07:44 amI've reduced my number of max connections to 10, running OK so far but monitoring it..How to do?
I've reduced my number of max connections to 10, running OK so far but monitoring it..
Should we panic?
>>>Program received signal SIGSEGV, Segmentation fault.[Switching to Thread 0x7fff87fff700 (LWP 4915)]std::_Hashtable<bts::net::item_id, bts::net::item_id, std::allocator<bts::net::item_id>, std::__detail::_Identity, std::equal_to<bts::net::item_id>, std::hash<bts::net::item_id>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, true, true> >::find ( this=this@entry=0x45944, __k=...) at /usr/include/c++/4.8/bits/hashtable.h:10241024 std::size_t __n = _M_bucket_index(__k, __code);(gdb) bt #0 std::_Hashtable<bts::net::item_id, bts::net::item_id, std::allocator<bts::net::item_id>, std::__detail::_Identity, std::equal_to<bts::net::item_id>, std::hash<bts::net::item_id>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, true, true> >::find ( this=this@entry=0x45944, __k=...) at /usr/include/c++/4.8/bits/hashtable.h:1024#1 0x0000000000a4ede5 in find (__x=..., this=0x45944) at /usr/include/c++/4.8/bits/unordered_set.h:517#2 bts::net::detail::node_impl::process_block_during_normal_operation (this=this@entry=0x5a7c2920, originating_peer=originating_peer@entry=0x7fff659dd210, block_message_to_process=..., message_hash=...) at /root/bitsharesx/libraries/net/node.cpp:2828#3 0x0000000000a5068b in bts::net::detail::node_impl::process_block_message (this=this@entry=0x5a7c2920, originating_peer=originating_peer@entry=0x7fff659dd210, message_to_process=..., message_hash=...) at /root/bitsharesx/libraries/net/node.cpp:2880#4 0x0000000000a51d03 in bts::net::detail::node_impl::on_message (this=0x5a7c2920, originating_peer=0x7fff659dd210, received_message=...) at /root/bitsharesx/libraries/net/node.cpp:1598#5 0x0000000000ac034a in bts::net::detail::message_oriented_connection_impl::read_loop (this=0x7fff65a19490) at /root/bitsharesx/libraries/net/message_oriented_connection.cpp:157#6 0x0000000000ac271c in operator() (__closure=<optimized out>) at /root/bitsharesx/libraries/net/message_oriented_connection.cpp:100#7 fc::detail::void_functor_run<bts::net::detail::message_oriented_connection_impl::accept()::__lambda0>::run(void *, void *) (functor=<optimized out>, prom=0x7fff65ab2120) at /root/bitsharesx/libraries/fc/include/fc/thread/task.hpp:83#8 0x00000000006bf323 in fc::task_base::run_impl (this=this@entry=0x7fff65ab2130) at /root/bitsharesx/libraries/fc/src/thread/task.cpp:43#9 0x00000000006bf9d5 in fc::task_base::run (this=this@entry=0x7fff65ab2130) at /root/bitsharesx/libraries/fc/src/thread/task.cpp:32#10 0x00000000006bd95b in run_next_task (this=0x7fff7c0008c0) at /root/bitsharesx/libraries/fc/src/thread/thread_d.hpp:415#11 fc::thread_d::process_tasks (this=this@entry=0x7fff7c0008c0) at /root/bitsharesx/libraries/fc/src/thread/thread_d.hpp:439#12 0x00000000006bdbe6 in fc::thread_d::start_process_tasks (my=140735273765056) at /root/bitsharesx/libraries/fc/src/thread/thread_d.hpp:395#13 0x0000000000f4628e in make_fcontext ()#14 0x00007fff7c0008c0 in ?? ()#15 0x00007fff7c068be0 in ?? ()#16 0x0000000000000000 in ?? ()