BitShares Forum

Main => General Discussion => Topic started by: freedom on May 10, 2016, 07:22:27 am

Title: BTS seed node is decling while STEEM seed node is rising
Post by: freedom on May 10, 2016, 07:22:27 am
All the developers moved to steem?

(http://i3.buimg.com/16863646530da531.png)
(http://i3.buimg.com/ec7641c06b43d72a.png)
Title: Re: BTS seed node is decling while STEEM seed node is rising
Post by: sudo on May 10, 2016, 07:36:22 am
 :'( :'( :'( :'( :'(
Title: Re: BTS seed node is decling while STEEM seed node is rising
Post by: ebit on May 10, 2016, 08:36:36 am
 :'( :'( :'( :'( :'(
Title: Re: BTS seed node is decling while STEEM seed node is rising
Post by: xeroc on May 10, 2016, 08:58:48 am
All the developers moved to steem?
you better follow the discussion and notice that there is a bug currently searched for in both, the steem backend node and bitshares that makes the nodes crash at some point ..
it just hasn't happend to the small steem network yet ..

//edit: BTS nodes are not run by the developers but by individuals!
Title: Re: BTS seed node is decling while STEEM seed node is rising
Post by: btswildpig on May 10, 2016, 09:06:38 am
All the developers moved to steem?
you better follow the discussion and notice that there is a bug currently searched for in both, the steem backend node and bitshares that makes the nodes crash at some point ..
it just hasn't happend to the small steem network yet ..

//edit: BTS nodes are not run by the developers but by individuals!

will running multiple instance of STEEM,MUSE,BTS on a same VPS cause that bug ?
Title: Re: BTS seed node is decling while STEEM seed node is rising
Post by: xeroc on May 10, 2016, 09:45:59 am
will running multiple instance of STEEM,MUSE,BTS on a same VPS cause that bug ?
only if you are using the same port and have too little RAM
Title: Re: BTS seed node is decling while STEEM seed node is rising
Post by: Thom on May 10, 2016, 11:50:27 am
will running multiple instance of STEEM,MUSE,BTS on a same VPS cause that bug ?
only if you are using the same port and have too little RAM

What do you base that on xeroc? Sounds like you've spent some time troubleshooting this problem, is that correct?

Actually from recent experience and observation the one factor that triggers this bug the most is request volume. Before I changed the port I was averaging around 50 connections pretty consistently on all of my nodes. Now it's up to 80 - 90. All but one of my servers are running low RAM, but even with the increased connections RAM use has remained flat without an increase.

Also, while switching all the nodes to use p2p port 1776 this weekend I had several seeds get stuck for some unknown reason. They've been solidly online for 2 days now, but nothing has changed so I suspect future failures will occur again.

I added abit's simple bash script to my list of monit tasks so I'll be notified if any of the nodes get stuck. Wackou's seed monitor is also useful for anyone that wants to see the status of all seed nodes. I haven't upgraded my nodes to his latest release of bts_tools yet, but I will soon.
Title: Re: BTS seed node is decling while STEEM seed node is rising
Post by: xeroc on May 10, 2016, 12:10:36 pm
will running multiple instance of STEEM,MUSE,BTS on a same VPS cause that bug ?
only if you are using the same port and have too little RAM

What do you base that on xeroc? Sounds like you've spent some time troubleshooting this problem, is that correct?

Well, statement number one: "same ports" is actually quite obvious
because you can only connect to either of the seed nodes and not all of
them. The second statement is more like an estimate. If you run a
BitShares seed/full node you need some RAM. Some more RAM for MUSE and
yet more RAM for STEEM.
If your machine has 32gigs of RAM that should do the trick, otherwise
you will end up with an unresponsive machine.

To clearify, I do not run a public seed node but multiple full nodes on
several machines

Quote
Actually from recent experience and observation the one factor that triggers this bug the most is request volume. Before I changed the port I was averaging around 50 connections pretty consistently on all of my nodes. Now it's up to 80 - 90. All but one of my servers are running low RAM, but even with the increased connections RAM use has remained flat without an increase.
RAM depends in fact on the use of BitShares. For instance open orders
are kept in RAM. Once the load in the DEX picks up, so will the
requirments for RAM.

Quote
Also, while switching all the nodes to use p2p port 1776 this weekend I had several seeds get stuck for some unknown reason. They've been solidly online for 2 days now, but nothing has changed so I suspect future failures will occur again.
My guess is that there is somethign weird happending after a few weeks
of runtime. And we haven't seen this kind of behavior in STEEM because
their have been plenty of hardforks and restarts.

Quote
I added abit's simple bash script to my list of monit tasks so I'll be notified if any of the nodes get stuck. Wackou's seed monitor is also useful for anyone that wants to see the status of all seed nodes. I haven't upgraded my nodes to his latest release of bts_tools yet, but I will soon.
+5%!
Title: Re: BTS seed node is decling while STEEM seed node is rising
Post by: cube on May 10, 2016, 05:41:52 pm
There is a bug identified earlier by @abit.  Perhaps abit can share more on is latest finding.
Title: Re: BTS seed node is decling while STEEM seed node is rising
Post by: abit on May 10, 2016, 08:55:39 pm

Quote
Also, while switching all the nodes to use p2p port 1776 this weekend I had several seeds get stuck for some unknown reason. They've been solidly online for 2 days now, but nothing has changed so I suspect future failures will occur again.
My guess is that there is somethign weird happending after a few weeks
of runtime. And we haven't seen this kind of behavior in STEEM because
their have been plenty of hardforks and restarts.

My node had been getting stuck every day in last week. I added some logging to the code and now waiting for next stuck so I can know more about the reason, but it hasn't been stuck for 2 days.. that sucks. Once we caught the bug, it's perhaps easy to fix though -- just catch one more exception, perhaps.

There is a bug identified earlier by @abit.  Perhaps abit can share more on is latest finding.