BitShares Forum

Main => General Discussion => Topic started by: bytemaster on December 05, 2015, 12:27:12 am

Title: Network Stability Under Graphene
Post by: bytemaster on December 05, 2015, 12:27:12 am
After almost 2 months of Graphene we have had a more stable network with higher participation than we ever had under BTS 0.x. 

Any witnesses having any problems with your nodes?   It seems like things just run smoothly!
Title: Re: Network Stability Under Graphene
Post by: CoinHoarder on December 05, 2015, 01:03:48 am
After almost 2 months of Graphene we have had a more stable network with higher participation than we ever had under BTS 0.x. 

Any witnesses having any problems with your nodes?   It seems like things just run smoothly!

Good work sir. Excuse the "debbie downers" and keep doing your thing. I love BTS 2.0.. its so fast, convenient, and sexy. The price, adoption, and volume will correct itself in time, we as a community just need to be patient.

Cheers
Title: Re: Network Stability Under Graphene
Post by: rgcrypto on December 05, 2015, 01:05:48 am
Agreed, I use it daily
Title: Re: Network Stability Under Graphene
Post by: rnglab on December 05, 2015, 01:24:20 am
It's been completely stable after witness withdraw fee was raised!

OT: witness rnglab was just left behind at previous maintenance interval, it's #24 from 23 signing witnesses now, please consider voting rnglab.
Title: Re: Network Stability Under Graphene
Post by: cube on December 05, 2015, 01:56:40 am
It has been stable. And I do not see any missing blocks since the last update (despite the short 3-second block time). 
Title: Re: Network Stability Under Graphene
Post by: Thom on December 05, 2015, 03:22:07 am
Indeed, very stable. Only rare occasional missed blocks.
Title: Re: Network Stability Under Graphene
Post by: donkeypong on December 05, 2015, 06:03:52 am
Beautiful piece of work. It runs great. Very, very close to prime time.
Title: Re: Network Stability Under Graphene
Post by: CalabiYau on December 05, 2015, 06:42:09 am
Beautiful piece of work. It runs great. Very, very close to prime time.

I completely agree.
Title: Re: Network Stability Under Graphene
Post by: pc on December 05, 2015, 07:45:10 am
Any witnesses having any problems with your nodes?

Some witnesses have reported missing random blocks due to high latencies. It seems to happen more often to some, less often to others (only once for me in the last 4 weeks or so).

IMO that's to be expected in a global distributed P2P network, but I thought I'd mention it.
Title: Re: Network Stability Under Graphene
Post by: merivercap on December 05, 2015, 09:00:07 am
 +5%

In general it seems to be operating well so congratulations to the dev team. 
Title: Re: Network Stability Under Graphene
Post by: triox on December 05, 2015, 10:45:51 am
Now that I'm voted out I can share a secret  ;)

Graphene has been so rock-solid that I was able to scale back my infrastructure to running a single t2.micro (plus a physical machine at home and another t2 ready to spin up in seconds).
Even on this low-end setup CPU utilization is close to zero, RAM is safely at around 60% and no latency issues.
During the two-months I've only had 33 missed blocks, all before the vesting-pay-bug fix. I now only scale up the machines during build.

Now, some witnesses are maintaining several four-core, 8GB machines, and you might think that it's somehow more secure, stable or scalable but the thing is, they have to dump their earned BTS to pay the bill. Meanwhile, I've never sold any BTS and am still able to easily scale up if/when blockchain usage picks up.

Graphene has exceeded expectations in terms of stability and footprint so much, that we ended up overpaying the witnesses. We can easily have twice as many witnesses or slash witness pay by half. And if the Blockchain starts demanding more resources we can always reevaluate.
Title: Re: Network Stability Under Graphene
Post by: pc on December 05, 2015, 04:31:48 pm
Thanks for stabbing our backs now that you're out.  >:(

Reality is, witness pay is a looong way from adequately paying for the TIME that we witnesses invest to give you guys a stable network.
Title: Re: Network Stability Under Graphene
Post by: abit on December 05, 2015, 06:47:58 pm
Now that I'm voted out I can share a secret  ;)

Graphene has been so rock-solid that I was able to scale back my infrastructure to running a single t2.micro (plus a physical machine at home and another t2 ready to spin up in seconds).
Even on this low-end setup CPU utilization is close to zero, RAM is safely at around 60% and no latency issues.
During the two-months I've only had 33 missed blocks, all before the vesting-pay-bug fix. I now only scale up the machines during build.

Now, some witnesses are maintaining several four-core, 8GB machines, and you might think that it's somehow more secure, stable or scalable but the thing is, they have to dump their earned BTS to pay the bill. Meanwhile, I've never sold any BTS and am still able to easily scale up if/when blockchain usage picks up.

Graphene has exceeded expectations in terms of stability and footprint so much, that we ended up overpaying the witnesses. We can easily have twice as many witnesses or slash witness pay by half. And if the Blockchain starts demanding more resources we can always reevaluate.
Just a note, while the block chain is growing fast, the witness_node consumes much more memory than the beginning.
Title: Re: Network Stability Under Graphene
Post by: Bhuz on December 05, 2015, 08:44:02 pm
I don't really think that running the very backbone of bitshares in a t2micro is a good practice at all, even if it seems to run "fine".

Witnesses should always be ready and able to deal with sudden increases in resources required by the network, and this is not possible with a t2micro.
What happens if a spam attack strike on bts? That low-end server will not be able to bear the attack.
What if all the witnesses run on a t2micro? Bitshares network dies.

And you can not even assume a priori that you will be always ready to manage your instance in time to mitigate an attack.
What if you are outside and can not connect to your servers or server's hosting? What if you are just sleeping?

Witnesses should always look at what could be the next required resources and their servers should always be capable of sustain a lot more requests that the current ones.


About the pay:
IIRC, bytemaster spoke about witness's job as a job that requires technical administration and managment skills.
He also spoke about a very well paid position to both stimulate competition among witnesses and also to be able to have a very strong and high quality backbone.

So no, the witness pay should not be only barely enough to pay the server's rent.
It should be high, to put more weight and responsability to the witnesses and to push them to do an exellent job.
Title: Re: Network Stability Under Graphene
Post by: maqifrnswa on December 05, 2015, 09:05:35 pm
I don't really think that running the very backbone of bitshares in a t2micro is a good practice at all, even if it seems to run "fine".

Witnesses should always be ready and able to deal with sudden increases in resources required by the network, and this is not possible with a t2micro.
What happens if a spam attack strike on bts? That low-end server will not be able to bear the attack.
What if all the witnesses run on a t2micro? Bitshares network dies.

And you can not even assume a priori that you will be always ready to manage your instance in time to mitigate an attack.
What if you are outside and can not connect to your servers or server's hosting? What if you are just sleeping?

Witnesses should always look at what could be the next required resources and their servers should always be capable of sustain a lot more requests that the current ones.
...
So no, the witness pay should not be only barely enough to pay the server's rent.
It should be high, to put more weight and responsability to the witnesses and to push them to do an exellent job.
+5%
witnesses need to be able to scale, and this can't be a hobby if it will succeed long term. If things go well, I anticipate witnesses actually being professional operations (like miners are really now large companies). And they must be paid accordingly well. We will hire/fire companies based on performance.
Title: Re: Network Stability Under Graphene
Post by: clayop on December 05, 2015, 09:26:33 pm
I don't really think that running the very backbone of bitshares in a t2micro is a good practice at all, even if it seems to run "fine".

Witnesses should always be ready and able to deal with sudden increases in resources required by the network, and this is not possible with a t2micro.
What happens if a spam attack strike on bts? That low-end server will not be able to bear the attack.
What if all the witnesses run on a t2micro? Bitshares network dies.

And you can not even assume a priori that you will be always ready to manage your instance in time to mitigate an attack.
What if you are outside and can not connect to your servers or server's hosting? What if you are just sleeping?

Witnesses should always look at what could be the next required resources and their servers should always be capable of sustain a lot more requests that the current ones.
+5% +5%

When spam attack happens, witnesses can easily scale up their witness servers. In my case, I regularly update my snapshot image, so I can upgrade my server to the highest one (e.g. 32 core, 208 G ram, 10 TB SSD) in a couple of hours.
Title: Re: Network Stability Under Graphene
Post by: triox on December 06, 2015, 09:30:45 am
I don't really think that running the very backbone of bitshares in a t2micro is a good practice at all, even if it seems to run "fine".

Witnesses should always be ready and able to deal with sudden increases in resources required by the network, and this is not possible with a t2micro.
What happens if a spam attack strike on bts? That low-end server will not be able to bear the attack.
What if all the witnesses run on a t2micro? Bitshares network dies.

And you can not even assume a priori that you will be always ready to manage your instance in time to mitigate an attack.
What if you are outside and can not connect to your servers or server's hosting? What if you are just sleeping?

Witnesses should always look at what could be the next required resources and their servers should always be capable of sustain a lot more requests that the current ones.
+5% +5%

When spam attack happens, witnesses can easily scale up their witness servers. In my case, I regularly update my snapshot image, so I can upgrade my server to the highest one (e.g. 32 core, 208 G ram, 10 TB SSD) in a couple of hours.

Exactly, that's the whole point of cloud computing: automatically scale resources based on demand. And currently we're mostly signing empty blocks, there's really no point overprovisioning. It's just bad cost management. Especially if we have to dump witness pay on the market to pay for it.
Title: Re: Network Stability Under Graphene
Post by: jsidhu on December 06, 2015, 06:33:59 pm
Is the dev team required to make changes to try to hit an internal network reaching 100k tps? Whats stopping us from demoing it? That would obvously bring in new investors,
Title: Re: Network Stability Under Graphene
Post by: bytemaster on December 06, 2015, 10:20:26 pm
Is the dev team required to make changes to try to hit an internal network reaching 100k tps? Whats stopping us from demoing it? That would obvously bring in new investors,

No changes required to hit that except perhaps a more efficient wallet for generating the transactions.
Title: Re: Network Stability Under Graphene
Post by: jsidhu on December 08, 2015, 04:18:35 pm
Is the dev team required to make changes to try to hit an internal network reaching 100k tps? Whats stopping us from demoing it? That would obvously bring in new investors,

No changes required to hit that except perhaps a more efficient wallet for generating the transactions.
Hmm even using the cli of desktop is a bottleneck? I thought adding fc would have solved that?
Title: Re: Network Stability Under Graphene
Post by: bytemaster on December 08, 2015, 04:26:41 pm
Is the dev team required to make changes to try to hit an internal network reaching 100k tps? Whats stopping us from demoing it? That would obvously bring in new investors,

No changes required to hit that except perhaps a more efficient wallet for generating the transactions.
Hmm even using the cli of desktop is a bottleneck? I thought adding fc would have solved that?

The desktop CLI is limited by RPC requests to the witness node.  Doing it all in the same process should give a 10x boost.
Title: Re: Network Stability Under Graphene
Post by: CLains on December 08, 2015, 04:32:52 pm
Silent running o/