Network Stability Under Graphene

[CLains]

Silent running o/

[bytemaster]

Is the dev team required to make changes to try to hit an internal network reaching 100k tps? Whats stopping us from demoing it? That would obvously bring in new investors,

No changes required to hit that except perhaps a more efficient wallet for generating the transactions.

Hmm even using the cli of desktop is a bottleneck? I thought adding fc would have solved that?

The desktop CLI is limited by RPC requests to the witness node.  Doing it all in the same process should give a 10x boost.

[jsidhu]

Is the dev team required to make changes to try to hit an internal network reaching 100k tps? Whats stopping us from demoing it? That would obvously bring in new investors,

No changes required to hit that except perhaps a more efficient wallet for generating the transactions.

Hmm even using the cli of desktop is a bottleneck? I thought adding fc would have solved that?

[bytemaster]

Is the dev team required to make changes to try to hit an internal network reaching 100k tps? Whats stopping us from demoing it? That would obvously bring in new investors,

No changes required to hit that except perhaps a more efficient wallet for generating the transactions.

[jsidhu]

Is the dev team required to make changes to try to hit an internal network reaching 100k tps? Whats stopping us from demoing it? That would obvously bring in new investors,

[triox]

I don't really think that running the very backbone of bitshares in a t2micro is a good practice at all, even if it seems to run "fine".

Witnesses should always be ready and able to deal with sudden increases in resources required by the network, and this is not possible with a t2micro.
What happens if a spam attack strike on bts? That low-end server will not be able to bear the attack.
What if all the witnesses run on a t2micro? Bitshares network dies.

And you can not even assume a priori that you will be always ready to manage your instance in time to mitigate an attack.
What if you are outside and can not connect to your servers or server's hosting? What if you are just sleeping?

Witnesses should always look at what could be the next required resources and their servers should always be capable of sustain a lot more requests that the current ones.

When spam attack happens, witnesses can easily scale up their witness servers. In my case, I regularly update my snapshot image, so I can upgrade my server to the highest one (e.g. 32 core, 208 G ram, 10 TB SSD) in a couple of hours.

Exactly, that's the whole point of cloud computing: automatically scale resources based on demand. And currently we're mostly signing empty blocks, there's really no point overprovisioning. It's just bad cost management. Especially if we have to dump witness pay on the market to pay for it.

[clayop]

I don't really think that running the very backbone of bitshares in a t2micro is a good practice at all, even if it seems to run "fine".

Witnesses should always be ready and able to deal with sudden increases in resources required by the network, and this is not possible with a t2micro.
What happens if a spam attack strike on bts? That low-end server will not be able to bear the attack.
What if all the witnesses run on a t2micro? Bitshares network dies.

And you can not even assume a priori that you will be always ready to manage your instance in time to mitigate an attack.
What if you are outside and can not connect to your servers or server's hosting? What if you are just sleeping?

Witnesses should always look at what could be the next required resources and their servers should always be capable of sustain a lot more requests that the current ones.

When spam attack happens, witnesses can easily scale up their witness servers. In my case, I regularly update my snapshot image, so I can upgrade my server to the highest one (e.g. 32 core, 208 G ram, 10 TB SSD) in a couple of hours.

[maqifrnswa]

I don't really think that running the very backbone of bitshares in a t2micro is a good practice at all, even if it seems to run "fine".

Witnesses should always be ready and able to deal with sudden increases in resources required by the network, and this is not possible with a t2micro.
What happens if a spam attack strike on bts? That low-end server will not be able to bear the attack.
What if all the witnesses run on a t2micro? Bitshares network dies.

And you can not even assume a priori that you will be always ready to manage your instance in time to mitigate an attack.
What if you are outside and can not connect to your servers or server's hosting? What if you are just sleeping?

Witnesses should always look at what could be the next required resources and their servers should always be capable of sustain a lot more requests that the current ones.
...
So no, the witness pay should not be only barely enough to pay the server's rent.
It should be high, to put more weight and responsability to the witnesses and to push them to do an exellent job.

witnesses need to be able to scale, and this can't be a hobby if it will succeed long term. If things go well, I anticipate witnesses actually being professional operations (like miners are really now large companies). And they must be paid accordingly well. We will hire/fire companies based on performance.

[Bhuz]

I don't really think that running the very backbone of bitshares in a t2micro is a good practice at all, even if it seems to run "fine".

Witnesses should always be ready and able to deal with sudden increases in resources required by the network, and this is not possible with a t2micro.
What happens if a spam attack strike on bts? That low-end server will not be able to bear the attack.
What if all the witnesses run on a t2micro? Bitshares network dies.

And you can not even assume a priori that you will be always ready to manage your instance in time to mitigate an attack.
What if you are outside and can not connect to your servers or server's hosting? What if you are just sleeping?

Witnesses should always look at what could be the next required resources and their servers should always be capable of sustain a lot more requests that the current ones.


About the pay:
IIRC, bytemaster spoke about witness's job as a job that requires technical administration and managment skills.
He also spoke about a very well paid position to both stimulate competition among witnesses and also to be able to have a very strong and high quality backbone.

So no, the witness pay should not be only barely enough to pay the server's rent.
It should be high, to put more weight and responsability to the witnesses and to push them to do an exellent job.

[abit]

Now that I'm voted out I can share a secret 

Graphene has been so rock-solid that I was able to scale back my infrastructure to running a single t2.micro (plus a physical machine at home and another t2 ready to spin up in seconds).
Even on this low-end setup CPU utilization is close to zero, RAM is safely at around 60% and no latency issues.
During the two-months I've only had 33 missed blocks, all before the vesting-pay-bug fix. I now only scale up the machines during build.

Now, some witnesses are maintaining several four-core, 8GB machines, and you might think that it's somehow more secure, stable or scalable but the thing is, they have to dump their earned BTS to pay the bill. Meanwhile, I've never sold any BTS and am still able to easily scale up if/when blockchain usage picks up.

Graphene has exceeded expectations in terms of stability and footprint so much, that we ended up overpaying the witnesses. We can easily have twice as many witnesses or slash witness pay by half. And if the Blockchain starts demanding more resources we can always reevaluate.

Just a note, while the block chain is growing fast, the witness_node consumes much more memory than the beginning.

[pc]

Thanks for stabbing our backs now that you're out. 

Reality is, witness pay is a looong way from adequately paying for the TIME that we witnesses invest to give you guys a stable network.

[triox]

Now that I'm voted out I can share a secret 

Graphene has been so rock-solid that I was able to scale back my infrastructure to running a single t2.micro (plus a physical machine at home and another t2 ready to spin up in seconds).
Even on this low-end setup CPU utilization is close to zero, RAM is safely at around 60% and no latency issues.
During the two-months I've only had 33 missed blocks, all before the vesting-pay-bug fix. I now only scale up the machines during build.

Now, some witnesses are maintaining several four-core, 8GB machines, and you might think that it's somehow more secure, stable or scalable but the thing is, they have to dump their earned BTS to pay the bill. Meanwhile, I've never sold any BTS and am still able to easily scale up if/when blockchain usage picks up.

Graphene has exceeded expectations in terms of stability and footprint so much, that we ended up overpaying the witnesses. We can easily have twice as many witnesses or slash witness pay by half. And if the Blockchain starts demanding more resources we can always reevaluate.

[merivercap]

 

In general it seems to be operating well so congratulations to the dev team. 

[pc]

Any witnesses having any problems with your nodes?

Some witnesses have reported missing random blocks due to high latencies. It seems to happen more often to some, less often to others (only once for me in the last 4 weeks or so).

IMO that's to be expected in a global distributed P2P network, but I thought I'd mention it.

[CalabiYau]

Beautiful piece of work. It runs great. Very, very close to prime time.

I completely agree.

[donkeypong]

Beautiful piece of work. It runs great. Very, very close to prime time.

[Thom]

Indeed, very stable. Only rare occasional missed blocks.

[cube]

It has been stable. And I do not see any missing blocks since the last update (despite the short 3-second block time). 

[rnglab]

It's been completely stable after witness withdraw fee was raised!

OT: witness rnglab was just left behind at previous maintenance interval, it's #24 from 23 signing witnesses now, please consider voting rnglab.

[rgcrypto]

Agreed, I use it daily

[CoinHoarder]

After almost 2 months of Graphene we have had a more stable network with higher participation than we ever had under BTS 0.x. 

Any witnesses having any problems with your nodes?   It seems like things just run smoothly!

Good work sir. Excuse the "debbie downers" and keep doing your thing. I love BTS 2.0.. its so fast, convenient, and sexy. The price, adoption, and volume will correct itself in time, we as a community just need to be patient.

Cheers

[bytemaster]

After almost 2 months of Graphene we have had a more stable network with higher participation than we ever had under BTS 0.x. 

Any witnesses having any problems with your nodes?   It seems like things just run smoothly!