Scheduling Proof of Scalability

[Thom]

 

To puppies & cube! You have cleared my fog on what you're trying to do (well, maybe still some mist in the air concerning the poll).

I'll help any way I can.

I will be more informed tomorrow at this time after I speak with wackou about our backbone and how we'll set it up to reduce latency and protect nodes from direct DDOS attacks.

A well balanced and thought out plan for the distribution of seed and backbone nodes should help minimize network / connection latencies.

[abit]

https://docs.google.com/spreadsheets/d/1amqTZZ0dllmEEONW6qvc_07CE1mqcEk0SYRPU4phgEc/edit?disco=AAAAAYfKzkI

1. Current version of Graphene has sync issues, especially when the network is under pressure (when there is transaction flooding). I think it's better to do this wider test after the sync issues fixed.
2. Current test network looks like stable, because it is highly centralized -- 80% of witnesses are running on one or two VPS(es) which controlled by BM/CryptoNomex, every forks made by other witnesses are simply discarded. In a decentralized network it's harder to decide which fork is effective (or say longest?). We'd better prove that it's stable on a decentralized network first.

BM has fixed the sync issues.  For witnesses, we could create a number of them and spread them out to different computers/instances.

Not yet fixed, at least when I wrote that post yesterday. 

Will see whether this commit https://github.com/cryptonomex/graphene/commit/ff2db08475908fad5e36df23d6c50256b9ab13f7 solves some of the issues. I'm running with a version which included that commit in current testnet now.

[kenCode]

I don't know if there is anything to learn from this.. but I thought I would share it in case it gives you all some ideas on ways to stress test the network:

https://bitcoinmagazine.com/21842/coinwallet-begins-pre-test-bitcoin-network-schedules-largest-stress-test-begin-september-10/

Anybody with bitcoins might want to brace yourselves for tomorrows bombardment.

Hope this helps in some way.

Tuck check this out ^^

[puppies]

That makes perfect sense Cube.  I think you have a better testing methodology that will result in a better final product.  If there are any simple tasks I can do please let me know.

I was thinking I would spin up a couple of instances and see what type of volume I could broadcast from a single machine using current spam techniques.  I get the feeling that we will need to develop better means of flooding.  Something like the build in flood_network command but with more sustained flooding.

[BunkerChainLabs-DataSecurityNode]

I don't know if there is anything to learn from this.. but I thought I would share it in case it gives you all some ideas on ways to stress test the network:

https://bitcoinmagazine.com/21842/coinwallet-begins-pre-test-bitcoin-network-schedules-largest-stress-test-begin-september-10/

Anybody with bitcoins might want to brace yourselves for tomorrows bombardment.

Hope this helps in some way.

[cube]

Thanks for the response cube. 

I was thinking that it would make more sense to start with a wide area network test.  I know this will not reach the actual potential throughput of graphene, and will be limited by the network speed, and hardware it is running on.  It would utilize volunteers rather than raising funds.  That is what would necessitate the ease of use, and multiple OS's.  This is also the reason why I would have it create an account.  So that users wouldn't have to. 

I am leaning towards thinking that we should stress test the existing test network rather than creating a new network to benchmark graphene. but if you think we should do the benchmark first and then work on improving the test networks performance later, thats not a big deal to me at all.

Thanks for clarifying.

I understand your objective now - to improve the test networks performance first and utilise greater number of volunteers. 

I think there are a few factors affecting the test network performance in a WAN - the graphene software (with its algorithm), the network limitation of individual nodes, and the cpu+ram resources of the individual nodes.  Assuming a scenario of a slow performance, it would be difficult to pin down where is the source of the slow down because of the various factors.  It could be because some witness nodes are 'far away' (ie many internet network hops),  some having a slow internet bandwidth/high latency connections, some running on old and slow computers, or some flaws in the graphene software codes.

However, if we can isolate the factors contributing from individual nodes, it become easier to say eg 'Ah the software starts to slow down when it is reaching X tps. I am pretty sure it is not due to the machines or the physical network (because they are top machines running on local fast network)'. 

BTW, I am not suggesting forgoing WAN test. In fact, we could proceed to do WAN test immediately after LAN test if resources and time allow.

We would still need a lot of help from volunteers eg vetting dummy account names and transactions in a database (to be used to blast the load).

Not sure how to work around this. Maybe have a global account registered and its
priv key distributed in the script. that way testers can register themselves
with that account.

What does everyone think?  Is this a good enough starting point? 

Absolutely. My problem is that I can only start working on this next week.

Great to have you with us.  Load testing is not an easy nor simple undetaking.  We should do it well and this takes time.  With your participation, we would be closer to the goal.

[xeroc]

Sorry for my late reply.

2) What is our goal?
     I propose a goal of 1000tps sustained for 5 minutes. 

Don't know if that may break the stats page. It's really put together quickly
and needs some more testing. Maybe even add BM's relay nodejs to harden it.

4) What do we need?
   a) compiled binaries for windows, osx, and linux
   b) instructions on how to download, intall, and start the witness_node,
      cli_wallet, and a python script (lets focus on getting everyone to the
      startup screen on cli_wallet, and take care of account creation, and
      transaction spamming with a script.
   c) a script that will:
     1c) create and unlock a wallet
     2c) Import a known test private key to get a balance
     3c) create an account and upgrade it to lifetime status
     4c) spam transactions
     5c) keep an eye on account balance
     6c) withdraw vesting funds so testing can continue longer
     7c) log anything that might be interesting.  Both to screen and to file.

Funds can only be important as a whole from a private key. Hence you either need
someone that can send you some funds or your need to distribute privkeys that
hold funds.

account creation is only possible if someone already registered pays the
registration fee. you cannot simply register on your own as far as I know.

Not sure how to work around this. Maybe have a global account registered and its
priv key distributed in the script. that way testers can register themselves
with that account.

What does everyone think?  Is this a good enough starting point? 

Absolutely. My problem is that I can only start working on this next week.

[puppies]

Thanks for the response cube. 

I was thinking that it would make more sense to start with a wide area network test.  I know this will not reach the actual potential throughput of graphene, and will be limited by the network speed, and hardware it is running on.  It would utilize volunteers rather than raising funds.  That is what would necessitate the ease of use, and multiple OS's.  This is also the reason why I would have it create an account.  So that users wouldn't have to. 

I am leaning towards thinking that we should stress test the existing test network rather than creating a new network to benchmark graphene. but if you think we should do the benchmark first and then work on improving the test networks performance later, thats not a big deal to me at all.

[cube]

Okay, here goes.  Please help me flesh this plan out.  Or alternately if you think its stupid, you could just tell me that.  Although I would appreciate it if you were a little bit polite when you made fun of my plan.

Thanks for an excellent plan.  I have a few comments/points I like to add:

1) What are we testing?
     For the first phase I propose that we attempt a stress test of the test network WAN.  This is a real world test of what we will be able to do when 2.0 launches. 

Do you mean LAN (Local Area Network) test to begin with?  If LAN test failed to achieve 100K tps then we know WAN test will fail too.  Besides, WAN depends on the individual witness node's speed and so there are more variables in the testing.

2) What is our goal?
     I propose a goal of 1000tps sustained for 5 minutes. 

Let's start with 5mins.  And let's see how long the network can hold before it breaks.

3) Why test? 
     First of all it will be good publicity to show what we can actually do, not just the theoretical limit.  Secondly, it will provide lots of information about how the binaries behave on different machines, and how the network deals with that amount of traffic.

Yes, an excellent way to 'show off' graphene's 100K tps real power.

4) What do we need?
   a) compiled binaries for windows, osx, and linux

I think for a start, and to make thing easier, choose one - Windows and/or Linux since they are available now.

   b) instructions on how to download, intall, and start the witness_node, cli_wallet, and a python script (lets focus on getting everyone to the startup screen on cli_wallet, and take care of account creation, and transaction spamming with a script.

This would be especially useful when we start running tests for WAN.

   c) a script that will:
     1c) create and unlock a wallet
     2c) Import a known test private key to get a balance
     3c) create an account and upgrade it to lifetime status
     4c) spam transactions
     5c) keep an eye on account balance
     6c) withdraw vesting funds so testing can continue longer

Having scripts to simulate as close to 'real life' transactions as possible - this list is excellent for that. But since writing scripts take time - a whole lot of time, I suggest we forcus on scripting tasks that most frequently used by users and those txs that take up most resources - cpu/mem/network. We want to see how much stress the network can bear. Import private keys are sensitive and private stuff, and so we may not want to script it.  Perhaps simplifying to:

     1c) create and unlock a wallet
     3c) create an account and upgrade it to lifetime status
     4c) transfer transactions

     7c) log anything that might be interesting.  Both to screen and to file.

Yes, we like to log the good results and show it to the world.

Alternately if we can set up a faucet the script can register the new accounts under a lifetime account, and our excess funds vesting will be in a centralized location.
It would be great if this script could be the interaction point for testers.  Asking any required information at the beginning, and displaying any information needed.  That would prevent testers from having to learn anything about the witness_node or cli_wallet

   d) Lots of people to test.  If installing 3 programs, and typing in a few commands sounds like something you can do, then you could help us make history. 

I am not sure if it is the right time now to 'teach' users who are not technical since preparing documentation/tutorials/simplying installation will take up lots of time and effort. But if we can accomplish that, it would be great.

If you can help us make this test a reality please let me know.  We already have binaries available, but will probably need them updated by the time we are ready to test.  I can help with instructions.  I could write the script, but it would take me 10 times longer than xeroc, and the final product would be 10 times worse than xerocs product would be.  I will also of course help run nodes for the test.  Both at home, and on some VPS's. 

Generating dummy user accounts and transactions will probably involve interacting with a database.  How easy to modify xeroc's scripts for that?

I am considering jmeter.  How about adding a custom plug-in to jmeter to perform the necessary rpc callings (these create the commands to the witness_node daemon to perform the transactions)?

Xeroc.  What do you think in regards to the script Idea I posted?  Would you be able to donate that script to the cause, or would you want to set a price up front?

I would like to hear from xeroc too.

No.  To save on fees.  The hardest part about testing so far has been getting your hands on enough core.  20bts per transaction adds up fast.  If the account is upgraded that is lowered to 4bts per transaction after rebate.  Since my goal is 1000tps for 5 minutes, if each node can handle 10tps for 5 minutes that would be a savings of 38k bts after the 10k bts fee to upgrade.  I hope that made sense.

If tx-fees is a concern, how about starting a new testnet for the load testing?  We can modify the source codes to generate tons and tons of CORE for the tests.

1. Current version of Graphene has sync issues, especially when the network is under pressure (when there is transaction flooding). I think it's better to do this wider test after the sync issues fixed.
2. Current test network looks like stable, because it is highly centralized -- 80% of witnesses are running on one or two VPS(es) which controlled by BM/CryptoNomex, every forks made by other witnesses are simply discarded. In a decentralized network it's harder to decide which fork is effective (or say longest?). We'd better prove that it's stable on a decentralized network first.

BM has fixed the sync issues.  For witnesses, we could create a number of them and spread them out to different computers/instances.

[puppies]

Very good !!

Just small extra points about the load testing script, If we identify the account names beforehand, nobody should ran out of core if is not for transaction fees, as we can all send each other.

I don't know if 1c is possible as you need to interact with an unlocked account.

Handle errors, for example if we don't have enough balance the script will fail (5c)

If 1c is possible we could build scripts that pick up usernames ids / balances from a server, and build an image (docker containers) that on start will pickup a user to setup.

If its possible for everyone to send transactions from the exact same account then we could skip all of that, and just have the script import the private keys of a throwaway account.  I am just trying to think of ways to make joining in the test very easy, so that people that are not super technical can help out.

Of course BMs new network protocol may complicate this.  It may not be possible to get the combination of witness node, cli wallet, and relay installed and running on windows in an easy to follow, hard to mess up way.

[clayop]

I'm curious that how many transactions init witnesses can process. I tried to flood network over 10 tps with two VPS but tps did not pass 10.

[liondani]

     3c) create an account and upgrade it to lifetime status

to see if it works?

No.  To save on fees.  The hardest part about testing so far has been getting your hands on enough core.  20bts per transaction adds up fast.  If the account is upgraded that is lowered to 4bts per transaction after rebate.  Since my goal is 1000tps for 5 minutes, if each node can handle 10tps for 5 minutes that would be a savings of 38k bts after the 10k bts fee to upgrade.  I hope that made sense.

Make sense (  i didn't make the math)

PS off topic:
now I realize the fees....  4 bts?(!)  (instead of 0.1 or 0.5 currently)
What if the market cap increases 10 fold for example or even more ?Is it not to much? (I assume "delegates" can change that) What about dynamic fees? A percentage like 0.2% for example?
Would it not be better so we are not in the position every now and the to change the fees?

[betax]

To summarise:

It needs a test plan (what are we testing, how are we testing it, what is what we want to achieved or why are we testing), environment/s setup, scripts to execute tests, and of course who is going to participate.

Once this is done, we can think what money we need to achieve this.

Okay, here goes.  Please help me flesh this plan out.  Or alternately if you think its stupid, you could just tell me that.  Although I would appreciate it if you were a little bit polite when you made fun of my plan.

1) What are we testing?
     For the first phase I propose that we attempt a stress test of the test network WAN.  This is a real world test of what we will be able to do when 2.0 launches. 
2) What is our goal?
     I propose a goal of 1000tps sustained for 5 minutes. 
3) Why test? 
     First of all it will be good publicity to show what we can actually do, not just the theoretical limit.  Secondly, it will provide lots of information about how the binaries behave on different machines, and how the network deals with that amount of traffic.
4) What do we need?
   a) compiled binaries for windows, osx, and linux
   b) instructions on how to download, intall, and start the witness_node, cli_wallet, and a python script (lets focus on getting everyone to the startup screen on cli_wallet, and take care of account creation, and transaction spamming with a script.
   c) a script that will:
     1c) create and unlock a wallet
     2c) Import a known test private key to get a balance
     3c) create an account and upgrade it to lifetime status
     4c) spam transactions
     5c) keep an eye on account balance
     6c) withdraw vesting funds so testing can continue longer
     7c) log anything that might be interesting.  Both to screen and to file.
Alternately if we can set up a faucet the script can register the new accounts under a lifetime account, and our excess funds vesting will be in a centralized location.
It would be great if this script could be the interaction point for testers.  Asking any required information at the beginning, and displaying any information needed.  That would prevent testers from having to learn anything about the witness_node or cli_wallet
   d) Lots of people to test.  If installing 3 programs, and typing in a few commands sounds like something you can do, then you could help us make history. 

What does everyone think?  Is this a good enough starting point? 

If you can help us make this test a reality please let me know.  We already have binaries available, but will probably need them updated by the time we are ready to test.  I can help with instructions.  I could write the script, but it would take me 10 times longer than xeroc, and the final product would be 10 times worse than xerocs product would be.  I will also of course help run nodes for the test.  Both at home, and on some VPS's. 

Xeroc.  What do you think in regards to the script Idea I posted?  Would you be able to donate that script to the cause, or would you want to set a price up front?

Either way if we end up doing this, please consider tipping xeroc, cube, and maqifrnswa.  Please let me know if I have missed any other tip worthy contributors.

Very good !!

Just small extra points about the load testing script, If we identify the account names beforehand, nobody should ran out of core if is not for transaction fees, as we can all send each other.

I don't know if 1c is possible as you need to interact with an unlocked account.

Handle errors, for example if we don't have enough balance the script will fail (5c)

If 1c is possible we could build scripts that pick up usernames ids / balances from a server, and build an image (docker containers) that on start will pickup a user to setup.

1. Current version of Graphene has sync issues, especially when the network is under pressure (when there is transaction flooding). I think it's better to do this wider test after the sync issues fixed.
2. Current test network looks like stable, because it is highly centralized -- 80% of witnesses are running on one or two VPS(es) which controlled by BM/CryptoNomex, every forks made by other witnesses are simply discarded. In a decentralized network it's harder to decide which fork is effective (or say longest?). We'd better prove that it's stable on a decentralized network first.

^^^^^ That will help too ^^^^^

[abit]

1. Current version of Graphene has sync issues, especially when the network is under pressure (when there is transaction flooding). I think it's better to do this wider test after the sync issues fixed.
2. Current test network looks like stable, because it is highly centralized -- 80% of witnesses are running on one or two VPS(es) which controlled by BM/CryptoNomex, every forks made by other witnesses are simply discarded. In a decentralized network it's harder to decide which fork is effective (or say longest?). We'd better prove that it's stable on a decentralized network first.

[puppies]

     3c) create an account and upgrade it to lifetime status

to see if it works?

No.  To save on fees.  The hardest part about testing so far has been getting your hands on enough core.  20bts per transaction adds up fast.  If the account is upgraded that is lowered to 4bts per transaction after rebate.  Since my goal is 1000tps for 5 minutes, if each node can handle 10tps for 5 minutes that would be a savings of 38k bts after the 10k bts fee to upgrade.  I hope that made sense.