Arguments Against 100 TPS

[fuzzy]

Ive been following this discussion intently and woul like to bring this up in todays hangout.  Will it be ok to carve some time out for this?

[testz]

I was recently asked by Testz to address this topic:

Thanks for clarification.

[Tuck Fheman]

I seem to have lost my bitcointalk login password, so can someone please cross-post this for me.  Thanks.

done

[bytemaster]

I was recently asked by Testz to address this topic:

https://bitcointalk.org/index.php?topic=1196533.msg12563741#msg12563741

ell seeing as Larimer himself said the solution is to "...keep everything in RAM..."  how much RAM to do think is required to keep up with a sustained 100,000 tps if it is indeed true?

You don't need to know anything about how that system works when these statements have been, 120 bytes per transaction * 100,000 transactions per second * 86400 seconds in a day = 1,036,800,000,000 bytes (almost 1TB) per day of data.  However you cut it that is a lot of data to manage efficiently in a vertically scaled system (which Bitshares is). 

If you have to go to disc even the fastest PCI SSDs will struggle to do 100,000 random reads per second after some moderate use, unless you are doing very aggressive and efficient disc management at a low level to ensure they are stored sequentially.  Even without considering transactions arriving out of order from the network, that in itself is no trivial task. 

Even batch writing large blocks of transactions to reduce the required IOPs might not cut it, as you have no guarantee that the SSD controller is going to (or even can) write them in one consecutive chunk.  If the controller wants to move data around so it can do this, that is additional overhead and your max IOPs will drop anyway.

If the RAM requirements are low, then you will be swapping out those transactions in RAM constantly, so then you are asking an already utilized IO stream to do 100,000 reads AND 100,000 writes each second on top of processing.

My issue isn't that 100,000 transactions can not be processed per second, my issue is that sustaining that processing capability with RAM and IO limitations is not as trivial as is being made out, and anyone with even a basic understanding of IO systems should be able to see it.

I believe this 100,000 tps relates to the systems ability to order match as he mentioned that specifically in the videos relating to this topic, and NOT a sustained network load of new transaction processing capability.

I want to address the MAJOR misconception and that is that we keep all transactions in RAM and that we need access to the full transaction history to process new transactions.

The default wallet has all transactions expiring just 15 seconds after they are signed which means that the network only has to consider 1,500,000 * 20 byte (trx id) => 3 MB of memory to protect against replay attacks of the same transaction.

The vast majority of all transactions simply modify EXISTING data structures (balances, orders, etc).   The only type of transaction that increases memory use permanently are account creation, asset creation, witness/worker/committee member creation. These particular operations COST much more than operations that modify existing data.  Their cost is derived from the need to keep them in memory for ever.   

So the system needs the ability to STREAM 11 MB per second of data to disk and over the network (assuming all transactions were 120 bytes). 

If there were 6 billion accounts and the average account had 1KB of data associated with it then the system would require 6000 GB or 6 TB of RAM... considering you can already buy motherboards supporting 2TB of ram and probably more if you look in the right places (http://www.eteknix.com/intels-new-serverboard-supports-dual-cpu-2tb-ram/)  I don't think it is unreasonable to require 1 TB per BILLION accounts. 

I seem to have lost my bitcointalk login password, so can someone please cross-post this for me.  Thanks.