BitShares Forum

Main => General Discussion => Topic started by: monsterer on May 22, 2015, 11:03:27 am

Title: Why does the bitshares_client use 10x the memory of bitcoind?
Post by: monsterer on May 22, 2015, 11:03:27 am
The high memory usage is probably a huge factor in the bad performance of the bitshares client. Why does it need so much RAM compared to bitcoind?
Title: Re: Why does the bitshares_client use 10x the memory of bitcoind?
Post by: xeroc on May 22, 2015, 11:09:26 am
The high memory usage is probably a huge factor in the bad performance of the bitshares client. Why does it need so much RAM compared to bitcoind?
It uses alot more indices than bitcoind ...
Recall that the BitShares client has a BUILT-IN block explorer!
Title: Re: Why does the bitshares_client use 10x the memory of bitcoind?
Post by: monsterer on May 22, 2015, 01:16:01 pm
It uses alot more indices than bitcoind ...
Recall that the BitShares client has a BUILT-IN block explorer!

Why do the indices need to be RAM resident?
Title: Re: Why does the bitshares_client use 10x the memory of bitcoind?
Post by: bytemaster on May 22, 2015, 01:18:15 pm
It uses alot more indices than bitcoind ...
Recall that the BitShares client has a BUILT-IN block explorer!

Why do the indices need to be RAM resident?

The bottle neck for BitShares is Disk IO.   BitShares uses a lot of RAM in an attempt to minimize Disk IO.   

Bitcoin has just a few LevelDB databases, BitShares has 2 dozen.    Each LevelDB has its own cache.

This is all old news anyway, soon BitShares will be using less RAM than your web browser.
Title: Re: Why does the bitshares_client use 10x the memory of bitcoind?
Post by: monsterer on May 22, 2015, 01:32:17 pm
The bottle neck for BitShares is Disk IO.   BitShares uses a lot of RAM in an attempt to minimize Disk IO.   

Bitcoin has just a few LevelDB databases, BitShares has 2 dozen.    Each LevelDB has its own cache.

This is all old news anyway, soon BitShares will be using less RAM than your web browser.

That bottle neck is probably the true bottle neck only for machines which aren't forced to swap the caches out to disk due to 'lower' real RAM than your development machines.

However, I am glad to hear you're already on it :)
Title: Re: Why does the bitshares_client use 10x the memory of bitcoind?
Post by: bytemaster on May 22, 2015, 01:40:28 pm
The bottle neck for BitShares is Disk IO.   BitShares uses a lot of RAM in an attempt to minimize Disk IO.   

Bitcoin has just a few LevelDB databases, BitShares has 2 dozen.    Each LevelDB has its own cache.

This is all old news anyway, soon BitShares will be using less RAM than your web browser.

That bottle neck is probably the true bottle neck only for machines which aren't forced to swap the caches out to disk due to 'lower' real RAM than your development machines.

However, I am glad to hear you're already on it :)

Even with 16 GB of RAM disk io is the bottleneck. 
Title: Re: Why does the bitshares_client use 10x the memory of bitcoind?
Post by: monsterer on May 22, 2015, 01:47:18 pm
Even with 16 GB of RAM disk io is the bottleneck.

My point was that with substantially less RAM than your development machines have (for example 2GB on a win7 machine), plain disk IO is unlikely to be the bottleneck... I'd expect it to be related to the swapping necessary to maintain the vast array of caches you mentioned.
Title: Re: Why does the bitshares_client use 10x the memory of bitcoind?
Post by: testz on May 22, 2015, 06:18:02 pm
It uses alot more indices than bitcoind ...
Recall that the BitShares client has a BUILT-IN block explorer!

Why do the indices need to be RAM resident?

The bottle neck for BitShares is Disk IO.   BitShares uses a lot of RAM in an attempt to minimize Disk IO.   

Bitcoin has just a few LevelDB databases, BitShares has 2 dozen.    Each LevelDB has its own cache.

This is all old news anyway, soon BitShares will be using less RAM than your web browser.

Oh, no, my web browser use 7 GB of RAM, let's say that "BitShares will be using less RAM than build-in calc"  :)
Title: Re: Why does the bitshares_client use 10x the memory of bitcoind?
Post by: karnal on May 23, 2015, 12:57:30 pm
This is all old news anyway, soon BitShares will be using less RAM than your web browser.

Really BM, now you mention it? I literally bought 8GB of RAM to be able to run the bitshares client without killing a few programs & swapping madly.

Seriously.

 ;D
Title: Re: Why does the bitshares_client use 10x the memory of bitcoind?
Post by: mangou007 on May 23, 2015, 02:26:19 pm

This is all old news anyway, soon BitShares will be using less RAM than your web browser.

Don't know why, but I am loving to see things like that :)  +5%
Title: Re: Why does the bitshares_client use 10x the memory of bitcoind?
Post by: julian1 on May 24, 2015, 02:04:14 am
I'm intensely curious to know what choices have been made here. 

Possibilities I thought off include consolidating all the leveldb databases by prefixing the leveldb keys with namespaces.

Or alternatively using a memory-mapped data-structure (btree) implementation.
Title: Re: Why does the bitshares_client use 10x the memory of bitcoind?
Post by: sittingduck on May 24, 2015, 03:17:04 pm

I'm intensely curious to know what choices have been made here. 

Possibilities I thought off include consolidating all the leveldb databases by prefixing the leveldb keys with namespaces.

Or alternatively using a memory-mapped data-structure (btree) implementation.

If disk io is the issue then neither of your suggestions would work.   


Sent from my iPhone using Tapatalk
Title: Re: Why does the bitshares_client use 10x the memory of bitcoind?
Post by: bytemaster on May 25, 2015, 12:11:55 pm
I'm intensely curious to know what choices have been made here. 

Possibilities I thought off include consolidating all the leveldb databases by prefixing the leveldb keys with namespaces.

Or alternatively using a memory-mapped data-structure (btree) implementation.

1 massive level DB database would still have a random access pattern and make every lookup take longer (the cost per lookup grows with log(n) the number of items). 

A memory mapped btree still requires disk IO (unless it is all cached).   Representing the data in a form that is suitable for storing on disk also slows down access. 

We load everything into memory on startup and keep them in boost::multi_index_container which allow both hashed and sorted indexes on multiple fields.  So lookup is constant time and updates involve mere pointer manipulation.    It takes mere seconds to load everything from disk to memory and then to save it again.   For now we are loading and saving to/from a Level DB database, but in the future it will probably just be a set of flat files.    Depending upon the type, we can get away with a "stack" based structure in a flat file. 
Title: Re: Why does the bitshares_client use 10x the memory of bitcoind?
Post by: abit on May 25, 2015, 02:58:44 pm
When talking about disk IO bottleneck, we always check I and O (reads and writes) separately. We can't avoid IO, we just reduce the frequency, at lease reduce random accessing.
There are still rooms available for performance optimizing with the use of leveldb, E.G.
* The optional FilterPolicy mechanism can be used to reduce the number of disk reads substantially.
* Using less quantity of level DB's will hopefully reduce competing on writes/compacts.
* Better design of keys (of values)
* Compressing and caching

Anyway, if we've already chosen another way, please go ahead.
Title: Re: Why does the bitshares_client use 10x the memory of bitcoind?
Post by: arubi on May 25, 2015, 09:51:11 pm
Ugh.

$ ps -xv | grep bitshares
 2797 pts/6    Sl+   18:01   4478 33385 1064798 955000 46.4 bitshares_client --server --httpport 9989


$ ps -xv | grep bitcoind
 6659 ?        Ssl  112:09   1230  4509 742338 595268 28.9 bitcoind


Both are accepting a maximum of 8 connections + both are not accepting incoming connections*
Bitshares also runs an http daemon (sure, it's technically nothing) but an extra 400~ MB in ram is acceptable (swap is set to 0 in this machine)

* - Don't worry, I have a different machine for seeding Bitcoin and Bitshares. I'm doing my part ;)

--

Oh, if you're interested what the "research" machine's stats are:

$ ps -xv | grep bitshares
 2341 pts/1    Sl+    6:02  24446 34648 2913743 1160992 14.1 bitshares_client --statistics-enabled --server --httpport 9989


$ ps -xv | grep bitcoind
11852 ?        SLsl  71:12   5032  4865 2020482 892552 10.8 bitcoind


* - This still isn't the seed machine. Both instances are full indexing nodes. Bitshares also runs the http daemon. Both are connected to 8 peers. No incoming connections.

--

Oh! I forgot to add, bitcoind listens as an RPC server in both instances.
Title: Re: Why does the bitshares_client use 10x the memory of bitcoind?
Post by: julian1 on May 26, 2015, 10:26:42 am


1 massive level DB database would still have a random access pattern and make every lookup take longer (the cost per lookup grows with log(n) the number of items). 

A memory mapped btree still requires disk IO (unless it is all cached).   Representing the data in a form that is suitable for storing on disk also slows down access. 

We load everything into memory on startup and keep them in boost::multi_index_container which allow both hashed and sorted indexes on multiple fields.  So lookup is constant time and updates involve mere pointer manipulation.    It takes mere seconds to load everything from disk to memory and then to save it again.   For now we are loading and saving to/from a Level DB database, but in the future it will probably just be a set of flat files.    Depending upon the type, we can get away with a "stack" based structure in a flat file.

Thanks. That sounds like a really good simplification and with a great performance payoff. I played with indexing bitcoin transactions using in-memory structures, and found it really quick (eg. 90 mins for utxo set 1.7 GB ram).