Author Topic: Why does the bitshares_client use 10x the memory of bitcoind?  (Read 3789 times)

0 Members and 1 Guest are viewing this topic.

julian1

  • Guest


1 massive level DB database would still have a random access pattern and make every lookup take longer (the cost per lookup grows with log(n) the number of items). 

A memory mapped btree still requires disk IO (unless it is all cached).   Representing the data in a form that is suitable for storing on disk also slows down access. 

We load everything into memory on startup and keep them in boost::multi_index_container which allow both hashed and sorted indexes on multiple fields.  So lookup is constant time and updates involve mere pointer manipulation.    It takes mere seconds to load everything from disk to memory and then to save it again.   For now we are loading and saving to/from a Level DB database, but in the future it will probably just be a set of flat files.    Depending upon the type, we can get away with a "stack" based structure in a flat file.

Thanks. That sounds like a really good simplification and with a great performance payoff. I played with indexing bitcoin transactions using in-memory structures, and found it really quick (eg. 90 mins for utxo set 1.7 GB ram).

Offline arubi

  • Sr. Member
  • ****
  • Posts: 209
    • View Profile
Ugh.

$ ps -xv | grep bitshares
 2797 pts/6    Sl+   18:01   4478 33385 1064798 955000 46.4 bitshares_client --server --httpport 9989


$ ps -xv | grep bitcoind
 6659 ?        Ssl  112:09   1230  4509 742338 595268 28.9 bitcoind


Both are accepting a maximum of 8 connections + both are not accepting incoming connections*
Bitshares also runs an http daemon (sure, it's technically nothing) but an extra 400~ MB in ram is acceptable (swap is set to 0 in this machine)

* - Don't worry, I have a different machine for seeding Bitcoin and Bitshares. I'm doing my part ;)

--

Oh, if you're interested what the "research" machine's stats are:

$ ps -xv | grep bitshares
 2341 pts/1    Sl+    6:02  24446 34648 2913743 1160992 14.1 bitshares_client --statistics-enabled --server --httpport 9989


$ ps -xv | grep bitcoind
11852 ?        SLsl  71:12   5032  4865 2020482 892552 10.8 bitcoind


* - This still isn't the seed machine. Both instances are full indexing nodes. Bitshares also runs the http daemon. Both are connected to 8 peers. No incoming connections.

--

Oh! I forgot to add, bitcoind listens as an RPC server in both instances.
« Last Edit: May 25, 2015, 10:17:41 pm by arubi »

Offline abit

  • Committee member
  • Hero Member
  • *
  • Posts: 4664
    • View Profile
    • Abit's Hive Blog
  • BitShares: abit
  • GitHub: abitmore
When talking about disk IO bottleneck, we always check I and O (reads and writes) separately. We can't avoid IO, we just reduce the frequency, at lease reduce random accessing.
There are still rooms available for performance optimizing with the use of leveldb, E.G.
* The optional FilterPolicy mechanism can be used to reduce the number of disk reads substantially.
* Using less quantity of level DB's will hopefully reduce competing on writes/compacts.
* Better design of keys (of values)
* Compressing and caching

Anyway, if we've already chosen another way, please go ahead.
BitShares committee member: abit
BitShares witness: in.abit

Offline bytemaster

I'm intensely curious to know what choices have been made here. 

Possibilities I thought off include consolidating all the leveldb databases by prefixing the leveldb keys with namespaces.

Or alternatively using a memory-mapped data-structure (btree) implementation.

1 massive level DB database would still have a random access pattern and make every lookup take longer (the cost per lookup grows with log(n) the number of items). 

A memory mapped btree still requires disk IO (unless it is all cached).   Representing the data in a form that is suitable for storing on disk also slows down access. 

We load everything into memory on startup and keep them in boost::multi_index_container which allow both hashed and sorted indexes on multiple fields.  So lookup is constant time and updates involve mere pointer manipulation.    It takes mere seconds to load everything from disk to memory and then to save it again.   For now we are loading and saving to/from a Level DB database, but in the future it will probably just be a set of flat files.    Depending upon the type, we can get away with a "stack" based structure in a flat file. 
For the latest updates checkout my blog: http://bytemaster.bitshares.org
Anything said on these forums does not constitute an intent to create a legal obligation or contract between myself and anyone else.   These are merely my opinions and I reserve the right to change them at any time.

Offline sittingduck

  • Sr. Member
  • ****
  • Posts: 246
    • View Profile

I'm intensely curious to know what choices have been made here. 

Possibilities I thought off include consolidating all the leveldb databases by prefixing the leveldb keys with namespaces.

Or alternatively using a memory-mapped data-structure (btree) implementation.

If disk io is the issue then neither of your suggestions would work.   


Sent from my iPhone using Tapatalk

julian1

  • Guest
I'm intensely curious to know what choices have been made here. 

Possibilities I thought off include consolidating all the leveldb databases by prefixing the leveldb keys with namespaces.

Or alternatively using a memory-mapped data-structure (btree) implementation.

Offline mangou007


This is all old news anyway, soon BitShares will be using less RAM than your web browser.

Don't know why, but I am loving to see things like that :)  +5%
BitShares French ConneXion, le portail francophone de BitShares
BitShares French ConneXion, the BitShares French gateway
www.bitsharesfcx.com

Offline karnal

  • Hero Member
  • *****
  • Posts: 1068
    • View Profile
This is all old news anyway, soon BitShares will be using less RAM than your web browser.

Really BM, now you mention it? I literally bought 8GB of RAM to be able to run the bitshares client without killing a few programs & swapping madly.

Seriously.

 ;D

Offline testz

It uses alot more indices than bitcoind ...
Recall that the BitShares client has a BUILT-IN block explorer!

Why do the indices need to be RAM resident?

The bottle neck for BitShares is Disk IO.   BitShares uses a lot of RAM in an attempt to minimize Disk IO.   

Bitcoin has just a few LevelDB databases, BitShares has 2 dozen.    Each LevelDB has its own cache.

This is all old news anyway, soon BitShares will be using less RAM than your web browser.

Oh, no, my web browser use 7 GB of RAM, let's say that "BitShares will be using less RAM than build-in calc"  :)

Offline monsterer

Even with 16 GB of RAM disk io is the bottleneck.

My point was that with substantially less RAM than your development machines have (for example 2GB on a win7 machine), plain disk IO is unlikely to be the bottleneck... I'd expect it to be related to the swapping necessary to maintain the vast array of caches you mentioned.
My opinions do not represent those of metaexchange unless explicitly stated.
https://metaexchange.info | Bitcoin<->Altcoin exchange | Instant | Safe | Low spreads

Offline bytemaster

The bottle neck for BitShares is Disk IO.   BitShares uses a lot of RAM in an attempt to minimize Disk IO.   

Bitcoin has just a few LevelDB databases, BitShares has 2 dozen.    Each LevelDB has its own cache.

This is all old news anyway, soon BitShares will be using less RAM than your web browser.

That bottle neck is probably the true bottle neck only for machines which aren't forced to swap the caches out to disk due to 'lower' real RAM than your development machines.

However, I am glad to hear you're already on it :)

Even with 16 GB of RAM disk io is the bottleneck. 
For the latest updates checkout my blog: http://bytemaster.bitshares.org
Anything said on these forums does not constitute an intent to create a legal obligation or contract between myself and anyone else.   These are merely my opinions and I reserve the right to change them at any time.

Offline monsterer

The bottle neck for BitShares is Disk IO.   BitShares uses a lot of RAM in an attempt to minimize Disk IO.   

Bitcoin has just a few LevelDB databases, BitShares has 2 dozen.    Each LevelDB has its own cache.

This is all old news anyway, soon BitShares will be using less RAM than your web browser.

That bottle neck is probably the true bottle neck only for machines which aren't forced to swap the caches out to disk due to 'lower' real RAM than your development machines.

However, I am glad to hear you're already on it :)
My opinions do not represent those of metaexchange unless explicitly stated.
https://metaexchange.info | Bitcoin<->Altcoin exchange | Instant | Safe | Low spreads

Offline bytemaster

It uses alot more indices than bitcoind ...
Recall that the BitShares client has a BUILT-IN block explorer!

Why do the indices need to be RAM resident?

The bottle neck for BitShares is Disk IO.   BitShares uses a lot of RAM in an attempt to minimize Disk IO.   

Bitcoin has just a few LevelDB databases, BitShares has 2 dozen.    Each LevelDB has its own cache.

This is all old news anyway, soon BitShares will be using less RAM than your web browser.
For the latest updates checkout my blog: http://bytemaster.bitshares.org
Anything said on these forums does not constitute an intent to create a legal obligation or contract between myself and anyone else.   These are merely my opinions and I reserve the right to change them at any time.

Offline monsterer

It uses alot more indices than bitcoind ...
Recall that the BitShares client has a BUILT-IN block explorer!

Why do the indices need to be RAM resident?
My opinions do not represent those of metaexchange unless explicitly stated.
https://metaexchange.info | Bitcoin<->Altcoin exchange | Instant | Safe | Low spreads

Offline xeroc

  • Board Moderator
  • Hero Member
  • *****
  • Posts: 12922
  • ChainSquad GmbH
    • View Profile
    • ChainSquad GmbH
  • BitShares: xeroc
  • GitHub: xeroc
The high memory usage is probably a huge factor in the bad performance of the bitshares client. Why does it need so much RAM compared to bitcoind?
It uses alot more indices than bitcoind ...
Recall that the BitShares client has a BUILT-IN block explorer!