Author Topic: [WorkerProposal]Data solution:data structured,flexible API and fast-built method  (Read 650 times)

0 Members and 1 Guest are viewing this topic.

Offline Digital Lucifer

  • Global Moderator
  • Full Member
  • *****
  • Posts: 123
  • 13 years of being Slackware abUser
    • View Profile
    • Bitshares DEX Thailand
  • BitShares: steem-not
  • GitHub: dls-cipher
If you want to get how much asset was issued in a period of time you can filter the elasticsearch operations by operation_type 14(ASSET ISSUE) and by  operation_history.op_object.asset_to_issue.asset_id in a period of time, please check this link:
http://148.251.96.48:5601/app/kibana#/discover?_g=()&_a=(columns:!(_source),index:'357b9f60-d5f8-11e8-bb51-9583fd938437',interval:auto,query:(language:lucene,query:'operation_type:%2014'),sort:!(block_data.block_time,desc))

If you want to go after the feed prices of a smart asset you can filter by operation type 19 and smart asset id: http://148.251.96.48:5601/app/kibana#/discover?_g=()&_a=(columns:!(_source),index:'357b9f60-d5f8-11e8-bb51-9583fd938437',interval:auto,query:(language:lucene,query:'operation_type:%2019%20AND%20%20operation_history.op_object.asset_id:%201.3.113'),sort:!(block_data.block_time,desc))

In both kibana links you can change the timeframe in the upper right.

If you want to get market price changes you can go after the fill order and so on.

Also, when operations are not enough there is the es_objects plugin that will allow to export certain(currently predefined) specific objects; however i am making some changes to try to make it work with any blockchain object so for example you could get the internal ticker objects for a pair and get the current and past market prices from there.

I strongly believe that those 2 plugins are in the same direction you want to go.

ES/Kibana plugin/UI has BEEN DEVELOPED, TESTED and have very GOOD and ON-GOING development/maintenance atm, 100% agree.

My personal and professional opinion on this would be: Someone stepped up already earlier and now we don't need another proposal that is "re-inventing the wheel" without proper diligence done towards achievements/implementations done in our eco-system.

Many thanks.

CheeĀ®s
« Last Edit: November 07, 2018, 05:19:34 am by Digital Lucifer »
Milos (Mike) Preocanin - General Manager @ Syntek Solutions
TOANDI Co., LTD. (BOI Approved) - TAX ID: 0205549016913 - 95/5 Moo 4 Siam Country Club Rd.
Nong Prue, Bang Lamung, Chonburi 20250, Thailand.

Offline vianull

  • Jr. Member
  • **
  • Posts: 23
    • View Profile
  • BitShares: vianull
Thank you for all your suggestions . I think we need to reassess our proposal and think it through.

Offline Digital Lucifer

  • Global Moderator
  • Full Member
  • *****
  • Posts: 123
  • 13 years of being Slackware abUser
    • View Profile
    • Bitshares DEX Thailand
  • BitShares: steem-not
  • GitHub: dls-cipher
In general i don't agree with this worker, here are some reasons:

- Cost. It cost almost 300k usd(288000 usd to be exact to do it).
- Limited. It is database specific, a new worker may came in saying mongodb or whatever is better for their specific needs.
- Closed to participate. No one except the bts.ai team can participate in the development.
- Closed source. It don't says anything that all the work will be open source and MIT license. Will bts.ai be open source and inside the bitshares organization ?
- Reinvent the wheel. The Elasticsearch plugin is working great and have all the data needed, the synchronization time is of 20 hours according to a last report, all the data inside operations is structured and available. That cost 0 to bitshares as it is already done. Doing the same from scratch using another database by a new team is IMHO a waste of resources. The core team, with the accumulated experience can do a postgres plugin in extremely reduced time if that is what the community needs. Also, the core team can pay a team or individual to do the plugin as plugins are core work and will need review and approval from core team members.
- Benefit. Besides having some better visualizations of some data which i think is important i don't see any other real benefit of the proposal.

In my opinion i will like to see some day a general worker that will do this and others in a bounty style, most of the API links mentioned are not being developed because there is  no funding to get developers on board. There is already a ruby project for bitshares at https://github.com/MatzFan/bitshares-ruby not being improved because of lack of funding, among many other dead projects.

I think that bitshares needs a worker proposal similar to the core worker where teams and individuals can participate in the development of different tools of the bitshares ecosystem.

It honestly looks to me like reinventing the wheel after all the work it has been done in this particular regard, discard everything and start from scratch instead of build on top of previous tools to save time, resources and advance.

Just my personal opinion, i don't have any voting power or influence to decide what is accepted or not.


The important thing is NOT PostgreSQL or ES or Mongo or anything . What we care most about is how to implement the requirment .

such as:
1) How to draw charts using specified data in a certain period, e.g. the average price of a market pair, the issue/burn number of an asset, and the feed price of a smart asset.
All these stuffs are unstructured by ES currently, which take some problems to our implements.
2) Bitshares is an exchange such that there may be more professional approach to the storage and analysis the transaction data in exchange viewpoint.

As mentioned in this worker, we will open source under MIT. If this worker is validated, then all future data APIs on bts.ai will be open source, which has been part of this worker.

2) BitShares is a Blockchain who have much brighter and wider future than just an Exchange.
Milos (Mike) Preocanin - General Manager @ Syntek Solutions
TOANDI Co., LTD. (BOI Approved) - TAX ID: 0205549016913 - 95/5 Moo 4 Siam Country Club Rd.
Nong Prue, Bang Lamung, Chonburi 20250, Thailand.

Offline oxarbitrage

Please don't get me wrong, i do think is a good idea to have ways to visualize the data with good graphics, API calls and others.

However, the worker do not consider there is already a framework available to do it, can be communication problems, last changes to make it all possible are relatively new, etc.

Another issue i have with this worker is the price, even if you have to make it all from scratch i think the worker price is at least 3 times above common sense.

I will be happy to support a worker that consider this 2 points.