Main > Stakeholder Proposals
[WorkerProposal]Data solution:data structured,flexible API and fast-built method
Digital Lucifer:
--- Quote from: oxarbitrage on November 06, 2018, 03:15:59 pm ---If you want to get how much asset was issued in a period of time you can filter the elasticsearch operations by operation_type 14(ASSET ISSUE) and by operation_history.op_object.asset_to_issue.asset_id in a period of time, please check this link:
http://148.251.96.48:5601/app/kibana#/discover?_g=()&_a=(columns:!(_source),index:'357b9f60-d5f8-11e8-bb51-9583fd938437',interval:auto,query:(language:lucene,query:'operation_type:%2014'),sort:!(block_data.block_time,desc))
If you want to go after the feed prices of a smart asset you can filter by operation type 19 and smart asset id: http://148.251.96.48:5601/app/kibana#/discover?_g=()&_a=(columns:!(_source),index:'357b9f60-d5f8-11e8-bb51-9583fd938437',interval:auto,query:(language:lucene,query:'operation_type:%2019%20AND%20%20operation_history.op_object.asset_id:%201.3.113'),sort:!(block_data.block_time,desc))
In both kibana links you can change the timeframe in the upper right.
If you want to get market price changes you can go after the fill order and so on.
Also, when operations are not enough there is the es_objects plugin that will allow to export certain(currently predefined) specific objects; however i am making some changes to try to make it work with any blockchain object so for example you could get the internal ticker objects for a pair and get the current and past market prices from there.
I strongly believe that those 2 plugins are in the same direction you want to go.
--- End quote ---
ES/Kibana plugin/UI has BEEN DEVELOPED, TESTED and have very GOOD and ON-GOING development/maintenance atm, 100% agree.
My personal and professional opinion on this would be: Someone stepped up already earlier and now we don't need another proposal that is "re-inventing the wheel" without proper diligence done towards achievements/implementations done in our eco-system.
Many thanks.
Chee®s
Fox:
- 1.1 Gather data from external APIs of CEX
- Support publishing open source tools/process to collect data from CEX
- Support collection of public data to be published as an open source dataset (may not meet EULA/TOS of provider(s))
- See [Bitshares-Core/1350](https://github.com/bitshares/bitshares-core/issues/1350): NVT Data Collection and Visualization
- 1.2 Data visualization
- Support the concept.
- Provided already by [Kibana](https://www.elastic.co/products/kibana) today (visualization tool for Elasticsearch)
- 2 Multi-dimensional query and data export from DEX
- Support.
- Provided already by Elasticsearch
- See [Bitshares-Core/1350](https://github.com/bitshares/bitshares-core/issues/1350): NVT Data Collection and Visualization
- 3 Data backup service
- Support the concept. Have open questions about the implementation:
- How does this differ from a seed node?
- If not seed node How can service provider attest data authenticity (trust required)
- Opinion: Best if block producers snapshot, attest and post to bitshares.org (caveat: loose history data)
- 4 Address identification and classification
- Concerned this may lead to censorship
- 5 Analysis of Witnesses (feeds, nodes, etc.)
- Support the concept
Suggestions for Other:
- Replace focus on Postgresql Plugin in favor of ZeroMQ implementation
- Focus on building out APIs and wrappers (using Elasticsearch) that UI Team can build from
- DevOp tools for node operators (seed, api, block producers)
I look forward to iterating on a proposal that advances development and adoption of the BitShares platform.
Best,
Fox
oxarbitrage:
If you want to get how much asset was issued in a period of time you can filter the elasticsearch operations by operation_type 14(ASSET ISSUE) and by operation_history.op_object.asset_to_issue.asset_id in a period of time, please check this link:
http://148.251.96.48:5601/app/kibana#/discover?_g=()&_a=(columns:!(_source),index:'357b9f60-d5f8-11e8-bb51-9583fd938437',interval:auto,query:(language:lucene,query:'operation_type:%2014'),sort:!(block_data.block_time,desc))
If you want to go after the feed prices of a smart asset you can filter by operation type 19 and smart asset id: http://148.251.96.48:5601/app/kibana#/discover?_g=()&_a=(columns:!(_source),index:'357b9f60-d5f8-11e8-bb51-9583fd938437',interval:auto,query:(language:lucene,query:'operation_type:%2019%20AND%20%20operation_history.op_object.asset_id:%201.3.113'),sort:!(block_data.block_time,desc))
In both kibana links you can change the timeframe in the upper right.
If you want to get market price changes you can go after the fill order and so on.
Also, when operations are not enough there is the es_objects plugin that will allow to export certain(currently predefined) specific objects; however i am making some changes to try to make it work with any blockchain object so for example you could get the internal ticker objects for a pair and get the current and past market prices from there.
I strongly believe that those 2 plugins are in the same direction you want to go.
vianull:
--- Quote from: oxarbitrage on November 05, 2018, 06:16:54 pm ---In general i don't agree with this worker, here are some reasons:
- Cost. It cost almost 300k usd(288000 usd to be exact to do it).
- Limited. It is database specific, a new worker may came in saying mongodb or whatever is better for their specific needs.
- Closed to participate. No one except the bts.ai team can participate in the development.
- Closed source. It don't says anything that all the work will be open source and MIT license. Will bts.ai be open source and inside the bitshares organization ?
- Reinvent the wheel. The Elasticsearch plugin is working great and have all the data needed, the synchronization time is of 20 hours according to a last report, all the data inside operations is structured and available. That cost 0 to bitshares as it is already done. Doing the same from scratch using another database by a new team is IMHO a waste of resources. The core team, with the accumulated experience can do a postgres plugin in extremely reduced time if that is what the community needs. Also, the core team can pay a team or individual to do the plugin as plugins are core work and will need review and approval from core team members.
- Benefit. Besides having some better visualizations of some data which i think is important i don't see any other real benefit of the proposal.
In my opinion i will like to see some day a general worker that will do this and others in a bounty style, most of the API links mentioned are not being developed because there is no funding to get developers on board. There is already a ruby project for bitshares at https://github.com/MatzFan/bitshares-ruby not being improved because of lack of funding, among many other dead projects.
I think that bitshares needs a worker proposal similar to the core worker where teams and individuals can participate in the development of different tools of the bitshares ecosystem.
It honestly looks to me like reinventing the wheel after all the work it has been done in this particular regard, discard everything and start from scratch instead of build on top of previous tools to save time, resources and advance.
Just my personal opinion, i don't have any voting power or influence to decide what is accepted or not.
--- End quote ---
The important thing is NOT PostgreSQL or ES or Mongo or anything . What we care most about is how to implement the requirment .
such as:
1) How to draw charts using specified data in a certain period, e.g. the average price of a market pair, the issue/burn number of an asset, and the feed price of a smart asset.
All these stuffs are unstructured by ES currently, which take some problems to our implements.
2) Bitshares is an exchange such that there may be more professional approach to the storage and analysis the transaction data in exchange viewpoint.
As mentioned in this worker, we will open source under MIT. If this worker is validated, then all future data APIs on bts.ai will be open source, which has been part of this worker.
oxarbitrage:
In general i don't agree with this worker, here are some reasons:
- Cost. It cost almost 300k usd(288000 usd to be exact to do it).
- Limited. It is database specific, a new worker may came in saying mongodb or whatever is better for their specific needs.
- Closed to participate. No one except the bts.ai team can participate in the development.
- Closed source. It don't says anything that all the work will be open source and MIT license. Will bts.ai be open source and inside the bitshares organization ?
- Reinvent the wheel. The Elasticsearch plugin is working great and have all the data needed, the synchronization time is of 20 hours according to a last report, all the data inside operations is structured and available. That cost 0 to bitshares as it is already done. Doing the same from scratch using another database by a new team is IMHO a waste of resources. The core team, with the accumulated experience can do a postgres plugin in extremely reduced time if that is what the community needs. Also, the core team can pay a team or individual to do the plugin as plugins are core work and will need review and approval from core team members.
- Benefit. Besides having some better visualizations of some data which i think is important i don't see any other real benefit of the proposal.
In my opinion i will like to see some day a general worker that will do this and others in a bounty style, most of the API links mentioned are not being developed because there is no funding to get developers on board. There is already a ruby project for bitshares at https://github.com/MatzFan/bitshares-ruby not being improved because of lack of funding, among many other dead projects.
I think that bitshares needs a worker proposal similar to the core worker where teams and individuals can participate in the development of different tools of the bitshares ecosystem.
It honestly looks to me like reinventing the wheel after all the work it has been done in this particular regard, discard everything and start from scratch instead of build on top of previous tools to save time, resources and advance.
Just my personal opinion, i don't have any voting power or influence to decide what is accepted or not.
Navigation
[0] Message Index
[#] Next page
[*] Previous page
Go to full version