Author Topic: [Worker Proposal]Data Visualization:Investigation, Design and Implement  (Read 4132 times)

0 Members and 1 Guest are viewing this topic.

Offline zapata42

  • Full Member
  • ***
  • Posts: 62
    • View Profile
  • BitShares: zapata42
Is there a plan for a bounty process as in the core and UI proposal, so external freelancers can help you?

Offline pc

  • Hero Member
  • *****
  • Posts: 1530
    • View Profile
    • Bitcoin - Perspektive oder Risiko?
  • BitShares: cyrano
In order to achieve the above objectives, our plan is:
    1. Investigate visualization requirements.
    Questionnaires will be launched on multiple sources such as WeChat, Forum, and Telegram. The requirements of BTS users for data visualization will be widely collected and analyzed for their rationality (see “User Demand Analysis” for current collections).

I find it strange that you list this investigation as part of the tasks you're going to fulfill, yet at the same time you list all the tasks needed to fulfill these requirements *and* you're able to put a price tag on it.
Bitcoin - Perspektive oder Risiko? ISBN 978-3-8442-6568-2

Offline vianull

  • Full Member
  • ***
  • Posts: 91
    • View Profile
  • BitShares: vianull
Thanks to all the suggestions in
We seriously discussed and rethough this proposal, and finally decided to propose a new one. Our main purpose is to "visualize" and to analyze the data on the chain without being entangled in the underlying implementation. In the new solution, we will take the ES solution as first choice to obtain the data on chain. In some special cases, we will consider using PostgreSQL to supplement it.
We are  team
Witness:  witness.hiblockchain
Standby Committee: btsai

Offline vianull

  • Full Member
  • ***
  • Posts: 91
    • View Profile
  • BitShares: vianull
BTS Worker 提案 - 数据可视化:调研,设计和实现
BTS Worker Proposal - Data Visualization:Investigation, Design and Implement



     Bitshares has been running stably for more than three years, and has accumulated 110W+ accounts, 3k+ user issued assets (UIA), and 200 million+ historical account data. As a decentralized exchange, BTS has achieved great progress for its high performance, privacy and flexibility. However, there are still some drawbacks to overcome.
     After investigated the current BTS blockchain browsers and data analysis products, we found that it lacks of a tool to in-depth analyze and visualize asset data and account activity in BTS community. We cannot intuitively identify high-quality assets or understand certain behavior of an account, and so on. The on-chain historical data is like a gold mine waiting to be discovered. In addition, a customized export tool of full historical operation records is also very important for users. Users can back up their historical operation data locally. On the other hand, they can perform additional analysis and processing according to their own requirements.
     For traditional developers, a complete set of on-chain data query and statistical results API is crucial for BTS blockchain based development. However, the API implementation schemes in the community are different such that there is no uniform standard. Moreover, the existing API solution does not collect the transaction data of CEX and DEX for overall analysis.
    To solve the above questions, we propose this worker to do our contributions for BTS community.

1)明确的可视化需求。根据我们的理解,已经列出了十余项的可视化需求。用户需求是本worker 的出发点与目标。
2)丰富实用的 API 接口,涵盖传统与所有可视化需求的接口,同时满足更深层次、定制化的数据查询需求。
To solve the above problems, we propose and will implement a solution about data visualization, API and data export solutions, including:
1) Clear visual requirements. According to our understanding, more than ten visualization requirements have been listed to meet user requirements.
2) A rich and practical API that covers the traditional and all visualization requirements as well as meet deeper, customized data query needs.
3) Data visualization Demo. Part of the Demo will be implemented for data visualization based on the above API.
4) Open source code. The API and Demo source code involved in the worker will open to the outside world using the MIT protocol, which is convenient for the community to improve it further.

    从微信、论坛、Telegram 等多个渠道发起问卷。广泛收集并分析BTS用户对数据可视化的需求,并分析其合理性(目前已经整理的需求见“用户需求分析”)。
    a)选择合适的数据存储方案。 针对具体的可视化需求选用合适的数据方案,包括但不限于 ES Plugin , PostgreSQL做部分数据支撑。
    a) 传统的Bitshares API 接口,包含目前的基础功能
    b) 可视化的数据 API,包含所有本次要做的可视化功能
    4.实现可视化 DEMO
    实现可视化需求的 DEMO,主要使用 echart.js,个别可能会采用d3.js,为其他开发人员提供参考
Our plan
In order to achieve the above objectives, our plan is:
    1. Investigate visualization requirements.
    Questionnaires will be launched on multiple sources such as WeChat, Forum, and Telegram. The requirements of BTS users for data visualization will be widely collected and analyzed for their rationality (see “User Demand Analysis” for current collections).
    2. Data storage and collection scheme
    a) Choose a proper data storage solution. Choose the good data solution for the specific visualization requirements, including but not limited to ES Plugin, PostgreSQL for partial data support.
    b) Design an external data collection scheme. The corresponding data collection method will be deployed according to different CEX. We will normalize the data before storage from different sources.
    3. Design and implement the API.
    a) Traditional Bitshares API interface, including current basis function
    b) Data visualization API, including all the visualization features to be done in this proposal
    4. Visualize DEMO
    Implement the visualization requirement DEMO, mainly using echart.js,  maybe use d3.js, the code will provide as a  reference for other developers

1. User requirements analysis

1.1. Support BTS data from centralized exchange (exchanges that provide data api)
     The blockchain is a decentralized global facility, but currently, most of the virtual currency transactions are generated on centralized exchanges, which results in highly diversified transaction data.  It is very difficult to evaluate the actual volume of transactions per day. By importing the CEX BTS transaction data, We can comprehensively analyze the data from CEX and DEX and provide more valuable and reliable analyze results. We will develop the following functions:

a. Summary of daily trading volume on major CEXs and DEX

In the above figure, we hope to show the daily trading volume of BTSs on different exchanges by the stacked histogram.  Ordinary users can analyze the daily trading volume at each CEXs and DEX. The whole trading volume is shown as well.
b、各大交易所及内盘交易对活跃度 (每秒交易数)
b. Trading activeness on major CEXs and DEX (number of transactions per second)

The above figure can show the relationship between number of transactions and time. It implies the daily trading activeness in each exchange, which is beneficial for ordinary users. Users can select exchange based on this chart. This figure is very helpful to analyze trading habits of different exchange users.

c. Historical price on major CEXs and DEX

As shown in the above figure, the BTS price comparison of different exchanges can indicate user habits and lowest price, which is beneficial for ordinary users.
d. Collection of trading depth on major CEXs and DEX

We will combine the market depth on major CEXs and DEX, so that it is easy to show the total depth of the current BTS market. Large value order may be benefited from this figure. Trading analysis can be done according to this figure as well.

1.2. Asset visualization analysis

Smart assets and customized assets are the advantages of BTS, but there are too many useless assets and trade markets. However, data visualization methods are very few. It is difficult for ordinary users to find qualified assets by themselves. The updated asset related webpage will add the following features:
a、智能资产供应量vs 时间图
a. Chart of smart asset supply vs time

智能资产供应量随时间的变化程度会反应用户抵押情绪,而其它智能资产的供应量(比如bitCNY和bitUSD) 会对BTS社区的投票和策略产生极大影响,它与历史数据的对比对于交易本身也非常有意义。
The above chart will show the user's debt sentiment. Furthermore, the supply of other smart assets (such as bitCNY and bitUSD) will impact the voting and strategy from BTS community. The comparison with historical data is very useful on sell/buy decision.

b.  Scatter chart on BitCNY, BitUSD's call price vs. number of debt asset

Scatter chart of BitCNY, BitUSD's call price vs. number of debted BTS can show the community’s debt sentiment. It also relates to the voting and strategy from BTS community.
c、支持活跃地址数(当天参与交易地址数) vs 时间图
                    休眠地址数(30天未交易) vs 时间图
                    新增持仓地址数 vs 时间图
c. Chart of active address number (the number of participating trading addresses per day) vs time
                     number of dormant addresses (30 days not traded) vs time
                     number of new users vs time

The number of active addresses reflects the trading enthusiasm of the DEX user. It also reflects whether current prices are acceptable. The number of position addresses and the number of new address are important indicators as well .
d. Ranking of position address

Positions are a very important indicator of asset distribution, and centralized assets are risky to trade. We will implement the rank of position address and account operation analysis as well.
e. Indicators on smart asserts

We well show the number of bitCNY, the debt number of bts, average number of bts vs bitCNY and median of bts vs bitCNY.
1.3、 账户分析
1.3.  Account analysis

a. Related account analyze according to historical trading data

根据历史交易信息, 找出与该账户互动最为活跃的账户列表, 包括但不限于转账操作, 注册操作等
Based on historical transaction information, find a list of accounts that are most active in interacting with an account, including but not limited to transfer operations, registration operations, etc.

b、分交易类型的账户历史交易量统计, 包括不同交易类型交易量以及占比
b. Account historical transaction volume statistics, including transaction volume and volume/ratio of different transaction types

对账户交易类型进行统计分析, 可以有效分析账户的操作偏好, 根据挂单量/成交量可以推断该账户是否为自动交易机器人等
Statistical analysis of the account transaction type, it can effectively show the operating habits of the account. According to the amount of pending orders/volume, it can be inferred whether the account is an automatic trading robot.

c. Draw a curve about account’s asset amount vs time

分析账户资产的持仓情况, 可以看出该账户对此资产的偏好程度
Analysis the position of account assets, one can see the user's preference of this assert.

1.4. Data visualization analyze

We are going to import CEX data, and then analysis according to these aspects.

a. Trading volume per day of the current market
b. Comparison with last day (Volume, Maximum transaction value, Number of active address)
c. Analysis of block trading accounts

Analyze the number of transactions in a period of time, an account list will be shown according to the trading volume.

1.5. Analysis of Witness's Feeding Behavior

There is almost no visual monitoring on the witness's feeding price. However, the feeding price is an extremely important part of the BTS economic system. It plays a critical role on smart asserts. We plan to visualize the feeding price of each witness and the average feeding price as a reference.
a. Witness average update rate

Update speed of witness feed price is a necessary reference for users, we plan to display this data as a field on the page.

b. A chart about how far witness feed price deviates from the system feed price

1.6. Availability analysis of each access node

The reliability and stability of the access point can be seen from its availability analysis.

1.7、 数据导出
1.7. Data export

基于需求,我们或用户就可以基于 BTS 的成交数据做更深层次的交易分析。 包括,某时间段内的交易概况; 某账号某资产的交易均价; 某账号的盈亏分析等等。

There are many different types of user operation records on chain, such as pending orders, transaction matching, transfer, voting, and so on. Some users want to  get historical transaction records based on time intervals, transaction pairs, transaction types and other keywords, and provide transaction data export functions to facilitate backup or analysis through other software such as Excel.
Based on the data export, user can perform better transaction analysis based on the BTS transaction data. Including, the overview of the transaction within a certain period of time; the average transaction price of an account on one assert;  the profit and loss analysis of an account.

2. Others
We will add some other functions according to comments from the community.

All above underlying data will be supported by our open-source project. The above functions may be modified as community comments and development progress.

负责团队: 团队


Our team: team

We have been a mature team; the major developers own a lot of experience on website development.  We will hire more developers in the future to speed up the development.
Connect us:

主要开发者: vianull, Chen188
设计: Zheng
数据分析师: tiancaomei
产品负责人: vianull
运维负责人: wukoo
Our team members include:
Core Developer: vianull, Chen188
Designer: Zheng
Data Analyst: tiancaomei
Product Manager: vianull
Operation and Maintenance Manager: wukoo
Tester: jie

The updated plan will start on December 5, 2018. It will last for about 4 months and end on April 5, 2019. Developers will start work as soon as possible from December 5st. Each of the functions listed above will be tested before online. Production report will be published every month. The development progress will be announced on the website every week. If the function is stable, we will update the webpage online immediately.
It is expected that the development will be on schedule, but there may be a delay during development due to force majeure and inscrutability. If it happens, we will announce the reason on the website and then re-estimate developing period.


收到的资金将被托管在datasolution-worker-escrow账号里,该账号由 @bitcrab  @jademont @abit 2/3多签,托管人每4周将收到的资金转换成等价值的bitCNY 并将所有多余的 BTS 返回资金池, 就像中描述的那样。为防止 BTS 下跌至0.3bitCNY/BTS造成的风险,我们每天的预算为20000 BTS,多余的会返回资金池。
The workload of this project is very large. There is a lot of data to be sorted out, but we are well prepared for it. We ask to be funded according to the following table.

Our received fund will be hosted in the datasolution-worker-escrow account, which will be 2/3 signed by @bitcrab @jademont @abit. The escrow account will exchange the received BTS into bitCNY and return all excess BTS to the pool, as explained in (Under Escrow Worker Model). In case BTS price below 0.3 bitCNY/BTS, we ask for 20000 BTS per day. All excess BTS will return to the pool.
Because the software development can only be estimated beforehand, our team will also face some risks as well. The multi-signed escrow takes response for evaluating the quality of the released software. All payments will be open and transparent.
« Last Edit: December 21, 2018, 07:36:40 am by vianull »
We are  team
Witness:  witness.hiblockchain
Standby Committee: btsai