Author Topic: Professional Price Feeds for DEX  (Read 1350 times)

0 Members and 1 Guest are viewing this topic.

Offline sahkan

  • Full Member
  • ***
  • Posts: 188
    • View Profile
    • BitShares DEX
Professional Price Feeds for DEX
« on: April 16, 2017, 12:37:29 am »
Team,

First off, Happy Easter everyone!

My witness got voted in to the active witnesses list recently and I started to pay more attention to the price feeds.
Clearly we should put some governance in place to get some consistency. For example:
1. What is the list of required asset feeds for the witnesses? (Some do 6 some 22, some in between)
2. What is the minimum frequency for posting the feeds (1 hour?)
3. What are the required sources? (If any)

I hope we can get the witnesses involved (both active and standby) and work this out with community's input.

I also see a need for a professional feed, I think it would be a positive thing for DEX. Since the price feed script was not originally developed with Graphene, there are few scripts out there that do the job.
I would be willing to pay for an Enterprise level subscription Plan from a provider like https://currencylayer.com/ and provide the feed to all active witnesses if everyone was willing to update their scripts and come up with a price feed schedule that would provide consistent price updates on DEX.

As far as the implementation timeline, early May would be a good target date as I know some of the guys are busy at work.

Also, since we have an international community here and some witnesses are not fluent in English, please have this translated for them if they are your friends. Let me know your thoughts!

Offline Thom

Re: Professional Price Feeds for DEX
« Reply #1 on: April 16, 2017, 05:02:56 pm »
Glad to see your desire to improve the quality of the witness role.

These are not new topics, but it seems to be very difficult to get traction. I believe it's gonna require coming up with a BSIP and then getting community to recognize it.

There don't seem to be many witnesses willing to discuss such details, let alone take the initiative to write a BSIP to formalize the conventions. I also don't recall if there is a formal approval process BSIPs must go through by committee or whoever. I think they're more a matter of formalizing proposals in a document. BSIPs are good standards to have.

Some reform / evolution of the witness role will require coding, for example to implement standards that can be measured and automatically removing or disabling witnesses when those standards are violated, such as when X number of blocks are missed, or fails to publish feeds at a minimum frequency, or the feed accuracy slips outside an certain deviation from other witness feeds.

We have discussed a worker proposal to get professional feed sources, but they are quite expensive from what I gather. There is also the issue of centralization. That's one reason why no single feed script should be relied upon. To my knowledge there are 3 primary scripts most witnesses use, however the parameters each witness uses for them can be different based on that witness' influence. There are few standards on setting those parameters.

There is a Telegram bot created by witness spartaco to notify of missed blocks, set time slots for when during the hour a witness should publish their feed (use /slots in Telegram to list which minute is assigned to what witness. There are 3 groups that bot works in: BitsharesDEX, Bitshares_Witnesses and BTS Witness Alerts). Here are the commands the bot responds to:

Quote
/help
/price [ASSET]
/missed WITNESS
/monitor WITNESS
/listmonitor WITNESS
/stopmonitor WITNESS
/status
/feed [ASSET]
/slots
/setslot WITNESS MIN
/rmslot WITNESS
/locations
/setlocation WITNESS LOCATION
/rmlocation WITNESS

Please use the BTS Witness Alerts channel for urgent matters only. It's kindof like the "Hotline" to all witnesses. Witness alt recently posted in the forum saying several witnesses were broadcasting double blocks back to back. If true that would be a major issue and could cause forking of the blockchain. It turned out to be a false claim however the best place to communicate such things is via this real-time Telegram channel.

I would like to hear more about your offer to buy a subscription to a professional price feed source. Your offer is generous but I don't think we should continue to rely on altruism to support necessary blockchain operations to publish accurate price feeds.

Price feeds and block production require 2 different skill sets and I suggested these should be separate roles. The general attitude I see expressed however is that "producing a price feed isn't very complicated" and thus need not be split out. I have a different perspective which I've expressed in more detail elsewhere but it seems I'm alone in it. There does seem to be some agreement on the idea of automating some type of witness monitoring, but when it comes to saying what corrective action to take mostly I hear crickets.

It's also important to note these issues are of interest primarily to witnesses, and many of them are not interested in them enough to contribute to the discussion.

Injustice anywhere is a threat to justice everywhere - MLK |  Verbaltech2 Witness Reports: https://bitsharestalk.org/index.php/topic,23902.0.html

Offline sahkan

  • Full Member
  • ***
  • Posts: 188
    • View Profile
    • BitShares DEX
Re: Professional Price Feeds for DEX
« Reply #2 on: April 16, 2017, 05:56:04 pm »
Quote
(use /slots in Telegram to list which minute is assigned to what witness. There are 3 groups that bot works in: BitsharesDEX, Bitshares_Witnesses and BTS Witness Alerts)

The slots are assigned to 59 witnesses and we have only 21 active.

Here is the real price feed feed:

datasecuritynode   9 minutes ago   94.2   89.5   1750   1100
xman   11 minutes ago   93.4   88.7   1750   1100
xn-delegate   11 minutes ago   87.3   86.4   1750   1100
verbaltech2   12 minutes ago   88.6   84.2   1750   1100
sahkan-bitshares   15 minutes ago   88.5   84   1750   1100
delegate.freedom   15 minutes ago   87.8   87   1750   1100
bhuz   18 minutes ago   91.9   87.3   1750   1100
witness.yao   19 minutes ago   87.3   86.5   1750   1100
witness.still   20 minutes ago   87.3   86.5   1750   1100
jerryliu   20 minutes ago   87.2   86.4   1750   1100
delegate.ihashfury   21 minutes ago   89.1   84.7   1750   1100
abc123   24 minutes ago   89.1   84.7   1750   1100
harvey-xts   25 minutes ago   87   86.2   1750   1100
in.abit   26 minutes ago   94.1   89.4   1750   1100
delegate.baozi   26 minutes ago   86.9   86   1750   1100
xeldal   27 minutes ago   91.4   90.5   1750   1100
fox   28 minutes ago   88   83.6   1750   1100
rnglab   29 minutes ago   92.7   88   1750   1100
wackou   52 minutes ago   87.3   83   1750   1100
roadscape   1 hour ago   87.3   83   1750   1100
delegate-clayop   8 days ago   116.3   116.3   1750   1100


Quote
We have discussed a worker proposal to get professional feed sources, but they are quite expensive from what I gather. There is also the issue of centralization. That's one reason why no single feed script should be relied upon.

This should be our endstate where we can provide real time ticker from a professional feed to DEX. But I think if we can get witnesses to cooperate what I proposed might work. Single script is ok, as long as witnesses can add/use multiple sources, as an example we could use yahoo, google and 1 professional feed.

Quote
I would like to hear more about your offer to buy a subscription to a professional price feed source. Your offer is generous but I don't think we should continue to rely on altruism to support necessary blockchain operations to publish accurate price feeds.

I looked at feasibility of running a professional feed only through my witness and publishing it every 60 seconds but came to a conclusion that if we don't get a buy in from everyone it will not make sense. If everyone upgrades their script and publishes feeds so we have only 1-2 intervals in between (now we have sometimes 9-10 minutes) I will pay for the yearly subscription and provide the API key in the witness channel. Running a witness should not be considered a business, but rather a service to the community and responsibility for providing capable infrastructure for DEX. You should have your main business stream on DEX (for example building referrals) or somewhere else, but unfortunately I get an impression that some witness scale back on publishing feeds to save on fees, and their equipment is being maxed out on performance and not future ready so they can make money out of it. So if we can reach a consensus on this I will absorb the cost as long as they are willing to implement and fix their feed frequency. Of course if everyone thinks that current feeds are acceptable we can stay in status quo.



Offline rnglab

  • Full Member
  • ***
  • Posts: 171
    • View Profile
  • BitShares: rnglab
Re: Professional Price Feeds for DEX
« Reply #3 on: April 17, 2017, 10:09:37 am »
I wouldn't care about the /slots feature from spartako-bot. It was added by spartako almost instantly when Dan asked us to provide price feeds and xeroc's script was released.  As we started being just a few witnesses publishing feeds until different timezones and avaiability allowed everyone to update nodes, making the time span as distributed as possible was one of the first actions.

If that information were up to date and leaked, it would be easy for every feedsource API provider to know most witnesses IP addresses.

The blockchain takes prices from the majority of witnesses that "agrees" on a median, withing the feed_lifetime parameter  (if someone can add a more accurate description would be apreciated).
It has prooved to be very resilent against wrong feeds, and less dependant on publishing frequency.

this are the time parameters from xeroc's pricefeeds script:
Code: [Select]
################################################################################
# Publishing Criteria
################################################################################
#
# Publish price if any of these are valid:
#
#  1.) price feed is older than maxAgeFeedInSeconds
#  2.) price has moved from my last published price by more than change_min
#
# Do NOT publish if
#
#  * Price has moved more than 'change_max'
#     AND
#  * Manual confirmation is 'false'
#
# A feed becomes invalid if it is older than 24h. Hence, it should be force
# published once a day (e.g. every 12h) Note that the script should be run more
# frequently and publishes only those prices that fit the publishing criteria
maxAgeFeedInSeconds          = 12 * 60 * 60  # max age of 12h
change_min                   = 0.5       # Percentage of price change to force an update
change_max                   = 5       # Percentage of price change to cause a warning

a low change_max helps to prevent market manipulation, specially black swan events. It will stop publishing until manual confirmation, if the value is exceeded in the time span in which you run the script.
with high volatility is where you need to fin tune this parameters. A low change_max without frequently running your script will ask for manual confirmation as false alarm too often to handle.

On the other hand, most if not all Chinese feed sources has a DDOS protection that blacklists your ip with *many* less API calls than the specified. Sometimes you can make reverse DNS loockups to find direct connection to an API, sometimes they provide a private IP if you mail them, but in both cases the addresses change often and you'll have to find out what to put in your hosts file very often if blacklisted by Baidu.

Feeds can be published from a non producing witness with the active key.
If  someone has troubles reaching btc38 at this moment,  producing feeds from a node in China may be a solution.

Offline Thom

Re: Professional Price Feeds for DEX
« Reply #4 on: April 17, 2017, 02:25:59 pm »
IOn the other hand, most if not all Chinese feed sources has a DDOS protection that blacklists your ip with *many* less API calls than the specified.

Feeds can be published from a non producing witness with the active key.

Interesting. Would their DDoS protection kick in from 1 request an hour? I presume when you say "feed sources" you're talking about exchanges?

I don't believe your last point is correct. The active OR owner keys can be used to publish feeds, but the API call will fail unless the account calling the publish_feeds API is voted in (active). Don't get the state of being "active" / voted in confused with the type of key used on the account calling the API.

I would like to see some type of method for non-active witnesses to publish "informal" or "unofficial" price feeds to show shareholders they are capable and what their accuracy would be. It would also provide potential witnesses a way to perfect their scripts.
Injustice anywhere is a threat to justice everywhere - MLK |  Verbaltech2 Witness Reports: https://bitsharestalk.org/index.php/topic,23902.0.html

Offline Customminer

  • Hero Member
  • *****
  • Posts: 535
  • Bitshares FTW!
    • View Profile
    • Gridcoin.US
  • GitHub: grctest
Re: Professional Price Feeds for DEX
« Reply #5 on: April 17, 2017, 02:33:13 pm »
Ideally all witnesses should be publishing feeds.
Hertz, Beyond Bitshares, Gridcoin!

Offline rnglab

  • Full Member
  • ***
  • Posts: 171
    • View Profile
  • BitShares: rnglab
Re: Professional Price Feeds for DEX
« Reply #6 on: April 18, 2017, 06:45:14 am »
IOn the other hand, most if not all Chinese feed sources has a DDOS protection that blacklists your ip with *many* less API calls than the specified.

Feeds can be published from a non producing witness with the active key.

Interesting. Would their DDoS protection kick in from 1 request an hour? I presume when you say "feed sources" you're talking about exchanges?

Yes I was talking about exchanges in this case, but the same may apply for any Chineese feed source.
One request per hour is a lot of time to trigger DOS protection, I have just a few random rejects fetching prices every  5 or 10 minutes.

Quote
I don't believe your last point is correct. The active OR owner keys can be used to publish feeds, but the API call will fail unless the account calling the publish_feeds API is voted in (active). Don't get the state of being "active" / voted in confused with the type of key used on the account calling the API.

 I think I was not clear.  Signing keys are not required to publish feeds, active (elected) witnesses can feed prices from any node that has its active key. We can produce blocks from one node and publish feeds from another.
Quote


I would like to see some type of method for non-active witnesses to publish "informal" or "unofficial" price feeds to show shareholders they are capable and what their accuracy would be. It would also provide potential witnesses a way to perfect their scripts.

Agree. I think the simplest way would be to do it on testnet. It would also help to keep the public testnet allways distributed, not just when we need to run specific networking tests.

Offline rnglab

  • Full Member
  • ***
  • Posts: 171
    • View Profile
  • BitShares: rnglab
Re: Professional Price Feeds for DEX
« Reply #7 on: April 18, 2017, 08:57:41 am »
About rising witness quality standards (with emphasis on accurate pricefeeds), I like some of the ideas that have been discussed in the last months, but most of them would need heavy development, after a long way to reach consensus. 

At this stage I think our best trade off between priorities and resources would be to fund development for an open source block explorer, with an initial focus on witness monitoring tools (like historical pricefeed charts).

I like this approach from [member=22138]lafona[/member]: https://lafona.shinyapps.io/Feed_History/
Just adding a highlighted line with the median price published by the blockchain would make a good enough reference to start monitoring feeds accuracy. Maybe another line with asset prices from a professional paid source, as an off chain reference.

The lack of both an open source block explorer codebase, and a simple way to monitor witnesses, are top prioity issues that will probably result more expensive if we wait for someone to solve them for free some day, than if we just fund its development as a priority.
[member=11456]svk[/member] [member=11456]svk[/member] [member=11456]svk[/member]   :o

Offline Thom

Re: Professional Price Feeds for DEX
« Reply #8 on: April 19, 2017, 02:04:45 am »
The easiest solution in the short term is to use the testnet to perfect witness feeds / tools. However, without standards and conventions the data collected would be of limited value. For example, if there are no monitoring tools for price feed data (such as the chart you linked by lafona) from which comparisons can be made how can accuracy be measured?

Probably not difficult to do, but coordination of testnet use is also important. Doesn't make a lot of sense to be testing feeds if a stress test was underway.

As to your point about witnesses producing feeds with different keys, yes, you could use a different key. However, the witness associated with that key must be in the active state. Witness nodes can use an arbitrary key pair generated by the cli wallet to run their node, but only the signing key can be used to produce blocks.  I don't recall the specifics on feed publishing, but I suspect the only restriction is that the account open in the wallet which the script connects through must be a voted in witness account.
Injustice anywhere is a threat to justice everywhere - MLK |  Verbaltech2 Witness Reports: https://bitsharestalk.org/index.php/topic,23902.0.html

Offline rnglab

  • Full Member
  • ***
  • Posts: 171
    • View Profile
  • BitShares: rnglab
Re: Professional Price Feeds for DEX
« Reply #9 on: April 19, 2017, 05:28:13 am »
As to your point about witnesses producing feeds with different keys, yes, you could use a different key. However, the witness associated with that key must be in the active state. Witness nodes can use an arbitrary key pair generated by the cli wallet to run their node, but only the signing key can be used to produce blocks.  I don't recall the specifics on feed publishing, but I suspect the only restriction is that the account open in the wallet which the script connects through must be a voted in witness account.

Right. In other words, witnesses need two types of keys to produce blocks: the usual active key and a special key to sign blocks.
It's a good practice to distribute different (non-active) signing key pairs between failover nodes, because the 'update_witness' command allows changing the signing key for a given witness account. Thus we can switch block production between nodes without the need to restart a witness_node to change its signing keys.

Signing keys are no required to publish feeds. You can have one node producing blocks and another feeding prices, independently.

Offline Thom

Re: Professional Price Feeds for DEX
« Reply #10 on: April 20, 2017, 12:46:34 am »
As to your point about witnesses producing feeds with different keys, yes, you could use a different key. However, the witness associated with that key must be in the active state. Witness nodes can use an arbitrary key pair generated by the cli wallet to run their node, but only the signing key can be used to produce blocks.  I don't recall the specifics on feed publishing, but I suspect the only restriction is that the account open in the wallet which the script connects through must be a voted in witness account.

Right. In other words, witnesses need two types of keys to produce blocks: the usual active key and a special key to sign blocks.
It's a good practice to distribute different (non-active) signing key pairs between failover nodes, because the 'update_witness' command allows changing the signing key for a given witness account. Thus we can switch block production between nodes without the need to restart a witness_node to change its signing keys.

Signing keys are no required to publish feeds. You can have one node producing blocks and another feeding prices, independently.

The wallet used by the publish script needs to be unlocked for the script to use it to publish feeds. All 5 of my nodes can act as a block producer or seed node and all provide some degree of failover support. Next month I will be upgrading 2 of the 5 nodes to dedicated severs with 16GB of RAM.

All servers run an identical software and host configuration that allows any one of them to become the active block producer for verbaltech2. I always lock the wallet except for the active witness node so only 1 node is publishing feeds, that being the witness node. When block production is switched to a different node feed production begins there. It doesn't stop on the previously active witness node until I lock or stop the wallet.
Injustice anywhere is a threat to justice everywhere - MLK |  Verbaltech2 Witness Reports: https://bitsharestalk.org/index.php/topic,23902.0.html

Offline roelandp

  • Full Member
  • ***
  • Posts: 85
  • Witness, dad, kitesurfer, event organiser
    • View Profile
    • RoelandP.nl
  • BitShares: roelandp
  • GitHub: roelandp
Re: Professional Price Feeds for DEX
« Reply #11 on: April 20, 2017, 02:18:23 am »
Hey @sakhan in my Witness proposal you asked me to reply. I was already monitoring this chat, but a bit busy with the pricefeed scripts themselves atm :) Btw, this discussion has been talked about in the forums before, see for example with some interesting chats and opinions: https://bitsharestalk.org/index.php/topic,23607.0.html

How I feel about pricescripts and how they should come to life? First off, as the white paper [section 2.2] states, pricefeeds are important to have a fair price for settlements of Smartcoins on the Dex. It is important to have accurate information, but the software has some build in fault resistance as you can conclude from that paragraph.

I think setting up a trustworthy pricefeed for your witness means:
1. Sourcing from markets with enough volume.
2. Sourcing from a multitude of markets.
3. Have limits in place to prevent black swan events or sudden spikes / drops.
4. Adjust parameters to your reasoning and be open about your bias.

In general I am a huge fan of [member=120]xeroc[/member]'s work in python. He made me dive into python (finally). I have been testing his scripts, mainly his multi-source pricefeed script, which had some recent edits by [member=2186]iHashFury[/member] incorporated here.

IMHO [member=120]xeroc[/member]'s script does this and enables you to add as many sources as you like, it also has built in limits which should be monitored and therefore I am writing to expand the script to report to me on my telegram bot with manual verification or rejection of the feed publishing.

Having a paid for account for 1 (live) source won't change that much seeing GRAPHENE_DEFAULT_PRICE_FEED_LIFETIME in the config of 24 hours imho and especially it won't be a good idea to make the price-feeds part of the DEX software being dependent on it, as it would mean having a single point of failure.

Nonetheless I totally follow [member=21490]Thom[/member] to have some decent witness performance overview in place, like we have for Steem / Golos: https://steemdb.com and https://golosdb.com where it is easy to see who is missing blocks and how many, and who is slow on updating pricefeeds. This could help judge witnesses and actually vote / unvote them based on factual data and performance.

Alternatively we could all put trust in a machine script / AI and proxy your votes into that dedicated account who then votes / unvotes witnesses based on performance parameters to be set in a BSIP. In Steem it happens all the time once an active witness starts to miss blocks, and he/she is unresponsive, immediately he/she is voted out for the time being to keep the network at 100% participation. After the event is resolved and explained, the witness most always regains their votes.

Offline Pheonike

Re: Professional Price Feeds for DEX
« Reply #12 on: April 20, 2017, 03:50:32 am »
Hey @sakhan in my Witness proposal you asked me to reply. I was already monitoring this chat, but a bit busy with the pricefeed scripts themselves atm :) Btw, this discussion has been talked about in the forums before, see for example with some interesting chats and opinions: https://bitsharestalk.org/index.php/topic,23607.0.html

How I feel about pricescripts and how they should come to life? First off, as the white paper [section 2.2] states, pricefeeds are important to have a fair price for settlements of Smartcoins on the Dex. It is important to have accurate information, but the software has some build in fault resistance as you can conclude from that paragraph.

I think setting up a trustworthy pricefeed for your witness means:
1. Sourcing from markets with enough volume.
2. Sourcing from a multitude of markets.
3. Have limits in place to prevent black swan events or sudden spikes / drops.
4. Adjust parameters to your reasoning and be open about your bias.

In general I am a huge fan of [member=120]xeroc[/member]'s work in python. He made me dive into python (finally). I have been testing his scripts, mainly his multi-source pricefeed script, which had some recent edits by [member=2186]iHashFury[/member] incorporated here.

IMHO [member=120]xeroc[/member]'s script does this and enables you to add as many sources as you like, it also has built in limits which should be monitored and therefore I am writing to expand the script to report to me on my telegram bot with manual verification or rejection of the feed publishing.

Having a paid for account for 1 (live) source won't change that much seeing GRAPHENE_DEFAULT_PRICE_FEED_LIFETIME in the config of 24 hours imho and especially it won't be a good idea to make the price-feeds part of the DEX software being dependent on it, as it would mean having a single point of failure.

Nonetheless I totally follow [member=21490]Thom[/member] to have some decent witness performance overview in place, like we have for Steem / Golos: https://steemdb.com and https://golosdb.com where it is easy to see who is missing blocks and how many, and who is slow on updating pricefeeds. This could help judge witnesses and actually vote / unvote them based on factual data and performance.

Alternatively we could all put trust in a machine script / AI and proxy your votes into that dedicated account who then votes / unvotes witnesses based on performance parameters to be set in a BSIP. In Steem it happens all the time once an active witness starts to miss blocks, and he/she is unresponsive, immediately he/she is voted out for the time being to keep the network at 100% participation. After the event is resolved and explained, the witness most always regains their votes.
That would be a great tool for the bitshares network.

Sent from my SM-N920T using Tapatalk


Offline xeroc

  • Board Moderator
  • Hero Member
  • *****
  • Posts: 12722
  • ChainSquad GmbH
    • View Profile
    • ChainSquad GmbH
  • BitShares: xeroc
  • GitHub: xeroc
Re: Professional Price Feeds for DEX
« Reply #13 on: April 20, 2017, 08:54:00 am »
I am looking for a single (active) witness that is willing to invest some time and try my new price feed script that I am currently working on:
https://github.com/xeroc/bitshares-pricefeed
It eats a YAML formatted configuration file like this:
https://github.com/xeroc/bitshares-pricefeed/blob/master/bitshares_pricefeed/config-example.yaml
.. makes use of pybitshares and has a separated standalone app.

After installation with `pip3 install bitshares-pricefeed`, a new command line app `bitshares-pricefeed` will be installed (in .local/bin) ... see `bitshares-pricefeed --help`
Give BitShares a try! Use the http://testnet.bitshares.eu provided by http://bitshares.eu powered by ChainSquad GmbH

Offline roelandp

  • Full Member
  • ***
  • Posts: 85
  • Witness, dad, kitesurfer, event organiser
    • View Profile
    • RoelandP.nl
  • BitShares: roelandp
  • GitHub: roelandp
Re: Professional Price Feeds for DEX
« Reply #14 on: April 20, 2017, 11:54:10 am »
I am looking for a single (active) witness that is willing to invest some time and try my new price feed script that I am currently working on:
https://github.com/xeroc/bitshares-pricefeed
It eats a YAML formatted configuration file like this:
https://github.com/xeroc/bitshares-pricefeed/blob/master/bitshares_pricefeed/config-example.yaml
.. makes use of pybitshares and has a separated standalone app.

After installation with `pip3 install bitshares-pricefeed`, a new command line app `bitshares-pricefeed` will be installed (in .local/bin) ... see `bitshares-pricefeed --help`

Hi X, cool I tried it (could not find it through pip3 though, did a manual git clone and install via setup.py) - I have sent you a little debug / bug thingie in regards to BTC38 which I also stumbled upon when trying your 'previous' pricefeeds with a specific pair and non-json response from the BTC38 API, check it out here: https://gist.github.com/roelandp/580d14fd9d866afdbca43efe92e91e2e . I think I hit that error also on your new script. Would love to implement it and run it so I will def give it some more testing! pybitshares ftw!