Author Topic: Professional Price Feeds for DEX  (Read 6482 times)

0 Members and 1 Guest are viewing this topic.

Offline roelandp

  • Full Member
  • ***
  • Posts: 114
  • Witness, dad, kitesurfer, event organiser
    • View Profile
    • RoelandP.nl
  • BitShares: roelandp
  • GitHub: roelandp

Offline xeroc

  • Board Moderator
  • Hero Member
  • *****
  • Posts: 12922
  • ChainSquad GmbH
    • View Profile
    • ChainSquad GmbH
  • BitShares: xeroc
  • GitHub: xeroc
Good point @abit, and with the new price feed script that I am working on, you don't even need a cli_wallet anymore. Just pure and plain python script, wallet, signing and communication managed by pybitshares.

btw, the new uptick release will have a `newfeed` call aswell .. it works like this:

    uptick newfeed USD 0.0124 USD/BTS

Offline abit

  • Committee member
  • Hero Member
  • *
  • Posts: 4664
    • View Profile
    • Abit's Hive Blog
  • BitShares: abit
  • GitHub: abitmore
If that information were up to date and leaked, it would be easy for every feedsource API provider to know most witnesses IP addresses.
I'm not sure if anyone has pointed out, but for security reasons, witnesses can and should have price feed scripts running NOT on the block producing node. For example one can run the script in her home laptop and publish feeds via public API servers.

//Edit: the scripts are client software, they talk to the API servers.

//Edit: as quoted below:
Feeds can be published from a non producing witness with the active key.
« Last Edit: April 20, 2017, 09:46:24 pm by abit »
BitShares committee member: abit
BitShares witness: in.abit

Offline roelandp

  • Full Member
  • ***
  • Posts: 114
  • Witness, dad, kitesurfer, event organiser
    • View Profile
    • RoelandP.nl
  • BitShares: roelandp
  • GitHub: roelandp
I am looking for a single (active) witness that is willing to invest some time and try my new price feed script that I am currently working on:
https://github.com/xeroc/bitshares-pricefeed
It eats a YAML formatted configuration file like this:
https://github.com/xeroc/bitshares-pricefeed/blob/master/bitshares_pricefeed/config-example.yaml
.. makes use of pybitshares and has a separated standalone app.

After installation with `pip3 install bitshares-pricefeed`, a new command line app `bitshares-pricefeed` will be installed (in .local/bin) ... see `bitshares-pricefeed --help`

Hi X, cool I tried it (could not find it through pip3 though, did a manual git clone and install via setup.py) - I have sent you a little debug / bug thingie in regards to BTC38 which I also stumbled upon when trying your 'previous' pricefeeds with a specific pair and non-json response from the BTC38 API, check it out here: https://gist.github.com/roelandp/580d14fd9d866afdbca43efe92e91e2e . I think I hit that error also on your new script. Would love to implement it and run it so I will def give it some more testing! pybitshares ftw!

Offline xeroc

  • Board Moderator
  • Hero Member
  • *****
  • Posts: 12922
  • ChainSquad GmbH
    • View Profile
    • ChainSquad GmbH
  • BitShares: xeroc
  • GitHub: xeroc
I am looking for a single (active) witness that is willing to invest some time and try my new price feed script that I am currently working on:
https://github.com/xeroc/bitshares-pricefeed
It eats a YAML formatted configuration file like this:
https://github.com/xeroc/bitshares-pricefeed/blob/master/bitshares_pricefeed/config-example.yaml
.. makes use of pybitshares and has a separated standalone app.

After installation with `pip3 install bitshares-pricefeed`, a new command line app `bitshares-pricefeed` will be installed (in .local/bin) ... see `bitshares-pricefeed --help`

Offline Pheonike

Hey @sakhan in my Witness proposal you asked me to reply. I was already monitoring this chat, but a bit busy with the pricefeed scripts themselves atm :) Btw, this discussion has been talked about in the forums before, see for example with some interesting chats and opinions: https://bitsharestalk.org/index.php/topic,23607.0.html

How I feel about pricescripts and how they should come to life? First off, as the white paper [section 2.2] states, pricefeeds are important to have a fair price for settlements of Smartcoins on the Dex. It is important to have accurate information, but the software has some build in fault resistance as you can conclude from that paragraph.

I think setting up a trustworthy pricefeed for your witness means:
1. Sourcing from markets with enough volume.
2. Sourcing from a multitude of markets.
3. Have limits in place to prevent black swan events or sudden spikes / drops.
4. Adjust parameters to your reasoning and be open about your bias.

In general I am a huge fan of @xeroc's work in python. He made me dive into python (finally). I have been testing his scripts, mainly his multi-source pricefeed script, which had some recent edits by @iHashFury incorporated here.

IMHO @xeroc's script does this and enables you to add as many sources as you like, it also has built in limits which should be monitored and therefore I am writing to expand the script to report to me on my telegram bot with manual verification or rejection of the feed publishing.

Having a paid for account for 1 (live) source won't change that much seeing GRAPHENE_DEFAULT_PRICE_FEED_LIFETIME in the config of 24 hours imho and especially it won't be a good idea to make the price-feeds part of the DEX software being dependent on it, as it would mean having a single point of failure.

Nonetheless I totally follow @Thom to have some decent witness performance overview in place, like we have for Steem / Golos: https://steemdb.com and https://golosdb.com where it is easy to see who is missing blocks and how many, and who is slow on updating pricefeeds. This could help judge witnesses and actually vote / unvote them based on factual data and performance.

Alternatively we could all put trust in a machine script / AI and proxy your votes into that dedicated account who then votes / unvotes witnesses based on performance parameters to be set in a BSIP. In Steem it happens all the time once an active witness starts to miss blocks, and he/she is unresponsive, immediately he/she is voted out for the time being to keep the network at 100% participation. After the event is resolved and explained, the witness most always regains their votes.
That would be a great tool for the bitshares network.

Sent from my SM-N920T using Tapatalk


Offline roelandp

  • Full Member
  • ***
  • Posts: 114
  • Witness, dad, kitesurfer, event organiser
    • View Profile
    • RoelandP.nl
  • BitShares: roelandp
  • GitHub: roelandp
Hey @sakhan in my Witness proposal you asked me to reply. I was already monitoring this chat, but a bit busy with the pricefeed scripts themselves atm :) Btw, this discussion has been talked about in the forums before, see for example with some interesting chats and opinions: https://bitsharestalk.org/index.php/topic,23607.0.html

How I feel about pricescripts and how they should come to life? First off, as the white paper [section 2.2] states, pricefeeds are important to have a fair price for settlements of Smartcoins on the Dex. It is important to have accurate information, but the software has some build in fault resistance as you can conclude from that paragraph.

I think setting up a trustworthy pricefeed for your witness means:
1. Sourcing from markets with enough volume.
2. Sourcing from a multitude of markets.
3. Have limits in place to prevent black swan events or sudden spikes / drops.
4. Adjust parameters to your reasoning and be open about your bias.

In general I am a huge fan of @xeroc's work in python. He made me dive into python (finally). I have been testing his scripts, mainly his multi-source pricefeed script, which had some recent edits by @iHashFury incorporated here.

IMHO @xeroc's script does this and enables you to add as many sources as you like, it also has built in limits which should be monitored and therefore I am writing to expand the script to report to me on my telegram bot with manual verification or rejection of the feed publishing.

Having a paid for account for 1 (live) source won't change that much seeing GRAPHENE_DEFAULT_PRICE_FEED_LIFETIME in the config of 24 hours imho and especially it won't be a good idea to make the price-feeds part of the DEX software being dependent on it, as it would mean having a single point of failure.

Nonetheless I totally follow @Thom to have some decent witness performance overview in place, like we have for Steem / Golos: https://steemdb.com and https://golosdb.com where it is easy to see who is missing blocks and how many, and who is slow on updating pricefeeds. This could help judge witnesses and actually vote / unvote them based on factual data and performance.

Alternatively we could all put trust in a machine script / AI and proxy your votes into that dedicated account who then votes / unvotes witnesses based on performance parameters to be set in a BSIP. In Steem it happens all the time once an active witness starts to miss blocks, and he/she is unresponsive, immediately he/she is voted out for the time being to keep the network at 100% participation. After the event is resolved and explained, the witness most always regains their votes.

Offline Thom

As to your point about witnesses producing feeds with different keys, yes, you could use a different key. However, the witness associated with that key must be in the active state. Witness nodes can use an arbitrary key pair generated by the cli wallet to run their node, but only the signing key can be used to produce blocks.  I don't recall the specifics on feed publishing, but I suspect the only restriction is that the account open in the wallet which the script connects through must be a voted in witness account.

Right. In other words, witnesses need two types of keys to produce blocks: the usual active key and a special key to sign blocks.
It's a good practice to distribute different (non-active) signing key pairs between failover nodes, because the 'update_witness' command allows changing the signing key for a given witness account. Thus we can switch block production between nodes without the need to restart a witness_node to change its signing keys.

Signing keys are no required to publish feeds. You can have one node producing blocks and another feeding prices, independently.

The wallet used by the publish script needs to be unlocked for the script to use it to publish feeds. All 5 of my nodes can act as a block producer or seed node and all provide some degree of failover support. Next month I will be upgrading 2 of the 5 nodes to dedicated severs with 16GB of RAM.

All servers run an identical software and host configuration that allows any one of them to become the active block producer for verbaltech2. I always lock the wallet except for the active witness node so only 1 node is publishing feeds, that being the witness node. When block production is switched to a different node feed production begins there. It doesn't stop on the previously active witness node until I lock or stop the wallet.
Injustice anywhere is a threat to justice everywhere - MLK |  Verbaltech2 Witness Reports: https://bitsharestalk.org/index.php/topic,23902.0.html

Offline rnglab

  • Full Member
  • ***
  • Posts: 171
    • View Profile
  • BitShares: rnglab
As to your point about witnesses producing feeds with different keys, yes, you could use a different key. However, the witness associated with that key must be in the active state. Witness nodes can use an arbitrary key pair generated by the cli wallet to run their node, but only the signing key can be used to produce blocks.  I don't recall the specifics on feed publishing, but I suspect the only restriction is that the account open in the wallet which the script connects through must be a voted in witness account.

Right. In other words, witnesses need two types of keys to produce blocks: the usual active key and a special key to sign blocks.
It's a good practice to distribute different (non-active) signing key pairs between failover nodes, because the 'update_witness' command allows changing the signing key for a given witness account. Thus we can switch block production between nodes without the need to restart a witness_node to change its signing keys.

Signing keys are no required to publish feeds. You can have one node producing blocks and another feeding prices, independently.

Offline Thom

The easiest solution in the short term is to use the testnet to perfect witness feeds / tools. However, without standards and conventions the data collected would be of limited value. For example, if there are no monitoring tools for price feed data (such as the chart you linked by lafona) from which comparisons can be made how can accuracy be measured?

Probably not difficult to do, but coordination of testnet use is also important. Doesn't make a lot of sense to be testing feeds if a stress test was underway.

As to your point about witnesses producing feeds with different keys, yes, you could use a different key. However, the witness associated with that key must be in the active state. Witness nodes can use an arbitrary key pair generated by the cli wallet to run their node, but only the signing key can be used to produce blocks.  I don't recall the specifics on feed publishing, but I suspect the only restriction is that the account open in the wallet which the script connects through must be a voted in witness account.
Injustice anywhere is a threat to justice everywhere - MLK |  Verbaltech2 Witness Reports: https://bitsharestalk.org/index.php/topic,23902.0.html

Offline rnglab

  • Full Member
  • ***
  • Posts: 171
    • View Profile
  • BitShares: rnglab
About rising witness quality standards (with emphasis on accurate pricefeeds), I like some of the ideas that have been discussed in the last months, but most of them would need heavy development, after a long way to reach consensus. 

At this stage I think our best trade off between priorities and resources would be to fund development for an open source block explorer, with an initial focus on witness monitoring tools (like historical pricefeed charts).

I like this approach from @lafona: https://lafona.shinyapps.io/Feed_History/
Just adding a highlighted line with the median price published by the blockchain would make a good enough reference to start monitoring feeds accuracy. Maybe another line with asset prices from a professional paid source, as an off chain reference.

The lack of both an open source block explorer codebase, and a simple way to monitor witnesses, are top prioity issues that will probably result more expensive if we wait for someone to solve them for free some day, than if we just fund its development as a priority.
@svk @svk @svk   :o

Offline rnglab

  • Full Member
  • ***
  • Posts: 171
    • View Profile
  • BitShares: rnglab
IOn the other hand, most if not all Chinese feed sources has a DDOS protection that blacklists your ip with *many* less API calls than the specified.

Feeds can be published from a non producing witness with the active key.

Interesting. Would their DDoS protection kick in from 1 request an hour? I presume when you say "feed sources" you're talking about exchanges?

Yes I was talking about exchanges in this case, but the same may apply for any Chineese feed source.
One request per hour is a lot of time to trigger DOS protection, I have just a few random rejects fetching prices every  5 or 10 minutes.

Quote
I don't believe your last point is correct. The active OR owner keys can be used to publish feeds, but the API call will fail unless the account calling the publish_feeds API is voted in (active). Don't get the state of being "active" / voted in confused with the type of key used on the account calling the API.

 I think I was not clear.  Signing keys are not required to publish feeds, active (elected) witnesses can feed prices from any node that has its active key. We can produce blocks from one node and publish feeds from another.
Quote


I would like to see some type of method for non-active witnesses to publish "informal" or "unofficial" price feeds to show shareholders they are capable and what their accuracy would be. It would also provide potential witnesses a way to perfect their scripts.

Agree. I think the simplest way would be to do it on testnet. It would also help to keep the public testnet allways distributed, not just when we need to run specific networking tests.

Offline R

  • Hero Member
  • *****
  • Posts: 1004
    • View Profile
Ideally all witnesses should be publishing feeds.

Offline Thom

IOn the other hand, most if not all Chinese feed sources has a DDOS protection that blacklists your ip with *many* less API calls than the specified.

Feeds can be published from a non producing witness with the active key.

Interesting. Would their DDoS protection kick in from 1 request an hour? I presume when you say "feed sources" you're talking about exchanges?

I don't believe your last point is correct. The active OR owner keys can be used to publish feeds, but the API call will fail unless the account calling the publish_feeds API is voted in (active). Don't get the state of being "active" / voted in confused with the type of key used on the account calling the API.

I would like to see some type of method for non-active witnesses to publish "informal" or "unofficial" price feeds to show shareholders they are capable and what their accuracy would be. It would also provide potential witnesses a way to perfect their scripts.
Injustice anywhere is a threat to justice everywhere - MLK |  Verbaltech2 Witness Reports: https://bitsharestalk.org/index.php/topic,23902.0.html

Offline rnglab

  • Full Member
  • ***
  • Posts: 171
    • View Profile
  • BitShares: rnglab
I wouldn't care about the /slots feature from spartako-bot. It was added by spartako almost instantly when Dan asked us to provide price feeds and xeroc's script was released.  As we started being just a few witnesses publishing feeds until different timezones and avaiability allowed everyone to update nodes, making the time span as distributed as possible was one of the first actions.

If that information were up to date and leaked, it would be easy for every feedsource API provider to know most witnesses IP addresses.

The blockchain takes prices from the majority of witnesses that "agrees" on a median, withing the feed_lifetime parameter  (if someone can add a more accurate description would be apreciated).
It has prooved to be very resilent against wrong feeds, and less dependant on publishing frequency.

this are the time parameters from xeroc's pricefeeds script:
Code: [Select]
################################################################################
# Publishing Criteria
################################################################################
#
# Publish price if any of these are valid:
#
#  1.) price feed is older than maxAgeFeedInSeconds
#  2.) price has moved from my last published price by more than change_min
#
# Do NOT publish if
#
#  * Price has moved more than 'change_max'
#     AND
#  * Manual confirmation is 'false'
#
# A feed becomes invalid if it is older than 24h. Hence, it should be force
# published once a day (e.g. every 12h) Note that the script should be run more
# frequently and publishes only those prices that fit the publishing criteria
maxAgeFeedInSeconds          = 12 * 60 * 60  # max age of 12h
change_min                   = 0.5       # Percentage of price change to force an update
change_max                   = 5       # Percentage of price change to cause a warning

a low change_max helps to prevent market manipulation, specially black swan events. It will stop publishing until manual confirmation, if the value is exceeded in the time span in which you run the script.
with high volatility is where you need to fin tune this parameters. A low change_max without frequently running your script will ask for manual confirmation as false alarm too often to handle.

On the other hand, most if not all Chinese feed sources has a DDOS protection that blacklists your ip with *many* less API calls than the specified. Sometimes you can make reverse DNS loockups to find direct connection to an API, sometimes they provide a private IP if you mail them, but in both cases the addresses change often and you'll have to find out what to put in your hosts file very often if blacklisted by Baidu.

Feeds can be published from a non producing witness with the active key.
If  someone has troubles reaching btc38 at this moment,  producing feeds from a node in China may be a solution.