Delegates Please Publish Prices for USD, BTC, and CNY

Started by bytemaster, August 31, 2014, 08:02:22 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

GaltReport

#91
I managed to hack up the price feed python script that was floating around so that I can schedule it using the Linux cron job scheduler as well as implementing the update on price variance and hours since last update.  I will put it below for anyone that may find it useful.

I'm no great python programmer so please validate and/or fix any bugs discovered. 

I modified the config.json.  I added 2 new variables:

variance: percentage price variance
maxhours: maximum time between updates

The program will check the delegate's current published price for each asset and publish a new price if the current price exceeds the variance or if it has been longer than maxhours since the price has been published.

It has only been through basic testing so be advised.  I am not particularly confident in the math and date manipulations but it seems to work for my limited test cases.

I store the config.json file below in /home/ubuntu.  You can move it somewhere else but will need to change the location in the code. 

In the config.json file below you need to modify the following:

1. Set 1434 to the port number used in the httpd_endpoint entry in your .BitSharesX/config.json file
2. Set rpc-username-goes-here to the rpc_user setting in .BitSharesX/config.json file
3. Set rpc-password-goes-here to the rpc_password setting in .BitSharesX/config.json file
4. Set delegate-name-goes-here to your delegate name.
5. Set variance and maxhours to what you want

config.json


{
  "bts_rpc": {
    "url": "http://localhost:1434/rpc",
    "username": "rpc-username-goes-here",
    "password": "rpc-password-goes-here"
  },
  "asset_list": ["USD","BTC","CNY"],
  "delegate_list": ["delegate-name-goes-here"],
  "variance": 1,
  "maxhours": 12
}


price_feed.py


import requests
import json
import sys
from math import fabs

import datetime, threading, time
from pprint import pprint


headers = {'content-type': 'application/json',
   'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:22.0) Gecko/20100101 Firefox/22.0'}

config_data = open('/home/ubuntu/config.json')
config = json.load(config_data)
config_data.close()

## -----------------------------------------------------------------------
## function about bts rpc
## -----------------------------------------------------------------------
auth = (config["bts_rpc"]["username"], config["bts_rpc"]["password"])
url = config["bts_rpc"]["url"]

asset_list = config["asset_list"]
init_asset_list = asset_list
delegate_list = config["delegate_list"]

def fetch_from_btc38():
  url="http://api.btc38.com/v1/ticker.php"
  while True:
     try:
       params = { 'c': 'btsx', 'mk_type': 'btc' }
       responce = requests.get(url=url, params=params, headers=headers)
       result = responce.json()
       price["BTC"].append(float(result["ticker"]["last"]))

       params = { 'c': 'btsx', 'mk_type': 'cny' }
       responce = requests.get(url=url, params=params, headers=headers)
       result = responce.json()
       price_cny = float(result["ticker"]["last"])
       price["CNY"].append(float(result["ticker"]["last"]))

       price["USD"].append(price_cny/rate_usd_cny)
       break
     except:
       e = sys.exc_info()[0]
       print "Error: fetch_from_btc38: retrying in 30 seconds", e
       time.sleep(30)
       continue

def fetch_from_bter():
  while True:
     try:
       url="http://data.bter.com/api/1/ticker/btsx_btc"
       responce = requests.get(url=url, headers=headers)
       result = responce.json()
       price["BTC"].append(float(result["last"]))

       url="http://data.bter.com/api/1/ticker/btsx_cny"
       responce = requests.get(url=url, headers=headers)
       result = responce.json()
       price_cny = float(result["last"])
       price["CNY"].append(float(result["last"]))
       price["USD"].append(price_cny/rate_usd_cny)
       break
     except:
       e = sys.exc_info()[0]
       print "Error: fetch_from_bter: retrying in 30 seconds", e
       time.sleep(30)
       continue

def get_rate_from_yahoo():
  global headers
  global rate_usd_cny, rate_xau_cny

  while True:
     try:
       url="http://download.finance.yahoo.com/d/quotes.csv"
       params = {'s':'USDCNY=X,XAUCNY=X','f':'l1','e':'.csv'}
       responce = requests.get(url=url, headers=headers,params=params)

       pos = posnext = 0
       posnext = responce.text.find("\n", pos)
       rate_usd_cny = float(responce.text[pos:posnext])
       print "Fetch: rate usd/cny", rate_usd_cny
       pos = posnext + 1
       posnext = responce.text.find("\n", pos)
       rate_xau_cny = float(responce.text[pos:posnext])
       print "Fetch: rate xau/cny", rate_xau_cny
       print
       break
     except:
       e = sys.exc_info()[0]
       print "Error: get_rate_from_yahoo:  try again after 30 seconds", e
       time.sleep(30)
       continue

def update_price(delegate,asset,price,feed):
      update_request = {
         "method": "wallet_publish_price_feed",
         "params": [delegate, price, asset],
         "jsonrpc": "2.0",
         "id": 1
      }

      present  = datetime.datetime.now()
      feed_price  = feed['price']
      symbol      = feed['asset_symbol']
      last_update = feed['last_update']

      lu_yr  = int(last_update[0:4])
      lu_mn  = int(last_update[4:6])
      lu_dy  = int(last_update[6:8])
      lu_hr  = int(last_update[9:11])
      lu_min = int(last_update[11:13])
      lu_sec = int(last_update[13:15])
      lu_d   = datetime.date(lu_yr,lu_mn,lu_dy)
      lu_t   = datetime.time(lu_hr,lu_min,lu_sec)
      lu_dt  = datetime.datetime.combine(lu_d,lu_t)

      # Calculate Price Variance
      if (price > feed_price):
            diff = 100 - (round((feed_price / price) * 100,0))
      else:
            diff = 100 - (round((price / feed_price) * 100,0))

      # Calculate Time Since Last Update
      tm_df  = present-lu_dt
      tm_mx  = datetime.timedelta(hours=config['maxhours'])

      print "   Delegate Price Feed: ",symbol,feed_price
      print "         Current Price: ",asset,price
      print " BC Last Update String: ",last_update
      print "      Last Update Date: ",lu_dt
      print "     Current Date/Time: ",present
      print "     Time Since Update: ",str(tm_df)
      print " Max Hrs Before Update: ",str(tm_mx)
      print "        Price Variance: ",int(diff)
      print "    Max Price Variance: ",config['variance']
      print "           Update Feed: ",

      while True:
           try:
               # Publish Asset Price If Maximum Price Variance or Maximum Time Are Exceeded
               if ((int(diff) >= config['variance']) or (tm_df > tm_mx)):
                  print "Yes",
                  responce = requests.post(url, data=json.dumps(update_request), headers=headers, auth=auth)
                  result = json.loads(vars(responce)["_content"])
                  print "-",delegate, price_average[asset], asset
               else:
                  print "No"
               print
               break

           except:
               e = sys.exc_info()[0]
               print "Warnning: Can't connect to rpc server or other error, (update_request) try again after 30 seconds", e
               time.sleep(30)

def update_feed(price, asset):
  for delegate in delegate_list:
     headers = {'content-type': 'application/json'}
     feed_request = {
         "method": "blockchain_get_feeds_from_delegate",
         "params": [delegate],
         "jsonrpc": "2.0",
         "id": 1
     }
     while True:
        try:
           # Get Delegate Price Feeds
           responce = requests.post(url, data=json.dumps(feed_request), headers=headers, auth=auth)
           result   = json.loads(vars(responce)['_content'])
           lresult  = result['result']
           for i in lresult:
              if (asset == i['asset_symbol']):
                 update_price(delegate,asset,price,i)
           break

        except:
           e = sys.exc_info()[0]
           print "Warnning: Can't connect to rpc server or other error, (get_feeds) try again after 30 seconds", e
   time.sleep(30)

def fetch_price():
  for asset in init_asset_list:
    price[asset] = []

  fetch_from_btc38()
  fetch_from_bter()

  for asset in asset_list:
    if len(price[asset]) == 0:
      print "Warning: can't get price of", asset
      continue
    price_average[asset] = sum(price[asset])/len(price[asset])
    if price_average_last[asset] != 0.0:
      change = 100.0 * (price_average[asset] - price_average_last[asset])/price_average_last[asset]
    else:
      change = 100.0

    update_feed(price_average[asset], asset)

print '=================', time.strftime("%Y%m%dT%H%M%S", time.localtime(time.time())), '=================='

rate_usd_cny = 0.0
rate_xau_cny = 0.0
get_rate_from_yahoo()

price = {}
price_average = {}
price_average_last = {}

for asset in init_asset_list:
  price[asset] = []
  price_average[asset] = 0.0
  price_average_last[asset] = 0.0

fetch_price()

print '=================', time.strftime("%Y%m%dT%H%M%S", time.localtime(time.time())), '=================='
print



Below crontab will run the script every 4 hours:

crontab


0 1,5,9,13,17,21 * * *  python /home/ubuntu/price_feed.py >> /home/ubuntu/price_feed.out


Log of actions will be stored in /home/ubuntu/price_feed.out

This may grow large after awhile so keep an eye on it and remove it when it get's too large or if you are familiar with it, you can use logrotate to maintain it

Edit: Edited to fix bug where it wouldn't work for multiple delegates.

Edit: Edited to change the retry Timer to 60 seconds instead of 1 second.  I ended up missing blocks because the program got into a tight 1 second retry loop and spiked my CPU.  I highly recommend changing this if you are using the original to something more than 1 second.  Otherwise, you have a chance of spiking your CPU and missing blocks if it continually retries every second.

The relevant lines of the code to change start with threading.Timer(1
Change the 1 to 30 or 60 maybe.

Edit: Edited to remove use of "threading.Timer" to avoid possible zombie process/threads that may use up memory/cpu.  Substituted while loop for retries.

happyshares

Quote from: bytemaster on September 04, 2014, 06:14:19 PM
Quote from: GaltReport on September 04, 2014, 04:41:18 PM
Quote from: Riverhead on September 04, 2014, 04:09:39 PM
Quote from: bytemaster on September 04, 2014, 03:44:52 PM
I think early on it will be helpful for feeds to be published every 4 hours.  The market is still thin and 4 hours is a lot of time with the volatility of BTSX.


I updated mine to every 4 hours and only if changed by more than 5% from previous update.

I updated mine to every 4 hours (and adjusted the fee  ;) )

I think we need tighter tolerances to enforce the peg better.   Update if changed by more than 1%.  Right now shorts are selling at 3% above.

This way i spent over 28 BTSX for updating the feed since midnight, using the python script. Quite expensive related to payrate.


alt

Quote from: CalabiYau on September 05, 2014, 06:27:06 AM
Quote from: alt on September 05, 2014, 02:13:29 AM
I suggest  to update the automatic feed price scripts.
the main different is use the median price to replace the average price.
you can change the pararm "median_length" to adjust the price change latency.
because version 0.4.12 will use feed price to decide  the min cover price,
we need to protect the cover order from a serial margin call,
maybe cause of a suddenly price drop with a short period at the central trade site.
my English is poor, maybe not say it clearly   ???
Quote from: alt on September 05, 2014, 02:06:35 AM
updated:
1. update config file
2. use median price to replace average price, to protect system from  a suddenly drop price with a short period at the central trade site.

Thank you alt, I will try the improved script.
"Neuron" sent you some compensation for your efforts.
yes, I have got your tips, thanks  :)

xeroc

Quote from: cryptillionaire on September 05, 2014, 08:45:52 AM
Can someone point me in the direction of how to set up feeds for currency values? I'm very interested in creating a delegate, and this is a gap in my knowledge at the moment.
Thanks :)
Theses are the scripts:
https://github.com/Bitsuperlab/operation_tools/tree/master/btsxfeed

R2

Can someone point me in the direction of how to set up feeds for currency values? I'm very interested in creating a delegate, and this is a gap in my knowledge at the moment.
Thanks :)

CalabiYau

Quote from: alt on September 05, 2014, 02:13:29 AM
I suggest  to update the automatic feed price scripts.
the main different is use the median price to replace the average price.
you can change the pararm "median_length" to adjust the price change latency.
because version 0.4.12 will use feed price to decide  the min cover price,
we need to protect the cover order from a serial margin call,
maybe cause of a suddenly price drop with a short period at the central trade site.
my English is poor, maybe not say it clearly   ???
Quote from: alt on September 05, 2014, 02:06:35 AM
updated:
1. update config file
2. use median price to replace average price, to protect system from  a suddenly drop price with a short period at the central trade site.

Thank you alt, I will try the improved script.
"Neuron" sent you some compensation for your efforts.

alt

I suggest  to update the automatic feed price scripts.
the main different is use the median price to replace the average price.
you can change the pararm "median_length" to adjust the price change latency.
because version 0.4.12 will use feed price to decide  the min cover price,
we need to protect the cover order from a serial margin call,
maybe cause of a suddenly price drop with a short period at the central trade site.
my English is poor, maybe not say it clearly   ???
Quote from: alt on September 05, 2014, 02:06:35 AM
updated:
1. update config file
2. use median price to replace average price, to protect system from  a suddenly drop price with a short period at the central trade site.

alt

Quote from: xeroc on September 04, 2014, 07:10:29 PM
The auto script of alt needs some more sanity checks .. I just pushed a feed with 0USD .. *strange* :)
It's very strange, I didn't  find the bug.
and thanks for your bitUSD tip :)

amencon

My feed should now also update when difference over 1%.

bytemaster

Quote from: xeroc on September 04, 2014, 07:10:29 PM
The auto script of alt needs some more sanity checks .. I just pushed a feed with 0USD .. *strange* :)

Working to add a RPC call to publish multiple feeds in a single transaction.
For the latest updates checkout my blog: [url="http://bytemaster.bitshares.org"]http://bytemaster.bitshares.org[/url]
Anything said on these forums does not constitute an intent to create a legal obligation or contract between myself and anyone else.   These are merely my opinions and I reserve the right to change them at any time.

xeroc

The auto script of alt needs some more sanity checks .. I just pushed a feed with 0USD .. *strange* :)


Riverhead

Quote from: bytemaster on September 04, 2014, 06:14:19 PM
I think we need tighter tolerances to enforce the peg better.   Update if changed by more than 1%.  Right now shorts are selling at 3% above.


Done.

GaltReport

Quote from: bytemaster on September 04, 2014, 06:14:19 PM
Quote from: GaltReport on September 04, 2014, 04:41:18 PM
Quote from: Riverhead on September 04, 2014, 04:09:39 PM
Quote from: bytemaster on September 04, 2014, 03:44:52 PM
I think early on it will be helpful for feeds to be published every 4 hours.  The market is still thin and 4 hours is a lot of time with the volatility of BTSX.


I updated mine to every 4 hours and only if changed by more than 5% from previous update.

I updated mine to every 4 hours (and adjusted the fee  ;) )

I think we need tighter tolerances to enforce the peg better.   Update if changed by more than 1%.  Right now shorts are selling at 3% above.

I update it regardless every 4 hours.  Is that okay?

bytemaster

Quote from: GaltReport on September 04, 2014, 04:41:18 PM
Quote from: Riverhead on September 04, 2014, 04:09:39 PM
Quote from: bytemaster on September 04, 2014, 03:44:52 PM
I think early on it will be helpful for feeds to be published every 4 hours.  The market is still thin and 4 hours is a lot of time with the volatility of BTSX.


I updated mine to every 4 hours and only if changed by more than 5% from previous update.

I updated mine to every 4 hours (and adjusted the fee  ;) )

I think we need tighter tolerances to enforce the peg better.   Update if changed by more than 1%.  Right now shorts are selling at 3% above.
For the latest updates checkout my blog: [url="http://bytemaster.bitshares.org"]http://bytemaster.bitshares.org[/url]
Anything said on these forums does not constitute an intent to create a legal obligation or contract between myself and anyone else.   These are merely my opinions and I reserve the right to change them at any time.