Delegates Please Publish Prices for USD, BTC, and CNY

[GaltReport]

I managed to hack up the price feed python script that was floating around so that I can schedule it using the Linux cron job scheduler as well as implementing the update on price variance and hours since last update.  I will put it below for anyone that may find it useful.

I'm no great python programmer so please validate and/or fix any bugs discovered. 

I modified the config.json.  I added 2 new variables:

variance: percentage price variance
maxhours: maximum time between updates

The program will check the delegate's current published price for each asset and publish a new price if the current price exceeds the variance or if it has been longer than maxhours since the price has been published.

It has only been through basic testing so be advised.  I am not particularly confident in the math and date manipulations but it seems to work for my limited test cases.

I store the config.json file below in /home/ubuntu.  You can move it somewhere else but will need to change the location in the code. 

In the config.json file below you need to modify the following:

1. Set 1434 to the port number used in the httpd_endpoint entry in your .BitSharesX/config.json file
2. Set rpc-username-goes-here to the rpc_user setting in .BitSharesX/config.json file
3. Set rpc-password-goes-here to the rpc_password setting in .BitSharesX/config.json file
4. Set delegate-name-goes-here to your delegate name.
5. Set variance and maxhours to what you want

config.json

Code: [Select]

{
  "bts_rpc": {
    "url": "http://localhost:1434/rpc",
    "username": "rpc-username-goes-here",
    "password": "rpc-password-goes-here"
  },
  "asset_list": ["USD","BTC","CNY"],
  "delegate_list": ["delegate-name-goes-here"],
  "variance": 1,
  "maxhours": 12
}

price_feed.py

Code: [Select]

import requests
import json
import sys
from math import fabs

import datetime, threading, time
from pprint import pprint


headers = {'content-type': 'application/json',
   'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:22.0) Gecko/20100101 Firefox/22.0'}

config_data = open('/home/ubuntu/config.json')
config = json.load(config_data)
config_data.close()

## -----------------------------------------------------------------------
## function about bts rpc
## -----------------------------------------------------------------------
auth = (config["bts_rpc"]["username"], config["bts_rpc"]["password"])
url = config["bts_rpc"]["url"]

asset_list = config["asset_list"]
init_asset_list = asset_list
delegate_list = config["delegate_list"]

def fetch_from_btc38():
  url="http://api.btc38.com/v1/ticker.php"
  while True:
     try:
       params = { 'c': 'btsx', 'mk_type': 'btc' }
       responce = requests.get(url=url, params=params, headers=headers)
       result = responce.json()
       price["BTC"].append(float(result["ticker"]["last"]))

       params = { 'c': 'btsx', 'mk_type': 'cny' }
       responce = requests.get(url=url, params=params, headers=headers)
       result = responce.json()
       price_cny = float(result["ticker"]["last"])
       price["CNY"].append(float(result["ticker"]["last"]))

       price["USD"].append(price_cny/rate_usd_cny)
       break
     except:
       e = sys.exc_info()[0]
       print "Error: fetch_from_btc38: retrying in 30 seconds", e
       time.sleep(30)
       continue

def fetch_from_bter():
  while True:
     try:
       url="http://data.bter.com/api/1/ticker/btsx_btc"
       responce = requests.get(url=url, headers=headers)
       result = responce.json()
       price["BTC"].append(float(result["last"]))

       url="http://data.bter.com/api/1/ticker/btsx_cny"
       responce = requests.get(url=url, headers=headers)
       result = responce.json()
       price_cny = float(result["last"])
       price["CNY"].append(float(result["last"]))
       price["USD"].append(price_cny/rate_usd_cny)
       break
     except:
       e = sys.exc_info()[0]
       print "Error: fetch_from_bter: retrying in 30 seconds", e
       time.sleep(30)
       continue

def get_rate_from_yahoo():
  global headers
  global rate_usd_cny, rate_xau_cny

  while True:
     try:
       url="http://download.finance.yahoo.com/d/quotes.csv"
       params = {'s':'USDCNY=X,XAUCNY=X','f':'l1','e':'.csv'}
       responce = requests.get(url=url, headers=headers,params=params)

       pos = posnext = 0
       posnext = responce.text.find("\n", pos)
       rate_usd_cny = float(responce.text[pos:posnext])
       print "Fetch: rate usd/cny", rate_usd_cny
       pos = posnext + 1
       posnext = responce.text.find("\n", pos)
       rate_xau_cny = float(responce.text[pos:posnext])
       print "Fetch: rate xau/cny", rate_xau_cny
       print
       break
     except:
       e = sys.exc_info()[0]
       print "Error: get_rate_from_yahoo:  try again after 30 seconds", e
       time.sleep(30)
       continue

def update_price(delegate,asset,price,feed):
      update_request = {
         "method": "wallet_publish_price_feed",
         "params": [delegate, price, asset],
         "jsonrpc": "2.0",
         "id": 1
      }

      present  = datetime.datetime.now()
      feed_price  = feed['price']
      symbol      = feed['asset_symbol']
      last_update = feed['last_update']

      lu_yr  = int(last_update[0:4])
      lu_mn  = int(last_update[4:6])
      lu_dy  = int(last_update[6:8])
      lu_hr  = int(last_update[9:11])
      lu_min = int(last_update[11:13])
      lu_sec = int(last_update[13:15])
      lu_d   = datetime.date(lu_yr,lu_mn,lu_dy)
      lu_t   = datetime.time(lu_hr,lu_min,lu_sec)
      lu_dt  = datetime.datetime.combine(lu_d,lu_t)

      # Calculate Price Variance
      if (price > feed_price):
            diff = 100 - (round((feed_price / price) * 100,0))
      else:
            diff = 100 - (round((price / feed_price) * 100,0))

      # Calculate Time Since Last Update
      tm_df  = present-lu_dt
      tm_mx  = datetime.timedelta(hours=config['maxhours'])

      print "   Delegate Price Feed: ",symbol,feed_price
      print "         Current Price: ",asset,price
      print " BC Last Update String: ",last_update
      print "      Last Update Date: ",lu_dt
      print "     Current Date/Time: ",present
      print "     Time Since Update: ",str(tm_df)
      print " Max Hrs Before Update: ",str(tm_mx)
      print "        Price Variance: ",int(diff)
      print "    Max Price Variance: ",config['variance']
      print "           Update Feed: ",

      while True:
           try:
               # Publish Asset Price If Maximum Price Variance or Maximum Time Are Exceeded
               if ((int(diff) >= config['variance']) or (tm_df > tm_mx)):
                  print "Yes",
                  responce = requests.post(url, data=json.dumps(update_request), headers=headers, auth=auth)
                  result = json.loads(vars(responce)["_content"])
                  print "-",delegate, price_average[asset], asset
               else:
                  print "No"
               print
               break

           except:
               e = sys.exc_info()[0]
               print "Warnning: Can't connect to rpc server or other error, (update_request) try again after 30 seconds", e
               time.sleep(30)

def update_feed(price, asset):
  for delegate in delegate_list:
     headers = {'content-type': 'application/json'}
     feed_request = {
         "method": "blockchain_get_feeds_from_delegate",
         "params": [delegate],
         "jsonrpc": "2.0",
         "id": 1
     }
     while True:
        try:
           # Get Delegate Price Feeds
           responce = requests.post(url, data=json.dumps(feed_request), headers=headers, auth=auth)
           result   = json.loads(vars(responce)['_content'])
           lresult  = result['result']
           for i in lresult:
              if (asset == i['asset_symbol']):
                 update_price(delegate,asset,price,i)
           break

        except:
           e = sys.exc_info()[0]
           print "Warnning: Can't connect to rpc server or other error, (get_feeds) try again after 30 seconds", e
   time.sleep(30)

def fetch_price():
  for asset in init_asset_list:
    price[asset] = []

  fetch_from_btc38()
  fetch_from_bter()

  for asset in asset_list:
    if len(price[asset]) == 0:
      print "Warning: can't get price of", asset
      continue
    price_average[asset] = sum(price[asset])/len(price[asset])
    if price_average_last[asset] != 0.0:
      change = 100.0 * (price_average[asset] - price_average_last[asset])/price_average_last[asset]
    else:
      change = 100.0

    update_feed(price_average[asset], asset)

print '=================', time.strftime("%Y%m%dT%H%M%S", time.localtime(time.time())), '=================='

rate_usd_cny = 0.0
rate_xau_cny = 0.0
get_rate_from_yahoo()

price = {}
price_average = {}
price_average_last = {}

for asset in init_asset_list:
  price[asset] = []
  price_average[asset] = 0.0
  price_average_last[asset] = 0.0

fetch_price()

print '=================', time.strftime("%Y%m%dT%H%M%S", time.localtime(time.time())), '=================='
print


Below crontab will run the script every 4 hours:

crontab

Code: [Select]

0 1,5,9,13,17,21 * * *  python /home/ubuntu/price_feed.py >> /home/ubuntu/price_feed.out

Log of actions will be stored in /home/ubuntu/price_feed.out

This may grow large after awhile so keep an eye on it and remove it when it get's too large or if you are familiar with it, you can use logrotate to maintain it

Edit: Edited to fix bug where it wouldn't work for multiple delegates.

Edit: Edited to change the retry Timer to 60 seconds instead of 1 second.  I ended up missing blocks because the program got into a tight 1 second retry loop and spiked my CPU.  I highly recommend changing this if you are using the original to something more than 1 second.  Otherwise, you have a chance of spiking your CPU and missing blocks if it continually retries every second.

The relevant lines of the code to change start with threading.Timer(1
Change the 1 to 30 or 60 maybe.

Edit: Edited to remove use of "threading.Timer" to avoid possible zombie process/threads that may use up memory/cpu.  Substituted while loop for retries.

[happyshares]

I think early on it will be helpful for feeds to be published every 4 hours.  The market is still thin and 4 hours is a lot of time with the volatility of BTSX.

I updated mine to every 4 hours and only if changed by more than 5% from previous update.

I updated mine to every 4 hours (and adjusted the fee  )

I think we need tighter tolerances to enforce the peg better.   Update if changed by more than 1%.  Right now shorts are selling at 3% above.

This way i spent over 28 BTSX for updating the feed since midnight, using the python script. Quite expensive related to payrate.

[alt]

I suggest  to update the automatic feed price scripts.
the main different is use the median price to replace the average price.
you can change the pararm "median_length" to adjust the price change latency.
because version 0.4.12 will use feed price to decide  the min cover price,
we need to protect the cover order from a serial margin call,
maybe cause of a suddenly price drop with a short period at the central trade site.
my English is poor, maybe not say it clearly   

updated:
1. update config file
2. use median price to replace average price, to protect system from  a suddenly drop price with a short period at the central trade site.

Thank you alt, I will try the improved script.
"Neuron" sent you some compensation for your efforts.

yes, I have got your tips, thanks 

[xeroc]

Can someone point me in the direction of how to set up feeds for currency values? I'm very interested in creating a delegate, and this is a gap in my knowledge at the moment.
Thanks

Theses are the scripts:
https://github.com/Bitsuperlab/operation_tools/tree/master/btsxfeed

[cryptillionaire]

Can someone point me in the direction of how to set up feeds for currency values? I'm very interested in creating a delegate, and this is a gap in my knowledge at the moment.
Thanks

[CalabiYau]

I suggest  to update the automatic feed price scripts.
the main different is use the median price to replace the average price.
you can change the pararm "median_length" to adjust the price change latency.
because version 0.4.12 will use feed price to decide  the min cover price,
we need to protect the cover order from a serial margin call,
maybe cause of a suddenly price drop with a short period at the central trade site.
my English is poor, maybe not say it clearly   

updated:
1. update config file
2. use median price to replace average price, to protect system from  a suddenly drop price with a short period at the central trade site.

Thank you alt, I will try the improved script.
"Neuron" sent you some compensation for your efforts.

[alt]

I suggest  to update the automatic feed price scripts.
the main different is use the median price to replace the average price.
you can change the pararm "median_length" to adjust the price change latency.
because version 0.4.12 will use feed price to decide  the min cover price,
we need to protect the cover order from a serial margin call,
maybe cause of a suddenly price drop with a short period at the central trade site.
my English is poor, maybe not say it clearly   

updated:
1. update config file
2. use median price to replace average price, to protect system from  a suddenly drop price with a short period at the central trade site.

[alt]

The auto script of alt needs some more sanity checks .. I just pushed a feed with 0USD .. *strange*

It's very strange, I didn't  find the bug.
and thanks for your bitUSD tip

[amencon]

My feed should now also update when difference over 1%.

[bytemaster]

The auto script of alt needs some more sanity checks .. I just pushed a feed with 0USD .. *strange*

Working to add a RPC call to publish multiple feeds in a single transaction.

[xeroc]

The auto script of alt needs some more sanity checks .. I just pushed a feed with 0USD .. *strange*

[happyshares]

Updated

[Riverhead]

I think we need tighter tolerances to enforce the peg better.   Update if changed by more than 1%.  Right now shorts are selling at 3% above.

Done.

[GaltReport]

I think early on it will be helpful for feeds to be published every 4 hours.  The market is still thin and 4 hours is a lot of time with the volatility of BTSX.

I updated mine to every 4 hours and only if changed by more than 5% from previous update.

I updated mine to every 4 hours (and adjusted the fee  )

I think we need tighter tolerances to enforce the peg better.   Update if changed by more than 1%.  Right now shorts are selling at 3% above.

I update it regardless every 4 hours.  Is that okay?

[bytemaster]

I think early on it will be helpful for feeds to be published every 4 hours.  The market is still thin and 4 hours is a lot of time with the volatility of BTSX.

I updated mine to every 4 hours and only if changed by more than 5% from previous update.

I updated mine to every 4 hours (and adjusted the fee  )

I think we need tighter tolerances to enforce the peg better.   Update if changed by more than 1%.  Right now shorts are selling at 3% above.