Yet Another Price Feed Mod

[xeroc]

Ok, so I guess there's not much point to running blocks and script on different machines then... it's more a question of protecting the delegate block production environment by not revealing owner keys?

I'm gonna take the opportunity to read up on owner keys, active keys and sign keys. I found a couple of your guides on the subject, i will follow these.
https://bitsharestalk.org/index.php?topic=14344.0
https://bitsharestalk.org/index.php?topic=14360.0

Thanks for the help!

The most important thing is the OWNER KEY ..

I recommend you read this:
http://wiki.bitshares.org/index.php/Delegate/Security

[Spectral]

I'm trying to run the script connecting to a wallet that is not the wallet of the delegate. The wallet has a 50BTS balance right now, so that should be enough.

So it seem your delegate is "running" on a different machine then?

Yes. The delegate with block production enabled is running on a different machine.

Does it mean that the script must connect to the delegate wallet itself and draw funds from that? If so, how can I do that without providing the delegate wallet password in cleartext in 'config.py'?

If you want to provide continuous updates for the feed, then you can only put the password in cleartext into the config. BUT, you should create a wallet that only contains the most recent active key and signing key of the delegate in it .. and no other account ...
Then you can do a rescan of the blockchain to catch up with the balance and run the script again

Ok, so I guess there's not much point to running blocks and script on different machines then... it's more a question of protecting the delegate block production environment by not revealing owner keys?

I'm gonna take the opportunity to read up on owner keys, active keys and sign keys. I found a couple of your guides on the subject, i will follow these.
https://bitsharestalk.org/index.php?topic=14344.0
https://bitsharestalk.org/index.php?topic=14360.0

Thanks for the help!

[xeroc]

I'm trying to run the script connecting to a wallet that is not the wallet of the delegate. The wallet has a 50BTS balance right now, so that should be enough.

So it seem your delegate is "running" on a different machine then?

Does it mean that the script must connect to the delegate wallet itself and draw funds from that? If so, how can I do that without providing the delegate wallet password in cleartext in 'config.py'?

If you want to provide continuous updates for the feed, then you can only put the password in cleartext into the config. BUT, you should create a wallet that only contains the most recent active key and signing key of the delegate in it .. and no other account ...
Then you can do a rescan of the blockchain to catch up with the balance and run the script again

[Spectral]

I'm trying to run the script connecting to a wallet that is not the wallet of the delegate. The wallet has a 50BTS balance right now, so that should be enough.

Does it mean that the script must connect to the delegate wallet itself and draw funds from that? If so, how can I do that without providing the delegate wallet password in cleartext in 'config.py'?

[xeroc]

It says:

20010 insufficient_funds: insufficient funds

So you delegate does not have funds (either in his active balance or in his delegates pay) that can be used to pay for the transaction fee. Each publishing transaction costs 0.5 BTS

[Spectral]

Hi guys, I have a few questions about price feeds that I was hoping to get help with...

I'm using a wallet with some funds (not the delegate wallet) to try to post feeds for the 'bitspace-clains' delegate.

I'm using xeroc's bitshares-pytools, and think I managed to get the dependencies right. At first run, I would always get the message 'no update required' and -1% change on all assets in the table, so I edited 'main.py' to remove the required update test (is that ok for first time running feed script?). Now it runs, and I get the following output:

Code: [Select]

Update required! Forcing now!
Unlocking wallet
publishing feeds for delegate: bitspace-clains
Traceback (most recent call last):
  File "main.py", line 473, in <module>
    update_feed(rpc,asset_list_final)
  File "main.py", line 273, in update_feed
    result = rpc.wallet_publish_feeds(delegate, assets)
  File "/home/manuel/.local/lib/python3.4/site-packages/bitsharesrpc/client.py", line 135, in method
    "id": 0
  File "/home/manuel/.local/lib/python3.4/site-packages/bitsharesrpc/client.py", line 78, in rpcexec
    raise RPCError(ret['error']['detail'])
bitsharesrpc.client.RPCError: 20010 insufficient_funds: insufficient funds
    {"from_account_name":"bitspace-clains","amount_to_withdraw":{"amount":50000,"asset_id":0},"balance_records":[]}
    th_a  wallet.cpp:68 withdraw_to_transaction

Do I understand correctly that the script is trying to draw funds from the delegate account? I'm guessing that means that a feed must be published from the delegate account itself...?

If so, how do you avoid inserting the wallet password into 'config.py' in cleartext?

Also, how much funds is required to post the delegate feeds?

[xeroc]

You've started a rewrite. Good.
However without test network for bitshares 2.0 it looks impractical, doesn't it ?

Yes ... that's why it mostly only contains the Base58 and Address stuff only ..
I will work alot more on this when we have a testnet/client

[emski]

You've started a rewrite. Good.
However without test network for bitshares 2.0 it looks impractical, doesn't it ?

[xeroc]

Most of the changes I've made in this script modification are now merged in xeroc's script.
There is no need to support this mod anymore.

Maybe you want to join python development?
I can definitely need your help!

I'll be happy to help whenever I can.

I have started a complete rewrite for BitShares 2.0 and published it over here:
https://github.com/xeroc/python-bitshareslib

I plan to update most scripts to use the new library with BitShares 2.0 once released but have to do more work on the the transactions module (lots of changes under the hood)
Feel free to read into the existing code and send pull request ..

However, note that I am unavailable for the next three weeks ... HOLIDAY!!!

[emski]

Most of the changes I've made in this script modification are now merged in xeroc's script.
There is no need to support this mod anymore.

Maybe you want to join python development?
I can definitely need your help!

I'll be happy to help whenever I can.

[xeroc]

Most of the changes I've made in this script modification are now merged in xeroc's script.
There is no need to support this mod anymore.

Maybe you want to join python development?
I can definitely need your help!

[emski]

Most of the changes I've made in this script modification are now merged in xeroc's script.
There is no need to support this mod anymore.

[xeroc]

I've pushed an update fixing the issue with yahoo.

Thanks xeroc for the support.
I was away snowboarding in the alps without reliable internet access.

I'll post some more updates when I'm able to catch up with all the events.

Dang .. I was in the alps snowboarding too, this weekend
got sun burned .. though the snow was awesome

[emski]

I've pushed an update fixing the issue with yahoo.

Thanks xeroc for the support.
I was away snowboarding in the alps without reliable internet access.

I'll post some more updates when I'm able to catch up with all the events.

[xeroc]

xeroc, why does your script lock the wallet after updating feed? the delegate needs an unlocked wallet to run?

Thats because my feed scripts are not run on the delegate machine ..

just remove the
rpc.lock()
.. it is either at the end of thw script or in the update routine ... cant check atm

... gonna add a switch to the config soon (tm)

[kokojie]

xeroc, why does your script lock the wallet after updating feed? the delegate needs an unlocked wallet to run?

[kokojie]

i'd guess you have installed prettytable into python2 .. maybe there ia a separate pip3 or so for python3 .. else you can try easy_install3 .. dependa on your linux distribution ..

Holy shit, there's indeed a "pip3" command.  Your script now works now for me. Thanks for all your help.

[cube]

I'm getting a "unknown error - yahoo" now when I run the script. Anyone else getting this?

You may try this change and see if it helps.

<code>

   #yahooprices =  response.text.split( '\r\n' )
   yahooprices =  response.text.replace('\r','').split( '\n' )

</code>

[xeroc]

i'd guess you have installed prettytable into python2 .. maybe there ia a separate pip3 or so for python3 .. else you can try easy_install3 .. dependa on your linux distribution ..

[kokojie]

xeroc, I did "pip install prettytable", and it tell me successfully installed prettytable

Then I get the same error when I run your script: No module named 'prettytable'

What did I do wrong?

[xeroc]

Yup .. i.stall ftom that repo .. also need prettytable ..
Read the README in bitshares-pytools too

[kokojie]

You need python3!!

ok so I needed to run "python3 main.py ALL", but then I get this error:
File "main.py", line 4, in <module>
import bitsharesrpc
ImportError: No module named 'bitsharesrpc'

Do I need to install this too? https://github.com/xeroc/python-bitsharesrpc

Also, it looks like I need a "PrettyTable" module too? where do I get this.

[xeroc]

You need python3!!

[kokojie]

Oh .. you run the script with

python main.py ALL

Will publish all prices implemented in the script

I certainly have to do more documentation on ALL my scripts .. give me some time please

I get an error after running that:
File "main.py", line 413
  print("Loading data: ", end="", flush=True)
                                                   ^
SyntaxError: invalid syntax

[xeroc]

Oh .. you run the script with

python main.py ALL

Will publish all prices implemented in the script

I certainly have to do more documentation on ALL my scripts .. give me some time please

[kokojie]

I'm getting a "unknown error - yahoo" now when I run the script. Anyone else getting this?

I made some modifications and moved the script into bitshares-pytools on github .. please try that version

I'm using emski's script, the problem is directed at emski.

btw, I can't get your script to work, can you write some documentation on github, like where am I suppose to put the asset symbols? I don't see them in the config-example.

[xeroc]

I'm getting a "unknown error - yahoo" now when I run the script. Anyone else getting this?

I made some modifications and moved the script into bitshares-pytools on github .. please try that version

[kokojie]

I'm getting a "unknown error - yahoo" now when I run the script. Anyone else getting this?

[xeroc]

EDIT: Just to note that currently bter returns an error so it is ignored by the script anyway

unless you set it to >0.7 or then the script will stop .. at least my implementation

[emski]

I recommend every delegate to set bter_trust_level to 0.0 in the config file

Given the recent events this might be a good idea.
Delegates are free to modify the configuration as they see need for it.

EDIT: Just to note that currently bter returns an error so it is ignored by the script anyway

[xeroc]

I recommend every delegate to set bter_trust_level to 0.0 in the config file

[emski]

I've pushed an update fixing Poloniex's asset name change.

same here ... i should open up my own thread for updates and not misuse yours

Given the fact both scripts are mostly identical it is not a big deal (:

[xeroc]

I've pushed an update fixing Poloniex's asset name change.

same here ... i should open up my own thread for updates and not misuse yours

[emski]

I've pushed an update fixing Poloniex's asset name change.

[clayop]

Added to my slate. Thanks!

[emski]

https://bitsharestalk.org/index.php?topic=9857.0

[clayop]

Can you add KRW to your script? There is a growing interest in BitKRW and I want to introduce Korean community how to use it. (Plus encouraging Korean exchanges to add BitKRW)

Done!

I didn't realise how much feed it already got. Also I see a lot of new markets were enabled recently. I'm updating my feeds so that I publish feed for each active market (and KRW as it currently has 49 feeds).

Thanks Emski!!! (are you running delegate? if so please let me know your delegate)

[emski]

Can you add KRW to your script? There is a growing interest in BitKRW and I want to introduce Korean community how to use it. (Plus encouraging Korean exchanges to add BitKRW)

Done!

I didn't realise how much feed it already got. Also I see a lot of new markets were enabled recently. I'm updating my feeds so that I publish feed for each active market (and KRW as it currently has 49 feeds).

[clayop]

Can you add KRW to your script? There is a growing interest in BitKRW and I want to introduce Korean community how to use it. (Plus encouraging Korean exchanges to add BitKRW)

[emski]

I've tested the script on v0.4.26 and it is working as designed.

[emski]

I was talking about how it was updating every single block. You're just wasting your own money if most delegates do it much less often.

It updates according to configuration. I intentionally configured it like that. If the price changes more than specified threshold it updates the feed no matter what. You can always specify different thresholds.

EDIT: Furthermore the script runs at more than 3 minutes intervals. It cannot update the feed in less than 3 minutes (at least on my side). What you observed in that transaction spam wasn't produced by single delegate. A lot of these messages were delegate version updates and public data fields.

I believe the script is performing as designed. If you dont like it we can discuss that.

[toast]

I was talking about how it was updating every single block. You're just wasting your own money if most delegates do it much less often.

[svk]

I'll stop it for some brief investigation.

I think it's simply due to all the missed blocks and the general state of the network, transactions end up as pending so to the script it appears as if there hasn't been any recent updates.

[emski]

I'll stop it for some brief investigation.

[emski]

Is this script spamming feed updates right now?

http://bitsharesblocks.com/blocks/block?id=1248105

On my end it is publishing feed due to market changes.

2 consequtive runs:

New Feed differs for EUR : Old:0.0144859928 ; New:0.0138386806 ; Diff:0.0006473123 ; Max allowed Diff:-0.0000289720 +0.0001448599 ; Force updating!
New Feed differs for BTC : Old:0.0000512105 ; New:0.0000490656 ; Diff:0.0000021449 ; Max allowed Diff:-0.0000001024 +0.0000005121 ; Force updating!
New Feed differs for GOLD : Old:0.0000147824 ; New:0.0000140996 ; Diff:0.0000006828 ; Max allowed Diff:-0.0000000296 +0.0000001478 ; Force updating!
New Feed differs for USD : Old:0.0180345335 ; New:0.0172444977 ; Diff:0.0007900359 ; Max allowed Diff:-0.0000360691 +0.0001803453 ; Force updating!
New Feed differs for CNY : Old:0.1116319582 ; New:0.1066865365 ; Diff:0.0049454217 ; Max allowed Diff:-0.0002232639 +0.0011163196 ; Force updating!


Loading data: рyahoo, BTC38, BTer, Poloniex, bittrex -- done. Calculating btsx feeds prices and checking publish rules.
============================================================================================================================================================================================================================
EUR|new: 0.0138385BTSX (e:0.0124887/0.0138792) (bc:0.0143143)  |  change: -0.0006%  |  change (to med.): -0.0476%  |  exchange (median): +0.2938%  |  exchange (range): -30.4115% to +0.8562%  |  last update: 0:00:45.946552 ago
BTC|new: 0.0000491BTSX (e:0.0000463/0.0000493) (bc:0.0000502)  |  change: +0.0000%  |  change (to med.): -0.0001%  |  exchange (median): +0.5070%  |  exchange (range): -30.5407% to .0706%  |  last update: 0:00:45.946819 ago
GOLD|new: 0.0000141BTSX (e:0.0000127/0.0000141) (bc:0.0000146)  |  change: -0.0000%  |  change (to med.): -0.0000%  |  exchange (median): +0.2939%  |  exchange (range): -31.9129% to +0.8563%  |  last update: 0:00:45.947105 ago
USD|new: 0.0172445BTSX (e:0.0155624/0.0172952) (bc:0.0178263)  |  change: -0.0005%  |  change (to med.): -0.0582%  |  exchange (median): +0.2938%  |  exchange (range): -30.4126% to +0.8562%  |  last update: 0:00:45.947383 ago
CNY|new: 0.1066865BTSX (e:0.0962801/0.1070000) (bc:0.1102914)  |  change: -0.0029%  |  change (to med.): -0.3605%  |  exchange (median): +0.2938%  |  exchange (range): -30.4126% to +0.8562%  |  last update: 0:00:45.947652 ago
no update required

[toast]

Is this script spamming feed updates right now?

http://bitsharesblocks.com/blocks/block?id=1248105

[emski]

Actually I've updated the script to work with 0.4.25.
You can safely update now.

[emski]

Think we need an update to handle the new date format, "oldtime" is no longer parsed correctly.  This seems to work but I'm not 100% sure:

Code: [Select]

oldtime[ f[ "asset_symbol" ] ] = datetime.strptime(f["last_update"],"%Y-%m-%dT%H:%M:%S")

I'll look into it after work. For now you can use the feed through the old client.

[svk]

Think we need an update to handle the new date format, "oldtime" is no longer parsed correctly.  This seems to work but I'm not 100% sure:

Code: [Select]

oldtime[ f[ "asset_symbol" ] ] = datetime.strptime(f["last_update"],"%Y-%m-%dT%H:%M:%S")

[emski]

I've pushed a fix to handle the btsx->bts name change on btc38 .

[emski]

I've pushed a check for malformed exchange reply (btc38 only for now).
It appears that there is an issue with btc38.
I'll investigate further tomorrow as it is 2:00 am here and I've driven about 600 km on Bulgarian roads today.

[xeroc]

cool, emski yours is working for me now

The error in xeroc's script you see might be related to improper config file.
There are different variables in both versions. Make sure you use proper version.

example:

Code: [Select]

{                                                                                                                                       
  "bts_rpc": {
    "url": "http://10.0.0.16:19988/rpc",
    "username": ""
    "password": ""
  }, 
  "delegate_list": [ "delegate.xeroc"],
  "payaccount" : "delegate.xeroc",
  "maxAgeFeedInSeconds" : 2700,
  "minValidAssetPrice" : 0.00001,
  "discount" : 0.995,
  "change_min" : 0.5,
  "btc38_trust_level" : 0.7,
  "bter_trust_level" : 1.0,
  "poloniex_trust_level" : 0.5,
  "bittrex_trust_level" : 0.5
}




[emski]

cool, emski yours is working for me now

The error in xeroc's script you see might be related to improper config file.
There are different variables in both versions. Make sure you use proper version.

[kokojie]

cool, emski yours is working for me now

[emski]

My bad. I've committed not working version.
Should be OK now.

[emski]

I still get errors with both emski and xeroc's updated script:

emski:
  File "./btsx_feed.py", line 125
    volume_in_btc[ mapAsset ].append(float(result[coin.lower()+"_btc"]["vol_btc"])*confi$
                                                                                                                                           ^
SyntaxError: invalid syntax


xeroc:
  File "/home/user/feed_xeroc/btsx_feed.py", line 433, in <module>
    if publish_rule() :
  File "/home/user/feed_xeroc/btsx_feed.py", line 47, in publish_rule
    elif fabs(myCurrentFeed[asset]-realPrice[asset])/realPrice[asset] > config["change_min"] and\
KeyError: 'change_min'

What arguments are you passing to the scripts ?
Example: "./btsx_feed.py GOLD BTC USD CNY EUR"

Are you sure you have the file as in github ?

[kokojie]

I still get errors with both emski and xeroc's updated script:

emski:
  File "./btsx_feed.py", line 125
    volume_in_btc[ mapAsset ].append(float(result[coin.lower()+"_btc"]["vol_btc"])*confi$
                                                                                                                                           ^
SyntaxError: invalid syntax


xeroc:
  File "/home/user/feed_xeroc/btsx_feed.py", line 433, in <module>
    if publish_rule() :
  File "/home/user/feed_xeroc/btsx_feed.py", line 47, in publish_rule
    elif fabs(myCurrentFeed[asset]-realPrice[asset])/realPrice[asset] > config["change_min"] and\
KeyError: 'change_min'

[xeroc]

Apparently BTER changed BTSX asset name to BTS.
I've pushed a quick patch to address that.
I'll try to make something more robust soon.

dito ... had my inbox filled with errors .. need to put hand on the script some more ..

[emski]

Apparently BTER changed BTSX asset name to BTS.
I've pushed a quick patch to address that.
I'll try to make something more robust soon.

[emski]

I've updated with changes related to v0.4.24 .
I've merged some bugfixes (thanks to xeroc (python identation is tricky)).

[xeroc]

I think there might be an error in discount implementation in the original script.
It increases the feed price instead of decreasing it.

assume discount = 0.995
Original code:

Code: [Select]

### Discount
  price_in_btsx_weighted[asset] = price_in_btsx_weighted[asset] / config["discount"]

My code:

Code: [Select]

### Discount
  price_in_btsx_weighted[asset] = price_in_btsx_weighted[asset] * config["discount"]


For sake of completeness

I fixed that issue bug in my code :-D

[emski]

I've merged the fix for btc38 failure (thanks alt).

[vikram]

Seems like it might be good to make a wiki page listing the different price feed scripts and their main differences. Something in the spirit of this: https://en.wikipedia.org/wiki/Tomato_%28firmware%29#Feature_comparison but of course not so complicated.

[emski]

I think there might be an error in discount implementation in the original script.
It increases the feed price instead of decreasing it.

assume discount = 0.995
Original code:

Code: [Select]

### Discount
  price_in_btsx_weighted[asset] = price_in_btsx_weighted[asset] / config["discount"]

My code:

Code: [Select]

### Discount
  price_in_btsx_weighted[asset] = price_in_btsx_weighted[asset] * config["discount"]


[liondani]

I as a user/owner have always advocated for weighing the price feed from different exchanges, so I am happy to see that and hope more delegates will start using this script.

The other features seem beneficial to the delegates themselves, so I hope for a fast adoption.

0.02 BTSX

until now I published the feeds manually...  With this approach I consider seriously to start using the script.
Thanks to alt, xeroc and emski for their efforts   

[tonyk]

I as a user/owner have always advocated for weighing the price feed from different exchanges, so I am happy to see that and hope more delegates will start using this script.

The other features seem beneficial to the delegates themselves, so I hope for a fast adoption.

0.02 BTSX

[emski]

I've done some modifications to xeroc's (which was derived from alt's version) version of the price feed script.
You can see it here: https://github.com/emilvelichkov/pytshares

Basically I've added weighting of each exchange so a delegate could select which exchange matters more.

Also I've changed the conditions when the feed is published:
Nothing except time restrictions and price difference is taken into account. If price differs more than [-max_negative_diff , max_positive_diff]*MyLastFeed feed is published. By default: [-0.2%, 0.5%].
Feeds cannot be published more frequently than minFeedAgeInSeconds.

Also I've changed the logic of price fetching to include additional validation and to ignore error from exchanges with low trust_level. This way failure to fetch price data from low trusted exchange will not prevent the calculation and publishing of feeds.

It is still experimental so some issues might arise. Some proof-read is welcome.

Here is the full changelog:
+ added minFeedAgeInSeconds (minimum time between feed publish)
+ added minValidAssetPrice (any feed for asset with lower price will be ignored)
+ added max_positive_diff/max_negative_diff (asset price should increase more than max_positive_diff in order to be published)
+ added exchange trust level (it multiplies the volume of the exchange.)
* changed publish_rule in the following way:
price is published if maxFeedAgeInSeconds have passed
price is published if price of any asset changes more than [-max_negative_diff, max_positive_diff]
* Errors fetching prices from any exchange are ignored unless exchange_trust_level > 0.8 (this is to prevent single small exchange from stopping all feeds)