BitShares Forum
Main => General Discussion => Topic started by: emski on October 04, 2014, 11:42:19 pm
-
I've done some modifications to xeroc's (which was derived from alt's version) version of the price feed script.
You can see it here: https://github.com/emilvelichkov/pytshares (https://github.com/emilvelichkov/pytshares)
Basically I've added weighting of each exchange so a delegate could select which exchange matters more.
Also I've changed the conditions when the feed is published:
Nothing except time restrictions and price difference is taken into account. If price differs more than [-max_negative_diff , max_positive_diff]*MyLastFeed feed is published. By default: [-0.2%, 0.5%].
Feeds cannot be published more frequently than minFeedAgeInSeconds.
Also I've changed the logic of price fetching to include additional validation and to ignore error from exchanges with low trust_level. This way failure to fetch price data from low trusted exchange will not prevent the calculation and publishing of feeds.
It is still experimental so some issues might arise. Some proof-read is welcome.
Here is the full changelog:
+ added minFeedAgeInSeconds (minimum time between feed publish)
+ added minValidAssetPrice (any feed for asset with lower price will be ignored)
+ added max_positive_diff/max_negative_diff (asset price should increase more than max_positive_diff in order to be published)
+ added exchange trust level (it multiplies the volume of the exchange.)
* changed publish_rule in the following way:
price is published if maxFeedAgeInSeconds have passed
price is published if price of any asset changes more than [-max_negative_diff, max_positive_diff]
* Errors fetching prices from any exchange are ignored unless exchange_trust_level > 0.8 (this is to prevent single small exchange from stopping all feeds)
-
I as a user/owner have always advocated for weighing the price feed from different exchanges, so I am happy to see that and hope more delegates will start using this script.
The other features seem beneficial to the delegates themselves, so I hope for a fast adoption.
0.02 BTSX
-
I as a user/owner have always advocated for weighing the price feed from different exchanges, so I am happy to see that and hope more delegates will start using this script.
The other features seem beneficial to the delegates themselves, so I hope for a fast adoption.
0.02 BTSX
until now I published the feeds manually... With this approach I consider seriously to start using the script.
Thanks to alt, xeroc and emski for their efforts ;)
-
I think there might be an error in discount implementation in the original script.
It increases the feed price instead of decreasing it.
assume discount = 0.995
Original code:
### Discount
price_in_btsx_weighted[asset] = price_in_btsx_weighted[asset] / config["discount"]
My code:
### Discount
price_in_btsx_weighted[asset] = price_in_btsx_weighted[asset] * config["discount"]
-
Seems like it might be good to make a wiki page listing the different price feed scripts and their main differences. Something in the spirit of this: https://en.wikipedia.org/wiki/Tomato_%28firmware%29#Feature_comparison but of course not so complicated.
-
I've merged the fix for btc38 failure (thanks alt).
-
I think there might be an error in discount implementation in the original script.
It increases the feed price instead of decreasing it.
assume discount = 0.995
Original code:
### Discount
price_in_btsx_weighted[asset] = price_in_btsx_weighted[asset] / config["discount"]
My code:
### Discount
price_in_btsx_weighted[asset] = price_in_btsx_weighted[asset] * config["discount"]
For sake of completeness :)
I fixed that issue bug in my code :-D
-
I've updated with changes related to v0.4.24 .
I've merged some bugfixes (thanks to xeroc (python identation is tricky)).
-
Apparently BTER changed BTSX asset name to BTS.
I've pushed a quick patch to address that.
I'll try to make something more robust soon.
-
Apparently BTER changed BTSX asset name to BTS.
I've pushed a quick patch to address that.
I'll try to make something more robust soon.
dito ... had my inbox filled with errors .. need to put hand on the script some more ..
-
I still get errors with both emski and xeroc's updated script:
emski:
File "./btsx_feed.py", line 125
volume_in_btc[ mapAsset ].append(float(result[coin.lower()+"_btc"]["vol_btc"])*confi$
^
SyntaxError: invalid syntax
xeroc:
File "/home/user/feed_xeroc/btsx_feed.py", line 433, in <module>
if publish_rule() :
File "/home/user/feed_xeroc/btsx_feed.py", line 47, in publish_rule
elif fabs(myCurrentFeed[asset]-realPrice[asset])/realPrice[asset] > config["change_min"] and\
KeyError: 'change_min'
-
I still get errors with both emski and xeroc's updated script:
emski:
File "./btsx_feed.py", line 125
volume_in_btc[ mapAsset ].append(float(result[coin.lower()+"_btc"]["vol_btc"])*confi$
^
SyntaxError: invalid syntax
xeroc:
File "/home/user/feed_xeroc/btsx_feed.py", line 433, in <module>
if publish_rule() :
File "/home/user/feed_xeroc/btsx_feed.py", line 47, in publish_rule
elif fabs(myCurrentFeed[asset]-realPrice[asset])/realPrice[asset] > config["change_min"] and\
KeyError: 'change_min'
What arguments are you passing to the scripts ?
Example: "./btsx_feed.py GOLD BTC USD CNY EUR"
Are you sure you have the file as in github ?
-
My bad. I've committed not working version.
Should be OK now.
-
cool, emski yours is working for me now
-
cool, emski yours is working for me now
The error in xeroc's script you see might be related to improper config file.
There are different variables in both versions. Make sure you use proper version.
-
cool, emski yours is working for me now
The error in xeroc's script you see might be related to improper config file.
There are different variables in both versions. Make sure you use proper version.
example:
{
"bts_rpc": {
"url": "http://10.0.0.16:19988/rpc",
"username": ""
"password": ""
},
"delegate_list": [ "delegate.xeroc"],
"payaccount" : "delegate.xeroc",
"maxAgeFeedInSeconds" : 2700,
"minValidAssetPrice" : 0.00001,
"discount" : 0.995,
"change_min" : 0.5,
"btc38_trust_level" : 0.7,
"bter_trust_level" : 1.0,
"poloniex_trust_level" : 0.5,
"bittrex_trust_level" : 0.5
}
-
I've pushed a check for malformed exchange reply (btc38 only for now).
It appears that there is an issue with btc38.
I'll investigate further tomorrow as it is 2:00 am here and I've driven about 600 km on Bulgarian roads today.
-
I've pushed a fix to handle the btsx->bts name change on btc38 .
-
Think we need an update to handle the new date format, "oldtime" is no longer parsed correctly. This seems to work but I'm not 100% sure:
oldtime[ f[ "asset_symbol" ] ] = datetime.strptime(f["last_update"],"%Y-%m-%dT%H:%M:%S")
-
Think we need an update to handle the new date format, "oldtime" is no longer parsed correctly. This seems to work but I'm not 100% sure:
oldtime[ f[ "asset_symbol" ] ] = datetime.strptime(f["last_update"],"%Y-%m-%dT%H:%M:%S")
I'll look into it after work. For now you can use the feed through the old client.
-
Actually I've updated the script to work with 0.4.25.
You can safely update now.
-
Is this script spamming feed updates right now?
http://bitsharesblocks.com/blocks/block?id=1248105
-
Is this script spamming feed updates right now?
http://bitsharesblocks.com/blocks/block?id=1248105
On my end it is publishing feed due to market changes.
2 consequtive runs:
New Feed differs for EUR : Old:0.0144859928 ; New:0.0138386806 ; Diff:0.0006473123 ; Max allowed Diff:-0.0000289720 +0.0001448599 ; Force updating!
New Feed differs for BTC : Old:0.0000512105 ; New:0.0000490656 ; Diff:0.0000021449 ; Max allowed Diff:-0.0000001024 +0.0000005121 ; Force updating!
New Feed differs for GOLD : Old:0.0000147824 ; New:0.0000140996 ; Diff:0.0000006828 ; Max allowed Diff:-0.0000000296 +0.0000001478 ; Force updating!
New Feed differs for USD : Old:0.0180345335 ; New:0.0172444977 ; Diff:0.0007900359 ; Max allowed Diff:-0.0000360691 +0.0001803453 ; Force updating!
New Feed differs for CNY : Old:0.1116319582 ; New:0.1066865365 ; Diff:0.0049454217 ; Max allowed Diff:-0.0002232639 +0.0011163196 ; Force updating!
Loading data: рyahoo, BTC38, BTer, Poloniex, bittrex -- done. Calculating btsx feeds prices and checking publish rules.
============================================================================================================================================================================================================================
EUR|new: 0.0138385BTSX (e:0.0124887/0.0138792) (bc:0.0143143) | change: -0.0006% | change (to med.): -0.0476% | exchange (median): +0.2938% | exchange (range): -30.4115% to +0.8562% | last update: 0:00:45.946552 ago
BTC|new: 0.0000491BTSX (e:0.0000463/0.0000493) (bc:0.0000502) | change: +0.0000% | change (to med.): -0.0001% | exchange (median): +0.5070% | exchange (range): -30.5407% to +1.0706% | last update: 0:00:45.946819 ago
GOLD|new: 0.0000141BTSX (e:0.0000127/0.0000141) (bc:0.0000146) | change: -0.0000% | change (to med.): -0.0000% | exchange (median): +0.2939% | exchange (range): -31.9129% to +0.8563% | last update: 0:00:45.947105 ago
USD|new: 0.0172445BTSX (e:0.0155624/0.0172952) (bc:0.0178263) | change: -0.0005% | change (to med.): -0.0582% | exchange (median): +0.2938% | exchange (range): -30.4126% to +0.8562% | last update: 0:00:45.947383 ago
CNY|new: 0.1066865BTSX (e:0.0962801/0.1070000) (bc:0.1102914) | change: -0.0029% | change (to med.): -0.3605% | exchange (median): +0.2938% | exchange (range): -30.4126% to +0.8562% | last update: 0:00:45.947652 ago
no update required
-
I'll stop it for some brief investigation.
-
I'll stop it for some brief investigation.
I think it's simply due to all the missed blocks and the general state of the network, transactions end up as pending so to the script it appears as if there hasn't been any recent updates.
-
I was talking about how it was updating every single block. You're just wasting your own money if most delegates do it much less often.
-
I was talking about how it was updating every single block. You're just wasting your own money if most delegates do it much less often.
It updates according to configuration. I intentionally configured it like that. If the price changes more than specified threshold it updates the feed no matter what. You can always specify different thresholds.
EDIT: Furthermore the script runs at more than 3 minutes intervals. It cannot update the feed in less than 3 minutes (at least on my side). What you observed in that transaction spam wasn't produced by single delegate. A lot of these messages were delegate version updates and public data fields.
I believe the script is performing as designed. If you dont like it we can discuss that.
-
I've tested the script on v0.4.26 and it is working as designed.
-
Can you add KRW to your script? There is a growing interest in BitKRW and I want to introduce Korean community how to use it. (Plus encouraging Korean exchanges to add BitKRW)
-
Can you add KRW to your script? There is a growing interest in BitKRW and I want to introduce Korean community how to use it. (Plus encouraging Korean exchanges to add BitKRW)
Done!
I didn't realise how much feed it already got. Also I see a lot of new markets were enabled recently. I'm updating my feeds so that I publish feed for each active market (and KRW as it currently has 49 feeds).
-
Can you add KRW to your script? There is a growing interest in BitKRW and I want to introduce Korean community how to use it. (Plus encouraging Korean exchanges to add BitKRW)
Done!
I didn't realise how much feed it already got. Also I see a lot of new markets were enabled recently. I'm updating my feeds so that I publish feed for each active market (and KRW as it currently has 49 feeds).
Thanks Emski!!! (are you running delegate? if so please let me know your delegate)
-
https://bitsharestalk.org/index.php?topic=9857.0 (https://bitsharestalk.org/index.php?topic=9857.0)
-
Added to my slate. Thanks!
-
I've pushed an update fixing Poloniex's asset name change.
-
I've pushed an update fixing Poloniex's asset name change.
same here ... i should open up my own thread for updates :) and not misuse yours 8)
-
I've pushed an update fixing Poloniex's asset name change.
same here ... i should open up my own thread for updates :) and not misuse yours 8)
Given the fact both scripts are mostly identical it is not a big deal (:
-
I recommend every delegate to set bter_trust_level to 0.0 in the config file
-
I recommend every delegate to set bter_trust_level to 0.0 in the config file
Given the recent events this might be a good idea.
Delegates are free to modify the configuration as they see need for it.
EDIT: Just to note that currently bter returns an error so it is ignored by the script anyway
-
EDIT: Just to note that currently bter returns an error so it is ignored by the script anyway
unless you set it to >0.7 or then the script will stop .. at least my implementation
-
I'm getting a "unknown error - yahoo" now when I run the script. Anyone else getting this?
-
I'm getting a "unknown error - yahoo" now when I run the script. Anyone else getting this?
I made some modifications and moved the script into bitshares-pytools on github .. please try that version
-
I'm getting a "unknown error - yahoo" now when I run the script. Anyone else getting this?
I made some modifications and moved the script into bitshares-pytools on github .. please try that version
I'm using emski's script, the problem is directed at emski.
btw, I can't get your script to work, can you write some documentation on github, like where am I suppose to put the asset symbols? I don't see them in the config-example.
-
Oh .. you run the script with
python main.py ALL
Will publish all prices implemented in the script
I certainly have to do more documentation on ALL my scripts .. give me some time please
-
Oh .. you run the script with
python main.py ALL
Will publish all prices implemented in the script
I certainly have to do more documentation on ALL my scripts .. give me some time please
I get an error after running that:
File "main.py", line 413
print("Loading data: ", end="", flush=True)
^
SyntaxError: invalid syntax
-
You need python3!!
-
You need python3!!
ok so I needed to run "python3 main.py ALL", but then I get this error:
File "main.py", line 4, in <module>
import bitsharesrpc
ImportError: No module named 'bitsharesrpc'
Do I need to install this too? https://github.com/xeroc/python-bitsharesrpc
Also, it looks like I need a "PrettyTable" module too? where do I get this.
-
Yup .. i.stall ftom that repo .. also need prettytable ..
Read the README in bitshares-pytools too
-
xeroc, I did "pip install prettytable", and it tell me successfully installed prettytable
Then I get the same error when I run your script: No module named 'prettytable'
What did I do wrong?
-
i'd guess you have installed prettytable into python2 .. maybe there ia a separate pip3 or so for python3 .. else you can try easy_install3 .. dependa on your linux distribution ..
-
I'm getting a "unknown error - yahoo" now when I run the script. Anyone else getting this?
You may try this change and see if it helps.
<code>
#yahooprices = response.text.split( '\r\n' )
yahooprices = response.text.replace('\r','').split( '\n' )
</code>
-
i'd guess you have installed prettytable into python2 .. maybe there ia a separate pip3 or so for python3 .. else you can try easy_install3 .. dependa on your linux distribution ..
Holy shit, there's indeed a "pip3" command. Your script now works now for me. Thanks for all your help.
-
xeroc, why does your script lock the wallet after updating feed? the delegate needs an unlocked wallet to run?
-
xeroc, why does your script lock the wallet after updating feed? the delegate needs an unlocked wallet to run?
Thats because my feed scripts are not run on the delegate machine ..
just remove the
rpc.lock()
.. it is either at the end of thw script or in the update routine ... cant check atm
... gonna add a switch to the config soon (tm)
-
I've pushed an update fixing the issue with yahoo.
Thanks xeroc for the support.
I was away snowboarding in the alps without reliable internet access.
I'll post some more updates when I'm able to catch up with all the events.
-
I've pushed an update fixing the issue with yahoo.
Thanks xeroc for the support.
I was away snowboarding in the alps without reliable internet access.
I'll post some more updates when I'm able to catch up with all the events.
Dang .. I was in the alps snowboarding too, this weekend 8)
got sun burned .. though the snow was awesome :)
-
Most of the changes I've made in this script modification are now merged in xeroc's script.
There is no need to support this mod anymore.
-
Most of the changes I've made in this script modification are now merged in xeroc's script.
There is no need to support this mod anymore.
Maybe you want to join python development?
I can definitely need your help!
-
Most of the changes I've made in this script modification are now merged in xeroc's script.
There is no need to support this mod anymore.
Maybe you want to join python development?
I can definitely need your help!
I'll be happy to help whenever I can.
-
Most of the changes I've made in this script modification are now merged in xeroc's script.
There is no need to support this mod anymore.
Maybe you want to join python development?
I can definitely need your help!
I'll be happy to help whenever I can.
I have started a complete rewrite for BitShares 2.0 and published it over here:
https://github.com/xeroc/python-bitshareslib
I plan to update most scripts to use the new library with BitShares 2.0 once released but have to do more work on the the transactions module (lots of changes under the hood)
Feel free to read into the existing code and send pull request ..
However, note that I am unavailable for the next three weeks ... HOLIDAY!!! :D
-
You've started a rewrite. Good.
However without test network for bitshares 2.0 it looks impractical, doesn't it ?
-
You've started a rewrite. Good.
However without test network for bitshares 2.0 it looks impractical, doesn't it ?
Yes ... that's why it mostly only contains the Base58 and Address stuff only ..
I will work alot more on this when we have a testnet/client
-
Hi guys, I have a few questions about price feeds that I was hoping to get help with...
I'm using a wallet with some funds (not the delegate wallet) to try to post feeds for the 'bitspace-clains' delegate.
I'm using xeroc's bitshares-pytools, and think I managed to get the dependencies right. At first run, I would always get the message 'no update required' and -1% change on all assets in the table, so I edited 'main.py' to remove the required update test (is that ok for first time running feed script?). Now it runs, and I get the following output:
Update required! Forcing now!
Unlocking wallet
publishing feeds for delegate: bitspace-clains
Traceback (most recent call last):
File "main.py", line 473, in <module>
update_feed(rpc,asset_list_final)
File "main.py", line 273, in update_feed
result = rpc.wallet_publish_feeds(delegate, assets)
File "/home/manuel/.local/lib/python3.4/site-packages/bitsharesrpc/client.py", line 135, in method
"id": 0
File "/home/manuel/.local/lib/python3.4/site-packages/bitsharesrpc/client.py", line 78, in rpcexec
raise RPCError(ret['error']['detail'])
bitsharesrpc.client.RPCError: 20010 insufficient_funds: insufficient funds
{"from_account_name":"bitspace-clains","amount_to_withdraw":{"amount":50000,"asset_id":0},"balance_records":[]}
th_a wallet.cpp:68 withdraw_to_transaction
Do I understand correctly that the script is trying to draw funds from the delegate account? I'm guessing that means that a feed must be published from the delegate account itself...?
If so, how do you avoid inserting the wallet password into 'config.py' in cleartext?
Also, how much funds is required to post the delegate feeds?
-
It says:
20010 insufficient_funds: insufficient funds
So you delegate does not have funds (either in his active balance or in his delegates pay) that can be used to pay for the transaction fee. Each publishing transaction costs 0.5 BTS
-
I'm trying to run the script connecting to a wallet that is not the wallet of the delegate. The wallet has a 50BTS balance right now, so that should be enough.
Does it mean that the script must connect to the delegate wallet itself and draw funds from that? If so, how can I do that without providing the delegate wallet password in cleartext in 'config.py'?
-
I'm trying to run the script connecting to a wallet that is not the wallet of the delegate. The wallet has a 50BTS balance right now, so that should be enough.
So it seem your delegate is "running" on a different machine then?
Does it mean that the script must connect to the delegate wallet itself and draw funds from that? If so, how can I do that without providing the delegate wallet password in cleartext in 'config.py'?
If you want to provide continuous updates for the feed, then you can only put the password in cleartext into the config. BUT, you should create a wallet that only contains the most recent active key and signing key of the delegate in it .. and no other account ...
Then you can do a rescan of the blockchain to catch up with the balance and run the script again
-
I'm trying to run the script connecting to a wallet that is not the wallet of the delegate. The wallet has a 50BTS balance right now, so that should be enough.
So it seem your delegate is "running" on a different machine then?
Yes. The delegate with block production enabled is running on a different machine.
Does it mean that the script must connect to the delegate wallet itself and draw funds from that? If so, how can I do that without providing the delegate wallet password in cleartext in 'config.py'?
If you want to provide continuous updates for the feed, then you can only put the password in cleartext into the config. BUT, you should create a wallet that only contains the most recent active key and signing key of the delegate in it .. and no other account ...
Then you can do a rescan of the blockchain to catch up with the balance and run the script again
Ok, so I guess there's not much point to running blocks and script on different machines then... it's more a question of protecting the delegate block production environment by not revealing owner keys?
I'm gonna take the opportunity to read up on owner keys, active keys and sign keys. I found a couple of your guides on the subject, i will follow these.
https://bitsharestalk.org/index.php?topic=14344.0
https://bitsharestalk.org/index.php?topic=14360.0
Thanks for the help! :)
-
Ok, so I guess there's not much point to running blocks and script on different machines then... it's more a question of protecting the delegate block production environment by not revealing owner keys?
I'm gonna take the opportunity to read up on owner keys, active keys and sign keys. I found a couple of your guides on the subject, i will follow these.
https://bitsharestalk.org/index.php?topic=14344.0
https://bitsharestalk.org/index.php?topic=14360.0
Thanks for the help! :)
The most important thing is the OWNER KEY ..
I recommend you read this:
http://wiki.bitshares.org/index.php/Delegate/Security