Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.


Messages - litepresence

Pages: [1] 2 3 4 5
1
General Discussion / [ANN] metaNODE = Bitshares_Trustless_Client()
« on: June 06, 2018, 01:52:45 pm »
metaNODE = Bitshares_Trustless_Client()

There were two ways to get data from Bitshares blockchain:

- a private node that uses lots of RAM, prefers its own machine, and is technical to tend
- a public node that is difficult to stay connected to and may provide rogue data

I've created a 3rd path; connect to several random nodes in the public network... ask them each for latest market data; then move on to other nodes in the network continually. Finally, perform statistical analysis on the combined feeds and maintain a streaming curated output file; the metaNODE.

python script and whitepaper, including usage:

https://github.com/litepresence/extinction-event/tree/master/metaNODE

Code: [Select]
LICENSE:

'litepresence 2018'

def WTFPL_v0_March_1765():
    if any([stamps, licenses, taxation, regulation, fiat, etat]):
        try:
            print('no thank you')
        except:
            return [tar, feathers]

Mission:

1/10 the RAM usage of a personal node
99.9999 six sigma uptime
99.9999 six sigma accurate data feed
less than 5000ms latency


metaNODE is a streaming text file that contains statistically curated dex data from all public nodes for a single market on a single account.  metaNODE currently curates the following feeds:

- last
- market history
- open orders
- orderbook
- market-account balance


Run time has been demonstrated in excess of 2 weeks. 
metaNODE.txt is updated about once per second with live data.

metaNODE
is entirely independent of pybitshares.
Public database calls are made with websocket-client.


This project has received funding through the DEXbot worker.  Additional funding has been earmarked for incorporation of metaNODE into future release of the DEXbot platform.

metaNODE10 has already been incorporated into the Extinction Event dex algo trading framework EV.py; learn more at

www.litepresence.com





2
New Version of Extinction Event has been released.

The project has progressed beyond the point of "alpha" and is now officially in beta testing stage; it is "feature complete" and appears to be able to run stable long term - days on end - without user input.

From here forward, EV.py will be dependent on the stand alone app metaNODE.py

The latest stack is

EV5/metaNODE10

and is available on my github

metaNODE whitepaper can be found here:

https://github.com/litepresence/extinction-event/blob/master/metaNODE/README.md

metaNODE app:
https://github.com/litepresence/extinction-event/blob/master/metaNODE/metaNODE.py

EV app:
https://github.com/litepresence/extinction-event/blob/master/EV/EV.py

You'll need to install my virtual environment per instructions in my exctinction-event github repo, then create a folder and put a copy of EV.py and metaNODE.py into it.  To run live, start the metaNODE first, let it warm up fully, then you can launch an EV session for paper trading, live, or test orders. 

3
General Discussion / Bitshares Database Pocket Reference - Python
« on: April 27, 2018, 08:43:56 pm »
Code: [Select]
#Python3

' Bitshares Database Api Websocket Calls Pocket Reference'

# https://github.com/bitshares/bitshares-core/blob/master/libraries/app/database_api.cpp#L77

'litepresence 2018'

import websocket  #pip install websocket-client
websocket.enableTrace(False)

# EXAMPLE DATABASE REQUEST PARAMETERS
node = 'wss://api.bts.mobi/wss'#'wss://bts-seoul.clockwork.gr' # websocket address
asset = 'BTS' # symbol
currency = 'OPEN.BTC' #symbol
start = '2017-01-01T11:22:33' # iso date
stop = '2018-01-01T11:22:33' # iso date
account_name = 'litepresence1' # string
account_names = '["xeroc", "litepresence1"]' # list of string names
account_id = '1.2.743179' # a.b.c vector
block_num = 26444444 # int block number
block_nums = [26444444, 26444445, 26444446] # list of block numbers
trx_in_block = 4 # int index of item
transaction_id = '1.7.66209049'
transaction_id_type = 'cancel'
skip_order_book = 'false' # bool - non pythonic 'true' or 'false'
limit = 5 # int depth of data called
object_id_type = '' # a.b.c vector
object_ids = []
asset_id = '1.3.0'
asset_id2 = '1.3.861'
subscribe = 'false' # bool - non pythonic 'true' or 'false'
public_key = "BTS7d4MpYzecprWMfso8f6o1Ln8fQyxuQGD5LM83PRTfQBkodk4Ck"
witness_id = '1.6.65'
trx = '{"expiration":"2018-04-27T14:16:35","extensions":[],"operation_results":[[1,"1.7.66407704"]],"operations":[[1,{"amount_to_sell":{"amount":71055254,"asset_id":"1.3.0"},"expiration":"2018-05-04T14:16:04","extensions":[],"fee":{"amount":578,"asset_id":"1.3.0"},"fill_or_kill":false,"min_to_receive":{"amount":1922091,"asset_id":"1.3.121"},"seller":"1.2.879926"}]],"ref_block_num":8413,"ref_block_prefix":4199813419}'

# '{"id":1,"method":"call","params":["database","get_ticker",["OPEN.BTC","BTS"]]}'
Z = '{"id":1,"method":"call","params":["database",'

'Objects'
get_objects                     = Z + ''

'Subscriptions'
set_subscribe_callback          = Z + ''
set_pending_transaction_callback = Z + ''
set_block_applied_callback      = Z + ''
cancel_all_subscriptions        = Z + ''

'Blocks and transactions'
get_block_header                = Z + '"get_block_header",["%s"]]}' % block_num
get_block_header_batch          = Z + '"get_block_header_batch",["%s"]]}' % (block_nums)
get_block                       = Z + '"get_block",["%s"]]}' % block_num
get_transaction                 = Z + '"get_transaction",["%s", "%s"]]}' % (block_num, trx_in_block)
get_recent_transaction_by_id    = Z + ''

'Globals' # DONE
get_chain_properties            = Z + '"get_chain_properties",[]]}'
get_global_properties           = Z + '"get_global_properties",[]]}'
get_config                      = Z + '"get_config",[]]}'
get_chain_id                    = Z + '"get_chain_id",[]]}'
get_dynamic_global_properties   = Z + '"dynamic_global_properties",[]]}'

'Keys'
get_key_references              = Z + '"get_key_references",[["%s",]]]}' % public_key
is_public_key_registered        = Z + '"is_public_key_registered",["%s"]]}' % public_key

'Accounts'
get_accounts                    = Z + '"get_accounts",[["%s",]]]}' % account_id
get_full_accounts               = Z + '"get_full_accounts",[["%s",],%s]]}' % (account_name, subscribe)
get_account_by_name             = Z + '"get_account_by_name",["%s"]]}' % account_name
get_account_references          = Z + '"get_account_references",["%s",]]}' % account_id
lookup_account_names            = Z + '"lookup_account_names",[%s]]}' % account_names
lookup_accounts                 = Z + '"lookup_accounts",["%s", "%s"]]}' % (account_name, limit)
get_account_count               = Z + '"get_account_count",[]]}'

'Balances'
get_account_balances            = Z + '"get_account_balances",["%s", [] ]]}' % (account_id)
get_named_account_balances      = Z + '"get_named_account_balances",["%s", [] ]]}' % (account_name)
get_balance_objects             = Z + ''
get_vested_balances             = Z + ''
get_vesting_balances            = Z + '"get_vesting_balances",["%s"]]}' % account_id

'Assets'
get_assets                      = Z + '"get_assets",[["%s",]]]}' % asset_id
list_assets                     = Z + '"list_assets",["%s","%s"]]}' % (asset, limit)
lookup_asset_symbols            = Z + '"lookup_asset_symbols",[["%s",]]]}' % asset

'Markets / feeds'
get_order_book                  = Z + '"get_order_book",["%s","%s","%s"]]}' % (currency, asset, limit)
get_limit_orders                = Z + '"get_limit_orders",["%s","%s","%s"]]}' % (asset_id, asset_id2, limit)
get_call_orders                 = Z + '"get_call_orders",["%s","%s"]]}' % (asset_id, limit)
get_settle_orders               = Z + '"get_settle_orders",["%s","%s"]]}' % (asset_id, limit)
get_margin_positions            = Z + '"get_margin_positions",["%s"]]}' % account_id
get_collateral_bids             = Z + ''
subscribe_to_market             = Z + ''
unsubscribe_from_market         = Z + ''
get_ticker                      = Z + '"get_ticker",["%s","%s","%s"]]}' % (currency, asset, skip_order_book)
get_24_volume                   = Z + '"get_24_volume",["%s","%s"]]}' % (currency, asset)
get_top_markets                 = Z + '"get_top_markets",["%s"]]}' % limit
get_trade_history               = Z + '"get_trade_history",["%s","%s","%s","%s","%s"]]}' % (currency, asset, start, stop, limit)
get_trade_history_by_sequence   = Z + ''

'Witnesses'
get_witnesses                   = Z + ''
get_witness_by_account          = Z + '"get_witness_by_account",["%s"]]}' % witness_id
lookup_witness_accounts         = Z + '"lookup_witness_accounts",["%s","%s"]]}' % (witness_id, limit)
get_witness_count               = Z + '"get_witness_count",[]]}'

'Committee members'
get_committee_members           = Z + ''
get_committee_member_by_account = Z + ''
lookup_committee_member_accounts = Z + ''
get_committee_count             = Z + '"get_committee_count",[]]}'

'Workers'
get_all_workers                 = Z + ''
get_workers_by_account          = Z + ''
get_worker_count                = Z + '"get_worker_count",[]]}'

'Votes'
lookup_vote_ids = Z + ''

'Authority / validation'
get_transaction_hex             = Z + '"get_transaction_hex",[%s]]}' % trx
get_required_signatures         = Z + ''
get_potential_signatures        = Z + '"get_potential_signatures",[%s]]}' % trx
get_potential_address_signatures  = Z + '"get_potential_address_signatures",[%s]]}' % trx
verify_authority                = Z + '"verify_authority",[%s]]}' % trx
verify_account_authority        = Z + ''
validate_transaction            = Z + '"validate_transaction",[%s]]}' % trx
get_required_fees               = Z + ''

'Proposed transactions'
get_proposed_transactions = Z + '"get_proposed_transactions",["%s"]]}' % account_id

'Blinded balances'
get_blinded_balances = Z + ''

'Withdrawals'
get_withdraw_permissions_by_giver = Z + ''
get_withdraw_permissions_by_recipient = Z + ''


all_calls = [get_objects,set_subscribe_callback,set_pending_transaction_callback,set_block_applied_callback,cancel_all_subscriptions,get_block_header,get_block_header_batch,get_block,get_transaction,get_recent_transaction_by_id,get_chain_properties,get_global_properties,get_config,get_chain_id,get_dynamic_global_properties,get_key_references,is_public_key_registered,get_accounts,get_full_accounts,get_account_by_name,get_account_references,lookup_account_names,lookup_accounts,get_account_count,get_account_balances,get_named_account_balances,get_balance_objects,get_vested_balances,get_vesting_balances,get_assets,list_assets,lookup_asset_symbols,get_order_book,get_limit_orders,get_call_orders,get_settle_orders,get_margin_positions,get_collateral_bids,subscribe_to_market,unsubscribe_from_market,get_ticker,get_24_volume,get_top_markets,get_trade_history,get_trade_history_by_sequence,get_witnesses,get_witness_by_account,lookup_witness_accounts,get_witness_count,get_committee_members,get_committee_member_by_account,lookup_committee_member_accounts,get_committee_count,get_all_workers,get_workers_by_account,get_worker_count,lookup_vote_ids,get_transaction_hex,get_required_signatures,get_potential_signatures,get_potential_address_signatures,verify_authority,verify_account_authority,validate_transaction,get_required_fees,get_proposed_transactions,get_blinded_balances,get_withdraw_permissions_by_giver,get_withdraw_permissions_by_recipient,]


object_calls = []

subscription_calls = []

block_calls =      [get_block,
                    get_block_header,
                    get_block_header_batch,
                    get_transaction]

global_calls =     [get_chain_properties,
                    get_global_properties,
                    get_config,
                    get_chain_id,
                    get_dynamic_global_properties]

key_calls =        [get_key_references,
                    is_public_key_registered]

account_calls =    [get_accounts,
                    get_full_accounts,
                    get_account_by_name,
                    get_account_references,
                    lookup_account_names,
                    lookup_accounts,
                    get_account_count]

asset_calls =      [get_assets,
                    list_assets,
                    lookup_asset_symbols]

balance_calls =    [get_account_balances,
                    get_named_account_balances,
                    get_vesting_balances]

market_calls =     [get_order_book,
                    get_ticker,
                    get_trade_history,
                    get_limit_orders,
                    get_call_orders,
                    get_settle_orders,
                    get_margin_positions,
                    get_24_volume,
                    get_top_markets, ]

witness_calls =    [get_witness_by_account,
                    lookup_witness_accounts,
                    get_witness_count]

committe_calls =   [get_committee_count]

worker_calls =     [get_worker_count]

vote_calls = []

authority_calls =  [get_transaction_hex,
                    get_potential_signatures,
                    get_potential_address_signatures,
                    verify_authority,
                    validate_transaction]

proposed_calls = []
blindied_calls = []
withdrawal_calls = []


for call in asset_calls:

    try:
        ws = websocket.create_connection(node)
        print('')
        print((call.split(',"params":')[1]).rstrip('}'))
        print('-----------------------------------------------------------')
        ws.send(call)
        ret = ws.recv()
        print (ret)
        ws.close()
    except Exception as e:
        print (e.args)
        pass

# you can also make https requests in this manner
'''
import requests
data = '{"id":1,"method":"call","params":["database","get_ticker",["CNY","USD"]]}'
response = requests.post('https://api.bts.mobi/wss;echo', data=data)
'''


also here:

https://github.com/litepresence/extinction-event
https://github.com/litepresence/extinction-event/blob/master/tools/bitshares-database-reference.py

4
get named balances returns an asset id without its human readable name....
and an amount - without the location of its decimal point!

this is kind of thing makes neckbeard itchy

Code: [Select]
["database","get_named_account_balances",["abc123", [] ]]
-----------------------------------------------------------
[{'asset_id': '1.3.0', 'amount': 12019967}, {'asset_id': '1.3.1382', 'amount': 100}, {'asset_id': '1.3.1578', 'amount': 100000000}, {'asset_id': '1.3.2419', 'amount': 500000}, {'asset_id': '1.3.2841', 'amount': 12000}, {'asset_id': '1.3.2931', 'amount': 10000}, {'asset_id': '1.3.3248', 'amount': 2580000}, {'asset_id': '1.3.3261', 'amount': 1000}, {'asset_id': '1.3.3279', 'amount': 10000}, {'asset_id': '1.3.3389', 'amount': 1000000000}, {'asset_id': '1.3.3431', 'amount': 1000}, {'asset_id': '1.3.3530', 'amount': 100000000}, {'asset_id': '1.3.3540', 'amount': 100000000}, {'asset_id': '1.3.3830', 'amount': '25000000000'}]

we can fix this... so get asset details

which is another api call and gives us "precision"; location of decimal, and the asset "symbol"; ie OPEN.BTC or BTS


but then we have to merge those dictionaries and they do not contain a common key; one has "asset_id" the other "id"

grrr

every call for what the asset is called and where its decimal point goes also contains bit of spam for every shitcoin donated to your account, on every bot tick, which brings in a huge page of asset banner ads like this silliness:


Code: [Select]
'{"main":"WARNING, DANGER, RED-ALERT, PANIC! PANIC! PANIC!\\n\\nYOU HAVE BEEN INFECTED,\\n\\nZOMBIES HAVE BEEN RELEASED! \\n\\nZombies are going to appear from nowhere & infest your portfolio,\\nSome zombies will come from your friends and family,\\nAND YES there are even some sickos out there that will want to buy Zombies too :s\\n\\n\\nThese VILE undead human corpses have risen, The Mission, BURN Zombies :)\\nHelp FILL THE POOL, Zombies are BURNED when there sent to the pool.\\n\\n\\nDisclaimer: NO REAL ZOMBIES WILL BE HURT IN THE CREATION OF THIS TOKEN ;)","short_name":"Zombies",

be thankful you're not on data plan!

nonetheless, hocus pocus account balances:

Code: [Select]
>>>
balances
{'TURION': 0.1, 'FREECOIN': 10000.0, 'SOLOMON': 0.1, 'BIRMINGHAMGOLD': 100.0, 'ZOMBIES': 12.0, 'DECENTRALIZED': 10.0, 'ANONYMOUS': 1.0, 'BTS': 120.19967, 'BTSJON': 1000.0, 'BADCOIN': 10000.0, 'HERTZ': 0.01, 'URTHONA': 50.0, 'SEED': 258.0, 'UNIVERSAL': 25000.0}


Code: [Select]
'litepresence 2018'

import websocket  #pip install websocket-client
websocket.enableTrace(False)
from ast import literal_eval as literal

def database_call(node, call):

    while 1:
        try:
            call = call.replace("'",'"') # never use single quotes
            ws = websocket.create_connection(node)
            print('')
            print((call.split(',"params":')[1]).rstrip('}'))
            print('-----------------------------------------------------------')
            ws.send(call)
            # 'result' key of literally evaluated
            # string representation of dictionary from websocket
            ret = literal(ws.recv())['result']
            print (ret)
            ws.close()
            return ret
        except Exception as e:
            print (e.args)
            pass

def account_balances(node, account_name):

    Z = '{"id":1,"method":"call","params":["database",'
    # make call for raw account balances as returned by api
    get_named_account_balances = Z + '"get_named_account_balances",["%s", [] ]]}' % (account_name)
    raw_balances = database_call(node, get_named_account_balances)
    # make list of asset_id's in raw account balances
    asset_ids = []
    for i in range(len(raw_balances)):
        asset_ids.append(raw_balances[i]['asset_id'])
    # make a second api request for additional data about each asset
    get_assets = Z + '"get_assets",[%s]]}' % asset_ids
    raw_assets = database_call(node, get_assets)
    # create a common key "asset_id" for both list of dicts
    # also extract the symbol and precision
    id_sym_prec = []
    for i in range(len(raw_assets)):
        id_sym_prec.append({'asset_id':raw_assets[i]['id'],
                            'symbol':raw_assets[i]['symbol'],
                            'precision':raw_assets[i]['precision'], })
    # merge the two list of dicts with common key "asset_id"
    data = {}
    lists = [raw_balances, id_sym_prec]
    for each_list in lists:
       for each_dict in each_list:
           data.setdefault(each_dict['asset_id'], {}).update(each_dict)
    # convert back to list
    data = list(data.values())
    # create a new dictionary containing only the symbol and quantity
    ret = {}
    for i in range(len(data)):
        qty = float(data[i]['amount'])/10**float(data[i]['precision'])
        ret[data[i]['symbol']] = qty
    return raw_balances, ret

#node = 'wss://bts-seoul.clockwork.gr' # websocket address
node = 'wss://api.bts.mobi/wss'
account_name = 'abc123' # string
raw_balances, balances = account_balances(node, account_name)


print('')
print('balances')
print(balances)

I'll also host this on my github

https://github.com/litepresence/extinction-event
https://github.com/litepresence/extinction-event/blob/master/tools/bitshares-account_balances.py

5
General Discussion / Re: [python-bitshares-qt] CITADEL Desktop Wallet
« on: April 14, 2018, 12:54:41 pm »
nice work! stoked to try it out!

6
General Discussion / Re: [ANN] microDEX - low latency minimalist UI
« on: April 09, 2018, 07:33:17 pm »
Seems very promising, looking forward to the v0.1 :D

moving up one satoshi version every few days, could be about 100,000 years before v0.1

u might want to give install a try now instead

:D

9
General Discussion / Re: [ANN] microDEX - low latency minimalist UI
« on: April 07, 2018, 08:42:24 pm »
v0.00000013 uploaded

was small bug in buy/sell/cancel where
Code: [Select]
attempt=0 on wrong line below
Code: [Select]
except:

10
General Discussion / Re: [ANN] microDEX - low latency minimalist UI
« on: April 07, 2018, 07:33:08 pm »
v0.00000012

- plots state machine
- checks all known servers for any suffix ( /, /ws, /wss) returns only fastest from each location
- improved exception logging for development


11
Technical Support / Re: Market history from blocks
« on: April 04, 2018, 04:45:55 pm »
I working on a trading bot and I want to test my strategies on historic datas.

check out the Extinction Event framework, it allows you to backtest against centralized exchange data which should be sufficient for most altcoin/bitcoin pairs as:

LTC/BTC on CEX very closely matches OPEN.LTC/OPEN.BTC on DEX; at least on daily candle scale.


By the way, public API servers (e.g. OpenLedger) by default only stores history of 3 days.

that's a crucial tidbit I didn't know


13
General Discussion / Re: [ANN] microDEX - low latency minimalist UI
« on: April 04, 2018, 03:28:10 am »
v0.00000011 uploaded to repo


install docs will be uploaded tomorrow

time for zzzzzz's

14
General Discussion / Re: [ANN] microDEX - low latency minimalist UI
« on: April 04, 2018, 01:04:02 am »
Pull request even for documentation improvements are much appreciated, after all its also open source :)

incoming ;D

https://github.com/bitshares/python-bitshares/issues/70

15
General Discussion / Re: [ANN] microDEX - low latency minimalist UI
« on: April 03, 2018, 07:26:35 pm »
Do you have a thorough logging already implemented? That would be awesome to find the root of appearing connection problems? If not I would be happy to assist!

that's the magic of public software; there are many eyes on it

v0.00000011 will release with much improved logging to 'microDEX_log.txt'

I'm certainly open to suggestion / assistance; but probably best to fork from 11 than 10 as I've shifted quite a bit around and have good protocol established; new version should be out soon; I'll post back here.

improvements soon to come on 11

better buy/sell/cancel exception handling
logging exceptions and trades to file
removal of the zprint() for "BOOK" and "CHART UPDATE"
elimination of unused nodes_update()
elimination of many unused print lines in nodes_process()
improved line comments
nodes process was running constantly will be every 10 minutes
concurrent orderbook children will live 10 minutes instead of 2
orderbook children will reconnect() w/ new node list upon first failed connect
added race_append() for logging errors and buys






Pages: [1] 2 3 4 5