Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.


Messages - litepresence

Pages: 1 2 3 4 5 6 7 [8] 9 10
106
updated repo again v0.00000004

improved handling of invalid orders
improved handling of orders in excess of funding
confirmation window now displays details of order being confirmed

eta, something went wrong when I uploaded earlier, patched @ 15:30 EST


107
updated repo

https://github.com/litepresence/extinction-event/blob/master/microDEX/microDEX.py

buy/sell/cancel confirmation popup window
login/logout capability
background process now maintains list of top ten nodes sorted by latency
better control of child processes to prevent too many processes error
added market history and seconds since last trade
added dictionary of complete holdings
improved login screen

low latency scheme works like this

55 known nodes are tested for ping; top ten are returned by background process
then another background process spawns 6 child processes
each of the children have a 60 second lifespan and continually update the orderbook whenever they have data
as they expire each is replaced by a new child so 6 "fresh" processes remain always
this means the book you see is actually flashing between the data from 6 nodes; so if one goes rogue you'll see it immediately



also... live tested w/ funds by my 8 year old

:D

108
General Discussion / [ANN] microDEX - low latency minimalist UI
« on: March 27, 2018, 12:14:18 am »

What if the UI maintained DEX connection for months at a time?
What if the orderbook was updated several times per second?
What if the data feed was not dependent on a single whitelisted node?
What if historical CEX data and latest DEX data were on the same chart?

What if you could read and trust all the code that makes it happen
over a single cup of coffee?



[ANN] microDEX

Code: [Select]
                                     ______   ________  ____  ____ 
                                    (_   _ `.(_   __  |(_  _)(_  _)
     __  __  ____  ___  ____   ___    | | `. \ | |_ \_|  \ \__/ /   
    (  \/  )(_  _)/ __)(  _ \ / _ \   | |  | | |  _) _    ) __ (   
     )    (  _||_( (__  )   /( (_) ) _| |_.' /_| |__/ | _/ /  \ \_ 
    (_/\/\_)(____)\___)(_)\_) \___/ (______.'(________|(____)(____)
   =================================================================
           microDEX v0.00000009 - low latency minimalist UI
   =================================================================


BUY/SELL/CANCEL
LIVE LOW LATENCY MULTI NODE BOOKS
NEVER OUT OF SYNC
ALWAYS BLOCKCHAIN CONNECTED


minimal dependencies:

python3
pybitshares
tkinter
matplotlib


Code:

https://github.com/litepresence/extinction-event/blob/master/microDEX/microDEX.py

Installation:

https://github.com/litepresence/extinction-event/blob/master/Installation




more coming soon!


'till then, need botscript?  check me out at


litepresence.com



ETA:



how's this for soon?  v0.00000009 is live:





The Fastest DEX Orderbooks in the Business!



All that and just 1300 fluffy lines of script :D



109
version 2 is out:

https://github.com/litepresence/extinction-event/blob/master/EV/EV0.00000002.py

Code: [Select]
    #===================================================================
    ''' FEATURES v0.00000002'''
    #===================================================================
    '''
   Rogue Node Immunity:
       A background daemon process maintains a list of low latency nodes
       for buy/sell/cancel/orders ops in a text file
       distributed exchange prices and orderbook are verified and curated
       using multinode statistical approach with daemon processes
       open orders are checked in triplicate on multiple nodes
       dex() definitions have been upgraded after consultation with
       Bitshares core developers and node admin
   Move to github:
       https://github.com/litepresence/extinction-event
   New MODES:
       SALES mode backtest only plots buy/sell actions; no state machine
       LATENCY mode connect to all nodes and reports on latency
       PAPER mode runs live, but does not trade

110
Technical Support / Re: Very slow market buy with pybitshares API
« on: March 18, 2018, 03:23:05 pm »
I don't know much about VS, but I can tell you that you can greatly improve signing speed by installing
pip3 install scrypt
and
pip3 install secp256k1

perhaps these should auto install w/ uptick and pybitshares?

are there any other unmentioned "dependencies"?

111
General Discussion / 5 Node Verified Price Feed - Python
« on: March 18, 2018, 01:39:28 am »
developing here:

https://github.com/litepresence/extinction-event/tree/master/EV


working under the assumption that any node can provide rogue price feeds at any given time,
I sort all nodes by latency every two minutes
then check last price on 5 low latency nodes to verify authenticity of the data every 30 seconds


5 NODE VERIFIED LAST PRICE

nodes.py and last.py should be run in seperate terminals

nodes.py creates file nodes.txt
last.py creates file last.txt and appends to file blacklist.txt

nodes.txt is the top 10 latency sorted nodes of about 50 known nodes; updated approx every 2 minutes

last.py uses nodes.txt to get price from 5 nodes, then makes a list "prices", updated approx ever 30 seconds

then process that list of prices from 5 different nodes pseudocode:


if the spread of the prices is too wide:

    append to file blacklist.txt the list of prices and the list of nodes used to gather them


if all prices are same:
   
    last = latest price

elif latest price less than 2% different than mean(prices):
   
    last = latest price
 
else:
    try: last = mode(prices)
    except: last = median(prices)

write last to last.txt



the good news:

I ran this all day on BTS/BTC and no set of 5 prices were more than 2% different... meaning I never encountered a rogue node.

the better news:

this price feed can be used with confidence to feed bots with latest price data without risk of node going stale or going rogue

the best news:

I work on bid/ask tomorrow

:D

112
I am not an expert here, I'm simply an algo trader looking to integrate to the Bishares DEX environment by connecting to existing node architecture - without running my own node.  I have asked a lot of questions lately to Node Admins and have concluded there is a good deal of terminology involved that needs clarification.   There are also risks when dealing with existing nodes, that need to be mitigated.  So, please if you are an expert, chime in and I'll update this OP to summarize key points.   

This OP was initial populated with cut/paste commentary via Node Admin channel on Telegram.


NODES


some universal truisms:

- nodes are in no way uniform
- however, all nodes store all blocks
- any node can go rogue at any time
- every node is a man-in-the-middle between you and the blockchain


A node can be:

P2P

RPC

API - exposed to a network, either LAN or WAN

FULL -  nodes which provide full histories;  nodes which provide full histories; needs several tens of gigs of RAM; x5670 with 100G ram should be good for full history node

SEED - full nodes with publicly reachable p2p port

WITNESS - is a matter of votes, but you can run a witness node wich locally validates the blocks

PRODUCTION

LOAD BALANCED

PUBLIC

DOCKER

PAID

DELAY NODE

OWN USE

BLOCK PRODUCING

WHITE LISTED

BLACK LISTED

ACTIVE

STANDBY

LOW LATENCY

UNRESPONSIVE

TESTNET - testnet is actually used to determine if it's mainnet or testnet

SYNCHRONIZED

DATACENTER

HARDENED

SSL

VPS

DELAYED

INITIAL

FAKE - fake.automatic-selection is a hard-coded psuedo server

BITSHARES OFFICIAL - deployed with bitshares UI

CRYPTOBRIDGE OFFICIAL - deployed with cryptobridge UI

UP TO DATE - has a head block that is less than 3 seconds old

STALE - has a head block that is greater than 3 seconds old

PROPERLY DEPLOYED

LATEST VERSION

VERSION 171105a - if there are many wicks in price chart, it's on this version

pre-VERSION 1802xx - if owner key signature is not able to change active permissions, it's before 1802xx

ROGUE - a node can delay, refuse, or reorder your buy/sell/cancel or other requests.   A rogue node can also return inaccurate information to all or any one individual.  A rogue node can NOT modify or create a buy/sell/cancel operation on your behalf.  A rogue node can provide false account ID in an attempt to get you to send funds to bad address.


WORKING

VERIFIED



WALLETS


CLI - command line interface
PYBITSHARES
LIGHT
WEB WALLET
FAUCET - a pool of funds used to pay the registration fee for the new accounts, there are currently several public faucets up



PASSWORDS and KEYS


BRAIN

PASSPHRASE


113
General Discussion / Re: When stop loss
« on: March 17, 2018, 03:05:51 am »
check here:

Wrapper for Common PyBitshares DEX Algo Trading API Calls
https://bitsharestalk.org/index.php?topic=25988.0

should be very easy to implement, like this:

Code: [Select]
while 1:
    if dex('last') < stoploss:
         dex('sell')

call it done

114
General Discussion / Re: (Python) latency sorted Bitshares nodes
« on: March 17, 2018, 12:03:24 am »
I've made considerable progress on this script check the github in OP

I've live tested it in a while loop 24 hours and it maintains an output file of latency tested "nodes.txt"

Code: [Select]
def nodes(timeout=20, pings=999999, crop=99, noprint=False, write=False,
          include=False, exclude=False, suffix=True, master=False):

    # timeout : seconds to ping until abort per node
    # pings   : number of good nodes to find until satisfied (0 none, 999 all)
    # suffix  : checks each node for no suffix plus with /ws or /wss
    # noprint : disables printing, only returns list of good nodes
    # master  : check only nodes listed in bitshares/ui/master
    # crop    : return only best nodes
    # write   : maintains an output file nodes.txt with list of best nodes

115
h/t @ Alex M

resolved was able to pass:

Code: [Select]
num_retries=0

Code: [Select]
chain = Blockchain(bitshares_instance=BitShares(n, num_retries=0), mode='head')
I had given up on this early on because I tried this without success:

Code: [Select]
chain = Blockchain(bitshares_instance=BitShares(n), num_retries=0, mode='head')

116
When I get this error and it attempts a retry, it never actually does connect because the node itself is down or having issues.

Code: [Select]
Retrying in 2 seconds
Retrying in 4 seconds
Retrying in 6 seconds
etc.

How can I "catch" this retry-attempt like an error? 
How can I provide an argument to prevent any retry at all?

currently the only way I have found, is to use a `multiprocess.Process` approach to enforce a timeout that overrides the retry attempt by starting a side process to contain the node connect attempt, then breaking that process if it takes too long.   

I would rather it just break as soon as it says
Code: [Select]
Retrying in 2 seconds
as I've yet to see a retry actually accomplish anything.

Using a timeout approach has limitations; the timeout must be long enough to account for latency fluctuation, while being short enough to not eat up too much script time.

From my perspective, it would be advantageous if the pybitshares module, rather than attempting a retry on a down node, throws an exception that I can handle with try/except.  Upon exception, I can then specify a new node to attempt... rather than waiting for a timeout to expire.



related pybitshares code:

https://github.com/xeroc/python-bitshares/blob/9250544ca8eadf66de31c7f38fc37294c11f9548/bitsharesapi/websocket.py

beginning line 291

Code: [Select]
    def run_forever(self):
        """ This method is used to run the websocket app continuously.
            It will execute callbacks as defined and try to stay
            connected with the provided APIs
        """
        cnt = 0
        while not self.run_event.is_set():
            cnt += 1
            self.url = next(self.urls)
            log.debug("Trying to connect to node %s" % self.url)
            try:
                # websocket.enableTrace(True)
                self.ws = websocket.WebSocketApp(
                    self.url,
                    on_message=self.on_message,
                    on_error=self.on_error,
                    on_close=self.on_close,
                    on_open=self.on_open
                )
                self.ws.run_forever()
            except websocket.WebSocketException as exc:
                if (self.num_retries >= 0 and cnt > self.num_retries):
                    raise NumRetriesReached()

                sleeptime = (cnt - 1) * 2 if cnt < 10 else 10
                if sleeptime:
                    log.warning(
                        "Lost connection to node during wsconnect(): %s (%d/%d) "
                        % (self.url, cnt, self.num_retries) +
                        "Retrying in %d seconds" % sleeptime
                    )
                    time.sleep(sleeptime)

            except KeyboardInterrupt:
                self.ws.keep_running = False
                raise

            except Exception as e:
                log.critical("{}\n\n{}".format(str(e), traceback.format_exc()))


this snippet here is a major source of brittleness:

Code: [Select]
"Lost connection to node during wsconnect(): %s (%d/%d) "
                        % (self.url, cnt, self.num_retries) +
                        "Retrying in %d seconds" % sleeptime

A better behavior than just waiting and retrying on the same failed node is to switch nodes, THEN retry.


or simply raise the exception and let the user handle the failure: switching nodes; perhaps also blacklisting the old, or ringing a bell:

Code: [Select]
except websocket.WebSocketException:
    self.ws.keep_running = False
    raise

where the user could command:

Code: [Select]

attempt = 1
while attempt:

     try:

      # make api call
     attempt = 0

    except websocket.WebSocketException:

      # blacklist this node
      # run routine to find another white-listed node with low latency and non-stale blocktime
      attempt +=1
      if attempt > n:
          # run failsafe routine to generate new node list
      else:
          # switch node to known good node and loop
 


would it be possible to import the websocket.WebsocketException from the bitshares module?

something like

Code: [Select]
from bitshares import websocket.WebsocketException
but that doesn't work

I've also tried

Code: [Select]
from bitshares.websocket import WebsocketException
from bitshares import websocket
from bitshares import WebsocketException
and attempted to import the class function:

Quote
from bitshares import BitSharesWebsocket

but none of them work either, don't mind my ignorance here... just throwing things at the wall to see what sticks

I've also tried importing the websocket module and preempting the exception

Code: [Select]
import websocket

try:
    # connect to api
except websocket.WebsocketException:
    # print ('hello world')

does not catch

118
Perhaps it's good tool for pybitshares. Can you make a PR in github? Or even create your own tool set.

I've decided to maintain my own tool set.

licence.txt

Code: [Select]
DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
                   Version 0, March 1765

TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION, AND MODIFICATION:

1) If it bears an official stamp, prepare to throw it overboard.

119
General Discussion / Re: More Nodes....
« on: March 12, 2018, 08:11:35 pm »

Please vote for this node "neoreel-1" currently ranked 98 with 50 millions votes...

where is this ranking listed?

120
General Discussion / Re: More Nodes....
« on: March 12, 2018, 12:05:00 am »
these 34 are known to be active with ping less than 5 seconds as of today, sorted by lowest latency to east coast US:


connected to wss://us-east-1.bts.crypto-bridge.org with latency 1.17

['wss://us-east-1.bts.crypto-bridge.org', 'wss://this.uptick.rocks/', 'wss://dexnode.net/ws', 'wss://eu.nodes.bitshares.ws', 'wss://eu.nodes.bitshares.works', 'wss://dex.rnglab.org', 'wss://node.market.rudex.org', 'wss://bitshares.crypto.fans/ws', 'wss://eu-west-1.bts.crypto-bridge.org', 'wss://kc-us-dex.xeldal.com/ws', 'wss://eu-central-1.bts.crypto-bridge.org', 'wss://us-west-1.bts.crypto-bridge.org', 'wss://bitshares.openledger.info/ws', 'wss://bts.proxyhosts.info/wss', 'wss://api-ru.bts.blckchnd.com', 'wss://api.bts.blckchnd.com', 'wss://us.nodes.bitshares.works', 'wss://eu.openledger.info/ws', 'wss://b.mrx.im/ws', 'wss://slovenia.bitshares.apasia.tech/ws', 'wss://node.bitshares.eu', 'wss://ap-northeast-1.bts.crypto-bridge.org', 'wss://sa-east-1.bts.crypto-bridge.org', 'wss://us.nodes.bitshares.ws', 'wss://la.dexnode.net/ws', 'wss://openledger.hk/ws', 'wss://ap-northeast-2.bts.crypto-bridge.org', 'wss://ap-southeast-1.bts.crypto-bridge.org', 'wss://sg.nodes.bitshares.works', 'wss://bitshares-api.wancloud.io/ws', 'wss://ws.winex.pro', 'wss://ws.gdex.top', 'wss://sg.nodes.bitshares.ws', 'wss://bit.btsabc.org/ws']

Pages: 1 2 3 4 5 6 7 [8] 9 10