Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.


Messages - litepresence

Pages: [1] 2 3 4 5 6
1
Technical Support / Re: Account Hijacked
« on: March 28, 2019, 02:17:45 pm »
I am documenting this issue here:

https://github.com/bitshares/bsips/issues/154

BSIP: Proposals Scam Prevention #154

2
General Discussion / Re: microDEX <-> metaNODE <-> manualSIGNING
« on: February 23, 2019, 10:32:53 pm »


1 week run time achieved and UI is still 100ms responsive to buy/sell/cancel requests.

I forgot to mention that all auth'd ops occur in parallel processes; so you can order several buy/sell operations prior to the first actually executing (which takes a few seconds).

3
Technical Support / 1000 Candles of Historical HLOC from the DEX
« on: February 17, 2019, 05:00:46 pm »
BTS/bitUSD



list of dicts of human readable numpy arrays

Code: [Select]
from websocket import create_connection as wss  # handshake to node
from json import dumps as json_dumps
from json import loads as json_loads
import matplotlib.pyplot as plt
from datetime import datetime
from pprint import pprint
import numpy as np
import time


def public_nodes():
    return [
        'wss://altcap.io/wss',
        'wss://api-ru.bts.blckchnd.com/ws',
        'wss://api.bitshares.bhuz.info/wss',
        'wss://api.bitsharesdex.com/ws',
        'wss://api.bts.ai/ws',
        'wss://api.bts.blckchnd.com/wss',
        'wss://api.bts.mobi/wss',
        'wss://api.bts.network/wss',
        'wss://api.btsgo.net/ws',
        'wss://api.btsxchng.com/wss',
        'wss://api.dex.trading/ws',
        'wss://api.fr.bitsharesdex.com/ws',
        'wss://api.open-asset.tech/wss',
        'wss://atlanta.bitshares.apasia.tech/wss',
        'wss://australia.bitshares.apasia.tech/ws',
        'wss://b.mrx.im/wss',
        'wss://bit.btsabc.org/wss',
        'wss://bitshares.crypto.fans/wss',
        'wss://bitshares.cyberit.io/ws',
        'wss://bitshares.dacplay.org/wss',
        'wss://bitshares.dacplay.org:8089/wss',
        'wss://bitshares.openledger.info/wss',
        'wss://blockzms.xyz/ws',
        'wss://bts-api.lafona.net/ws',
        'wss://bts-seoul.clockwork.gr/ws',
        'wss://bts.liuye.tech:4443/wss',
        'wss://bts.open.icowallet.net/ws',
        'wss://bts.proxyhosts.info/wss',
        'wss://btsfullnode.bangzi.info/ws',
        'wss://btsws.roelandp.nl/ws',
        'wss://chicago.bitshares.apasia.tech/ws',
        'wss://citadel.li/node/wss',
        'wss://crazybit.online/wss',
        'wss://dallas.bitshares.apasia.tech/wss',
        'wss://dex.iobanker.com:9090/wss',
        'wss://dex.rnglab.org/ws',
        'wss://dexnode.net/ws',
        'wss://england.bitshares.apasia.tech/ws',
        'wss://eu-central-1.bts.crypto-bridge.org/wss',
        'wss://eu.nodes.bitshares.ws/ws',
        'wss://eu.openledger.info/ws',
        'wss://france.bitshares.apasia.tech/ws',
        'wss://frankfurt8.daostreet.com/wss',
        'wss://japan.bitshares.apasia.tech/wss',
        'wss://kc-us-dex.xeldal.com/ws',
        'wss://kimziv.com/ws',
        'wss://la.dexnode.net/ws',
        'wss://miami.bitshares.apasia.tech/ws',
        'wss://na.openledger.info/ws',
        'wss://ncali5.daostreet.com/wss',
        'wss://netherlands.bitshares.apasia.tech/ws',
        'wss://new-york.bitshares.apasia.tech/ws',
        'wss://node.bitshares.eu/ws',
        'wss://node.market.rudex.org/wss',
        'wss://nohistory.proxyhosts.info/wss',
        'wss://openledger.hk/wss',
        'wss://paris7.daostreet.com/wss',
        'wss://relinked.com/wss',
        'wss://scali10.daostreet.com/wss',
        'wss://seattle.bitshares.apasia.tech/wss',
        'wss://sg.nodes.bitshares.ws/ws',
        'wss://singapore.bitshares.apasia.tech/ws',
        'wss://status200.bitshares.apasia.tech/wss',
        'wss://us-east-1.bts.crypto-bridge.org/ws',
        'wss://us-la.bitshares.apasia.tech/ws',
        'wss://us-ny.bitshares.apasia.tech/ws',
        'wss://us.nodes.bitshares.ws/wss',
        'wss://valley.bitshares.apasia.tech/ws',
        'wss://ws.gdex.io/ws',
        'wss://ws.gdex.top/wss',
        'wss://ws.hellobts.com/wss',
        'wss://ws.winex.pro/wss'
    ]


def wss_handshake(node):
    global ws
    ws = wss(node, timeout=5)


def wss_query(params):
    query = json_dumps({"method": "call",
                        "params": params,
                        "jsonrpc": "2.0",
                        "id": 1})
    ws.send(query)
    ret = json_loads(ws.recv())
    try:
        return ret['result']  # if there is result key take it
    except:
        return ret


def rpc_market_history(currency_id, asset_id, period, start, stop):

    ret = wss_query(["history",
                     "get_market_history",
                     [currency_id,
                      asset_id,
                      period,
                      to_iso_date(start),
                      to_iso_date(stop)]])
    return ret


def chartdata(pair, start, stop, period):
    pass  # as per extinctionEVENT cryptocompare call


def rpc_lookup_asset_symbols(asset, currency):
    ret = wss_query(['database',
                     'lookup_asset_symbols',
                     [[asset, currency]]])
    asset_id = ret[0]['id']
    asset_precision = ret[0]['precision']
    currency_id = ret[1]['id']
    currency_precision = ret[1]['precision']

    return asset_id, asset_precision, currency_id, currency_precision

def backtest_candles(raw):  # HLOCV numpy arrays

    # gather complete dataset so only one API call is required
    d = {}
    d['unix'] = []
    d['high'] = []
    d['low'] = []
    d['open'] = []
    d['close'] = []
    for i in range(len(raw)):
        d['unix'].append(raw[i]['time'])
        d['high'].append(raw[i]['high'])
        d['low'].append(raw[i]['low'])
        d['open'].append(raw[i]['open'])
        d['close'].append(raw[i]['close'])
    del raw
    d['unix'] = np.array(d['unix'])
    d['high'] = np.array(d['high'])
    d['low'] = np.array(d['low'])
    d['open'] = np.array(d['open'])
    d['close'] = np.array(d['close'])

    # normalize high and low data
    for i in range(len(d['close'])):
        if d['high'][i] > 2 * d['close'][i]:
            d['high'][i] = 2 * d['close'][i]
        if d['low'][i] < 0.5 * d['close'][i]:
            d['low'][i] = 0.5 * d['close'][i]

    return d

def from_iso_date(date):  # returns unix epoch given iso8601 datetime
    return int(time.mktime(time.strptime(str(date),
                                         '%Y-%m-%dT%H:%M:%S')))


def to_iso_date(unix):  # returns iso8601 datetime given unix epoch
    return datetime.utcfromtimestamp(int(unix)).isoformat()


def parse_market_history():

    ap = asset_precision  # quote
    cp = currency_precision  # base
    history = []
    for i in range(len(g_history)):
        h = ((float(int(g_history[i]['high_quote'])) / 10 ** cp) /
            (float(int(g_history[i]['high_base'])) / 10 ** ap))
        l = ((float(int(g_history[i]['low_quote'])) / 10 ** cp) /
            (float(int(g_history[i]['low_base'])) / 10 ** ap))
        o = ((float(int(g_history[i]['open_quote'])) / 10 ** cp) /
            (float(int(g_history[i]['open_base'])) / 10 ** ap))
        c = ((float(int(g_history[i]['close_quote'])) / 10 ** cp) /
            (float(int(g_history[i]['close_base'])) / 10 ** ap))
        cv = (float(int(g_history[i]['quote_volume'])) / 10 ** cp)
        av = (float(int(g_history[i]['base_volume'])) / 10 ** ap)
        vwap = cv / av
        t = int(min(time.time(),
               (from_iso_date(g_history[i]['key']['open']) + 86400)))
        history.append({'high': h,
                        'low': l,
                        'open': o,
                        'close': c,
                        'vwap': vwap,
                        'currency_v': cv,
                        'asset_v': av,
                        'time': t})
    return history

print("\033c")

node_id = 2
calls = 5  # number of requests
candles = 200  # candles per call
period = 86400  # data resolution
asset = 'BTS'
currency = 'USD'

# fetch node list
nodes = public_nodes()
# select one node from list
wss_handshake(nodes[node_id])
# gather cache data to describe asset and currency
asset_id, asset_precision, currency_id, currency_precision = (
    rpc_lookup_asset_symbols(asset, currency))
print(asset_id, asset_precision, currency_id, currency_precision)


full_history = []
now = time.time()
window = period * candles
for i in range((calls - 1), -1, -1):
    print('i', i)
    currency_id = '1.3.121'
    asset_id = '1.3.0'
    start = now - (i + 1) * window
    stop = now - i * window
    g_history = rpc_market_history(currency_id,
                                   asset_id,
                                   period,
                                   start,
                                   stop)
    print(g_history)
    history = parse_market_history()
    full_history += history

pprint(full_history)
print(len(full_history))

data = backtest_candles(full_history)

fig = plt.figure()
ax = plt.axes()
fig.patch.set_facecolor('black')
ax.patch.set_facecolor('0.1')
ax.yaxis.tick_right()
ax.spines['bottom'].set_color('0.5')
ax.spines['top'].set_color(None)
ax.spines['right'].set_color('0.5')
ax.spines['left'].set_color(None)
ax.tick_params(axis='x', colors='0.7', which='both')
ax.tick_params(axis='y', colors='0.7', which='both')
ax.yaxis.label.set_color('0.9')
ax.xaxis.label.set_color('0.9')
plt.yscale('log')

x = data['unix']
plt.plot(x, data['high'], color='white')
plt.plot(x, data['low'], color='white')
plt.plot(x, data['open'], color='aqua')
plt.plot(x, data['close'], color='blue')

plt.show()

RPC call to public node

get_market_history()

Is in graphene format; ie no decimal places, no human readable pricing

(min_to_receive/10^receive_precision) / (amount_to_sell/10^sell_precision)

gives you this:



I give you this:



crypto long, moar coinz short!

- uncle lp

4
Technical Support / Re: Assert Exception: min_to_receive.amount > 0:
« on: February 14, 2019, 08:53:42 pm »
on the backend there are no decimals in prices OR amounts

everything is integers in this format:

(amount_to_sell / 10^precision_of_asset1)
(min_to_receive / 10^precision_of_asset2)

each asset has a precision... if you get too close to its precision... ie trying to sell dust then this will get rounded to zero and rejected.

5
latencyTEST.py

github/litepresence/extinction-event/EV

6
General Discussion / Re: API Node Latency Map
« on: February 14, 2019, 07:08:50 pm »
You can still use this script on your own machine, but I am no longer keeping this live updated; my internet bandwidth is very limited.  Please contact me on telegram @litepresence if you would like to keep a public version alive and I'll point this thread at it.

my github has recently been updated to newer version, you no longer need the basemap, the script will aquire it externally from imgur

7
General Discussion / microDEX <-> metaNODE <-> manualSIGNING
« on: February 14, 2019, 05:29:58 pm »


I have set out to ensure UI buy/sell/cancel ALWAYS works when I push buttons and never have to think about which node to pick, slow connectivity, idiosyncrasies of graphene speak, or the bugs of "full featured software". 

Just on demand buy/sell/cancel.

Go to my github repo...

`github/litepresence/extinction-event/EV`

You get copy of each of these and plop in one folder

microDEX.py
manualSIGNING.py
metaNODE.py


each script is about 50kb; 1500 lines; very small, quick/easy to read

all dependencies are `pip3 install` the stack does NOT require pybitshares

Code: [Select]
SYSTEM REQUIREMENTS
linux / python3

MODULES YOU WILL NEED
ecdsa
numpy
tkinter
requests
secp256k1
websocket-client


metaNODE
is statistically curated public api data
you run this in a separate terminal before launching microDEX

manualSIGNING
is your private authenticated ops api
microDEX imports the broker(order) method from manualSIGNING to sign transactions with a WIF key.  manualSIGNING is a purpose built fork of pybitshares for signing limit orders about 1/40th the size of the full package.

microDEX
is your buy/sell/cancel user interface
built on tkinter, it communicates with metaNODE at nanosecond speed and places orders via manualSIGNING to the fastest nodes in the network.

it runs 24/7 maintaining 100ms response time on button clicks
all buy/sell/cancel ops occur in timed out parallel process to the UI process; nothing can get hung
extremely reactive, 99.99% uptime, never rogue or stale; statistically curated public api data
when new version is available microDEX will inform you at startup and allow in script update

I maintain a "heavily commented", "minimal dependencies", "minimal viable product", "pep8", and "procedural-style" script.

you can demo the platform without providing any acct details

there is also metaNODEwhitepaper and manualSIGNGwhitepaper in my repo

enjoy, pro bono

crypto long, moar coinz short!

-uncle lp





POST SCRIPT:

YES! this stack does take your wif key, and performs ECDSA. 
<THIS SHOULD CONCERN YOU>
Scammers can use these methods to take all ur cryptoz!!!
<THIS SHOULD CONCERN YOU>
You should NOT blindly trust ANYONE to write a GUI wallet for you. 
You should read ANY script with access to your funds before giving it your keys!

8
General Discussion / Re: API Node Latency Map
« on: February 14, 2019, 05:17:34 pm »
What does the large pulsating circle around geographical points indicate? Activity?

the bigger the circle the faster the response time to USA east coast

9
General Discussion / Re: API Node Latency Map
« on: December 13, 2018, 01:05:42 am »
updated repo

- removed whitespace from uploaded images for easier incorporation into user websites
- uploaded basemap.png to repo
- added list of seed nodes to jsonbin under key 'SEEDS' with data ('ip', 'ping', 'geolocation')
- plotting seed nodes on top with yellow dots
- added UTC timestamp to images
- removed misleading green dots in africa from the lens flare in the base map

new sample:




this should satisfy most requirements of issue 1348

https://github.com/bitshares/bitshares-core/issues/1348


10
General Discussion / API Node Latency Map
« on: December 07, 2018, 10:30:21 pm »
for data you go here:

https://api.jsonbin.io/b/5c06e4f71deea01014bd4261/latest#Bitshares_Latency

This jsonbin api is at a stationary location.

I update the latency info hourly via python script; I'm on east coast US.

at the link you will find json dictionary with keys:

['GEO', 'COUNT', 'PING', 'URLS', 'LIVE', 'SOURCE_CODE', 'MAP_URL', 'MISSION', 'UTC', 'UNIX', 'OWNER', 'LOCATION', 'UNIVERSE']

UNIVERSE is all known bitshares public api nodes mentioned in various github projects; I suspect I have the largest known list of bitshares public api domains

URLS are github urls where node lists can be found; this tool does not contain a list of nodes.  Instead it goes to many different repos on github (such as bitshares, cryptobridge, rudex) etc.  where people have configuration files with lists of nodes; if they update their repo with new nodes, the script finds them with web scraping methods

LIVE is a list of tested nodes that respond within 10 seconds and have a head block number that is not stale. 

PING is list of tuples of ping time for each LIVE node

GEO is geo location data (country, city, longitude, latitude) for each LIVE node as provided by ip-api.com or ipinfo.io

COUNT is number of live nodes vs number of nodes in universe

SOURCE_CODE is my github were you can get this tool and run it yourself to produce your own web hosted latency map in distributed fashion.  The latency script will also give you information as to why each down node failed to connect.  A full report looks like this:

https://pastebin.com/JL4DZzwY

You can also pick up copy of metaNODE and ExtinctionEvent while you're there :D

UNIX and UTC are time stamp when this jsonbin was last updated

MAP_URL is a link to a visual image of the data on the jsonbin;
the map url will change every hour!


it will be something like:
https://vgy.me/dgCE1C.png


and look like one frame of this gif





source code is here, WTFPL:

https://github.com/litepresence/extinction-event/blob/master/EV/bitshares-latency.py

todo list... upload 24hr animations instead of still images


thanks to free and easy to use api services that make this project is possible:

ip-api.com
jsonbin.io
vgy.me



11


I've tightened my margins... let the scalp wars begin.

blue is dex bid/ask; tan is dex last

the margins and price line are now very dynamic on the dex, much like external cex prices for bitshares vs btc



:D

same market, matching 24 hour period, 60 days later... bts:open.btc dex prices and book have gone from bar code to liquid


12
EV6 has been released




this chart is 30 days ago
magenta/yellow is cex price
blue/white is dex price


Even without my personal EV running, the BTS:OPEN.BTC bid / ask and priceline are no longer stepwise like this image from 30 days ago.     
The margins are tight, dynamic like the cex markets... they look to be EV controlled by some other operator running a different tune from my own.

I've tightened my margins... let the scalp wars begin.

:D

13
EV7.py and metaNODE11.py released to github

lots of new niceties related to plotting and logging, improved control over scalping

Live session now rotates through 3 views every 5 minutes.

The EV7/metaNODE11 stack has been

LIVE TESTED WITH FUNDS

for a 1,000,000 second (12 day) run without hiccup or user intervention; buy/sell/cancel and replace all scalp and primary orders every 5 minutes; maintaining a perfect count of 5 minute ticks during the ongoing run. 

I quite literally locked a linux box in a room running algo for 2 weeks entirely unattended; it said BTS/BTC bear market when I left.  When I returned it said bull market and had 13% more btc than starting portfolio.  Log file clean.  8)

PRIMARY LIVE VIEW


TRADE DESION LIVE VIEW


SCALPING LIVE VIEW


it ships with a 120X stock 1000 day BTS/BTC tune; my personal tune backtests over 7000X during the same period

BACKTEST

14
EV6 has been released

improved scalp logic offsets thresholds above or below cex market prices based on primary indicator market cross

previously when creating simple moving averages from cex data there was a degree of stepwise high frequency "wiggle" in the indications where the moving averages would oscillate up and down +/- 1 satoshi per hour; this is a result of candle aggregation methods in the data arriving from cryptocompare.  I have resolved this using polynomial regression of the last 24 hours of plotted moving average.  So now bot plots moving average from cex data, then performs poly, then replots the moving average "smoothly"; the state machine is then fed the smoothed indication.

additionally; the new regression allows for moving averages to be extrapolated into the future; these moving average predictions are in yellow and are deleted and redrawn every 5 minutes; the tip of the prediction (24 hours into the future) is plotted as a single magenta dot and it stays on the chart.

finally, in the log some information about primary moving average slope and concavity is provided.

you can see the older stepwise moving average here; the step size is about one satoshi



this shows new poly regressions of moving averages extending into future


this is zoomed in on the 50 day average


zoomed in further



this is data provided in the new log

15
General Discussion / [ANN] metaNODE = Bitshares_Trustless_Client()
« on: June 06, 2018, 01:52:45 pm »
metaNODE = Bitshares_Trustless_Client()

There were two ways to get data from Bitshares blockchain:

- a private node that uses lots of RAM, prefers its own machine, and is technical to tend
- a public node that is difficult to stay connected to and may provide rogue data

I've created a 3rd path; connect to several random nodes in the public network... ask them each for latest market data; then move on to other nodes in the network continually. Finally, perform statistical analysis on the combined feeds and maintain a streaming curated output file; the metaNODE.

python script and whitepaper, including usage:

https://github.com/litepresence/extinction-event/tree/master/metaNODE

Code: [Select]
LICENSE:

'litepresence 2018'

def WTFPL_v0_March_1765():
    if any([stamps, licenses, taxation, regulation, fiat, etat]):
        try:
            print('no thank you')
        except:
            return [tar, feathers]

Mission:

1/10 the RAM usage of a personal node
99.9999 six sigma uptime
99.9999 six sigma accurate data feed
less than 5000ms latency


metaNODE is a streaming text file that contains statistically curated dex data from all public nodes for a single market on a single account.  metaNODE currently curates the following feeds:

- last
- market history
- open orders
- orderbook
- market-account balance


Run time has been demonstrated in excess of 2 weeks. 
metaNODE.txt is updated about once per second with live data.

metaNODE
is entirely independent of pybitshares.
Public database calls are made with websocket-client.


This project has received funding through the DEXbot worker.  Additional funding has been earmarked for incorporation of metaNODE into future release of the DEXbot platform.

metaNODE10 has already been incorporated into the Extinction Event dex algo trading framework EV.py; learn more at

www.litepresence.com





Pages: [1] 2 3 4 5 6