Author Topic: [python] bts_tools witness monitoring tools for BitShares 2.0  (Read 2521 times)

0 Members and 1 Guest are viewing this topic.

Offline wackou

[python] bts_tools witness monitoring tools for BitShares 2.0
« on: October 27, 2015, 09:41:41 pm »
For those that don't know the bts_tools package, the best description can probably be given with screenshots (a picture is worth a thousand words!): http://bts-tools.readthedocs.org/en/latest/monitor.html#screenshots

-------------------------------------

A bit later than what I originally hoped for, here comes the bts_tools package ported to support graphene clients, in this case BitShares 2.0.

Given the new asynchronous model of communication via websockets, I had to refactor more than originally planned, but the bulk of it is now done, and I learnt quite a great deal in the process (wanted to look into async programming in python for some time now, and this was the perfect occasion!). Thanks go to Xeroc for writing his python-graphenelib module, that was extremely helpful in getting me bootstrapped and understanding how it all worked.

Please bear in mind that there are probably still a few things not working, or not working properly. The basic functionality is there, though:
 - cpu/ram/network connections graph
 - feeds publishing, using either a time interval (as before) or a time slot (as currently setup in the telegram chat)
 - some other plugins: wallet_state, voted_in, online, network_connections

notably missing:
 - detection of blocks missed

So for the non-faint of heart that don't mind a few quirks, you can install the tools as usual with:

Code: [Select]
$ pip3 install bts_tools
and you should get version >= 0.3. The documentation, although not updated yet, should still mostly apply to everything: http://bts-tools.readthedocs.org/

What you can do then:
 - bts2 build [tag]: builds the latest version, or a given version
 - bts2 run: runs the witness client
 - bts2 run_cli: runs the cli wallet
 - bts2 monitor: launch the monitoring tools

Things to watch out for in this release:
 - the config file is still sensitive, ie: you need to configure everything properly, there is very little checking. The default config.yaml file comes with the minimum required fields you need to configure to get something working, edit at will. If you had a previous config.yaml file, it is highly recommended to delete it beforehand.
 - in particular, you need to setup correctly an api_access.json file (as described here: https://github.com/BitShares/bitshares-2#accessing-restricted-apis - the saltpass.py script can be found here: https://github.com/BitShares/bitshares-2/blob/bitshares/programs%2Fwitness_node%2Fsaltpass.py) and configure your config.ini file to use it. This is needed to get access to restricted APIs of the client such as the network one.

What I plan to do next is to iron out the support, port the remaining monitoring plugins and stabilize the code base. Support for Muse clients should also come very soon :)

Let me know what you think!
Please vote for witness wackou! More info at http://digitalgaia.io

Offline bytemaster

Re: [python] bts_tools witness monitoring tools for BitShares 2.0
« Reply #1 on: October 27, 2015, 09:49:00 pm »
Great work!  Very excited! Much Wow!
For the latest updates checkout my blog: http://bytemaster.bitshares.org
Anything said on these forums does not constitute an intent to create a legal obligation or contract between myself and anyone else.   These are merely my opinions and I reserve the right to change them at any time.

Offline xeroc

  • Board Moderator
  • Hero Member
  • *****
  • Posts: 12915
  • ChainSquad GmbH
    • View Profile
    • ChainSquad GmbH
  • BitShares: xeroc
  • GitHub: xeroc
Re: [python] bts_tools witness monitoring tools for BitShares 2.0
« Reply #2 on: October 28, 2015, 08:20:57 am »
Very cool
Give BitShares a try! Use the http://testnet.bitshares.eu provided by http://bitshares.eu powered by ChainSquad GmbH

Offline emski

  • Hero Member
  • *****
  • Posts: 1282
    • View Profile
    • http://lnkd.in/nPbhxG
Re: [python] bts_tools witness monitoring tools for BitShares 2.0
« Reply #3 on: October 28, 2015, 08:32:39 am »
Congratulations! You've persistently improved these tools starting from ground level. Good Job!

Offline abit

  • Committee member
  • Hero Member
  • *
  • Posts: 4473
    • View Profile
    • Abit's Hive Blog
  • BitShares: abit
  • GitHub: abitmore
Re: [python] bts_tools witness monitoring tools for BitShares 2.0
« Reply #4 on: October 28, 2015, 10:34:59 am »
 +5%
BTS account: abit
BTS committee member: abit
BTS witness: in.abit

Offline cube

  • Hero Member
  • *****
  • Posts: 1404
  • Bit by bit, we will get there!
    • View Profile
  • BitShares: bitcube
Re: [python] bts_tools witness monitoring tools for BitShares 2.0
« Reply #5 on: October 28, 2015, 01:53:02 pm »
For those that don't know the bts_tools package, the best description can probably be given with screenshots (a picture is worth a thousand words!): http://bts-tools.readthedocs.org/en/latest/monitor.html#screenshots

Let me know what you think!

The screenshots are outdated.  The dates are in 2014 and the terms used are for BTS1.  Could you please put up screenshots for BTS2?

ID: bitcube
bitcube is a dedicated witness and committe member. Please vote for bitcube.

Offline roadscape

Re: [python] bts_tools witness monitoring tools for BitShares 2.0
« Reply #6 on: October 28, 2015, 05:32:05 pm »
Nice! +5%
http://cryptofresh.com  |  witness: roadscape

Offline wackou

Re: [python] bts_tools witness monitoring tools for BitShares 2.0
« Reply #7 on: October 28, 2015, 09:52:04 pm »
The screenshots are outdated.  The dates are in 2014 and the terms used are for BTS1.  Could you please put up screenshots for BTS2?

I haven't gotten around to making new screenshots yet, was planning to do it after fixing everything. Currently the screenshots I could take don't show everything or some wrong fields, and I prefer to leave old screenshots for now than put broken ones. They would be very similar in spirit, though, as the UI hasn't changed much since bts1.
Please vote for witness wackou! More info at http://digitalgaia.io

Offline cube

  • Hero Member
  • *****
  • Posts: 1404
  • Bit by bit, we will get there!
    • View Profile
  • BitShares: bitcube
Re: [python] bts_tools witness monitoring tools for BitShares 2.0
« Reply #8 on: October 29, 2015, 04:12:44 am »
The screenshots are outdated.  The dates are in 2014 and the terms used are for BTS1.  Could you please put up screenshots for BTS2?

I haven't gotten around to making new screenshots yet, was planning to do it after fixing everything. Currently the screenshots I could take don't show everything or some wrong fields, and I prefer to leave old screenshots for now than put broken ones. They would be very similar in spirit, though, as the UI hasn't changed much since bts1.

Looking forward to the new fields being ready.  It would be a great tool to use.
ID: bitcube
bitcube is a dedicated witness and committe member. Please vote for bitcube.

Offline wackou

Re: [python] bts_tools witness monitoring tools for BitShares 2.0
« Reply #9 on: November 01, 2015, 10:52:15 am »
Released 0.3.1 which fixes a few things, has supports for Muse (should mostly work, but not extensively tested), and has enhanced feeds support. From the changelog:

Code: [Select]
* support for Muse clients
* better feeds script. Process is now the following:
  - get the BTS/BTC valuation from Poloniex, CCEDK, Bter, Btc38 (configurable)
  - get the BTC/USD valuation from BitcoinAverage, with fallback on Bitfinex and Bitstamp
  - get the BTS valuation in other fiat currencies using Yahoo forex rates
  - get market indices using Yahoo, Google, Bloomberg (configurable)
* can specify 'boost_root' option in build environment in config.yaml
* minor bugfixes everywhere

When using multiple feed providers, price is always weighted by the volume of the exchange. Screenshots coming soon now, once I fix feeds display (in info screen) and "blocks produced / blocks missed" (always at 0 for now)
Please vote for witness wackou! More info at http://digitalgaia.io

Offline wackou

Re: [python] bts_tools witness monitoring tools for BitShares 2.0
« Reply #10 on: December 06, 2015, 04:31:02 pm »
Released 0.3.2 as an interim release to fix the CNY feed publishing. Price is now taken from pure BTS/CNY markets on Btc38, Bter and Yunbi, weighed by volume. There are (a lot!) more fixes in master, but those will have to wait for the next release of the BitShares backend.
Please vote for witness wackou! More info at http://digitalgaia.io

Offline Bhuz

  • Committee member
  • Sr. Member
  • *
  • Posts: 467
    • View Profile
  • BitShares: bhuz
Re: [python] bts_tools witness monitoring tools for BitShares 2.0
« Reply #11 on: December 06, 2015, 04:45:36 pm »
Released 0.3.2 as an interim release to fix the CNY feed publishing. Price is now taken from pure BTS/CNY markets on Btc38, Bter and Yunbi, weighed by volume. There are (a lot!) more fixes in master, but those will have to wait for the next release of the BitShares backend.
Bter do not trade bts 2.0
Please do not include Bter until it will honor bts 2.0

Offline wackou

Re: [python] bts_tools witness monitoring tools for BitShares 2.0
« Reply #12 on: December 06, 2015, 07:30:13 pm »
Released 0.3.2 as an interim release to fix the CNY feed publishing. Price is now taken from pure BTS/CNY markets on Btc38, Bter and Yunbi, weighed by volume. There are (a lot!) more fixes in master, but those will have to wait for the next release of the BitShares backend.
Bter do not trade bts 2.0
Please do not include Bter until it will honor bts 2.0

oh, I didn't know they hadn't upgraded yet... Anyone using my script can remove it themselves by not including it in the list of active feed providers in the config.yaml file. I will remove it from the default config for next version.
Please vote for witness wackou! More info at http://digitalgaia.io

Offline wackou

Re: [python] bts_tools witness monitoring tools for BitShares 2.0
« Reply #13 on: December 10, 2015, 11:17:13 pm »
I just released 0.3.3, which fixes quite a new things overall and general stability should be much better now. There are still some minor features missing, but in general this should work as well as it did for BitShares 0.9.x.

note: this requires the latest bitshares client to work  (v2.0.151209)

Code: [Select]
0.3.3 (2015-12-10)
------------------

* NOTE: requires the latest version (v2.0.151209) of the BitShares witness client
* better integration with the websocket event loop
* network views available again
* overall lots of minor fixes and general stability improvements
Please vote for witness wackou! More info at http://digitalgaia.io

Offline wackou

Re: [python] bts_tools witness monitoring tools for BitShares 2.0
« Reply #14 on: December 21, 2015, 04:17:48 pm »
0.3.4 has a few fixes, is more robust when checking feeds on the chinese exchanges (btc38, yunbi), publishes the "correct" price for BTS/CNY and also publishes feed price for the TCNY asset with correct parameters.

Code: [Select]
0.3.4 (2015-12-21)
------------------

* added support for publishing TCNY feed; generally more robust feeds fetching/publishing
* renamed 'bts2' commandline tool to 'bts'. 'bts2' still working for convenience, old 'bts' is still available as 'bts1'.
* change in the config.yaml format: https://github.com/wackou/bts_tools/commit/24e962820775a8a23e0b45d26c501aa7e723ff64

Please vote for witness wackou! More info at http://digitalgaia.io