Author Topic: Tool / website: Bitshares Witness Log  (Read 3482 times)

0 Members and 1 Guest are viewing this topic.

Offline roelandp

  • Full Member
  • ***
  • Posts: 111
  • Witness, dad, kitesurfer, event organiser
    • View Profile
    • RoelandP.nl
  • BitShares: roelandp
  • GitHub: roelandp
Tool / website: Bitshares Witness Log
« on: May 01, 2017, 10:13:47 pm »
Last week I released a witness monitor tool, meant for personal usage by witnesses, to monitor the availability of seednode, pricefeed latency in updates and missing blocks. From various members I received requests to expand on that monitoring tool and start to log the data and turn it into a Bitshares Witness Log. I found it a challenge in my python skills, so I sharpened my knife:



What does it do?

* Update Frequency
Three times per hour (2nd,22nd,42nd minute) the backend checks this witness data by using Python-Bitshares (thank you [member=120]xeroc[/member]!) and storing it in an MySQL database.

Three times per hour (5th, 25th, 45th minute) the backend publishes a new version as a static JSONP file with the latest data details as compiled from this database as a source-feed for this overview, using Bootstrap-Table to display it. This means the data you see can be maximum 23 minutes old before fresh data is displayed. If this is not the case and the top "last updated at" mentions way more time since that, please notify me as my server must have crashed!

Insights & Analyse
  • This witness log gives historical insights in basic witness tasks: producing blocks & regular price feed publications.
  • The font and opacity of price feed entries get smaller the older the feed gets. Once the feed is older than 24 hours it gets a red font-color.
  • The markets are chosen and sorted by market 24 hour market volume.
  • Expand individual rows by hitting the '+'-sign for a detailed witness pricefeed log including variance of the settlement price.
  • The overview table shows 'red' warning fonts whenever:
    • Feed is not available.
    • Feed is older than 24 hours.
    • Feed price variance is more than +/- 5% of current settlement price.
And more...
  • Export data via the top-right export menu.
  • Customise the table via the top-right export menu and exclude certain columns.
  • Search the table.
  • Sort the table by many of the columns.
  • Hover over the price feed entries to see 'Last Published' time
Todos:
  • adding public seednodes of every witness (1 per witness) so I can also monitor the uptime.
  • After 1 week of logging I will also add a 'difference indicator', showing how many new blocks are missed (for 1 week displayed) per witness.
Click here to go to the Bitshares Witness Log
http://roelandp.nl/bitshareswitnesslog


ps: I know I am a very new witness, and I don't want to be telling anyone what to do, I feel humble in my position: nonetheless I noticed that some of the witnesses are missing their 'urls'-part in the witness entry. Would be great if you could 'update_witness' with an URL![/list]
« Last Edit: May 02, 2017, 03:41:01 pm by roelandp »

Offline alt

  • Hero Member
  • *****
  • Posts: 2806
    • View Profile
  • BitShares: baozi
Re: Bitshares Witness Log
« Reply #1 on: May 02, 2017, 12:53:47 am »
good job, thank you

Offline sahkan

  • Sr. Member
  • ****
  • Posts: 233
    • View Profile
    • BitShares DEX
Re: Bitshares Witness Log
« Reply #2 on: May 02, 2017, 05:00:44 am »
 +5%

Offline Chris4210

  • Sr. Member
  • ****
  • Posts: 431
  • Keep Building!
    • View Profile
    • www.payger.com
  • BitShares: chris4210
Re: Bitshares Witness Log
« Reply #3 on: May 02, 2017, 05:12:18 am »
Thank you Roelandp for this cool new tool. That will help to monitor the witness activity more accurate. It is also great to see what kind of tools can be built thanks to Xeroc´s Python-Libary!  +5%
Vote Chris4210 for Committee Member http://bit.ly/1WKC03B! | www.Payger.com - Payments + Messenger | www.BitShareshub.io - Community based fanpage for the BitShares Blockchain

Offline fav

  • Administrator
  • Hero Member
  • *****
  • Posts: 4272
  • No Pain, No Gain
    • View Profile
    • Follow Me!
  • BitShares: fav
Re: Bitshares Witness Log
« Reply #4 on: May 02, 2017, 06:43:12 am »
awesome!!
Join us on Telegram: https://t.me/btstalk

Offline roelandp

  • Full Member
  • ***
  • Posts: 111
  • Witness, dad, kitesurfer, event organiser
    • View Profile
    • RoelandP.nl
  • BitShares: roelandp
  • GitHub: roelandp
Re: Bitshares Witness Log
« Reply #5 on: May 02, 2017, 07:55:52 am »
thanks for the replies [member=4465]alt[/member] [member=38926]Chris4210[/member] @sakhan [member=31]fav[/member] !

Updated with some cosmetic changes:



Note if you hit the (+) button in front of each witness you can see a detailed view

http://roelandp.nl/bitshareswitnesslog

Offline Yao

  • Hero Member
  • *****
  • Posts: 532
  • QQ/WeChat(微信):664349247
    • View Profile
  • BitShares: yao
  • GitHub: imYao
Re: Bitshares Witness Log
« Reply #6 on: May 02, 2017, 08:15:49 am »
Great job!  +5%
BTS witness: witness.yao
BTS Proxy: yao

Offline roelandp

  • Full Member
  • ***
  • Posts: 111
  • Witness, dad, kitesurfer, event organiser
    • View Profile
    • RoelandP.nl
  • BitShares: roelandp
  • GitHub: roelandp
Re: Bitshares Witness Log
« Reply #7 on: May 02, 2017, 03:40:24 pm »
Updates
  • Under the detail-pane (hit (+) button in front of any witness) you now find all (other) feeds published by the given witness (provided I track them, if you miss any, hit me up))
  • In the main overview with the 8-10 main markets (based on 24h volume) the 'median' price feed is highlighted with 'green + bold' in each column.
  • Some minor additional cosmetic changes to make err'thing look a little more slick-rick.
Let me know any feedback! Thanks!
« Last Edit: May 02, 2017, 06:27:28 pm by roelandp »

Offline roelandp

  • Full Member
  • ***
  • Posts: 111
  • Witness, dad, kitesurfer, event organiser
    • View Profile
    • RoelandP.nl
  • BitShares: roelandp
  • GitHub: roelandp
Re: Tool / website: Bitshares Witness Log
« Reply #8 on: May 02, 2017, 09:45:30 pm »
Update:
Added the 'This week missed' count next to the 'total blocks missed' per witness. The difference between the current missed blocks or the missed_blocks 7 days ago (or oldest registered if no 7 days exist) is being displayed. This adds to the realtime data and says more than the total_missed blocks IMHO.

Offline tbone

  • Hero Member
  • *****
  • Posts: 632
    • View Profile
  • BitShares: tbone2
Re: Tool / website: Bitshares Witness Log
« Reply #9 on: May 02, 2017, 11:33:33 pm »
[member=43607]roelandp[/member], this is really excellent work!  A tool like this has been sorely needed.  There was a nice discussion a while back about what features would be useful for such a tool. I can't find it right now, but beyond what you've done so far, some of the ideas I can remember included monitoring latency and uptime (which you may have in the works already), measuring server processor speed and memory, and there was discussion about combining all of the various factors into a score.  And finally, there was talk about creating some charts in order to help visual the data.  I hope this gives you some ideas to continue taking this excellent tool forward!

Offline xeroc

  • Board Moderator
  • Hero Member
  • *****
  • Posts: 12897
  • ChainSquad GmbH
    • View Profile
    • ChainSquad GmbH
  • BitShares: xeroc
  • GitHub: xeroc
Re: Tool / website: Bitshares Witness Log
« Reply #10 on: May 03, 2017, 08:52:41 am »
Excellent work!
Give BitShares a try! Use the http://testnet.bitshares.eu provided by http://bitshares.eu powered by ChainSquad GmbH

Offline roelandp

  • Full Member
  • ***
  • Posts: 111
  • Witness, dad, kitesurfer, event organiser
    • View Profile
    • RoelandP.nl
  • BitShares: roelandp
  • GitHub: roelandp
Re: Tool / website: Bitshares Witness Log
« Reply #11 on: May 03, 2017, 09:24:18 am »
    [member=43607]roelandp[/member], this is really excellent work!  A tool like this has been sorely needed.  There was a nice discussion a while back about what features would be useful for such a tool. I can't find it right now, but beyond what you've done so far, some of the ideas I can remember included monitoring latency and uptime (which you may have in the works already), measuring server processor speed and memory, and there was discussion about combining all of the various factors into a score.  And finally, there was talk about creating some charts in order to help visual the data.  I hope this gives you some ideas to continue taking this excellent tool forward!

    thx tbone.
    • I will add a monitoring for latency & uptime of the public seednodes provided i can find them by the witnesses posts.
    • Monitoring individual block producing servers for the network can be done as well however for security reasons witnesses should not declare their public nodes' IP / hostnames, so I cannot and/or won't tie those IP addresses to witness names.
    • Measuring server processor speed & memory would need a program (which would monitor and submit) run on the client side of the witness nodes, which is a bit to much to ask and also puts trust onto the witness to install this program onto their actual producing nodes and not some high powered machine to modify stats, so I will leave this out for now. I think the server speed / memory etc is not that important either, as long as witnesses are reliable block producers. If the chain grows and tx's requires better servers they will notice as they might start missing blocks and operate accoringly to upgrade system (or not).
    • Charts: [member=22138]lafona[/member] made some charts already... Let me know what kind of charts you want to see based on the data of http://roelandp.nl/bitshareswitnesslog ! I currently log the follow data 3 times per hour:
      • Pricefeeds per witness
      • Missed blocks
    [/list]
    « Last Edit: May 03, 2017, 09:26:00 am by roelandp »

    Offline Thom

    Re: Tool / website: Bitshares Witness Log
    « Reply #12 on: May 03, 2017, 03:52:45 pm »
    When showing "last update time" there are 2 timers to consider that operate asynchronously: 1) timer for when feeds are published, controlled by witness and 2) the timer your tool uses between updates.

    Thus when you say a feed is an hour old, I presume you mean that is the publish time you obtained in your most recent query of the blockchain. However, due to these 2 asynchronous timers there is inaccuracy in reporting.

    My witness publishes every hour ~2 minutes after the hour. I looked at your witness log at 10 minutes after the hour, after I published, and it reported the feed was not updated when it was.

    I don't think this is a major issue at all. You may already display the last time the blockchain was sampled and I just didn't notice it. Might be nice to show if it's not now.

    In my case if you sample 3 times near 0, 20 and 40 minutes past the hour the most out of date report would be less than 20 minutes old if requested between 2 and 19 minutes past the hour.
    Injustice anywhere is a threat to justice everywhere - MLK |  Verbaltech2 Witness Reports: https://bitsharestalk.org/index.php/topic,23902.0.html

    Offline roelandp

    • Full Member
    • ***
    • Posts: 111
    • Witness, dad, kitesurfer, event organiser
      • View Profile
      • RoelandP.nl
    • BitShares: roelandp
    • GitHub: roelandp
    Re: Tool / website: Bitshares Witness Log
    « Reply #13 on: May 03, 2017, 04:40:29 pm »
    Thus when you say a feed is an hour old, I presume you mean that is the publish time you obtained in your most recent query of the blockchain. However, due to these 2 asynchronous timers there is inaccuracy in reporting.

    My witness publishes every hour ~2 minutes after the hour. I looked at your witness log at 10 minutes after the hour, after I published, and it reported the feed was not updated when it was.

    Hi [member=21490]Thom[/member] the 'X time ago' feeds are based on the time registered on the blockchain as the 'feed publication' time. They are stored in the database, and then published in the JSON I use for the website. That JSON is parsed with moment.js to display the X-time ago based on your computer's time setting.

    Here is the part in the code grabbing that timestamp from the blockchain and prepping it as a datetime for mysql:
    Code: [Select]
    for feedarrays in asset['bitasset_data']['feeds']:
                feedtime = datetime.datetime.strptime(feedarrays[1][0], "%Y-%m-%dT%H:%M:%S")

    I use this combo feedtime + id_asset + id_account to compare feeds in my database so I only log new feed publications and not every time the same feeds when my tool does another round of data syncing.

    What is happening instead and why your feed is not showing up: My tool gets at exactly (NTP) minute 2, 22, 42 cronjobs the current status from the blockchain. Most probably you are then still executing your feed publications :)

    But yeah there is indeed a discrepancy... the 'last updated time' in the top of the page says whenever the last time the datafile was created. That is done on 5, 25, 45. But that is indeed based on the poll from 2,22,42. Let me see if I can align that more... I wanted to keep the polling & creating seperated, but I might just copy everything over into 1 script so I executes in 1 flow.

    TLDR; the website shows the times based on blockchain.
    « Last Edit: May 03, 2017, 06:15:28 pm by roelandp »

    Offline roelandp

    • Full Member
    • ***
    • Posts: 111
    • Witness, dad, kitesurfer, event organiser
      • View Profile
      • RoelandP.nl
    • BitShares: roelandp
    • GitHub: roelandp
    Re: Tool / website: Bitshares Witness Log
    « Reply #14 on: May 03, 2017, 06:25:49 pm »
    [member=21490]Thom[/member] I have updated the scripts now:

    After the logging of the latest blockchain data the script immediately executes the creation of the new cache-file (JSONP) for the display on the website. I also intensified the logging (and therefore updating of the site) having it 4 times per hour 5,20,35,50 minute.

    thanks for your input!