BitShares Forum
Main => Stakeholder Proposals => Topic started by: roelandp on May 01, 2017, 10:13:47 pm
-
Last week I released a witness monitor tool (https://bitsharestalk.org/index.php/topic,24042.msg305149.html#msg305149), meant for personal usage by witnesses, to monitor the availability of seednode, pricefeed latency in updates and missing blocks. From various members I received requests to expand on that monitoring tool and start to log the data and turn it into a Bitshares Witness Log. I found it a challenge in my python skills, so I sharpened my knife:
(http://i.imgur.com/KKgPCRT.png) (http://roelandp.nl/bitshareswitnesslog)
What does it do?
* Update Frequency
Three times per hour (2nd,22nd,42nd minute) the backend checks this witness data by using Python-Bitshares (https://python-bitshares.readthedocs.io) (thank you @xeroc!) and storing it in an MySQL database.
Three times per hour (5th, 25th, 45th minute) the backend publishes a new version as a static JSONP file with the latest data details as compiled from this database as a source-feed for this overview (http://roelandp.nl/bitshareswitnesslog), using Bootstrap-Table (https://github.com/wenzhixin/bootstrap-table) to display it. This means the data you see can be maximum 23 minutes old before fresh data is displayed. If this is not the case and the top "last updated at" mentions way more time since that, please notify me as my server must have crashed!
Insights & Analyse
- This witness log gives historical insights in basic witness tasks: producing blocks & regular price feed publications.
- The font and opacity of price feed entries get smaller the older the feed gets. Once the feed is older than 24 hours it gets a red font-color.
- The markets are chosen and sorted by market 24 hour market volume.
- Expand individual rows by hitting the '+'-sign for a detailed witness pricefeed log including variance of the settlement price.
- The overview table shows 'red' warning fonts whenever:
- Feed is not available.
- Feed is older than 24 hours.
- Feed price variance is more than +/- 5% of current settlement price.
And more...
- Export data via the top-right export menu.
- Customise the table via the top-right export menu and exclude certain columns.
- Search the table.
- Sort the table by many of the columns.
- Hover over the price feed entries to see 'Last Published' time
Todos:
- adding public seednodes of every witness (1 per witness) so I can also monitor the uptime.
- After 1 week of logging I will also add a 'difference indicator', showing how many new blocks are missed (for 1 week displayed) per witness.
Click here to go to the Bitshares Witness Log (http://roelandp.nl/bitshareswitnesslog)
http://roelandp.nl/bitshareswitnesslog
ps: I know I am a very new witness, and I don't want to be telling anyone what to do, I feel humble in my position: nonetheless I noticed that some of the witnesses are missing their 'urls'-part in the witness entry. Would be great if you could 'update_witness' with an URL![/list]
-
good job, thank you
-
+5%
-
Thank you Roelandp for this cool new tool. That will help to monitor the witness activity more accurate. It is also great to see what kind of tools can be built thanks to Xeroc´s Python-Libary! +5%
-
awesome!!
-
thanks for the replies @alt @Chris4210 @sakhan @fav !
Updated with some cosmetic changes:
(http://i.imgur.com/yRDqfRK.png) (http://roelandp.nl/bitshareswitnesslog)
Note if you hit the (+) button in front of each witness you can see a detailed view
http://roelandp.nl/bitshareswitnesslog
-
Great job! +5%
-
Updates
- Under the detail-pane (hit (+) button in front of any witness) you now find all (other) feeds published by the given witness (provided I track them, if you miss any, hit me up))
- In the main overview with the 8-10 main markets (based on 24h volume) the 'median' price feed is highlighted with 'green + bold' in each column.
- Some minor additional cosmetic changes to make err'thing look a little more slick-rick.
Let me know any feedback! Thanks!
-
Update:
Added the 'This week missed' count next to the 'total blocks missed' per witness. The difference between the current missed blocks or the missed_blocks 7 days ago (or oldest registered if no 7 days exist) is being displayed. This adds to the realtime data and says more than the total_missed blocks IMHO.
-
@roelandp, this is really excellent work! A tool like this has been sorely needed. There was a nice discussion a while back about what features would be useful for such a tool. I can't find it right now, but beyond what you've done so far, some of the ideas I can remember included monitoring latency and uptime (which you may have in the works already), measuring server processor speed and memory, and there was discussion about combining all of the various factors into a score. And finally, there was talk about creating some charts in order to help visual the data. I hope this gives you some ideas to continue taking this excellent tool forward!
-
Excellent work!
-
@roelandp, this is really excellent work! A tool like this has been sorely needed. There was a nice discussion a while back about what features would be useful for such a tool. I can't find it right now, but beyond what you've done so far, some of the ideas I can remember included monitoring latency and uptime (which you may have in the works already), measuring server processor speed and memory, and there was discussion about combining all of the various factors into a score. And finally, there was talk about creating some charts in order to help visual the data. I hope this gives you some ideas to continue taking this excellent tool forward!
thx tbone.
- I will add a monitoring for latency & uptime of the public seednodes provided i can find them by the witnesses posts.
- Monitoring individual block producing servers for the network can be done as well however for security reasons witnesses should not declare their public nodes' IP / hostnames, so I cannot and/or won't tie those IP addresses to witness names.
- Measuring server processor speed & memory would need a program (which would monitor and submit) run on the client side of the witness nodes, which is a bit to much to ask and also puts trust onto the witness to install this program onto their actual producing nodes and not some high powered machine to modify stats, so I will leave this out for now. I think the server speed / memory etc is not that important either, as long as witnesses are reliable block producers. If the chain grows and tx's requires better servers they will notice as they might start missing blocks and operate accoringly to upgrade system (or not).
- Charts: @lafona made some charts already... Let me know what kind of charts you want to see based on the data of http://roelandp.nl/bitshareswitnesslog ! I currently log the follow data 3 times per hour:
- Pricefeeds per witness
- Missed blocks
[/list]
-
When showing "last update time" there are 2 timers to consider that operate asynchronously: 1) timer for when feeds are published, controlled by witness and 2) the timer your tool uses between updates.
Thus when you say a feed is an hour old, I presume you mean that is the publish time you obtained in your most recent query of the blockchain. However, due to these 2 asynchronous timers there is inaccuracy in reporting.
My witness publishes every hour ~2 minutes after the hour. I looked at your witness log at 10 minutes after the hour, after I published, and it reported the feed was not updated when it was.
I don't think this is a major issue at all. You may already display the last time the blockchain was sampled and I just didn't notice it. Might be nice to show if it's not now.
In my case if you sample 3 times near 0, 20 and 40 minutes past the hour the most out of date report would be less than 20 minutes old if requested between 2 and 19 minutes past the hour.
-
Thus when you say a feed is an hour old, I presume you mean that is the publish time you obtained in your most recent query of the blockchain. However, due to these 2 asynchronous timers there is inaccuracy in reporting.
My witness publishes every hour ~2 minutes after the hour. I looked at your witness log at 10 minutes after the hour, after I published, and it reported the feed was not updated when it was.
Hi @Thom the 'X time ago' feeds are based on the time registered on the blockchain as the 'feed publication' time. They are stored in the database, and then published in the JSON I use for the website. That JSON is parsed with moment.js to display the X-time ago based on your computer's time setting.
Here is the part in the code grabbing that timestamp from the blockchain and prepping it as a datetime for mysql:
for feedarrays in asset['bitasset_data']['feeds']:
feedtime = datetime.datetime.strptime(feedarrays[1][0], "%Y-%m-%dT%H:%M:%S")
I use this combo feedtime + id_asset + id_account to compare feeds in my database so I only log new feed publications and not every time the same feeds when my tool does another round of data syncing.
What is happening instead and why your feed is not showing up: My tool gets at exactly (NTP) minute 2, 22, 42 cronjobs the current status from the blockchain. Most probably you are then still executing your feed publications :)
But yeah there is indeed a discrepancy... the 'last updated time' in the top of the page says whenever the last time the datafile was created. That is done on 5, 25, 45. But that is indeed based on the poll from 2,22,42. Let me see if I can align that more... I wanted to keep the polling & creating seperated, but I might just copy everything over into 1 script so I executes in 1 flow.
TLDR; the website (http://roelandp.nl/bitshareswitnesslog) shows the times based on blockchain.
-
@Thom I have updated the scripts now:
After the logging of the latest blockchain data the script immediately executes the creation of the new cache-file (JSONP) for the display on the website. I also intensified the logging (and therefore updating of the site) having it 4 times per hour 5,20,35,50 minute.
thanks for your input!
-
Looks great!
1 little nit - The text under the table "Update Frequency" reflects 3 times an hour, but now you're updating 4 times an hour; do I have that right?
-
Looks great!
1 little nit - The text under the table "Update Frequency" reflects 3 times an hour, but now you're updating 4 times an hour; do I have that right?
yes i was so lazy I still have to update the frontend. They are seperate. For these kind of tools, the way how I like it is: the frontends are static hosted pages on AWS and the backend is a complete different server rendering static data files from the database and pushing it to AWS. This way no need to setup NGINX and support server loads. How i like it even better would be to not even have a database but host everything on Google Spreadsheet :P
But asap I will push some new stuff to the frontend and overwrite that info :)
-
@Thom updated the explainer on the site...
Additionally added feature to highlight + deeplink to a witness in the list: http://roelandp.nl/bitshareswitnesslog/?witness=roelandp
+ a fix for bug submitted by @Bhuz : sorting was off for 'missed blocks' column.
-
Updates
Updates to http://roelandp.nl/bitshareswitnesslog/ with regards to GUI:
- Made high variance difference not coloured red any longer. High Variance in a pricefeed of a given witness can also mean markets are moving fast and therefore a higher % off the median can also mean the given witness is already moving along to the new direction, whilst others not yet... New objective indicator for that... (tiny triangle top right of the cell)
- Also Median is no longer green (is subjective too), but now a colored background cell.
- Additionally onhover pricefeed cells now shows instant tooltips to reveal 'published ago + variance percentage'
- Added the 'market volumes' to the header titles
- Added some explainer stuff in the bottom
Hard refresh to see.