Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.


Messages - roelandp

Pages: 1 2 3 4 5 [6] 7 8
76
Stakeholder Proposals / Re: [Witness Proposal] roelandp
« on: May 07, 2017, 10:27:30 pm »
Hi Sakhan, that page is just a viewer for the datafile which is being created and stored here: http://s3.amazonaws.com/roelandp-nl/witnesslog.json every 15 minutes. The source for the logger is a mysql database in which I store the assets and basically the results from the other script I wrote, but then for all witnesses. IMHO a block explorer would require a different set of logging also logging different calls and update on each block. The code for the Bitshares Witness Log logger into mysql is not really worth publishing as I am still learning python :P

77
Stakeholder Proposals / Re: [Witness Proposal] roelandp
« on: May 07, 2017, 10:14:29 pm »
You are such a great witness! No wonder I voted for you  :D

I was thinking that maybe over time we can morph it into a BTS block explorer....

:D thank you! but https://cryptofresh.com ?

78
To avoid that you need a way to make sure the old witness is definitely is dead with no chance of coming back online while the new witness takes over. To do that you need some smarts in the cooperating failsafe nodes to determine each node's state. Some type of heartbeat so that if the node producing blocks does NOT hear heartbeats from at least 2 other nodes it will cease block production. The producing node needs to verify it can communicate with the other witnesses, particularly the failsafe nodes.

Hi @Thom we briefly discussed this in telegram (i think) but I still feel the setup with having multiple servers with each its own private  / public key (use
Code: [Select]
suggest_brain_key) and the witness_node software running is the way to go. As soon as the blockchain starts logging missing blocks for your witness, you know it is malfunctioning. If you then issue an
Code: [Select]
update_witness command to switch to the other public signing key it doesn't matter if the other node might be coming back online, because it then still would try to sign blocks with the (by then) outdated public key.

I wrote a paragraph in this update for the witness docs (not yet committed): https://github.com/roelandp/docs.bitshares.eu/commit/75f56c50caeddf1e34c548c005443d726d6ab509#diff-c4ebae0b7f619df56e73bcea77eb3fe1R235

79
Stakeholder Proposals / Re: [Witness Proposal] roelandp
« on: May 07, 2017, 10:04:20 pm »
Made a witness monitor script to check the 3 core witness tasks on regular intervals, see: https://bitsharestalk.org/index.php/topic,24042.0.html
Would you be willing to make this open source?

It was already as announced in the post you link to in your quote :P
https://github.com/roelandp/Bitshares-Witness-Monitor

80
Stakeholder Proposals / Re: Tool / website: Bitshares Witness Log
« on: May 05, 2017, 09:02:44 pm »
@Thom updated the explainer on the site...

Additionally added feature to highlight + deeplink to a witness in the list: http://roelandp.nl/bitshareswitnesslog/?witness=roelandp

+ a fix for bug submitted by @Bhuz : sorting was off for 'missed blocks' column.


81
Stakeholder Proposals / Re: Tool / website: Bitshares Witness Log
« on: May 04, 2017, 08:11:15 pm »
Looks great!

1 little nit - The text under the table "Update Frequency" reflects 3 times an hour, but now you're updating 4 times an hour; do I have that right?

yes i was so lazy I still have to update the frontend. They are seperate. For these kind of tools, the way how I like it is: the frontends are static hosted pages on AWS and the backend is a complete different server rendering static data files from the database and pushing it to AWS. This way no need to setup NGINX and support server loads. How i like it even better would be to not even have a database but host everything on Google Spreadsheet :P

But asap I will push some new stuff to the frontend and overwrite that info :)


82
Stakeholder Proposals / Re: Tool / website: Bitshares Witness Log
« on: May 03, 2017, 06:25:49 pm »
@Thom I have updated the scripts now:

After the logging of the latest blockchain data the script immediately executes the creation of the new cache-file (JSONP) for the display on the website. I also intensified the logging (and therefore updating of the site) having it 4 times per hour 5,20,35,50 minute.

thanks for your input!

83
Stakeholder Proposals / Re: Tool / website: Bitshares Witness Log
« on: May 03, 2017, 04:40:29 pm »
Thus when you say a feed is an hour old, I presume you mean that is the publish time you obtained in your most recent query of the blockchain. However, due to these 2 asynchronous timers there is inaccuracy in reporting.

My witness publishes every hour ~2 minutes after the hour. I looked at your witness log at 10 minutes after the hour, after I published, and it reported the feed was not updated when it was.

Hi @Thom the 'X time ago' feeds are based on the time registered on the blockchain as the 'feed publication' time. They are stored in the database, and then published in the JSON I use for the website. That JSON is parsed with moment.js to display the X-time ago based on your computer's time setting.

Here is the part in the code grabbing that timestamp from the blockchain and prepping it as a datetime for mysql:
Code: [Select]
for feedarrays in asset['bitasset_data']['feeds']:
            feedtime = datetime.datetime.strptime(feedarrays[1][0], "%Y-%m-%dT%H:%M:%S")

I use this combo feedtime + id_asset + id_account to compare feeds in my database so I only log new feed publications and not every time the same feeds when my tool does another round of data syncing.

What is happening instead and why your feed is not showing up: My tool gets at exactly (NTP) minute 2, 22, 42 cronjobs the current status from the blockchain. Most probably you are then still executing your feed publications :)

But yeah there is indeed a discrepancy... the 'last updated time' in the top of the page says whenever the last time the datafile was created. That is done on 5, 25, 45. But that is indeed based on the poll from 2,22,42. Let me see if I can align that more... I wanted to keep the polling & creating seperated, but I might just copy everything over into 1 script so I executes in 1 flow.

TLDR; the website shows the times based on blockchain.

84
Stakeholder Proposals / Re: Tool / website: Bitshares Witness Log
« on: May 03, 2017, 09:24:18 am »
    @roelandp, this is really excellent work!  A tool like this has been sorely needed.  There was a nice discussion a while back about what features would be useful for such a tool. I can't find it right now, but beyond what you've done so far, some of the ideas I can remember included monitoring latency and uptime (which you may have in the works already), measuring server processor speed and memory, and there was discussion about combining all of the various factors into a score.  And finally, there was talk about creating some charts in order to help visual the data.  I hope this gives you some ideas to continue taking this excellent tool forward!

    thx tbone.
    • I will add a monitoring for latency & uptime of the public seednodes provided i can find them by the witnesses posts.
    • Monitoring individual block producing servers for the network can be done as well however for security reasons witnesses should not declare their public nodes' IP / hostnames, so I cannot and/or won't tie those IP addresses to witness names.
    • Measuring server processor speed & memory would need a program (which would monitor and submit) run on the client side of the witness nodes, which is a bit to much to ask and also puts trust onto the witness to install this program onto their actual producing nodes and not some high powered machine to modify stats, so I will leave this out for now. I think the server speed / memory etc is not that important either, as long as witnesses are reliable block producers. If the chain grows and tx's requires better servers they will notice as they might start missing blocks and operate accoringly to upgrade system (or not).
    • Charts: @lafona made some charts already... Let me know what kind of charts you want to see based on the data of http://roelandp.nl/bitshareswitnesslog ! I currently log the follow data 3 times per hour:
      • Pricefeeds per witness
      • Missed blocks
    [/list]

    85
    Stakeholder Proposals / Re: Tool / website: Bitshares Witness Log
    « on: May 02, 2017, 09:45:30 pm »
    Update:
    Added the 'This week missed' count next to the 'total blocks missed' per witness. The difference between the current missed blocks or the missed_blocks 7 days ago (or oldest registered if no 7 days exist) is being displayed. This adds to the realtime data and says more than the total_missed blocks IMHO.

    86
    Stakeholder Proposals / Re: Bitshares Witness Log
    « on: May 02, 2017, 03:40:24 pm »
    Updates
    • Under the detail-pane (hit (+) button in front of any witness) you now find all (other) feeds published by the given witness (provided I track them, if you miss any, hit me up))
    • In the main overview with the 8-10 main markets (based on 24h volume) the 'median' price feed is highlighted with 'green + bold' in each column.
    • Some minor additional cosmetic changes to make err'thing look a little more slick-rick.
    Let me know any feedback! Thanks!

    87
    Stakeholder Proposals / Re: [Witness Proposal] roelandp
    « on: May 02, 2017, 07:57:47 am »
    +5%
    Glad to see the witness team injected fresh blood.

    In-depth study of BitShares 2.0, I believe you will fall in love with her, to play your talent, to expand the ecosystem of BitShares.

    I'm a proxy, you have my vote!
    Voting status will be updated in 16:00 ( UTC+8 ).

    EPIC thx Yao! fyi, i was already supporting Bitshares with a public seednode for about half a year, but now wanted to be more serious. Thanks for your vote!

    88
    Stakeholder Proposals / Re: Bitshares Witness Log
    « on: May 02, 2017, 07:55:52 am »
    thanks for the replies @alt @Chris4210 @sakhan @fav !

    Updated with some cosmetic changes:



    Note if you hit the (+) button in front of each witness you can see a detailed view

    http://roelandp.nl/bitshareswitnesslog

    89
    Stakeholder Proposals / Re: [Witness Proposal] roelandp
    « on: May 02, 2017, 07:52:32 am »
    I released a new Bitshares tool today: Public Bitshares Witness Log: Check it out here: https://bitsharestalk.org/index.php/topic,24069.msg305401.html#msg305401
    It displays current pricefeeds for witnesses and missed blocks and updates 3 times per hour. Direct link: http://roelandp.nl/bitshareswitnesslog


    90
    Stakeholder Proposals / Tool / website: Bitshares Witness Log
    « on: May 01, 2017, 10:13:47 pm »
    Last week I released a witness monitor tool, meant for personal usage by witnesses, to monitor the availability of seednode, pricefeed latency in updates and missing blocks. From various members I received requests to expand on that monitoring tool and start to log the data and turn it into a Bitshares Witness Log. I found it a challenge in my python skills, so I sharpened my knife:



    What does it do?

    * Update Frequency
    Three times per hour (2nd,22nd,42nd minute) the backend checks this witness data by using Python-Bitshares (thank you @xeroc!) and storing it in an MySQL database.

    Three times per hour (5th, 25th, 45th minute) the backend publishes a new version as a static JSONP file with the latest data details as compiled from this database as a source-feed for this overview, using Bootstrap-Table to display it. This means the data you see can be maximum 23 minutes old before fresh data is displayed. If this is not the case and the top "last updated at" mentions way more time since that, please notify me as my server must have crashed!

    Insights & Analyse
    • This witness log gives historical insights in basic witness tasks: producing blocks & regular price feed publications.
    • The font and opacity of price feed entries get smaller the older the feed gets. Once the feed is older than 24 hours it gets a red font-color.
    • The markets are chosen and sorted by market 24 hour market volume.
    • Expand individual rows by hitting the '+'-sign for a detailed witness pricefeed log including variance of the settlement price.
    • The overview table shows 'red' warning fonts whenever:
      • Feed is not available.
      • Feed is older than 24 hours.
      • Feed price variance is more than +/- 5% of current settlement price.
    And more...
    • Export data via the top-right export menu.
    • Customise the table via the top-right export menu and exclude certain columns.
    • Search the table.
    • Sort the table by many of the columns.
    • Hover over the price feed entries to see 'Last Published' time
    Todos:
    • adding public seednodes of every witness (1 per witness) so I can also monitor the uptime.
    • After 1 week of logging I will also add a 'difference indicator', showing how many new blocks are missed (for 1 week displayed) per witness.
    Click here to go to the Bitshares Witness Log
    http://roelandp.nl/bitshareswitnesslog


    ps: I know I am a very new witness, and I don't want to be telling anyone what to do, I feel humble in my position: nonetheless I noticed that some of the witnesses are missing their 'urls'-part in the witness entry. Would be great if you could 'update_witness' with an URL![/list]

    Pages: 1 2 3 4 5 [6] 7 8