Yeah, just goes to show we're not getting the whole story.

Looks great!

1 little nit - The text under the table "Update Frequency" reflects 3 times an hour, but now you're updating 4 times an hour; do I have that right?

Well, my block producing node got stuck due to insufficient disk space (filled by p2p log) a few hours ago, while I'm sleeping. Unfortunately my phone was set to vibration mode, although it was notifying me all the time, I didn't wake up. I missed 177 new blocks (133 -> 310). Quite ironic. I won't be always lucky. I think it's time to setup an fail-over script.

Sorry to hear that. So you & roelandp are working on automatic failover. I hope one of you can perfect it. I have discussed that idea elsewhere, but it seems not many believe the risks are significant. All it takes to mess up the chain is for 2 nodes to broadcast signed transactions for the same witness. Fork city. To avoid that you need a way to make sure the old witness is definitely is dead with no chance of coming back online while the new witness takes over. To do that you need some smarts in the cooperating failsafe nodes to determine each node's state. Some type of heartbeat so that if the node producing blocks does NOT hear heartbeats from at least 2 other nodes it will cease block production. The producing node needs to verify it can communicate with the other witnesses, particularly the failsafe nodes.

In regards to bills, at first I was running nodes in China with less cost. We didn't have that many transactions in the early days, so network latency was not a big issue. After Steem blockchain was launched, I got some compensation there, then setup a few nodes in AWS (as my main BitShares block producing nodes) after latency became an issue, still, had been compensated by Steem witness pay for quite some months until recently. My AWS instances are mostly r3.large (15G RAM, 2 cores, 32G local SSD), the cost per month is around 150$ each (including additional cost for more disk spaces, data transmission and etc).

Thanks for this info. This confirms that until recently witness pay barely covered the cost of servers. Essentially it was altruism (fueled by the belief the platform was worth subsidizing) that kept the network operating while we all hoped that eventually we would reach much higher adoption.

Last night I attempted to put that node into use as the block producing witness the same way I always do, but it missed 2 blocks in under a minute.

I restarted the node with different witness / cli binaries and it's working fine since yesterday. So it may be a compiler issue or missing dynamically link library (if any of those are used in the build process). I will rerun the build and carefully review the logs for errors.

Didn't work for me either.

Takes a very long time to load, then presents the settings page to select a different API node.

Always found that page confusing. It can't connect (which is why that page is shown), yet it says "Full node connection status: CONNECTED".

When I use Sahkan's full node address (wss://dexnode.net/ws) it worked just fine.

Your server cost estimate is too low as is your hourly time to maintain and upgrade witness servers.

You might be able to run two 8GB shared VPS servers for $100 / month, IF you look REAL HARD. However, that would leave NOTHING to pay for a witness's time. Plus, keep in mind 8GB is not enough memory for reliable and stable service at the current rate of network use. I am moving to dedicated servers with at least 16GB of RAM for all future node installations. Seed nodes can get by right now with only 8GB as long as adequate swap is configured.

I have also rented a 32GB server for use on the testnet.

And as for the amount of time per month I devote to witness work: It has been far more than 5 hours a month since February. Here is my estimate of hours spent this month alone (April 2017):

Hrs         Task
-----         -------
3             Hosting Service Research (locations, cost, owned by, complaints / community opinion)
8             Community involvement (Friday mumble, reading forum, Telegram etc)
10           Server upgrades & installation
4             Software updates and feed tweaking

To provide reasonable compensation for server costs 3 servers should be factored in MINIMUM: 1 Witness node, 1 Seed node and 1 node for the testnet. The testnet doesn't have to be rented continuously, it can be spun up for testing and destroyed when testing is not underway.

It's there for all to see, very few witnesses run seed nodes in addition to their witness node. A seed node can serve as a hot backup if configured with adequate RAM.

Another factor not taken into account is how often feeds are published. Verbaltech2 holds the record for the most operations performed which is due to the number of assets I post feeds for as well as their publication frequency.

I will also say this again, that few people are considering the architecture, reliability and decentralization aspects of the network like Wackou has. Nobody seems interested in looking at potential attack vectors and how to protect witness nodes from being discovered and DDoS'ed. Wackou's backbone scheme is the ONLY work I've ever seen that addresses such issues. Where is any of that in this discussion? Security of the blockchain is obviously extrememly important, so why is the job of witnesses treated with such little respect?

B/c many witnesses do not live up to their responsibilities and many shareholders don't bother to vote them out.

The fact that the number of witnesses does NOT impact the cost of security to the network and shareholders is not intuitive. People think the expense is Pay per block * number of witnesses == cost of network's security. That is FALSE, it actually is:

(Pay per block * ( ( number of seconds in maintenance interval ) / 3 ) ) / number of active witnesses. The witness pay pool is set entirely by the pay per block (currently 3 BTS) and the length of the maintenance interval. Increasing the number of witnesses can only reduce each witnesses' pay and does not increase the direct cost of securing the network.

Quote from: Customminer on April 26, 2017, 09:35:15 pm

We should be due for testnet soon, no?

So far, no one has contacted me yet for updating the testnet to support the "new style" of doing STEALTH.
The former style of stealth is available on the testnet, but not exposed in the UI

When you say "new style" are you talking about kencode's work? I hear bits & pieces that stealth will be going to test on the testnet "soon", "in a few weeks" etc. but I've seen nothing definitive from kencode yet. On other posts I've gotten the impression it could be weeks and months away.

Stealth aside weren't there other objectives to determine where the limiting factor in achieving higher performance is? Have you identified areas in the code that deserve more scrutiny and possible instrumentation to provide data improve your understanding?

 

You have been a great witness and it is strange you have been unelected.

Shareholders need to vote you back in again ASAP.

I agree. I fail to understand the rational that has led to your being voted out.

Let this be a wake up call to all witnesses that you need to up your game and at least begin to engage the community and provide regular reports on your activities.

"Campaigning" never ends!