Quote from: Thom on April 25, 2017, 03:23:56 am

Quote from: roelandp on April 24, 2017, 08:38:52 pm

1. Monitor missing blocks
Whenever a new block is missed you will get a notification. This part of the script can (and will) be extended towards automated switching to the backup witness signing key once a threshold is passed.

Several witnesses attempted to code an automatic failover algorithm but I don't believe any were successful without introducing new problems.

One important thing to consider is you absolutely do NOT want 2 nodes producing blocks for the same witness, as that is sure to cause havoc and fork the network.

Whenever I switch production using the "update_witness" API call I manually make sure both the old witness node and the new witness node are both listening and in sync before I execute the call. I usually submit the call on the old witness going out of production, not the new node going into production. I can then use the get_witness API call to verify the signing key for the new node in in effect before I shut down the old witness node.

The difficulty is in coming up with a reliable way to know for certain the node you want to take out of production will not be able to generate blocks after you switch production to another node. If the "aberrant" node has not crashed, is still running but cut off from the net (or the watchdog listener is cut off from that node) but the watchdog node falsely concludes it is dead,  it may broadcast a new signing key, causing a new node to take over, but then the network to the aberrant server is restored and resumes network communications still thinking it is the block producer and so generates a block along with the failover node.  As far as the aberrant node is concerned it never saw the new signing key, never thought it was offline and continues to generate a block whenever its time to do so comes around.

When the block producer fails it may not be possible to determine for certain why or get confirmation it will not resume block production. You will need to determine if the OS for the failing node is responding but not the app, in which case failover may be possible if you build in some type of communication to restart the witness_node app or restart the entire OS. The issue is what if you can't communicate with the failing node? Is it dead or just temporarily cut off? Will it fork the network if it should come back online?

I was hoping wackou & I could have implemented the backbone architecture and a failover protocol along with it, but there wasn't enough funding and wackou's time was very scarce (and still is actually). If this ecosystem is going to survive a frontal attack the witness nodes need to be protected from direct public access. Seed nodes and API servers should be the route available for public access, leaving witnesses alone to process and generate blocks quickly with minimum latency.

Looking at the stats i think @abit has a script that detects failing witness and issues a transaction to the network to update his signing key; this would allow him to run two witnesses both with different signing keys and auto switch based on any issues. - this is only speculation; i have no idea really. But i all his time of being a witness he has only missed 133 blocks, and you can see updates of signing key on his account.

That's a very good point @GChicken, I have often wondered how he has been able to achieve such low missed block numbers.

@roelandp you're correct in your understanding of how update_witness functions. However in the scenario I tried to describe, wherein an active witness has a network infrastructure failure (not an app failure or host failure such as out of diskspace or memory) and due to that doesn't see the transaction transmitted by the monitor to switch signing keys, if the network is restored and the witness is reconnected to the network, it will continue to sign blocks for that witness but with an incorrect signing key, thus creating the real possibility of forking the network.

I know that @puppies spent  some time working on an automatic failover algo and people found holes in it and I don't think his approach caught on due to the shortcomings raised. I am all for improving the robustness of our network, and hope a solid algo can be developed to automatically switch in redundant nodes and disable failed nodes. The testnet is a perfect context to work out such an algorithm and observe the affects. The exact case of a witness missing an update_witness transaction can be tested without risking a fork in production. 

This "whales" group could be seen as centralized control, or at least spun as that by competitors to BitShares.

However it's the nature of PoS. If these "whales" gain consensus among themselves and use that influence to steer BitShares that is where centralization of power can occur. All it takes is money to gain control of this ship. I'm not saying it could happen overnight, but if your aim is to gain control and money is no concern it could be done.

It will be interesting to see what comes of this whales group and whether they will get consensus on the things that are hindering our growth, and if such a consensus will translate into actions that result in growth.

1. Monitor missing blocks
Whenever a new block is missed you will get a notification. This part of the script can (and will) be extended towards automated switching to the backup witness signing key once a threshold is passed.

Several witnesses attempted to code an automatic failover algorithm but I don't believe any were successful without introducing new problems.

One important thing to consider is you absolutely do NOT want 2 nodes producing blocks for the same witness, as that is sure to cause havoc and fork the network.

Whenever I switch production using the "update_witness" API call I manually make sure both the old witness node and the new witness node are both listening and in sync before I execute the call. I usually submit the call on the old witness going out of production, not the new node going into production. I can then use the get_witness API call to verify the signing key for the new node in in effect before I shut down the old witness node.

The difficulty is in coming up with a reliable way to know for certain the node you want to take out of production will not be able to generate blocks after you switch production to another node. If the "aberrant" node has not crashed, is still running but cut off from the net (or the watchdog listener is cut off from that node) but the watchdog node falsely concludes it is dead,  it may broadcast a new signing key, causing a new node to take over, but then the network to the aberrant server is restored and resumes network communications still thinking it is the block producer and so generates a block along with the failover node.  As far as the aberrant node is concerned it never saw the new signing key, never thought it was offline and continues to generate a block whenever its time to do so comes around.

When the block producer fails it may not be possible to determine for certain why or get confirmation it will not resume block production. You will need to determine if the OS for the failing node is responding but not the app, in which case failover may be possible if you build in some type of communication to restart the witness_node app or restart the entire OS. The issue is what if you can't communicate with the failing node? Is it dead or just temporarily cut off? Will it fork the network if it should come back online?

I was hoping wackou & I could have implemented the backbone architecture and a failover protocol along with it, but there wasn't enough funding and wackou's time was very scarce (and still is actually). If this ecosystem is going to survive a frontal attack the witness nodes need to be protected from direct public access. Seed nodes and API servers should be the route available for public access, leaving witnesses alone to process and generate blocks quickly with minimum latency.

I understand the motivation for the LTM fees structure but never liked the breakdown. Here is an idea.

10%/40%/50%  All Referrers(weighted)/LTM/network

There seems to be no shortage of ideas on how to distribute funds. Getting consensus on which to implement AND finding devs interested and competent enough to create a proposal and do the work are as scarce as grandma day traders in the crypto space. This is just talk, churning the waters, getting nowhere.

What will bring consensus and workers? When I see answers to that I'll begin to think progress is being made. Until then, not so much.

here's how I would do it:

1. update website
2. update marketing material (landing pages tailored for specific topics like bitUSD, trading etc - make it ?ref friendly & catch emails)
3. discuss a marketing campaign (what social networks? bloggers? influencers? ad networks?)
4. define budget
5. run campaign
.
.
.
- hire a "face"

get the foundation ready first and in order.

@fav, I consider this one of the most constructive suggestions I've heard that provide realistic short term goals with its focus on marketing.

I would disagree with you about when the "face" is hired however. IMO the marketing campaign needs to have that face. I would put hiring the face after step 3 & before step 4 to define the budget. Good faces don't usually work for free.

Other worthwhile activities related to these would be work on documentation (@xeroc already has a great start on that), forum migration to a modern forum like discourse, and a barrage of short 10 - 60 seconds in length videos that focus on one specific feature at a very high level, not detailed, but to say what the feature is and hype its value.

Why wait for the foundation to be setup when we can do these things now. It's taking way too long for all the pieces to come together for the foundation, and I see more effort to define it, it's role, and providing some type of coherent statement about it to shareholders will be necessary. Perhaps some of that is ongoing, but there are no timeframes for its completion that I'm aware of.

In the mean time marketing opportunities like the decline of Tether are slipping away.

This is type of short term plan I want to see happen, let say in the next 90 days. We are at step 3 right now, but steps 1 & 2 have yet to be scheduled.

I am calling for everybody to describe in the same fashion fav did what your plan for the next 90 days would be if you were king. You should NOT include in your plan anything big that can't realistically be accomplished in the next 90 days, so forget adding new features or restructuring fees to pay interest. Stay away from the shortcomings and issues that need to be dealt with that require full time backend / frontend devs to do. Focus on a 90 day plan, the low hanging fruit you see and clearly separate longer term things you feel are important to be included in another plan for later.

Whatever your suggestions are, their effectiveness must be measurable. It would be helpful if you mentioned the metrics you would use to see how well your plan is working. Without that how will you know if your plan should be funded?

Quote from: Thom on April 23, 2017, 04:42:20 pm

Curious, WHY are there only 19 witnesses, did the committee vote to change that from 21 to 19?

From 24 to 21 to 19. Who or what is driving these changes?

The number of witness slots is the weighted average of the number of witnesses voted for by a given voter (proxy). Proxies voting for fewer witnesses will reduce the witness slots, which is what recently happened, resulting in two slots being eliminated. The number of witness slots is always an odd number and may not be less than 11. The committee has no impact on witness slots.

Thanks for that fox, I had no idea. Somewhere along the way I got it in my head it was just another committee parameter.

So if I understood you correctly the count is directly related to how many witnesses each vote proxy votes for. What do you mean by "weighted" average? And how do votes from individual accounts factor in?

In essence reduction of the number of witnesses gives the remaining witnesses more opportunities to produce a block and hence pay increases.

Curious, WHY are there only 19 witnesses, did the committee vote to change that from 21 to 19?

From 24 to 21 to 19. Who or what is driving these changes?

We do have 2 new witnesses (rolandp and sahkan) who I believe will be great additions. I am curious tho on the drop to 19.