Randomness of Witness Scheduling

[bytemaster]

Imagine for a second that our goal is "1 second blocks" so we have "instant" confirmation.  Imagine if we simply allowed every witness to produce 10 blocks in a row and then rotate?   This would give us instant confirmations while not reducing the security from what BitShares is today and has the benefit of having 0 latency between blocks produced in "groups" of 10.

I don't like this idea from a security POV. Right now, if I wait for 10 blocks, I have 10% of the network confirming my transaction, in this new scheme, I have 1%.

I agree.  I think that 1 second blocks with the same witness doing 10 in a row is actually just the same to me as 10 second block time.

There are two factors here:

1. every block produced is irreversible so you *KNOW* your transaction has been included and is unlikely to be reversed except potentially the last block produced by the witness if the next witness didn't see it in time. 
2. you still have to wait for 10*NUM_WITNESSES blocks to know for absolute certainty.
3. for markets the witness is committing to the order of transactions sooner rather than delaying.

So there is a significant difference in the normal situation of honest witnesses.

[Ander]

Imagine for a second that our goal is "1 second blocks" so we have "instant" confirmation.  Imagine if we simply allowed every witness to produce 10 blocks in a row and then rotate?   This would give us instant confirmations while not reducing the security from what BitShares is today and has the benefit of having 0 latency between blocks produced in "groups" of 10.

I don't like this idea from a security POV. Right now, if I wait for 10 blocks, I have 10% of the network confirming my transaction, in this new scheme, I have 1%.

I agree.  I think that 1 second blocks with the same witness doing 10 in a row is actually just the same to me as 10 second block time.

[monsterer]

Imagine for a second that our goal is "1 second blocks" so we have "instant" confirmation.  Imagine if we simply allowed every witness to produce 10 blocks in a row and then rotate?   This would give us instant confirmations while not reducing the security from what BitShares is today and has the benefit of having 0 latency between blocks produced in "groups" of 10.

I don't like this idea from a security POV. Right now, if I wait for 10 blocks, I have 10% of the network confirming my transaction, in this new scheme, I have 1%.

[clayop]

Security comes first. If revised witness node schedule does not harm any security, it sounds great.

[Ander]

Thre are a ton of useful application for having provably fair random number generation built into the blockchain.  I wouldnt want to see that go unless the savings were HUGE. 

Alos, having a witness make 10 blocks in a row is scary.  Essentially this means that you have to wait 10 blocks instead of 1 before you get a confirmation, because that 1 witness could be screwing you. 

[Ben Mason]

I would like to revise my statements:

Ben and I have had further discussions and have managed to achieve the following:

1. removing random number generation from witnesses which will reduce block header size by 46% and saving an "empty" blockchain 1.2GB per year.
2. keep rounds and prevent the same witness from going more than once
3. dramatically improve performance.

A separate issue of whether or not to have a witness produce N blocks in a row before rotating to the next witness is something to be considered separately.   In theory, allowing each witness to produce 2 blocks in a row would allow sub-second confirmation times without impacting security so long as we enforce the simple rule that a witness cannot miss their own block.   The benefits of sub-second confirmation time are that there would be less transactions "in-flight" at any given point in time.

If security is not impacted then the performance benefits seem compelling enough to risk needing to provide an explanation (on how the risk is mitigated) or two.

[merivercap]

As to the question about the benefits for the "average user" it comes down to our bottleneck being single-core CPU time and if we can eliminate a 0.0002 seconds from the mandatory per-block CPU time then we gain the potential for an extra 20 transactions per second.  In other words, this bottleneck was consuming enough CPU time to allow us to improve potential throughput by 2x bitcoin's transaction rate

 

Yeah the updated adjustments sounds good.

[bytemaster]

I think he means skipping the previous witnesses blocks, and making it look like they are not doing their job, thus getting them fired for missing blocks. 

I like the random numbers built into the blocks and would love to find a way to keep them, which I think requires no witness gets to sign two blocks in a row.  I just think having a provably fair RNG built into the blockchain is a good thing.  I would consider it worth a couple of hours of compute time on a witness node per year.

In the spirit of separation of powers, witnesses should not participate in random number generation.  There are many better schemes for random number generation than having the witnesses do it.  If the random numbers produced by witnesses were ever used for something that depended on "TRUE RANDOMNESS" , such as gambling, then witnesses would no longer be neutral.  The more neutral we can make witnesses the better off things will be from a legal point of view.   Plus the randomness comes at a high cost in terms of download / sync time of 1.2 GB per year.

true 
I am curious what the better schemes are.
I think you're right and that we can add rng back in as a worker proposal later.  I would guess it will consume at least the same overhead when it is introduced, but perhaps we can charge fees for it.

You will love my ideas on how to do randomness for gambling    But for now I care about getting BTS 2.0 right and simple.

[puppies]

I think he means skipping the previous witnesses blocks, and making it look like they are not doing their job, thus getting them fired for missing blocks. 

I like the random numbers built into the blocks and would love to find a way to keep them, which I think requires no witness gets to sign two blocks in a row.  I just think having a provably fair RNG built into the blockchain is a good thing.  I would consider it worth a couple of hours of compute time on a witness node per year.

In the spirit of separation of powers, witnesses should not participate in random number generation.  There are many better schemes for random number generation than having the witnesses do it.  If the random numbers produced by witnesses were ever used for something that depended on "TRUE RANDOMNESS" , such as gambling, then witnesses would no longer be neutral.  The more neutral we can make witnesses the better off things will be from a legal point of view.   Plus the randomness comes at a high cost in terms of download / sync time of 1.2 GB per year.

true 
I am curious what the better schemes are.
I think you're right and that we can add rng back in as a worker proposal later.  I would guess it will consume at least the same overhead when it is introduced, but perhaps we can charge fees for it.

[bytemaster]

I think he means skipping the previous witnesses blocks, and making it look like they are not doing their job, thus getting them fired for missing blocks. 

I like the random numbers built into the blocks and would love to find a way to keep them, which I think requires no witness gets to sign two blocks in a row.  I just think having a provably fair RNG built into the blockchain is a good thing.  I would consider it worth a couple of hours of compute time on a witness node per year.

In the spirit of separation of powers, witnesses should not participate in random number generation.  There are many better schemes for random number generation than having the witnesses do it.  If the random numbers produced by witnesses were ever used for something that depended on "TRUE RANDOMNESS" , such as gambling, then witnesses would no longer be neutral.  The more neutral we can make witnesses the better off things will be from a legal point of view.   Plus the randomness comes at a high cost in terms of download / sync time of 1.2 GB per year.

[bytemaster]

As to the question about the benefits for the "average user" it comes down to our bottleneck being single-core CPU time and if we can eliminate a 0.0002 seconds from the mandatory per-block CPU time then we gain the potential for an extra 20 transactions per second.  In other words, this bottleneck was consuming enough CPU time to allow us to improve potential throughput by 2x bitcoin's transaction rate

[puppies]

I think he means skipping the previous witnesses blocks, and making it look like they are not doing their job, thus getting them fired for missing blocks. 

I like the random numbers built into the blocks and would love to find a way to keep them, which I think requires no witness gets to sign two blocks in a row.  I just think having a provably fair RNG built into the blockchain is a good thing.  I would consider it worth a couple of hours of compute time on a witness node per year.

[bytemaster]

I would like to revise my statements:

Ben and I have had further discussions and have managed to achieve the following:

1. removing random number generation from witnesses which will reduce block header size by 46% and saving an "empty" blockchain 1.2GB per year.
2. keep rounds and prevent the same witness from going more than once
3. dramatically improve performance.

A separate issue of whether or not to have a witness produce N blocks in a row before rotating to the next witness is something to be considered separately.   In theory, allowing each witness to produce 2 blocks in a row would allow sub-second confirmation times without impacting security so long as we enforce the simple rule that a witness cannot miss their own block.   The benefits of sub-second confirmation time are that there would be less transactions "in-flight" at any given point in time.

[merivercap]

I think it's better to be conservative for the blockchain.

Not sure I like the idea of having multiple blocks produced by the same witness in a row.  What is meant by 'picking' on the witness that goes before them.  Do you mean 'picking' a witness to collude?  Why not  just a round-robin and occasional random reshuffle of the order after X number of blocks with the same restriction that 'no witness may produce a block less than Y blocks from their last block'.

How much improvement do you get from the old vs new design for block times?
What is the primary benefit of vastly increased reindexing performance for the average user?

thx.

[Thom]

I would like to see the results of a Monte Carlo style simulation or similar analysis of this algo to see how block production is distributed over the pool of witnesses. It would be great to see the results as a series of histogram plots with witnesses on the X axis and number of occurrences on Y. The variables being number of active witnesses and number of iterations for the simulation.

Statisticians can probably make quick work of such an analysis but I prefer to visualize it like this.

I want to be convinced that the distribution is sufficiently random over a specific period of time (# of blocks) so no witness has an unfair advantage.

In general your approach sounds very reasonable. You might see some concerns raised from some that see the removal of the random number generation from each witness as a valuable feature they don't want to disappear, so consider being very explicit about the impact of removal and quantitative benefit to performance.