BitShares Forum

Main => General Discussion => Topic started by: bytemaster on October 14, 2015, 08:48:20 pm

Title: Recent Network Split Analysis
Post by: bytemaster on October 14, 2015, 08:48:20 pm
Around 2015-10-14T19:21:39  dele-puppy attempted to claim a vesting balance (his witness pay) and as a result the network split 60/40 because 40% of the witnesses thought the transaction was invalid.

As a result the last irreversible block hung around block 33845 (the block prior to the one that forked the network).   The blockchain grew to 269 blocks before 2/3's of the witnesses managed to agree and extend the last irreversible block back to the typical 20 to 30 behind the head block number.

In this situation everyone on the network was accurately reporting the state and the user interfaces indicated to the users that their operations were pending confirmation.   Any / all exchanges following the last irreversible block would have been OK.

We have determined the cause is due to some non-deterministic behavior in witness pay that resolves itself with a replay.   After replaying the blockchain all witnesses agreed the transaction was valid. 

We will continue to monitor the blockchain for issues as we look for the underlying cause.   In the meantime this particular failure can be mitigated by witnesses periodically replaying their blockchain. 

Despite the "bug" everything is working just fine and your funds are safe.



Title: Re: Recent Network Split Analysis
Post by: lakerta06 on October 14, 2015, 08:53:18 pm
Around 2015-10-14T19:21:39  dele-puppy attempted to claim a vesting balance (his witness pay) and as a result the network split 60/40 because 40% of the witnesses thought the transaction was invalid.

As a result the last irreversible block hung around block 33845 (the block prior to the one that forked the network).   The blockchain grew to 269 blocks before 2/3's of the witnesses managed to agree and extend the last irreversible block back to the typical 20 to 30 behind the head block number.

In this situation everyone on the network was accurately reporting the state and the user interfaces indicated to the users that their operations were pending confirmation.   Any / all exchanges following the last irreversible block would have been OK.

We have determined the cause is due to some non-deterministic behavior in witness pay that resolves itself with a replay.   After replaying the blockchain all witnesses agreed the transaction was valid. 

We will continue to monitor the blockchain for issues as we look for the underlying cause.   In the meantime this particular failure can be mitigated by witnesses periodically replaying their blockchain. 

Despite the "bug" everything is working just fine and your funds are safe.

Would love to hear details.
Title: Re: Recent Network Split Analysis
Post by: betax on October 14, 2015, 08:53:51 pm
So every 12 hours we can put another witness up, with a fresh database.
Title: Re: Recent Network Split Analysis
Post by: wackou on October 14, 2015, 09:29:42 pm
Around 2015-10-14T19:21:39  dele-puppy attempted to claim a vesting balance (his witness pay) and as a result the network split 60/40 because 40% of the witnesses thought the transaction was invalid.

As a result the last irreversible block hung around block 33845 (the block prior to the one that forked the network).   The blockchain grew to 269 blocks before 2/3's of the witnesses managed to agree and extend the last irreversible block back to the typical 20 to 30 behind the head block number.

In this situation everyone on the network was accurately reporting the state and the user interfaces indicated to the users that their operations were pending confirmation.   Any / all exchanges following the last irreversible block would have been OK.

We have determined the cause is due to some non-deterministic behavior in witness pay that resolves itself with a replay.   After replaying the blockchain all witnesses agreed the transaction was valid. 

We will continue to monitor the blockchain for issues as we look for the underlying cause.   In the meantime this particular failure can be mitigated by witnesses periodically replaying their blockchain. 

Despite the "bug" everything is working just fine and your funds are safe.

Would love to hear details.

me too, got me really curious there... :o Although I would understand if we had to wait until it is properly resolved, as this might be a security issue I imagine...
Title: Re: Recent Network Split Analysis
Post by: bytemaster on October 14, 2015, 09:36:09 pm
I don't expect this to be a security issue because the only people that can trigger this particular bug are "trusted" witnesses.

I define security issue as something that would allow someone to get "something for nothing".   

We have a plan in place to catch this bug if it happens again.   
Title: Re: Recent Network Split Analysis
Post by: puppies on October 15, 2015, 01:15:09 am
lol.  The only reason I withdrew anything was to check if it was working for mindphlux.  In the short term I just won't withdraw any witness pay.  Does this bug affect other vesting balances?
Title: Re: Recent Network Split Analysis
Post by: xeroc on October 15, 2015, 02:24:33 am
Good thing we have delayed nodes ..

+5%
Title: Re: Recent Network Split Analysis
Post by: mindphlux on October 15, 2015, 05:33:33 am
It just happened again. I withdraw vested balance and 20% of the witnesses forked.

I will restrain from vesting until BM tells us to do so or has fixed the bug.
Title: Re: Recent Network Split Analysis
Post by: karnal on October 15, 2015, 09:20:14 am
Any cause for alarm here? (BTS vanishing, etc)
Title: Re: Recent Network Split Analysis
Post by: cube on October 15, 2015, 09:22:25 am
Any cause for alarm here? (BTS vanishing, etc)

No cause for alarm.  It only affects the witnesses in turn of them having to 'restart' their witness nodes if a split occurred.  No funds are affected in any way.
Title: Re: Recent Network Split Analysis
Post by: monsterer on October 15, 2015, 09:23:57 am
No cause for alarm.  It only affects the witnesses in turn of them having to 'restart' their witness nodes if a split occurred.  No funds are affected in any way.

Doesn't that depend on whether exchanges are following the 2/3 confirmation time rule? If an exchange deposits a transaction on a fork which gets rolled back, that would be a serious problem.
Title: Re: Recent Network Split Analysis
Post by: xeroc on October 15, 2015, 09:27:02 am
No cause for alarm.  It only affects the witnesses in turn of them having to 'restart' their witness nodes if a split occurred.  No funds are affected in any way.

Doesn't that depend on whether exchanges are following the 2/3 confirmation time rule? If an exchange deposits a transaction on a fork which gets rolled back, that would be a serious problem.
Thata why they are supposed to run the delayed node .. the all they see is FINAL and IRREVERSIBLE
Title: Re: Recent Network Split Analysis
Post by: cube on October 15, 2015, 09:29:10 am
No cause for alarm.  It only affects the witnesses in turn of them having to 'restart' their witness nodes if a split occurred.  No funds are affected in any way.

Doesn't that depend on whether exchanges are following the 2/3 confirmation time rule? If an exchange deposits a transaction on a fork which gets rolled back, that would be a serious problem.

Isn't it the exchanges' responsibility to be allowing tx only when they are on the main chain? Or run delayed nodes?  If an exchange did not bother to check whether it was on a folk, it could only expect to risk losing funds.
Title: Re: Recent Network Split Analysis
Post by: monsterer on October 15, 2015, 10:03:17 am
Isn't it the exchanges' responsibility to be allowing tx only when they are on the main chain? Or run delayed nodes?  If an exchange did not bother to check whether it was on a folk, it could only expect to risk losing funds.

It is yes. I'm just pointing out the facts.