We implemented a change in how we check for duplicate transactions.  Tested it.  It worked.
We applied the change to the main network and discovered that there was one transaction that was considered a duplicate under the new system that was processed by the old system.
Vikram disabled duplicate checking until the scheduled fork block so that we could validate the old chain.
Vikram tested syncing up with the old chain and it worked. 
Vikram tested producing a block with the upgraded version and it worked.
Announce upgrade.

Delegates started upgrading.   Duplicate transaction checking was "disabled" until the specified fork block.  Delegates started producing blocks that included duplicate transactions and thus were rejected by the non upgraded clients.   

So the testing we needed was "dev share" test net because the bug was is the "handoff / upgrade" code that didn't reveal itself until after we had a chain with multiple delegates running the new code. 

In retrospect the issue could have been easily avoided with a little thinking, but we were rushing to get a security fix in.

We are reviewing our options, but the conclusion is clear.   DevShares needs to come out as soon as possible because it is the most likely way we would have caught this bug.

Quote from: abit on December 12, 2014, 06:21:36 pm

My (standby) delegate a.delegate.abit running on v0.4.25-RC1 now..

blockchain_average_delegate_participation 16.64%

is it okay?

Ya. People are slowly switching over.