--> All Delegates: Please upgrade to 0.4.27.1 ASAP <--

[bytemaster]

I have a working fix in place.  There is no security issue.

The issue involves false positives on duplicate transactions when switching from a "short fork" to the "long fork" because our cache of "known transaction ids" wasn't properly cleared when unwinding the short fork.  When attempting to apply the longer fork it failed due to "duplicate transaction".   This means that what use to be a "small issue" of a single missed block could turn into a "large issue" of delegates going down two different paths. 

We also have the issue that some delegates have automatic backup nodes that prevent them from missing blocks.  These backup nodes would activate after they went down a different fork than the master and thus both forks add a cumulative participation greater than 100%.   As a result we produced a few very long (longer than 404 blocks) minority forks.   These long minority forks exceeded the undo history which meant that once your client went down that minority fork it was a dead end with no turning back except to attempt a fresh download of the entire chain.

We have put in a fix for the false positive issue and increased the undo history to 1616 blocks.  Delegates should upgrade to help reduce forking.  This is not a required upgrade for end users, but it will help them get on the right fork.   

[theoreticalbts]

Sounds juicy!

In this case, it's more of an abundance of caution.  I personally don't think this bug is exploitable (or at any rate that exploitation will be able to do anything worse than what's already happening naturally).  But we're still waiting until the fix is released to discuss details.

[monsterer]

We think we understand what has been causing the network to fork repeatedly over the past 12 hours or so.  We're currently working on implementing a fix.  We prefer not to publicly explain what's going on until the fix is released.

Sounds juicy!

[theoreticalbts]

We think we understand what has been causing the network to fork repeatedly over the past 12 hours or so.  We're currently working on implementing a fix.  We prefer not to publicly explain what's going on until the fix is released.

[Riverhead]

Even though these delegates were informed about the dangers of multiple chain block signing
and about the flaws of current public automated delegate backup system implementations they still used it.

I agree. However multiple chain block signing is a real risk now that version updates are longer than round times. In my case today the double signing was due to efforts to have the delegate survive a full machine crash and/or reboot by the vendor. Sadly it worked =/. I have it disabled on the backup VPS and only enabled on the live VPS.

To further prevent this from happening again I have moved the wallets directory out of the data-dir so even if it comes online it won't try to sign blocks.

[emski]

Even though these delegates were informed about the dangers of multiple chain block signing
and about the flaws of current public automated delegate backup system implementations they still used it.

Perhaps this should be made a bit more clear.

Anyway it looks like everyone is working to resolve the issues and there is no harm done.

[cube]

Aren't they supposed to be fired for this? Lol

The developers that introduced the forking code to make the network more robust or the delegates who are quickly adapting to keep the network running while the kinks are worked out?

Who would you have replace them that you think would do a better job?

Everyone is doing their best to get it working.  There is no need to fire anyone. :)

[Riverhead]

Aren't they supposed to be fired for this? Lol

The developers that introduced the forking code to make the network more robust or the delegates who are quickly adapting to keep the network running while the kinks are worked out?

Who would you have replace them that you think would do a better job?

[NewMine]

1283481 2014-12-17T11:33:40 backbone.riverhead              0       166     544     0.002572         d310b14962c87e4a97600e6f6b6d0d72e6039cad
you are signing the wrong chain
[13:49:25] Emil Velichkov: crazibyt, chinesecommunity, bm.payroll.riverhead, rose.ebit, titan.crazybit
[13:49:59] Emil Velichkov: including x.ebit
[13:50:00] Emil Velichkov: fox
[13:50:15] Emil Velichkov: *.helloworld
[13:50:23] Emil Velichkov: del0.cass
[13:50:29] Emil Velichkov: marketing.methodx
[13:50:59] Emil Velichkov: riverhead-del-server-1
[13:51:30] Emil Velichkov: anchor.crazybit
[13:52:00] Emil Velichkov: dc-delegate
[13:52:20] Emil Velichkov: there are still seed nodes on the forked chain and some delegates actively producing blocks there
[13:53:04] Emil Velichkov: if someone can reach out the the above-mentioned delegates that would be cool

Aren't they supposed to be fired for this? Lol

[Riverhead]

I had a backup server come online after a restart. The situation has been corrected and updated to not repeat.

[ebit]

Thanks,I will try

[liondani]

if you delete "chain" folder this message will not show again  if I am right ;)


PS make a restart and try again before you take my advice (maybe you save the resync time)

[ebit]

when run :  ./bitshares_client

------------ error --------------
10 assert_exception: Assert Exception
_map->is_open(): Database is not open!
    {}
    th_a  level_map.hpp:305 commit
error applying batch
    {}
    th_a  level_map.hpp:314 commit

    {}
    th_a  cached_level_map.hpp:52 flush

    {}
    th_a  cached_level_map.hpp:28 close

    {}
    th_a  chain_database.cpp:1485 close

    {"data_dir":"/root/.BitShares/chain"}
    th_a  chain_database.cpp:1434 open

    {"data_dir":"/root/.BitShares"}
    th_a  client.cpp:1365 open

[CalabiYau]

[13:44:18] Emil Velichkov: so checkpoints and --resync-blockchain does NOT work

Confirmed !

[emski]

1283481 2014-12-17T11:33:40 backbone.riverhead              0       166     544     0.002572         d310b14962c87e4a97600e6f6b6d0d72e6039cad
you are signing the wrong chain
[13:49:25] Emil Velichkov: crazibyt, chinesecommunity, bm.payroll.riverhead, rose.ebit, titan.crazybit
[13:49:59] Emil Velichkov: including x.ebit
[13:50:00] Emil Velichkov: fox
[13:50:15] Emil Velichkov: *.helloworld
[13:50:23] Emil Velichkov: del0.cass
[13:50:29] Emil Velichkov: marketing.methodx
[13:50:59] Emil Velichkov: riverhead-del-server-1
[13:51:30] Emil Velichkov: anchor.crazybit
[13:52:00] Emil Velichkov: dc-delegate
[13:52:20] Emil Velichkov: there are still seed nodes on the forked chain and some delegates actively producing blocks there
[13:53:04] Emil Velichkov: if someone can reach out the the above-mentioned delegates that would be cool