BitShares Forum
Main => General Discussion => Topic started by: cube on March 25, 2015, 09:48:00 am
-
About 25% delegates are affected by it. My delegate went into a folked chain. Can the devs look into it?
-
I am in the main fork (77.69% participation) try to remove the chain directory (~/.BitShares/chain/) and change the checkpoints.json in this way (added a recent block in main chain) and restart the client:
[[
2117977,
"1f9cdf643f1e21af79a18fe35751cd1593252014"
],[
2104000,
"147f68b04701f176cab47a512430c9a82c78e217"
],[
2071001,
"58847f93f0d1846cb9250d7ba342bebda6e698e4"
],[
991701,
"2e48f0a16d2107b9ab3d1c87e803ee70d3c91ce0"
],[
640001,
"11c748cdbfc445396deb6d24908521701aaa80aa"
],[
820201,
"904278b4bd585e3055c092aada1e030fbdf69a0e"
],[
613201,
"3317ea0a272976a10ef6593e840f9ff2032affad"
],[
1315315,
"baede744122d4b8d296ab3497bf5c849d1131466"
],[
871001,
"137ee1e9171a7ed1fe9cc58536d8e20749c3f199"
],[
578901,
"add6131c7bc6f1c9470de209b0e9c257d664f1c9"
],[
494001,
"59836f51f13e07be1de734a02360742c7b5f0dd6"
],[
451501,
"42e519db48f09570fcc02f38288648a92789dcb3"
],[
554801,
"4a992cc282a0024d393c6389cf84cbb8df1fc839"
],[
408751,
"12481e818abe9bc86069e45586df11d2cff7dbb8"
],[
357001,
"d240848ca9bf4be30204eded02fb35ce2e215d41"
],[
340001,
"d3858436100abe3ad7ee9f026e5f7f4781732e06"
],[
1772201,
"5832516564d86e86edbbb501cabec98c77504307"
],[
316002,
"206b7e6574019f4352515bd3d96162fb40a1b18a"
],[
274001,
"f46c109cfb1bac323122ae59b08edc23328d880c"
],[
1575501,
"52c5e5764fba4c876cde8c80e598c89ee5c35d9f"
],[
1,
"8abcfb93c52f999e3ef5288c4f837f4f15af5521"
]
]
-
confirming main chain
blockchain_get_block 2117977
{
"previous": "ade121bc11a8a05d75e1089f5185b6a45905ac91",
"block_num": 2117977,
"timestamp": "2015-03-25T10:24:20",
"transaction_digest": "c8cf12fe3180ed901a58a0697a522f1217de72d04529bd255627a4ad6164f0f0",
"next_secret_hash": "ce025dbda2034e7de6d03664b8e80eadb057dc42",
"previous_secret": "3ab66005d6814ac3505c81045eba7e68bbeeb639",
"delegate_signature": "20248cc81adc718234f6c18584dd2a4e85e5b2a31c8b9b0cd5a096933a07c3cb914abd9d4acedf77b51c49505bf1d0e4b753d114f576b9a8c070a842d5fc1b29ae",
"user_transaction_ids": [],
"id": "1f9cdf643f1e21af79a18fe35751cd1593252014",
"block_size": 166,
"latency": 0,
"signee_shares_issued": 150000,
"signee_fees_collected": 13860,
"signee_fees_destroyed": 448141,
"random_seed": "c8f24a8127e2f8d782830623f27c6e84a213566b",
"processing_time": 91567
}
-
Why is this a problem?
Isn't this the perfect time to hire some delegates? :D
-
confirming main chain
blockchain_get_block 2117977
{
"previous": "ade121bc11a8a05d75e1089f5185b6a45905ac91",
"block_num": 2117977,
"timestamp": "2015-03-25T10:24:20",
"transaction_digest": "c8cf12fe3180ed901a58a0697a522f1217de72d04529bd255627a4ad6164f0f0",
"next_secret_hash": "ce025dbda2034e7de6d03664b8e80eadb057dc42",
"previous_secret": "3ab66005d6814ac3505c81045eba7e68bbeeb639",
"delegate_signature": "20248cc81adc718234f6c18584dd2a4e85e5b2a31c8b9b0cd5a096933a07c3cb914abd9d4acedf77b51c49505bf1d0e4b753d114f576b9a8c070a842d5fc1b29ae",
"user_transaction_ids": [],
"id": "1f9cdf643f1e21af79a18fe35751cd1593252014",
"block_size": 166,
"latency": 0,
"signee_shares_issued": 150000,
"signee_fees_collected": 13860,
"signee_fees_destroyed": 448141,
"random_seed": "c8f24a8127e2f8d782830623f27c6e84a213566b",
"processing_time": 91567
}
Is there a simple and quick way to bring a delegate on a forked chain back to the main (besides removing the chain folder and resyn)?
-
maybe with the "checkpoints.json" file (see above) .. can't tell ..
BM stated somewhere that he had to reset the chain-folder
-
delete your data folder keep your wallet and redownload chain, its faster now... BM also had to do this, I did and now I think im on the right chain.
-
delete your data folder keep your wallet and redownload chain, its faster now... BM also had to do this, I did and now I think im on the right chain.
I removed the chain and redownloaded. After a long while, it resynced and was back to the main chain. But it just went into the wrong chain yet again. How to avoid getting into a fork? How to recover from a fork quickly?
-
delete your data folder keep your wallet and redownload chain, its faster now... BM also had to do this, I did and now I think im on the right chain.
I removed the chain and redownloaded. After a long while, it resynced and was back to the main chain. But it just went into the wrong chain yet again. How to avoid getting into a fork? How to recover from a fork quickly?
had you deleted the "peers.leveldb" folder when you removed the chain ?
if not ,try it
-
delete your data folder keep your wallet and redownload chain, its faster now... BM also had to do this, I did and now I think im on the right chain.
I removed the chain and redownloaded. After a long while, it resynced and was back to the main chain. But it just went into the wrong chain yet again. How to avoid getting into a fork? How to recover from a fork quickly?
had you deleted the "peers.leveldb" folder when you removed the chain ?
if not ,try it
Yes, I did. And it did not really help.
-
delete your data folder keep your wallet and redownload chain, its faster now... BM also had to do this, I did and now I think im on the right chain.
I removed the chain and redownloaded. After a long while, it resynced and was back to the main chain. But it just went into the wrong chain yet again. How to avoid getting into a fork? How to recover from a fork quickly?
had you deleted the "peers.leveldb" folder when you removed the chain ?
if not ,try it
Yes, I did. And it did not really help.
If you're on version 8.1 and you re-synched, we don't really expect it to happen again, so we need to see your logs. Can you zip up your p2p.log file and post it somewhere for us to look at?
-
delete your data folder keep your wallet and redownload chain, its faster now... BM also had to do this, I did and now I think im on the right chain.
I removed the chain and redownloaded. After a long while, it resynced and was back to the main chain. But it just went into the wrong chain yet again. How to avoid getting into a fork? How to recover from a fork quickly?
had you deleted the "peers.leveldb" folder when you removed the chain ?
if not ,try it
Yes, I did. And it did not really help.
If you're on version 8.1 and you re-synched, we don't really expect it to happen again, so we need to see your logs. Can you zip up your p2p.log file and post it somewhere for us to look at?
I removed the whole .BitShares folder, except wallets subfolder, for the current resync. If it happened again, I would send the log.
-
Just woke up to find that my delegate had lost all connections once again!
-
Just woke up to find that my delegate had lost all connections once again!
Same here .. almost ... 2 out of 3 machines had 0 connections ... luckly my backup delegate is still 'on-track' so I switched over to that one to produce blocks ..
When I restarted the other clients, they initiated a resync automatically?! never saw that happen before
-
Just woke up to find that my delegate had lost all connections once again!
Same here .. almost ... 2 out of 3 machines had 0 connections ... luckly my backup delegate is still 'on-track' so I switched over to that one to produce blocks ..
When I restarted the other clients, they initiated a resync automatically?! never saw that happen before
Yea that's happened twice to me as well, the automatic clearing of the blockchain and resyncing. It's very annoying cause it takes most of a day to get synced back up..
-
Not quite sure what is going on with my delegate, got this in the output:
--- there are now 44 active connections to the p2p network
--- syncing with p2p network, 13991 blocks left to fetch
--- in sync with p2p network
--- syncing with p2p network, 121260 blocks left to fetch
--- in sync with p2p network
which suggests it's having trouble trying to figure out which fork to sync to, yet info returns this:
"blockchain_average_delegate_participation": "83.47 %",
which seems to imply its on the main fork...
-
Not quite sure what is going on with my delegate, got this in the output:
--- there are now 44 active connections to the p2p network
--- syncing with p2p network, 13991 blocks left to fetch
--- in sync with p2p network
--- syncing with p2p network, 121260 blocks left to fetch
--- in sync with p2p network
which suggests it's having trouble trying to figure out which fork to sync to, yet info returns this:
"blockchain_average_delegate_participation": "83.47 %",
which seems to imply its on the main fork...
I'm seeing this too. My guess is that forked clients want to tell us about their alternative chain and our client drops the connection once it sees their chain is alternative to our own
-
My guess is that forked clients want to tell us about their alternative chain and our client drops the connection once it sees their chain is alternative to our own
That's correct. It doesn't mean that it is switching to that fork, it just means it has connected to someone with the fork and it will start downloading and indexing those blocks until it hits the first one it considers invalid, then it disconnects.
I've made a checkpoints.json here (https://gist.github.com/emfrias/c529156e40024364db36) with ~hourly checkpoints since the last hard fork. None of my clients have yet wandered off onto a minority fork so I'm not sure where things are going astray. Depending on when the minority fork(s?) appeared and how many blocks they contain, one checkpoint might not be enough to force your client to sync to the right fork. I *think* the json file hash enough checkpoints that you could reindex instead of redownload, but I'd just as soon download from scratch.
-
how do I know if I'm on the wrong chain?
-
how do I know if I'm on the wrong chain?
That's what I get:
"blockchain_average_delegate_participation": "90.99 %"
My guess is that forked clients want to tell us about their alternative chain and our client drops the connection once it sees their chain is alternative to our own
That's correct. It doesn't mean that it is switching to that fork, it just means it has connected to someone with the fork and it will start downloading and indexing those blocks until it hits the first one it considers invalid, then it disconnects.
I've made a checkpoints.json here (https://gist.github.com/emfrias/c529156e40024364db36) with ~hourly checkpoints since the last hard fork. None of my clients have yet wandered off onto a minority fork so I'm not sure where things are going astray. Depending on when the minority fork(s?) appeared and how many blocks they contain, one checkpoint might not be enough to force your client to sync to the right fork. I *think* the json file hash enough checkpoints that you could reindex instead of redownload, but I'd just as soon download from scratch.
I still don't understand why minority fork clients don't just discard whatever short chain they're on and join the main chain where we have >90% delegates participation. We can't rely on checkpoints for long :(
-
I still don't understand why minority fork clients don't just discard whatever short chain they're on and join the main chain where we have >90% delegates participation. We can't rely on checkpoints for long :(
We only save the undo state for about an hour's worth of blocks (maintaining it is expensive). For short forks this works out well, but if you get on a fork with >360 blocks on it, you won't have enough undo history available to roll your blockchain back to the fork point, thus you'll never be able to rejoin the main chain even if your client knows it is better and has downloaded all the blocks on the main chain. If you don't have the undo history to get there, the only other way to get back to the fork point is to reindex or re-download the blockchain from the beginning. And when re-downloading, as long as there are publicly-accessible nodes out there serving up the wrong blockchain, there's a chance you'll connect to them first and download the wrong blockchain and end up back in the same state, unless you force your client to reject the wrong chain with checkpoints. Fortunately, as more public clients switch over to the main chain, the chance anyone will end up syncing to a minority fork decreases.
-
I still don't understand why minority fork clients don't just discard whatever short chain they're on and join the main chain where we have >90% delegates participation. We can't rely on checkpoints for long :(
We only save the undo state for about an hour's worth of blocks (maintaining it is expensive). For short forks this works out well, but if you get on a fork with >360 blocks on it, you won't have enough undo history available to roll your blockchain back to the fork point, thus you'll never be able to rejoin the main chain even if your client knows it is better and has downloaded all the blocks on the main chain. If you don't have the undo history to get there, the only other way to get back to the fork point is to reindex or re-download the blockchain from the beginning. And when re-downloading, as long as there are publicly-accessible nodes out there serving up the wrong blockchain, there's a chance you'll connect to them first and download the wrong blockchain and end up back in the same state, unless you force your client to reject the wrong chain with checkpoints. Fortunately, as more public clients switch over to the main chain, the chance anyone will end up syncing to a minority fork decreases.
so .. Is there a reason a node on a minority (<50%) should publish blocks to other nodes?
-
We only save the undo state for about an hour's worth of blocks (maintaining it is expensive). For short forks this works out well, but if you get on a fork with >360 blocks on it, you won't have enough undo history available to roll your blockchain back to the fork point, thus you'll never be able to rejoin the main chain even if your client knows it is better and has downloaded all the blocks on the main chain. If you don't have the undo history to get there, the only other way to get back to the fork point is to reindex or re-download the blockchain from the beginning. And when re-downloading, as long as there are publicly-accessible nodes out there serving up the wrong blockchain, there's a chance you'll connect to them first and download the wrong blockchain and end up back in the same state, unless you force your client to reject the wrong chain with checkpoints. Fortunately, as more public clients switch over to the main chain, the chance anyone will end up syncing to a minority fork decreases.
I see. Thanks.
Hopefully in the future clients could perform a re-index from a specific block height (like Bitcoin's re-org) on their own without the user's intervention.
-
so .. Is there a reason a node on a minority (<50%) should publish blocks to other nodes?
I guess I'd say that just because you see <50% delegates, that doesn't necessarily mean it's not the best fork out there. There could be a three-way split, or there could really be 60% of delegates offline (maybe a crash that only happens on one OS).
The other way of dealing with the problem that comes to mind is don't publish your blocks if you know there's a longer fork out there, but that is vulnerable to attack because the only way to verify that the longer fork is valid is to roll back to the fork point and try to switch to the fork (and we can't do that when the fork point was too far in the past).
Hopefully in the future clients could perform a re-index from a specific block height (like Bitcoin's re-org) on their own without the user's intervention.
It sounds nice, but my gut feeling (it's not really my area of the code) is that it would be a lot of work to reorganize the database to support that.
-
I see .. fork resolution seems to be way over my head .. good to see smarter people around!
-
This is a very important thread to understand for ALL delegates.
I must admit I don't fully understand how to handle the "forking" discussed here. I'm quite technical but there are many delegates that are not. I suspect many marketing delegates like fuzzy & methodx will struggle to understand this without help from the stronger technically savvy delegates.
I understand what a transaction fork is. I was under the impression such forking is natural in bitcoin blockchains where you have zillions of minors asynchronously trying to find the next puzzle key, but in dpos it's a round robin process, not a competition. So how to do these forks occur on our dpos blockchain? Should they? Is it a bug or an attack?
I have been led to believe that aside from the typical unix admin issues one generally needs to be competent to handle if running a delegate and just paying attention to version releases, the client runs and automatically takes care of itself. I've always had reservations about those claims (even BM himself has made statements to that it's pretty easy to run a delegate) and this thread heightens my suspicions.
delegate.verbaltech has yet to be voted in, so I don't check it every day. I have to jump thru some hoops in an effort to minimize / obscure access to the delegate node. So I am quite interested in how issues such as this one can be detected and resolved.
I see info like this:
"blockchain_average_delegate_participation": "90.99 %"
But I don't have the cmd line commands memorized to know how to obtain this info, is it simply the info cmd or something else? It would be good to know.
Is this cmd sufficient to detect being on the correct fork? If so tools like wackou's bts_tools could monitor that and send a notification the node is on a minority fork.
-
Paging @xeroc or other devs, can you answer the questions I had above? I did notice a post by Vikram that he is checking into a forking issue, so this is beginning to sound like a bug...
-
I still don't understand why minority fork clients don't just discard whatever short chain they're on and join the main chain where we have >90% delegates participation. We can't rely on checkpoints for long :(
We only save the undo state for about an hour's worth of blocks (maintaining it is expensive). For short forks this works out well, but if you get on a fork with >360 blocks on it, you won't have enough undo history available to roll your blockchain back to the fork point, thus you'll never be able to rejoin the main chain even if your client knows it is better and has downloaded all the blocks on the main chain. If you don't have the undo history to get there, the only other way to get back to the fork point is to reindex or re-download the blockchain from the beginning. And when re-downloading, as long as there are publicly-accessible nodes out there serving up the wrong blockchain, there's a chance you'll connect to them first and download the wrong blockchain and end up back in the same state, unless you force your client to reject the wrong chain with checkpoints. Fortunately, as more public clients switch over to the main chain, the chance anyone will end up syncing to a minority fork decreases.
I haven't read the rest of the thread but the above is the primary problem, and is common whenever we hardfork because we get long minority chains from delegates who haven't updated.
There is also a second problem where blocks after the hardfork are not necessarily invalid for old clients. So they might upgrade after the hardfork and if they don't reindex, they will have an inconsistent state but cannot tell until they finally start rejecting blocks.
It seems like we can mitigate the most common issues by a combination of:
- Force all blocks after a hardfork to be invalid for old clients
- Always reindex for hardfork releases
- Always release a version with a checkpoint right after the hardfork as soon as possible. And force a reindex on startup if any checkpoints dont match the current state
Also theoretical had an idea to stop old clients syncing before a hardfork. I am not convinced it is worth the effort, but here is part of our conversation:
[3/26/15, 4:49:46 PM] Vikram Rajkumar: there is a remaining problem due to our implementation—some delegates will not upgrade and so old clients will go down that minority chain—once they go too far and are past the undo limit, and upgrade to say 0.8 for example which has the new rules but no new checkpoints after the hardfork—they are still stuck. it will not resync or reindex
[3/26/15, 4:49:56 PM] Vikram Rajkumar: automatically. so not sure how to address that
[3/26/15, 4:54:09 PM] theoretical bts: i think we talked about this the other day...have delegates publish in public_data the block number of next hardfork as well as the version they are running
[3/26/15, 4:54:53 PM] theoretical bts: then if client notices majority of delegates claim there is a hardfork at block #X but block #X does not exist in local hardfork database, client warns user that they are out of date and stops syncing at block #X
[3/26/15, 4:57:30 PM] Vikram Rajkumar: yea i guess that’s better than what we currently do is which is just disconnect all old clients from the main chain
-
Both of my api servers for Bitsharesblocks just found themselves on forks and when I tried to restart the client they just deleted the blockchain and started syncing from scratch. So bitsharesblocks will be out of service for another 7 hours or so.... This is getting pretty frustrating tbh.
-
Both of my api servers for Bitsharesblocks just found themselves on forks and when I tried to restart the client they just deleted the blockchain and started syncing from scratch. So bitsharesblocks will be out of service for another 7 hours or so.... This is getting pretty frustrating tbh.
Its alot faster to sync this version
-
Both of my api servers for Bitsharesblocks just found themselves on forks and when I tried to restart the client they just deleted the blockchain and started syncing from scratch. So bitsharesblocks will be out of service for another 7 hours or so.... This is getting pretty frustrating tbh.
Its alot faster to sync this version
Not for me, I'm also getting segmentation faults, random crashes and sync stopping for no reason :( I left the western api server to sync last night, woke up to find it stuck at block 990000..
-
Both of my api servers for Bitsharesblocks just found themselves on forks and when I tried to restart the client they just deleted the blockchain and started syncing from scratch. So bitsharesblocks will be out of service for another 7 hours or so.... This is getting pretty frustrating tbh.
Its alot faster to sync this version
Not for me, I'm also getting segmentation faults, random crashes and sync stopping for no reason :( I left the western api server to sync last night, woke up to find it stuck at block 990000..
* delete the .BitShares/logs and .BitShares/exceptions folders
* use chainserver: http://wiki.bitshares.org/index.php/BitShares/Chain-Server
-
zillions of minors asynchronously trying to find the next puzzle key
(http://25.media.tumblr.com/tumblr_lta3b1KmAv1qii6tmo1_500.gif)
-
Strangely I haven't seen any issues...
Neither on delegates nor on the seed node.
I hope I'll not see what you describe (:
-
Bitsharesblocks is finally up and running, hopefully it won't go out of sync again!
-
Bitsharesblocks is finally up and running, hopefully it won't go out of sync again!
What would be necessary to add one of my machines as api server?
-
Bitsharesblocks is finally up and running, hopefully it won't go out of sync again!
What would be necessary to add one of my machines as api server?
I'm also interested in this (:.
The more nodes we could pick the better.
-
Adding the checkpoints and re-indexing hasn't worked. If I am to start from scratch is it advisable to use a chain server? I was using one and ended up on a minority fork so...
-
i have the same problem
will the next upgrade solve this?
i use now 0.8.1 and try for weeks now to get it synced
after days of loading the blockchain i end always stuck
and the del paticipation got never over 30%
it loses networkconections
or wallet does not response errors
and i have to quit the client in task manager
will there be a solution for this soon ?
-
i have the same problem
will the next upgrade solve this?
i use now 0.8.1 and try for weeks now to get it synced
after days of loading the blockchain i end always stuck
and the del paticipation got never over 30%
it loses networkconections
or wallet does not response errors
and i have to quit the client in task manager
will there be a solution for this soon ?
For me the major issue was running out of disk space on the VPS due to exploding log files, did you make sure you have enough disk space?
-
For me the major issue was running out of disk space on the VPS due to exploding log files, did you make sure you have enough disk space?
[/quote]
is 64 gb enough?
-
off topic
i wonder what imapct for mass adoption the growing sice of the wallet will have
if a average useres pc with lets say 4 to 8 gb working space and a 120 gb disc and a series 4 procesor cant handle it
my procesor is while downloading the blockchain most of the time at 100% and i added already a extra cooling pad
right now the bitshare is less then a year runnning and only a few use it
what size will it have when millions use it , trade ........... every day
and if there is a upgrade for the wallet you need to resync the chain from day one?
i daubt many average useres or new users will have the patience to wait for days till its synct
i wonder if there is a way to make it smaler and faster
i read something about ethereum that they use a snapshot on which all sort of delegates or miners agree and put it in a codepoint
i am not a expert but as i understood it :
make a picture of the agreed chain part seel it and use that as a startpoint for the wallet
it reminds me of music files and how they developed mp3 technique i wonder if that is posible with the blockchain
-
off topic
i wonder what imapct for mass adoption the growing sice of the wallet will have
if a average useres pc with lets say 4 to 8 gb working space and a 120 gb disc and a series 4 procesor cant handle it
my procesor is while downloading the blockchain most of the time at 100% and i added already a extra cooling pad
right now the bitshare is less then a year runnning and only a few use it
what size will it have when millions use it , trade ........... every day
and if there is a upgrade for the wallet you need to resync the chain from day one?
i daubt many average useres or new users will have the patience to wait for days till its synct
i wonder if there is a way to make it smaler and faster
i read something about ethereum that they use a snapshot on which all sort of delegates or miners agree and put it in a codepoint
i am not a expert but as i understood it :
make a picture of the agreed chain part seel it and use that as a startpoint for the wallet
it reminds me of music files and how they developed mp3 technique i wonder if that is posible with the blockchain
It's said that average users will use the light wallet or web wallet in the future.