BitShares Forum
Main => General Discussion => Topic started by: Riverhead on September 22, 2014, 11:38:19 am
-
One of the good things about this alpha stage is that it's allowing the delegates to shake out system issues, build tools around their delegates, and better understand the technology.
So in that spirit I'm looking for some insight into what happened a few hours ago that caused me to miss 16 consecutive blocks. Is it related to the forks in the list after the picture? I noticed that after my fork the next delegate has a huge latency spike. Am I on the wrong fork because of them or visa versa? Why did it eventually figure itself out?
(http://i.imgur.com/gS2zEX5.png)
FORKED BLOCK FORKING BLOCK ID SIGNING DELEGATE TXN COUNT SIZE TIMESTAMP LATENCY VALID IN CURRENT CHAIN
--------------------------------------------------------------------------------------------------------------------------------------------------------------
544146
6dfadfc4e391f46184fcae08fce1eceb627a762f a.delegate.xeroc 0 166 2014-09-21T13:38:40 598 YES YES
954e5975848e98aba6ee7f80bc95fb9ad764d440 delegate.liondani 13 3019 2014-09-21T13:55:00 0 N/A NO
544359
30b0a1bf7566055f8cbabd8435d937d513934ea9 dele-puppy 0 166 2014-09-21T14:15:10 609 YES YES
4ea6c8ee71f7b6f98a4e70678f8724d304bc9818 www2.minebitshares-com 13 3374 2014-09-21T14:31:50 0 N/A NO
544512
ab2897abecd0d35b8c00af8a70a576395b0b1201 delegate.taolje 0 166 2014-09-21T14:40:50 0 YES YES
90b8609d8ec0ac76adc8bff9c7b897c22fd06cd5 fox 7 1930 2014-09-21T14:49:00 0 N/A NO
545307
21431f4eeb608faec4598af0a31d14a504491ae4 delegate.taolje 1 410 2014-09-21T16:54:00 0 YES YES
07e77010de0e66e766737774571c38083917c5b2 fox 13 4778 2014-09-21T17:10:30 1 N/A NO
545552
26e8fb9cae7f31249e1314c34fba192fa8f8c345 spartako1 1 410 2014-09-21T17:35:30 0 YES YES
ac8ba2a73a3dac799fe54cb1ad7fccb5bc29a8f1 www2.minebitshares-com 3 2267 2014-09-21T17:37:50 0 N/A NO
545580
7682aacc16c2b4077ff37ab36febe07e658d81cb riverhead-del-server-1 5 1276 2014-09-21T17:46:20 0 YES NO
34136cdfc995af6b79e59a86336e26da04ad974c dpos.crazybit 1 382 2014-09-21T17:40:20 3921 YES YES
545636
b34c1c611e50cf400c2e023d876ac7810dac87c3 btsx.chinesecommunity 0 166 2014-09-21T17:50:10 0 YES YES
d9b679d434be6ef11576deac06bf1569b21b0bfd delegate.webber 2 722 2014-09-21T17:54:30 0 N/A NO
546260
6cc0550eb4e5ea13c642b1098815916823e2b6ed titan.crazybit 1 410 2014-09-21T19:37:10 0 YES YES
0cde6d50bd417fc07dbad0c8c90daf4a55463230 calabiyau 2 2019 2014-09-21T19:49:20 -1 N/A NO
546293
fa4ac65214a818f387bb8eb91ee548985d871151 delegate.cgafeng 0 166 2014-09-21T19:42:50 0 YES YES
993f7692d68e136ff2575c9b1bca63f99ae271b0 maqifrnswa 1 1775 2014-09-21T19:48:30 0 N/A NO
548904
6ba84dc7b49762b367c9a63d1b9148f7c8ad8f4e delegate.liondani 0 166 2014-09-22T03:05:20 0 YES YES
34cce5013e5c865e0c1b018a45e042a247870a93 delegate.cgafeng 9 5576 2014-09-22T03:21:00 1 N/A NO
549041
de000ca0a862b328c05d92b765b5bae42d1a864a crazybit 1 506 2014-09-22T03:28:20 0 YES YES
e688612bc696b009d65449902e9dbaf7f4592e9d delegate.bitder 4 1430 2014-09-22T03:33:40 0 N/A NO
549231
76c478280584e04b46b376e516eb8daff3f8941c riverhead-del-server-1 2 2217 2014-09-22T04:08:10 0 YES NO
e5e9ddc2b67817f0d9edaf5d3bf5744cb8eab0ed cny.bts500 0 166 2014-09-22T04:01:10 15660 YES YES
549287
cbbd03bc312c9267a1a2803c90d2959461cadc0a delegate-alt 0 166 2014-09-22T04:11:00 0 YES YES
0747752a078447aa26685890164e000f62e9a46b delegate1-galt 0 166 2014-09-22T04:17:00 0 N/A NO
549429
421abe39c5f1769c22c8d42385558f15b6e2a08f coolspeed 0 166 2014-09-22T04:36:50 0 YES YES
7edfbf3eb8d94bc6e8df66166c71803af18716ba delegate.liondani 6 3162 2014-09-22T04:48:30 0 N/A NO
549526
72c89d29eb6a586db10dbe0c88394bf4f8b7979b dc-delegate 0 166 2014-09-22T04:54:20 0 YES YES
f9d5c5270fd95335a27907b1a72d4b23edb0c3ed chinese 3 1172 2014-09-22T04:58:50 0 N/A NO
549641
c8dbea7384f7c834ff92d7e7d178608bfc9cbc25 delegate-baozi 0 166 2014-09-22T05:15:40 0 YES YES
9a295c98a77be97b4fbf6b72268ccc2ba58f32ad kokojie 6 3186 2014-09-22T05:31:30 0 N/A NO
549700
56f4c2bcf413df9cb4ad047ffcdc3f624fd31363 sun.delegate.service 1 410 2014-09-22T05:26:10 1 YES YES
8a40ccba6e6dcc85d4749358c24443215e74e37e delegate.bitder 2 567 2014-09-22T05:26:40 0 N/A NO
550492
8db516f5dfe768a524abc00375c24cc705c5afbf dpos.crazybit 0 166 2014-09-22T07:51:00 0 YES YES
da70aa6eabbb31b2f8c2b9711feab65705cc68aa delegate.bitder 0 166 2014-09-22T07:51:20 0 N/A NO
-
Interesting, this happened to me a few times as well and would be nice to understand why. BM said at the time it could be due to the delegate behind or in front of me having a high latency and thus ignoring my block, so guess that could be why in your case.
I looked up the blocks where your fork happened on my site (545580+1 and 549231+1), and I've got them down as signed by "dpos.crazybit" and "cny.bts500" with a latency of 0s. It looks like you tried to produce that first block 6 minutes after and the second 7 minutes after it was produced, which I guess is why you created forks.
Maybe the latency spike was actually on your end, or your system clock was unsynchronized?
-
Thank you for the research, svk. That is an interesting point as a few times I have found that on another wallet (where I'm playing with bots) I sometimes see that my client is five to 10 minutes behind.
So I guess the next question is....why is my client experiencing these stoppages in block syncing? I wonder if there's enough information in the logs to correlate connection count and the missing blocks.
-
Thank you for the research, svk. That is an interesting point as a few times I have found that on another wallet (where I'm playing with bots) I sometimes see that my client is five to 10 minutes behind.
So I guess the next question is....why is my client experiencing these stoppages in block syncing? I wonder if there's enough information in the logs to correlate connection count and the missing blocks.
Experiencing similar things since some delegates upgraded to 0.4.16 (cr1/2) with my non delegate client. My client was still on 0.4.15; Now with me on 0.4.16 I have seen this behavior 3 or 4 times. Restarting seems to fix the problem for me but then again it is easy when one is not delegate... Will post more info the next time this happens...will be good if somebody points me to what info will be helpful.
My initial post/info: https://bitsharestalk.org/index.php?topic=9099.msg118456#msg118456
-
Thanks tonyk, glad I'm not alone on this. The reason I noticed it with my bots is that they would a long time without a price update. When I checked the wallet I noticed it was minutes behind. A restart fixed it then as well. However, it seems the client does eventually catch up on its own.
-
Just thought of something, are you running your client with "--accept-incoming-connections false"? That's how I missed blocks last time, after a certain time you start losing connections and your client gets out of sync. That doesn't explain the latency though..
-
I am running with it that way; but I always have been (it's behind a strict firewall). Perhaps things have changes. I'll try it with that flag set to true and see what happens.
Update: it is set to true...must have missed that in an upgrade. Anyway, I get this at start which looks normal.
./run_delegate.sh
Loading blockchain from: .delegate_0.4.16/chain
Loading config from file: .delegate_0.4.16/config.json
Starting JSON RPC server on port 49333 (localhost only)
Starting HTTP JSON RPC server on port 1579 (localhost only)
UPnP: ExternalIPAddress = xxx.xxx.xxx.xxx
UPnP Port Mapping successful
-
Just caught this. Restarted and didn't miss a block. I'll have to keep an eye on it until we figure out what's going on.
"blockchain_head_block_age": "14 minutes old",
info
{
"blockchain_head_block_num": 552313,
"blockchain_head_block_age": "14 minutes old",
"blockchain_head_block_timestamp": "2014-09-22T13:03:30",
"blockchain_average_delegate_participation": "55.19 %",
"blockchain_confirmation_requirement": 1,
"blockchain_delegate_pay_rate": "2.23912 BTSX",
"blockchain_share_supply": "1,999,861,866.73016 BTSX",
"blockchain_blocks_left_in_round": 56,
"blockchain_next_round_time": "at least 9 minutes in the future",
"blockchain_next_round_timestamp": "2014-09-22T13:26:30",
"blockchain_random_seed": "61c71a80b3c2f54aedc92f0c72c8f321352fcd0b",
"client_data_dir": "/home/riverhead/.delegate_0.4.16",
"client_version": "v0.4.16",
"network_num_connections": 0,
"network_num_connections_max": 200,
"ntp_time": "2014-09-22T13:17:16",
"ntp_time_error": -0.022846000000000002,
"wallet_open": true,
"wallet_unlocked": true,
"wallet_unlocked_until": "3 years 2 months in the future",
"wallet_unlocked_until_timestamp": "2017-11-22T22:50:43",
"wallet_last_scanned_block_timestamp": "2014-07-21T07:02:30",
"wallet_scan_progress": "? %",
"wallet_block_production_enabled": true,
"wallet_next_block_production_time": "at least 9 minutes in the future",
"wallet_next_block_production_timestamp": "2014-09-22T13:33:50"
-
I meant to say I missed blocks when I set it to true, sorry about that. It's when you're blocking incoming connections that you end up losing connections and getting out of sync.
-
Thank you for the research, svk. That is an interesting point as a few times I have found that on another wallet (where I'm playing with bots) I sometimes see that my client is five to 10 minutes behind.
So I guess the next question is....why is my client experiencing these stoppages in block syncing? I wonder if there's enough information in the logs to correlate connection count and the missing blocks.
Experiencing similar things since some delegates upgraded to 0.4.16 (cr1/2) with my non delegate client. My client was still on 0.4.15; Now with me on 0.4.16 I have seen this behavior 3 or 4 times. Restarting seems to fix the problem for me but then again it is easy when one is not delegate... Will post more info the next time this happens...will be good if somebody points me to what info will be helpful.
My initial post/info: https://bitsharestalk.org/index.php?topic=9099.msg118456#msg118456
I can confirm that since upgrading to 0.4.16 my delegate has missed blocks.
My non-delegate client (also running 0.4.16) stopped syncing.
"blockchain_head_block_age": "5 hours old",
"client_version": "v0.4.16",
"network_num_connections": 5,
Restarted the non-delegate client and it made some progress but stopped syncing again
"blockchain_head_block_num": 552498,
"blockchain_head_block_age": "22 minutes old",
"blockchain_head_block_timestamp": "2014-09-22T13:34:40",
"client_version": "v0.4.16",
"network_num_connections": 14,
"network_num_connections_max": 200,
"ntp_time": "2014-09-22T13:57:02",
"ntp_time_error": -0.024471,
-
Thank you for the research, svk. That is an interesting point as a few times I have found that on another wallet (where I'm playing with bots) I sometimes see that my client is five to 10 minutes behind.
So I guess the next question is....why is my client experiencing these stoppages in block syncing? I wonder if there's enough information in the logs to correlate connection count and the missing blocks.
Experiencing similar things since some delegates upgraded to 0.4.16 (cr1/2) with my non delegate client. My client was still on 0.4.15; Now with me on 0.4.16 I have seen this behavior 3 or 4 times. Restarting seems to fix the problem for me but then again it is easy when one is not delegate... Will post more info the next time this happens...will be good if somebody points me to what info will be helpful.
My initial post/info: https://bitsharestalk.org/index.php?topic=9099.msg118456#msg118456
I can confirm that since upgrading to 0.4.16 my delegate has missed blocks.
My non-delegate client (also running 0.4.16) stopped syncing.
"blockchain_head_block_age": "5 hours old",
"client_version": "v0.4.16",
"network_num_connections": 5,
Restarted the non-delegate client and it made some progress but stopped syncing again
"blockchain_head_block_num": 552498,
"blockchain_head_block_age": "22 minutes old",
"blockchain_head_block_timestamp": "2014-09-22T13:34:40",
"client_version": "v0.4.16",
"network_num_connections": 14,
"network_num_connections_max": 200,
"ntp_time": "2014-09-22T13:57:02",
"ntp_time_error": -0.024471,
Restarted my non-delegate client again and it caught up.
-
Thank you for the research, svk. That is an interesting point as a few times I have found that on another wallet (where I'm playing with bots) I sometimes see that my client is five to 10 minutes behind.
So I guess the next question is....why is my client experiencing these stoppages in block syncing? I wonder if there's enough information in the logs to correlate connection count and the missing blocks.
Experiencing similar things since some delegates upgraded to 0.4.16 (cr1/2) with my non delegate client. My client was still on 0.4.15; Now with me on 0.4.16 I have seen this behavior 3 or 4 times. Restarting seems to fix the problem for me but then again it is easy when one is not delegate... Will post more info the next time this happens...will be good if somebody points me to what info will be helpful.
My initial post/info: https://bitsharestalk.org/index.php?topic=9099.msg118456#msg118456
I can confirm that since upgrading to 0.4.16 my delegate has missed blocks.
My non-delegate client (also running 0.4.16) stopped syncing.
"blockchain_head_block_age": "5 hours old",
"client_version": "v0.4.16",
"network_num_connections": 5,
Restarted the non-delegate client and it made some progress but stopped syncing again
"blockchain_head_block_num": 552498,
"blockchain_head_block_age": "22 minutes old",
"blockchain_head_block_timestamp": "2014-09-22T13:34:40",
"client_version": "v0.4.16",
"network_num_connections": 14,
"network_num_connections_max": 200,
"ntp_time": "2014-09-22T13:57:02",
"ntp_time_error": -0.024471,
Can you try deleting the node_config.json in your data directory for that client and see if the problem goes away? I want to check if it's this issue: https://github.com/BitShares/bitshares_toolkit/issues/794
-
552367
2c084d3ebd8bbfdbdfdbd7bf8785d3c83778af4e a.delegate.xeroc 0 166 2014-09-22T13:12:40 0 YES YES
b7c66ee7c18d952c2243e7254265b12e28bb6b3f anchor.crazybit 0 166 2014-09-22T13:14:20 0 N/A NO
552563
eae05293ddcb22a2cef8b1e196fe646f69c99441 delegate.adam 0 166 2014-09-22T13:45:40 0 YES YES
422b027cf7424c9e08c7b9aef35dc79522eddec5 dc-delegate 13 2344 2014-09-22T13:59:40 1 N/A NO
553071
f83e68347c0cbc04270023632ad3364acfa831ba crazybit 1 410 2014-09-22T15:11:10 1 YES YES
67036b28d3890e11c18b096e1d4b0016d90adf01 delegate.cgafeng 1 410 2014-09-22T15:12:00 0 N/A NO
553310
0c0b4b133d29340801c1a10dc1950d1127d8b014 calabiyau 8 9577 2014-09-22T16:03:40 0 N/A NO
da5e0d99b7dbff3524ba6244b0cac9ea8d24b465 delegate.nathanhourt.com 0 166 2014-09-22T15:51:20 1701 YES YES
for the second time in 24 hours the delegate client lost the chain and was missing a block.
Deleted node_config.json and restart.
Watching.
-
Can you try deleting the node_config.json in your data directory for that client and see if the problem goes away? I want to check if it's this issue: https://github.com/BitShares/bitshares_toolkit/issues/794
Deleted.
-
>> get_info
{
"blockchain_head_block_num": 554852,
"blockchain_head_block_age": "6 minutes old",
"blockchain_head_block_timestamp": "2014-09-22T20:10:50",
"blockchain_average_delegate_participation": "74.26 %",
"blockchain_confirmation_requirement": 1,
"blockchain_delegate_pay_rate": "2.20136 BTSX",
"blockchain_share_supply": "2,000,187,518.50463 BTSX",
"blockchain_blocks_left_in_round": 42,
"blockchain_next_round_time": "at least 7 minutes in the future",
"blockchain_next_round_timestamp": "2014-09-22T20:23:40",
"blockchain_random_seed": "cca1278fda16821b701399d4b061a331c259008d",
"client_data_dir": "C:/Users/Toni/AppData/Roaming/BITSHA~1",
"client_version": "v0.4.16",
"network_num_connections": 14,
"network_num_connections_max": 200,
"ntp_time": "2014-09-22T20:16:42",
"ntp_time_error": 1.9602310000000001,
"wallet_open": true,
"wallet_unlocked": true,
"wallet_unlocked_until": "12 days in the future",
"wallet_unlocked_until_timestamp": "2014-10-04T08:49:10",
"wallet_last_scanned_block_timestamp": "2014-09-22T20:10:50",
"wallet_scan_progress": "100.00 %",
"wallet_block_production_enabled": false,
"wallet_next_block_production_time": null,
"wallet_next_block_production_timestamp": null
}
Deleted.
-
After dilation...
win7/32bit if this matters.
>> get_info
{
"blockchain_head_block_num": 555302,
"blockchain_head_block_age": "6 minutes old",
"blockchain_head_block_timestamp": "2014-09-22T21:35:00",
"blockchain_average_delegate_participation": "59.76 %",
"blockchain_confirmation_requirement": 50,
"blockchain_delegate_pay_rate": "2.19326 BTSX",
"blockchain_share_supply": "2,000,000,006.04770 BTSX",
"blockchain_blocks_left_in_round": 97,
"blockchain_next_round_time": "at least 16 minutes in the future",
"blockchain_next_round_timestamp": "2014-09-22T21:57:00",
"blockchain_random_seed": "3fec7728c2a94f0a0d10e75e7b572393280acbad",
"client_data_dir": "C:/Users/Toni/AppData/Roaming/BITSHA~1",
"client_version": "v0.4.16",
"network_num_connections": 14,
"network_num_connections_max": 200,
"ntp_time": "2014-09-22T21:40:50",
"ntp_time_error": 2.0256059999999998,
"wallet_open": true,
"wallet_unlocked": true,
"wallet_unlocked_until": "12 days in the future",
"wallet_unlocked_until_timestamp": "2014-10-04T10:08:52",
"wallet_last_scanned_block_timestamp": "2014-09-22T21:35:00",
"wallet_scan_progress": "100.00 %",
"wallet_block_production_enabled": false,
"wallet_next_block_production_time": null,
"wallet_next_block_production_timestamp": null
}
Will limit the max connections to 8. This never happened to me before when the default was 8 connections.
{UPDATE} Setting the max to 7 connections reproduced the issue (after restart) in less then 15min.
>> get_info
{
"blockchain_head_block_num": 555441,
"blockchain_head_block_age": "6 minutes old",
"blockchain_head_block_timestamp": "2014-09-22T21:55:50",
"blockchain_average_delegate_participation": "71.63 %",
-
My client is not synching any longer. Missed a bunch of blocks a couple of hours ago and it is in limbo.
It seems to not count the 30 odd blocks that the client missed. Last block before it started missing blocks was 554889 and after 30 odd missed blocks it got block 554890.
After this no more synching...
Deleting the node_config.json file does not help.
Has anybody found a way to get the client synching again.
I am on the latest client version. Thx.