The telegram chat is great and the bot works wonderfully but it's too general for an emergency channel. I often find myself muting it for 8 hours at a time, because there's just too much chatter to keep up with. The same happened to the nullstreet slack group back in the day, it became impossible and pointless to follow. An emergency/alert system has to be heavily curated, used very scarcely and impossible to ignore. A restricted forum thread that witnesses subscribe to fits the bill.

A few minutes ago a witness asked on Telegram why was he missing blocks. I've found that he was getting negative latency arround -500ms, asked him if hi had ntp installed and he didn't. A minute later (and about 15 minutes after he switched to another node and stoped missing blocks) I missed 3 blocks in a row and witness started showing super high latency's from all witnesses (above 300 seconds?), replayed blockchain and started producing again:

Code: [Select]

1125246ms th_a       application.cpp:458           handle_block         ] Got block: #72017 time: 2015-10-16T04:18:45 latency: 246 ms from:
holytransaction  irreversible: 71988 (-29)
1128179ms th_a       application.cpp:458           handle_block         ] Got block: #72018 time: 2015-10-16T04:18:48 latency: 179 ms from:
init10  irreversible: 71990 (-28)
1131131ms th_a       application.cpp:458           handle_block         ] Got block: #72019 time: 2015-10-16T04:18:51 latency: 131 ms from:
delegate-1.lafona  irreversible: 71991 (-28)
1134292ms th_a       application.cpp:458           handle_block         ] Got block: #72020 time: 2015-10-16T04:18:54 latency: 292 ms from:
bitcube  irreversible: 71995 (-25)
1137159ms th_a       application.cpp:458           handle_block         ] Got block: #72021 time: 2015-10-16T04:18:57 latency: 159 ms from:
init3  irreversible: 71996 (-25)
1140059ms th_a       application.cpp:458           handle_block         ] Got block: #72022 time: 2015-10-16T04:19:00 latency: 59 ms from: s
pectral  irreversible: 71997 (-25)
1143163ms th_a       application.cpp:458           handle_block         ] Got block: #72023 time: 2015-10-16T04:19:03 latency: 163 ms from:
init6  irreversible: 71998 (-25)
1146460ms th_a       application.cpp:458           handle_block         ] Got block: #72024 time: 2015-10-16T04:19:06 latency: 460 ms from:
delegate-clayop  irreversible: 71999 (-25)
1149207ms th_a       application.cpp:458           handle_block         ] Got block: #72025 time: 2015-10-16T04:19:09 latency: 206 ms from:
init7  irreversible: 72000 (-25)
1152026ms th_a       application.cpp:458           handle_block         ] Got block: #72026 time: 2015-10-16T04:19:12 latency: 26 ms from: s
partako  irreversible: 72001 (-25)
1155266ms th_a       application.cpp:458           handle_block         ] Got block: #72027 time: 2015-10-16T04:19:15 latency: 266 ms from:
delegated-proof-of-steak  irreversible: 72002 (-25)
1158043ms th_a       application.cpp:458           handle_block         ] Got block: #72028 time: 2015-10-16T04:19:18 latency: 43 ms from: c
yrano  irreversible: 72003 (-25)
1160989ms th_a       application.cpp:458           handle_block         ] Got block: #72029 time: 2015-10-16T04:19:21 latency: -10 ms from: riverhead  irreversible: 72004 (-25)
1164001ms th_a       witness.cpp:179               block_production_loo ] Generated block #72030 with timestamp 2015-10-16T04:19:24 at time 2015-10-16T04:19:24
1167204ms th_a       application.cpp:458           handle_block         ] Got block: #72031 time: 2015-10-16T04:19:27 latency: 204 ms from: init5  irreversible: 72006 (-25)
1169258ms th_a       application.cpp:778           get_blockchain_synop ] synopsis: ["00011944cd05a88b065c97459067e9433cf8a365","00011952892be327107879d8403f9cea3e6a2137","00011959da81c677a2df19da61c6c4aee44f95a4","0001195d1d6ea4375ae4f68399e9df2ed2b61105","0001195f39e0d57cedc4fdca4ad4dfcf585bdd99"]
1170130ms th_a       application.cpp:458           handle_block         ] Got block: #72032 time: 2015-10-16T04:19:30 latency: 130 ms from: delegate-1.lafona  irreversible: 72007 (-25)
1173026ms th_a       application.cpp:458           handle_block         ] Got block: #72033 time: 2015-10-16T04:19:33 latency: 26 ms from: bhuz  irreversible: 72007 (-26)
1176166ms th_a       application.cpp:458           handle_block         ] Got block: #72034 time: 2015-10-16T04:19:36 latency: 165 ms from: harvey-xts  irreversible: 72008 (-26)
1179163ms th_a       application.cpp:458           handle_block         ] Got block: #72035 time: 2015-10-16T04:19:39 latency: 163 ms from: init2  irreversible: 72008 (-27)
1182153ms th_a       application.cpp:458           handle_block         ] Got block: #72036 time: 2015-10-16T04:19:42 latency: 153 ms from: init0  irreversible: 72009 (-27)
1185002ms th_a       witness.cpp:179               block_production_loo ] Generated block #72037 with timestamp 2015-10-16T04:19:45 at time 2015-10-16T04:19:45
1188022ms th_a       application.cpp:458           handle_block         ] Got block: #72038 time: 2015-10-16T04:19:48 latency: 22 ms from: mindphlux.witness  irreversible: 72010 (-28)
1191282ms th_a       application.cpp:458           handle_block         ] Got block: #72039 time: 2015-10-16T04:19:51 latency: 282 ms from: bitcube  irreversible: 72011 (-28)
1194157ms th_a       application.cpp:458           handle_block         ] Got block: #72040 time: 2015-10-16T04:19:54 latency: 157 ms from: init3  irreversible: 72011 (-29)
1197042ms th_a       application.cpp:458           handle_block         ] Got block: #72041 time: 2015-10-16T04:19:57 latency: 42 ms from: cyrano  irreversible: 72011 (-30)
1200339ms th_a       application.cpp:458           handle_block         ] Got block: #72042 time: 2015-10-16T04:20:00 latency: 339 ms from: bue  irreversible: 72011 (-31)
1203185ms th_a       application.cpp:458           handle_block         ] Got block: #72043 time: 2015-10-16T04:20:03 latency: 185 ms from: init10  irreversible: 72012 (-31)
1206022ms th_a       application.cpp:458           handle_block         ] Got block: #72044 time: 2015-10-16T04:20:06 latency: 22 ms from: wackou  irreversible: 72012 (-32)
1209141ms th_a       application.cpp:458           handle_block         ] Got block: #72045 time: 2015-10-16T04:20:09 latency: 141 ms from: mr.agsexplorer  irreversible: 72013 (-32)
1496796ms th_a       application.cpp:458           handle_block         ] Got block: #72045 time: 2015-10-16T04:20:09 latency: 287796 ms from: mr.agsexplorer  irreversible: 72015 (-30)
1496796ms th_a       application.cpp:458           handle_block         ] Got block: #72045 time: 2015-10-16T04:20:09 latency: 287796 ms from: mr.agsexplorer  irreversible: 72015 (-30)
1496796ms th_a       application.cpp:458           handle_block         ] Got block: #72046 time: 2015-10-16T04:20:12 latency: 284796 ms from: riverhead  irreversible: 72015 (-31)
1496797ms th_a       application.cpp:458           handle_block         ] Got block: #72047 time: 2015-10-16T04:20:15 latency: 281797 ms from: init4  irreversible: 72015 (-32)
1496797ms th_a       application.cpp:458           handle_block         ] Got block: #72047 time: 2015-10-16T04:20:15 latency: 281797 ms from: init4  irreversible: 72016 (-31)
1496797ms th_a       application.cpp:458           handle_block         ] Got block: #72047 time: 2015-10-16T04:20:15 latency: 281797 ms from: init4  irreversible: 72016 (-31)
1496798ms th_a       application.cpp:507           handle_transaction   ] Got 1 transactions from network
1496798ms th_a       application.cpp:458           handle_block         ] Got block: #72048 time: 2015-10-16T04:20:18 latency: 278798 ms from: delegate-clayop  irreversible: 72016 (-32)

1496799ms th_a       application.cpp:458           handle_block         ] Got block: #72048 time: 2015-10-16T04:20:18 latency: 278799 ms from: delegate-clayop  irreversible: 72016 (-32)
1496799ms th_a       application.cpp:458           handle_block         ] Got block: #72048 time: 2015-10-16T04:20:18 latency: 278799 ms from: delegate-clayop  irreversible: 72016 (-32)
1496799ms th_a       application.cpp:458           handle_block         ] Got block: #72049 time: 2015-10-16T04:20:21 latency: 275799 ms from: init8  irreversible: 72016 (-33)
1496800ms th_a       application.cpp:458           handle_block         ] Got block: #72049 time: 2015-10-16T04:20:21 latency: 275800 ms from: init8  irreversible: 72017 (-32)
1496800ms th_a       application.cpp:458           handle_block         ] Got block: #72049 time: 2015-10-16T04:20:21 latency: 275800 ms from: init8  irreversible: 72017 (-32)
1496801ms th_a       application.cpp:458           handle_block         ] Got block: #72050 time: 2015-10-16T04:20:24 latency: 272801 ms from: triox-delegate  irreversible: 72017 (-33)
1496801ms th_a       application.cpp:458           handle_block         ] Got block: #72051 time: 2015-10-16T04:20:27 latency: 269801 ms from: init6  irreversible: 72022 (-29)
1496802ms th_a       application.cpp:458           handle_block         ] Got block: #72051 time: 2015-10-16T04:20:27 latency: 269802 ms from: init6  irreversible: 72022 (-29)
1496802ms th_a       application.cpp:458           handle_block         ] Got block: #72051 time: 2015-10-16T04:20:27 latency: 269802 ms from: init6  irreversible: 72022 (-29)
1496803ms th_a       application.cpp:458           handle_block         ] Got block: #72052 time: 2015-10-16T04:20:30 latency: 266803 ms from: delegated-proof-of-steak  irreversible: 72022 (-30)
1496804ms th_a       application.cpp:458           handle_block         ] Got block: #72053 time: 2015-10-16T04:20:33 latency: 263804 ms from: dele-puppy  irreversible: 72022 (-31)
1496805ms th_a       application.cpp:458           handle_block         ] Got block: #72053 time: 2015-10-16T04:20:33 latency: 263805 ms from: dele-puppy  irreversible: 72025 (-28)
1496805ms th_a       application.cpp:458           handle_block         ] Got block: #72054 time: 2015-10-16T04:20:36 latency: 260805 ms from: verbaltech2  irreversible: 72025 (-29)
1496806ms th_a       application.cpp:458           handle_block         ] Got block: #72054 time: 2015-10-16T04:20:36 latency: 260806 ms from: verbaltech2  irreversible: 72026 (-28)
1496806ms th_a       application.cpp:458           handle_block         ] Got block: #72054 time: 2015-10-16T04:20:36 latency: 260806 ms from: verbaltech2  irreversible: 72026 (-28)
1496806ms th_a       application.cpp:458           handle_block         ] Got block: #72055 time: 2015-10-16T04:20:39 latency: 257806 ms from: in.abit  irreversible: 72026 (-29)
1496807ms th_a       application.cpp:458           handle_block         ] Got block: #72055 time: 2015-10-16T04:20:39 latency: 257807 ms from: in.abit  irreversible: 72031 (-24)
1496807ms th_a       application.cpp:458           handle_block         ] Got block: #72055 time: 2015-10-16T04:20:39 latency: 257807 ms from: in.abit  irreversible: 72031 (-24)
1496807ms th_a       application.cpp:458           handle_block         ] Got block: #72056 time: 2015-10-16T04:20:42 latency: 254807 ms from: spartako  irreversible: 72031 (-25)
1496808ms th_a       application.cpp:458           handle_block         ] Got block: #72056 time: 2015-10-16T04:20:42 latency: 254808 ms from: spartako  irreversible: 72032 (-24)
1496808ms th_a       application.cpp:458           handle_block         ] Got block: #72056 time: 2015-10-16T04:20:42 latency: 254808 ms from: spartako  irreversible: 72032 (-24)
1496808ms th_a       application.cpp:458           handle_block         ] Got block: #72057 time: 2015-10-16T04:20:45 latency: 251808 ms from: delegate.ihashfury  irreversible: 72032 (-25)
1496810ms th_a       application.cpp:458           handle_block         ] Got block: #72058 time: 2015-10-16T04:20:48 latency: 248810 ms from: maqifrnswa  irreversible: 72033 (-25)
1496811ms th_a       application.cpp:458           handle_block         ] Got block: #72058 time: 2015-10-16T04:20:48 latency: 248811 ms from: maqifrnswa  irreversible: 72034 (-24)
1496811ms th_a       application.cpp:458           handle_block         ] Got block: #72059 time: 2015-10-16T04:20:51 latency: 245811 ms from: roadscape  irreversible: 72034 (-25)
1496812ms th_a       application.cpp:458           handle_block         ] Got block: #72059 time: 2015-10-16T04:20:51 latency: 245812 ms from: roadscape  irreversible: 72035 (-24)
1496812ms th_a       application.cpp:458           handle_block         ] Got block: #72060 time: 2015-10-16T04:20:54 latency: 242812 ms from: elmato  irreversible: 72035 (-25)
1496814ms th_a       application.cpp:458           handle_block         ] Got block: #72060 time: 2015-10-16T04:20:54 latency: 242814 ms from: elmato  irreversible: 72036 (-24)
1496814ms th_a       application.cpp:458           handle_block         ] Got block: #72061 time: 2015-10-16T04:20:57 latency: 239814 ms from: fox  irreversible: 72036 (-25)
1496815ms th_a       application.cpp:458           handle_block         ] Got block: #72062 time: 2015-10-16T04:21:00 latency: 236815 ms from: holytransaction  irreversible: 72037 (-25)
1496816ms th_a       application.cpp:458           handle_block         ] Got block: #72062 time: 2015-10-16T04:21:00 latency: 236816 ms from: holytransaction  irreversible: 72038 (-24)
1496816ms th_a       application.cpp:458           handle_block         ] Got block: #72062 time: 2015-10-16T04:21:00 latency: 236816 ms from: holytransaction  irreversible: 72038 (-24)
1496817ms th_a       application.cpp:458           handle_block         ] Got block: #72063 time: 2015-10-16T04:21:03 latency: 233817 ms from: xeldal  irreversible: 72038 (-25)
1496818ms th_a       application.cpp:458           handle_block         ] Got block: #72064 time: 2015-10-16T04:21:06 latency: 230818 ms from: spectral  irreversible: 72039 (-25)
1496819ms th_a       application.cpp:458           handle_block         ] Got block: #72065 time: 2015-10-16T04:21:09 latency: 227819 ms from: init7  irreversible: 72040 (-25)
1496820ms th_a       application.cpp:458           handle_block         ] Got block: #72066 time: 2015-10-16T04:21:12 latency: 224820 ms from: bitcube  irreversible: 72041 (-25)
1496821ms th_a       application.cpp:458           handle_block         ] Got block: #72066 time: 2015-10-16T04:21:12 latency: 224821 ms from: bitcube  irreversible: 72042 (-24)
1496821ms th_a       application.cpp:458           handle_block         ] Got block: #72067 time: 2015-10-16T04:21:15 latency: 221821 ms from: bhuz  irreversible: 72042 (-25)
1496822ms th_a       application.cpp:458           handle_block         ] Got block: #72068 time: 2015-10-16T04:21:18 latency: 218822 ms from: init4  irreversible: 72043 (-25)
1496822ms th_a       application.cpp:458           handle_block         ] Got block: #72068 time: 2015-10-16T04:21:18 latency: 218822 ms from: init4  irreversible: 72043 (-25)
1496830ms th_a       websocket_api.cpp:120         on_message           ] e.to_detail_string(): 10 assert_exception: Assert Exception
!ec: websocket send failed: invalid state
{"msg":"invalid state"}
th_a  websocket.cpp:164 send_message
1496830ms th_a       database_api.cpp:200          ~database_api_impl   ] freeing database api 166198384
1496831ms th_a       database_api.cpp:200          ~database_api_impl   ] freeing database api 218846336

Produced a few blocks and again, missed another block, high latency's again but this time things came back to normal without restarting the witness:

Code: [Select]

huz  irreversible: 72427 (-29)
2452531ms th_a       application.cpp:778           get_blockchain_synop ] synopsis: ["00011aeb78343c7bbd2c7ab4798208a27da4aad4","00011afa837
d1e88f1cfeedd86602831a9590572","00011b02964b12ec207adae71bd2b13a4a0a9d7f","00011b0689886d20ab9f0dc27ad97a66999f121e","00011b08a9ac06e2c6ab01
f5b508a4654518e7fc"]
2454393ms th_a       application.cpp:458           handle_block         ] Got block: #72457 time: 2015-10-16T04:40:54 latency: 393 ms from:
in.abit  irreversible: 72428 (-29)
2457171ms th_a       application.cpp:458           handle_block         ] Got block: #72458 time: 2015-10-16T04:40:57 latency: 170 ms from:
init7  irreversible: 72429 (-29)
2460020ms th_a       application.cpp:458           handle_block         ] Got block: #72459 time: 2015-10-16T04:41:00 latency: 20 ms from: s
partako  irreversible: 72430 (-29)
2463039ms th_a       application.cpp:458           handle_block         ] Got block: #72460 time: 2015-10-16T04:41:03 latency: 39 ms from: w
ackou  irreversible: 72430 (-30)
2466273ms th_a       application.cpp:458           handle_block         ] Got block: #72461 time: 2015-10-16T04:41:06 latency: 273 ms from:
maqifrnswa  irreversible: 72430 (-31)
2469135ms th_a       application.cpp:458           handle_block         ] Got block: #72462 time: 2015-10-16T04:41:09 latency: 135 ms from:
mr.agsexplorer  irreversible: 72431 (-31)
2470809ms th_a       application.cpp:507           handle_transaction   ] Got 1 transactions from network
2472120ms th_a       application.cpp:458           handle_block         ] Got block: #72463 time: 2015-10-16T04:41:12 latency: 120 ms from: init0  irreversible: 72433 (-30)^[[B^[[B
2554005ms th_a       application.cpp:458           handle_block         ] Got block: #72463 time: 2015-10-16T04:41:12 latency: 82005 ms from: init0  irreversible: 72434 (-29)
2554005ms th_a       application.cpp:458           handle_block         ] Got block: #72464 time: 2015-10-16T04:41:15 latency: 79005 ms from: xeldal  irreversible: 72434 (-30)
2554006ms th_a       application.cpp:458           handle_block         ] Got block: #72464 time: 2015-10-16T04:41:15 latency: 79006 ms from: xeldal  irreversible: 72434 (-30)
2554012ms th_a       application.cpp:458           handle_block         ] Got block: #72465 time: 2015-10-16T04:41:18 latency: 76012 ms from: bue  irreversible: 72434 (-31)
2554014ms th_a       application.cpp:458           handle_block         ] Got block: #72465 time: 2015-10-16T04:41:18 latency: 76014 ms from: bue  irreversible: 72435 (-30)
2554014ms th_a       application.cpp:458           handle_block         ] Got block: #72465 time: 2015-10-16T04:41:18 latency: 76014 ms from: bue  irreversible: 72435 (-30)
2554014ms th_a       application.cpp:458           handle_block         ] Got block: #72465 time: 2015-10-16T04:41:18 latency: 76014 ms from: bue  irreversible: 72435 (-30)
2554020ms th_a       application.cpp:458           handle_block         ] Got block: #72466 time: 2015-10-16T04:41:24 latency: 70020 ms from
: harvey-xts  irreversible: 72435 (-31)
2554021ms th_a       application.cpp:458           handle_block         ] Got block: #72466 time: 2015-10-16T04:41:24 latency: 70021 ms from
: harvey-xts  irreversible: 72435 (-31)
2554022ms th_a       application.cpp:458           handle_block         ] Got block: #72467 time: 2015-10-16T04:41:27 latency: 67022 ms from: init4  irreversible: 72435 (-32)
2554023ms th_a       application.cpp:458           handle_block         ] Got block: #72467 time: 2015-10-16T04:41:27 latency: 67023 ms from: init4  irreversible: 72436 (-31)
2554023ms th_a       application.cpp:458           handle_block         ] Got block: #72468 time: 2015-10-16T04:41:30 latency: 64023 ms from: spectral  irreversible: 72436 (-32)
2554025ms th_a       application.cpp:458           handle_block         ] Got block: #72468 time: 2015-10-16T04:41:30 latency: 64025 ms from: spectral  irreversible: 72439 (-29)
2554025ms th_a       application.cpp:458           handle_block         ] Got block: #72468 time: 2015-10-16T04:41:30 latency: 64025 ms from: spectral  irreversible: 72439 (-29)
2554025ms th_a       application.cpp:458           handle_block         ] Got block: #72469 time: 2015-10-16T04:41:33 latency: 61025 ms from: dele-puppy  irreversible: 72439 (-30)
2554026ms th_a       application.cpp:458           handle_block         ] Got block: #72470 time: 2015-10-16T04:41:36 latency: 58026 ms from
2554028ms th_a       application.cpp:458           handle_block         ] Got block: #72471 time: 2015-10-16T04:41:39 latency: 55028 ms from: init3  irreversible: 72441 (-30)
2554035ms th_a       application.cpp:458           handle_block         ] Got block: #72472 time: 2015-10-16T04:41:42 latency: 52035 ms from: init8  irreversible: 72441 (-31)
2554036ms th_a       application.cpp:458           handle_block         ] Got block: #72473 time: 2015-10-16T04:41:45 latency: 49036 ms from: riverhead  irreversible: 72443 (-30)
2554038ms th_a       application.cpp:458           handle_block         ] Got block: #72474 time: 2015-10-16T04:41:48 latency: 46038 ms from: init5  irreversible: 72443 (-31)
2554039ms th_a       application.cpp:458           handle_block         ] Got block: #72475 time: 2015-10-16T04:41:51 latency: 43039 ms from: delegate-1.lafona  irreversible: 72446 (-29)
[[B2563531ms th_a       application.cpp:778           get_blockchain_synop ] synopsis: ["00011afa837d1e88f1cfeedd86602831a9590572","00011b0be306f248278d8bfc0197490b71fed46e","00011b14920371991c39b89346a7ba38e1ff7d3a","00011b18e99b70aab449c520e2a32b8876ea1b30","00011b1a8c7f9c65faf922cdec51403ad4906d7e","00011b1b640dc038ce0ab632469edcf1d3b65478"]
2563541ms th_a       application.cpp:778           get_blockchain_synop ] synopsis: ["00011afa837d1e88f1cfeedd86602831a9590572","00011b0be306f248278d8bfc0197490b71fed46e","00011b14920371991c39b89346a7ba38e1ff7d3a","00011b18e99b70aab449c520e2a32b8876ea1b30","00011b1a8c7f9c65faf922cdec51403ad4906d7e","00011b1b640dc038ce0ab632469edcf1d3b65478"]
2563549ms th_a       application.cpp:778           get_blockchain_synop ] synopsis: ["00011afa837d1e88f1cfeedd86602831a9590572","00011b0be306f248278d8bfc0197490b71fed46e","00011b14920371991c39b89346a7ba38e1ff7d3a","00011b18e99b70aab449c520e2a32b8876ea1b30","00011b1a8c7f9c65faf922cdec51403ad4906d7e","00011b1b640dc038ce0ab632469edcf1d3b65478"]
2563558ms th_a       application.cpp:778           get_blockchain_synop ] synopsis: ["00011afa837d1e88f1cfeedd86602831a9590572","00011b0be306f248278d8bfc0197490b71fed46e","00011b14920371991c39b89346a7ba38e1ff7d3a","00011b18e99b70aab449c520e2a32b8876ea1b30","00011b1a8c7f9c65faf922cdec51403ad4906d7e","00011b1b640dc038ce0ab632469edcf1d3b65478"]
2563614ms th_a       application.cpp:778           get_blockchain_synop ] synopsis: ["00011b0d956a01f4b06ecab56e024757167c0cea","00011b1d15be486b5977df8f47f22ec3fe4b2cbf","00011b25829cea17b02b78c6168ca0eee65a724d","00011b2927707acd5e5d0fd6bb4b5f5321c1df5f","00011b2b3e7377c6f75c178587286331461be982","00011b2c3a103e5a3d35cd6dcaace58b1e5d3308"]
2563616ms th_a       application.cpp:778           get_blockchain_synop ] synopsis: ["00011b0d956a01f4b06ecab56e024757167c0cea","00011b1d15be486b5977df8f47f22ec3fe4b2cbf","00011b25829cea17b02b78c6168ca0eee65a724d","00011b2927707acd5e5d0fd6bb4b5f5321c1df5f","00011b2b3e7377c6f75c178587286331461be982","00011b2c3a103e5a3d35cd6dcaace58b1e5d3308"]
2563621ms th_a       application.cpp:778           get_blockchain_synop ] synopsis: ["00011b0d956a01f4b06ecab56e024757167c0cea","00011b1d15be486b5977df8f47f22ec3fe4b2cbf","00011b25829cea17b02b78c6168ca0eee65a724d","00011b2927707acd5e5d0fd6bb4b5f5321c1df5f","00011b2b3e7377c6f75c178587286331461be982","00011b2c3a103e5a3d35cd6dcaace58b1e5d3308"]
2563625ms th_a       application.cpp:778           get_blockchain_synop ] synopsis: ["00011b0d956a01f4b06ecab56e024757167c0cea","00011b1d15be486b5977df8f47f22ec3fe4b2cbf","00011b25829cea17b02b78c6168ca0eee65a724d","00011b2927707acd5e5d0fd6bb4b5f5321c1df5f","00011b2b3e7377c6f75c178587286331461be982","00011b2c3a103e5a3d35cd6dcaace58b1e5d3308"]
2563678ms th_a       application.cpp:778           get_blockchain_synop ] synopsis: ["00011b0d956a01f4b06ecab56e024757167c0cea","00011b1d15be486b5977df8f47f22ec3fe4b2cbf","00011b25829cea17b02b78c6168ca0eee65a724d","00011b2927707acd5e5d0fd6bb4b5f5321c1df5f","00011b2b3e7377c6f75c178587286331461be982","00011b2c3a103e5a3d35cd6dcaace58b1e5d3308"]
2563771ms th_a       application.cpp:778           get_blockchain_synop ] synopsis: ["00011b0d956a01f4b06ecab56e024757167c0cea","00011b1d15be486b5977df8f47f22ec3fe4b2cbf","00011b25829cea17b02b78c6168ca0eee65a724d","00011b2927707acd5e5d0fd6bb4b5f5321c1df5f","00011b2b3e7377c6f75c178587286331461be982","00011b2c3a103e5a3d35cd6dcaace58b1e5d3308"]
2563771ms th_a       application.cpp:778           get_blockchain_synop ] synopsis: ["00011b0d956a01f4b06ecab56e024757167c0cea","00011b1d15be486b5977df8f47f22ec3fe4b2cbf","00011b25829cea17b02b78c6168ca0eee65a724d","00011b2927707acd5e5d0fd6bb4b5f5321c1df5f","00011b2b3e7377c6f75c178587286331461be982","00011b2c3a103e5a3d35cd6dcaace58b1e5d3308"]
2563795ms th_a       application.cpp:778           get_blockchain_synop ] synopsis: ["00011b0d956a01f4b06ecab56e024757167c0cea","00011b1d15be486b5977df8f47f22ec3fe4b2cbf","00011b25829cea17b02b78c6168ca0eee65a724d","00011b2927707acd5e5d0fd6bb4b5f5321c1df5f","00011b2b3e7377c6f75c178587286331461be982","00011b2c3a103e5a3d35cd6dcaace58b1e5d3308"]
2563966ms th_a       application.cpp:778           get_blockchain_synop ] synopsis: ["00011b0d956a01f4b06ecab56e024757167c0cea","00011b1d15be486b5977df8f47f22ec3fe4b2cbf","00011b25829cea17b02b78c6168ca0eee65a724d","00011b2927707acd5e5d0fd6bb4b5f5321c1df5f","00011b2b3e7377c6f75c178587286331461be982","00011b2c3a103e5a3d35cd6dcaace58b1e5d3308"]
2564145ms th_a       application.cpp:778           get_blockchain_synop ] synopsis: ["00011b0d956a01f4b06ecab56e024757167c0cea","00011b1d15be486b5977df8f47f22ec3fe4b2cbf","00011b25829cea17b02b78c6168ca0eee65a724d","00011b2927707acd5e5d0fd6bb4b5f5321c1df5f","00011b2b3e7377c6f75c178587286331461be982","00011b2c3a103e5a3d35cd6dcaace58b1e5d3308"]
2565061ms th_a       application.cpp:778           get_blockchain_synop ] synopsis: ["00011b0d956a01f4b06ecab56e024757167c0cea","00011b1d15be486b5977df8f47f22ec3fe4b2cbf","00011b25829cea17b02b78c6168ca0eee65a724d","00011b2927707acd5e5d0fd6bb4b5f5321c1df5f","00011b2b3e7377c6f75c178587286331461be982","00011b2c3a103e5a3d35cd6dcaace58b1e5d3308"]
2565231ms th_a       application.cpp:507           handle_transaction   ] Got 1 transactions from network
2565345ms th_a       application.cpp:458
any clue?

I originally built from Vagrantfile, and it indeed doesn't inlcude ntp in dependencies. So after installing ntp per rnglab's advice, I updated witness to switch back to the main node and at that point I noticed on the spare box all blocks delivered with huge negative latencies.

Edit: the anomaly on my end was in the -500ms range

triox-delegate updated without missing a block. I love this tech!

Following; please, add me to your slate. I did take part in the final testnet and with decent reliability too.

Quote from: triox on October 13, 2015, 05:01:00 pm

Seed node 52.28.180.40:35681 up on the production chain.

can't connect

nc -vz 52.28.180.40 35681
Connection to 52.28.180.40 35681 port [tcp/*] succeeded!

weird

I'll be running 52.28.180.40:35681 as a dedicated seed node.

Quote from: triox on October 10, 2015, 06:26:31 am

Quote from: rnglab on October 10, 2015, 05:52:43 am

Is @triox running the webwallet node? he is the only witness on that chain.  Stuck ~2000 blocks behind so far

I'm not running it.

I'm back on the main chain now.

Thanks for the fast answer. Could you share your log? It might help as it seems that only you and the wallet node gone into that fork. did you have to replay or resync blockchain or it just recovered alone? on what commit are you running your witness?

Your post was the first thing I saw after waking up, so thanks : ) 
I recovered by - first - springing out of bed, then deleting the blockchain folder and starting with --replay-blockchain.
I'm on
git describe --tags
test6-30-g7fe0e64commit 7fe0e64a5e38df88c5dea32ae35f371e5c425010
Author: Eric Frias <efrias@syncad.com>
Date:   Fri Oct 9 11:42:56 2015 -0400
Tell me what to look for in the logs.

Updated.

Is it still the case that "total_votes" shows zero until the candidate is voted in?