My observer node got out of sync as well.

3194926ms th_a       fork_database.cpp:57          push_block           ] Pushing block to fork database that failed to link: 00011598c1a67b75aca7dc775fcc5b782ca0b7ff, 71064
3194926ms th_a       fork_database.cpp:58          push_block           ] Head: 71282, 00011672235d57fc149ead58202df2cf641d3d303194926ms th_a       application.cpp:422           handle_block         ] Error when pushing block:
3080000 unlinkable_block_exception: unlinkable block
block does not link to known chain
    {}
    th_a  fork_database.cpp:78 _push_block

    {"new_block":{"previous":"00011597d7e7e8b0cd788f563e024a0f9ce9e686","timestamp":"2015-10-08T05:55:15","witness":"1.6.26","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"1f729beb5475b4d0f378f0e8bf7420404d88b765b4a507ac38f1989e6ca23c62940987e48052760d61ef2fe387d99af55fe6f3f1f0760675418d08bf8c659f2d15","transactions":[]}}
    th_a  db_block.cpp:197 _push_block

...
...
3480154ms th_a       application.cpp:522           get_item             ] Couldn't find block 000114d83dc5f7ad03d3c90fbdc74d1d6d3e2288 -- corresponding ID in our chain is 000114d83dc5f7ad03d3c90fbdc74d1d6d3e2288



Looks like it's stuck at the fork produced by delegate.baozi  ???

Quote from: puppies on October 08, 2015, 08:57:52 PM
here is my automatic failover script powered by xerocs python-graphenelib.  I am a noob at teh programming, and if I did anything wrong I would love the feedback.

The wallet this script is running through needs to be unlocked, and must contain the owner key (I think its the owner key, rather than the active key) of the witness.

Awesome!  This gives me an idea for an "industrial strength" version.

The active key should be sufficient.

Quote from: wuyanren on October 08, 2015, 02:46:38 PM
Every day so many witnesses are not online, what is the reason?

We've identified an issue where the assignment of new object ID's is inconsistent (we use a hashed index with an undefined iteration order).  Fortunately the testnet's main fork has them in increasing order (I suspect it has to do with the memory allocator tending to put things allocated later at higher addresses), in the next hardfork this condition will be enforced (by replacing most of our hashed indexes with ordered indexes).  We suspect some witnesses are getting their state wrong when they assign an ID differently from the main network, if someone publishes a transaction using that ID then the wrong state turns into a desync (they'll no longer sign or pay attention to blocks on the main fork because they think the main fork has an illegal transaction.)

If you get stuck and the first error message mentions a vesting balance object, this is almost certainly what's happening to you.

The next hardfork will fix this issue, and in addition, I'm working on a tool to help us find if there are any similar issues.

The transactions being executed by the community testers have been incredibly helpful to us.

here is my automatic failover script powered by xerocs python-graphenelib.  I am a noob at teh programming, and if I did anything wrong I would love the feedback.

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import sys
import json
from grapheneapi import GrapheneWebsocket, GrapheneWebsocketProtocol
import time
import config

rpc = GrapheneWebsocket("localhost", 8092, "", "")
publickeys = config.publickeys
witnessname = config.witnessname


def getmissed(witnessname):
    witness = rpc.get_witness(witnessname)
    missed = witness["total_missed"]
    return missed

missed = getmissed(config.witnessname)


def switch(witnessname, publickeys, missed):
    keynumber = missed % len(publickeys)
    key = publickeys[keynumber]
    rpc.update_witness(witnessname, "", key, "true")
    print("witness down switching to " + key)
print ("Nothing in this window will change unless a block is missed")

while True:
    if missed < getmissed(config.witnessname):
        missed = getmissed(config.witnessname)
        switch(witnessname, publickeys, missed)
    else:
        time.sleep(3)

You need to add two lines to your config.py file.  witnessname as a string and publickeys as strings in a tuple of the public keys you want to switch between.  those lines look like this in my case.

witnessname = "dele-puppy"
publickeys = ("GPH75xxKG4ZeztPpnhmFch99smunUWMvDy9mB6Le497vpAA3XUXaD", "GPH8Sj81ncf8PK5QwJZvhWsvwp6m4bXzTFvp4bKycNNFvHaJpZSnH")


The wallet this script is running through needs to be unlocked, and must contain the owner key (I think its the owner key, rather than the active key) of the witness.

Quote from: puppies on October 08, 2015, 08:23:32 PM
dele-puppy will be intermittently missing  blocks as I am testing a failover script.

Im interested in that script, will you make it public?

Quote from: Bhuz on October 08, 2015, 08:09:29 PM
I got a couples of:
getObjects without subscribe callback??
What is it?

Don't worry with that message.

The witness node logs that when there is no subscription and a getobject requirements comes in.

Witness node crashed.

463719ms th_a       fork_database.cpp:57          push_block           ] Pushing block to fork database that failed to link: 0001159f760df617a20b8b8091820f229d707e17, 71071
463719ms th_a       fork_database.cpp:58          push_block           ] Head: 71200, 00011620289c0a9af56a59ba52e88f1841fcc5bf
463719ms th_a       application.cpp:422           handle_block         ] Error when pushing block:
3080000 unlinkable_block_exception: unlinkable block
block does not link to known chain
    {}
    th_a  fork_database.cpp:78 _push_block

    {"new_block":{"previous":"0001159efd84ad55e9ec8ececf862be1b33b5d40","timestamp":"2015-10-08T05:55:45","witness":"1.6.1","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"1f2d5c37139be7915ed09fe135a7364a97b3194ac66dfdfa273aa18b30c87d80fd39a12848c7ca16ad03305c376544d39bb469e8b88a18c758a6bc1023212d9fd8","transactions":[]}}
    th_a  db_block.cpp:197 _push_block
witness_node: /app/bts/graphene-test6.1/libraries/net/node.cpp:1594: void graphene::net::detail::node_impl::schedule_peer_for_deletion(const peer_connection_ptr&): Assertion `_closing_connections.find(peer_to_delete) == _closing_connections.end()' failed.

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffff3581700 (LWP 7034)]
0x00007ffff6c01cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56



dele-puppy will be intermittently missing  blocks as I am testing a failover script. 

I got a couples of:
getObjects without subscribe callback??
What is it?

Quote from: bytemaster on October 08, 2015, 07:44:07 PM
Please hold off testing the current network.   We are working on some things we would like to deploy before the next set of forks.

Ok I stopped my servers.

Please hold off testing the current network.   We are working on some things we would like to deploy before the next set of forks.

100 TPS / 1 hour test will begin at 10/8 20:00 UTC (in 25 min)

Flooding nodes are forked :'( I have to setup them again... I will notice the next test schedule soon.

Quote from: bytemaster on October 08, 2015, 12:25:46 PM

Quote from: liondani on October 08, 2015, 10:54:44 AM
is the "irreversible" output something it must noticed?

Code Select Expand


3173604ms th_a       application.cpp:394           handle_block         ] Got block: #75925 time: 2015-10-08T10:52:54 latency: -398 ms from: triox-delegate  irreversible: 75892 (-33)
3180219ms th_a       application.cpp:394           handle_block         ] Got block: #75926 time: 2015-10-08T10:53:00 latency: 216 ms from: init10  irreversible: 75892 (-34)
3185689ms th_a       application.cpp:394           handle_block         ] Got block: #75927 time: 2015-10-08T10:53:06 latency: -313 ms from: delegate-1.lafona  irreversible: 75893 (-34)
3189138ms th_a       application.cpp:394           handle_block         ] Got block: #75928 time: 2015-10-08T10:53:09 latency: 135 ms from: mindphlux.witness  irreversible: 75893 (-35)
3192219ms th_a       application.cpp:394           handle_block         ] Got block: #75929 time: 2015-10-08T10:53:12 latency: 217 ms from: init2  irreversible: 75893 (-36)
3195793ms th_a       application.cpp:394           handle_block         ] Got block: #75930 time: 2015-10-08T10:53:15 latency: 790 ms from: fox  irreversible: 75893 (-37)
3198140ms th_a       application.cpp:394           handle_block         ] Got block: #75931 time: 2015-10-08T10:53:18 latency: 137 ms from: jtm1  irreversible: 75893 (-38)
3201413ms th_a       application.cpp:394           handle_block         ] Got block: #75932 time: 2015-10-08T10:53:21 latency: 410 ms from: bitcube  irreversible: 75894 (-38)
3204146ms th_a       application.cpp:394           handle_block         ] Got block: #75933 time: 2015-10-08T10:53:24 latency: 143 ms from: spectral  irreversible: 75894 (-39)
3207154ms th_a       applic


The irreversible block is the last block that your node has that has been confirmed by 2/3 of the witnesses.  The "-X" represents how many blocks ago that was.

Recent witness output. What does the irreversible message indicate?

609200ms th_a       application.cpp:394           handle_block         ] Got block: #82384 time: 2015-10-08T17:10:09 latency: 197 ms from: init3  irreversible: 82359 (-25)
612313ms th_a       application.cpp:394           handle_block         ] Got block: #82385 time: 2015-10-08T17:10:12 latency: 310 ms from: bue  irreversible: 82360 (-25)
618195ms th_a       application.cpp:394           handle_block         ] Got block: #82386 time: 2015-10-08T17:10:18 latency: 193 ms from: init10  irreversible: 82361 (-25)
621168ms th_a       application.cpp:394           handle_block         ] Got block: #82387 time: 2015-10-08T17:10:21 latency: 166 ms from: bitshares-argentina  irreversible: 82362 (-25)
639346ms th_a       application.cpp:394           handle_block         ] Got block: #82392 time: 2015-10-08T17:10:39 latency: 344 ms from: xeldal  irreversible: 82367 (-25)00141cd4896630ecde14ee302b91402b54b14b8","000141d1a0843adbfe97ac2bb9dd09972af2e06e","000141d3e35ca391661bb1a33c30d8c86aaf2bc9"]
627190ms th_a       application.cpp:394           handle_block         ] Got block: #82388 time: 2015-10-08T17:10:27 latency: 187 ms from: init2  irreversible: 82363 (-25)
630351ms th_a       application.cpp:394           handle_block         ] Got block: #82389 time: 2015-10-08T17:10:30 latency: 348 ms from: roadscape  irreversible: 82364 (-25)
633198ms th_a       application.cpp:394           handle_block         ] Got block: #82390 time: 2015-10-08T17:10:33 latency: 195 ms from: init9  irreversible: 82365 (-25)
642055ms th_a       application.cpp:394           handle_block         ] Got block: #82393 time: 2015-10-08T17:10:42 latency: 52 ms from: bhuz  irreversible: 82368 (-25)


Edit: Sorry for repost