It's as easy as adding this to the witness_node command:
--checkpoint '[40902,"00009fc6d4e24db66e13af461b94063165999bfa"]' --enable-stale-production

But it seems too late to rescue the network, I still get this error:
1122000ms th_a       witness.cpp:240               block_production_loo ] slot: 11203 scheduled_witness: 1.6.5247 scheduled_[967/1963]
-08-20T10:18:42 now: 2015-08-20T10:18:42
1122000ms th_a       witness.cpp:243               block_production_loo ] Witness 1.6.5247 production slot has arrived; generating a b
lock now...
1122001ms th_a       db_block.cpp:170              _push_block          ] Failed to push new block:
3070000 undo_database_exception: undo database exception
The database does not have enough undo history to support a blockchain with so many missed blocks. Please add a checkpoint if you woul
d like to continue applying blocks beyond this point.
    {"recently_missed":44813,"max_undo":1000}
    th_a  db_update.cpp:72 update_global_dynamic_data

    {"next_block.block_num()":40903}
    th_a  db_block.cpp:439 _apply_block
1122001ms th_a       witness.cpp:266               block_production_loo ] Got exception while generating block:
3070000 undo_database_exception: undo database exception
The database does not have enough undo history to support a blockchain with so many missed blocks. Please add a checkpoint if you woul
d like to continue applying blocks beyond this point.
    {"recently_missed":44813,"max_undo":1000}
    th_a  db_update.cpp:72 update_global_dynamic_data

    {"next_block.block_num()":40903}
    th_a  db_block.cpp:439 _apply_block

    {"new_block":{"previous":"00009fc6d4e24db66e13af461b94063165999bfa","timestamp":"2015-08-20T10:18:42","witness":"1.6.5247","next_s
ecret_hash":"a151bcbe08eac49feb4eb86a5a68f332b4e7cf9d","previous_secret":"2fa1e05db52f0c421e335648c21bb3707383c1d6","transaction_merkl
e_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"206ea3856d3cf6317f19ab6b5d3af6648409ac092eae9c
c8d943d7357e3ac9c753141733131d2312c4644d44125835afce24190d295c21a6a620920ab660ec5f5c","transactions":[]}}
    th_a  db_block.cpp:176 _push_block

    {"witness_id":"1.6.5247"}
    th_a  db_block.cpp:315 _generate_block



I'm stuck at 40902 now. Looks like the testnet is dead again?

{
  "head_block_num": 40902,
  "head_block_id": "00009fc6d4e24db66e13af461b94063165999bfa",
  "head_block_age": "4 minutes old",
  "next_maintenance_time": "31 seconds ago",
  "chain_id": "5508f5f743717fe2c78445364f62a72badd7532974d26f089af2062228b532eb",
  "active_witnesses": [
    "1.6.0",
    "1.6.1",
    "1.6.2",
    "1.6.1525",
    "1.6.1536",
    "1.6.4435",
    "1.6.5245",
    "1.6.5246",
    "1.6.5247",
    "1.6.5248",
    "1.6.5249"
  ],
  "active_committee_members": [
    "1.5.0",
    "1.5.1",
    "1.5.2",
    "1.5.3",
    "1.5.4",
    "1.5.5",
    "1.5.6",
    "1.5.7",
    "1.5.8",
    "1.5.9"
  ],
  "entropy": "8927f3f36c4599a6a7af5a3cdfb70a8f57d463ae"
}

2015-08-20T07:11:59 th_a:invoke handle_block         handle_block ] Got block #40902 from network                       application.cp
p:343
2015-08-20T07:12:00 th_a:Witness Block Production block_production_loo ] slot: 1 scheduled_witness: 1.6.4435 scheduled_time: 2015-08-2
0T07:12:00 now: 2015-08-20T07:12:00                       witness.cpp:240
2015-08-20T07:12:01 th_a:Witness Block Production block_production_loo ] slot: 2 scheduled_witness: 1.6.5245 scheduled_time: 2015-08-2
0T07:12:01 now: 2015-08-20T07:12:01                       witness.cpp:240
2015-08-20T07:12:02 th_a:Witness Block Production block_production_loo ] slot: 3 scheduled_witness: 1.6.1 scheduled_time: 2015-08-20T0
7:12:02 now: 2015-08-20T07:12:02                  witness.cpp:240
2015-08-20T07:12:03 th_a:Witness Block Production block_production_loo ] slot: 4 scheduled_witness: 1.6.5247 scheduled_time: 2015-08-2
0T07:12:03 now: 2015-08-20T07:12:03                       witness.cpp:240
2015-08-20T07:12:03 th_a:Witness Block Production block_production_loo ] Witness 1.6.5247 production slot has arrived; generating a bl
ock now...                        witness.cpp:243
2015-08-20T07:12:03 th_a:Witness Block Production          _push_block ] Failed to push new block:
3070000 undo_database_exception: undo database exception
The database does not have enough undo history to support a blockchain with so many missed blocks. Please add a checkpoint if you would like to continue applying blocks beyond this point.
    {"recently_missed":1006,"max_undo":1000}
    th_a  db_update.cpp:72 update_global_dynamic_data

    {"next_block.block_num()":40903}
    th_a  db_block.cpp:439 _apply_block                 db_block.cpp:170


Happy forking 
After forked, if restart without --resync-blockchain, then my node won't sync.
If restart with --resync-blockchain, it won't produce block after in sync.

So it's best remove the data directory before start.

I'm online again now.

Quote from: puppies on August 19, 2015, 08:26:31 pm

Quote from: betax on August 19, 2015, 08:24:02 pm

Here we are...

Awesome, how did you build?

Using the graphene-ui build instructions in github.
As I am using ubuntu in a vps I changed server.js to use 0.0.0.0, also to use prod.
My witnesses uses for rpc 0.0.0.0 instead localhost / 127.0.0.1

Do you have a URL so others could access the GUI/webwallet?

My node is online now. Witness id 1.6.5247.
But number of active witnesses is still 11?
{
  "head_block_num": 11773,
  "head_block_id": "00002dfdab35043defebc42a6ae9d5a96767fd97",
  "head_block_age": "1 second old",
  "next_maintenance_time": "82 seconds in the future",
  "chain_id": "5508f5f743717fe2c78445364f62a72badd7532974d26f089af2062228b532eb",
  "active_witnesses": [
    "1.6.0",
    "1.6.1",
    "1.6.2",
    "1.6.3",
    "1.6.4",
    "1.6.5",
    "1.6.6",
    "1.6.1525",
    "1.6.4435",
    "1.6.5246",
    "1.6.5247"
  ],
  "active_committee_members": [
    "1.5.0",
    "1.5.1",
    "1.5.2",
    "1.5.3",
    "1.5.4",
    "1.5.5",
    "1.5.6",
    "1.5.7",
    "1.5.8",
    "1.5.9"
  ],
  "entropy": "73f47170212775453a20de79b10d927e9d3b761d"
}

//Edit: it's 11, not 10

Quote from: abit on August 19, 2015, 12:53:53 pm

Is this command useful? @Riverhead

Code: [Select]

rm -rf object_database*


A good suggestion but it did not take me off 9c. It did however widen my scope outside of the data directory for my reset script so thanks for that .

Is your wallet launched with this command?
./cli_wallet -w wallet-testnet-puppies.json --chain-id 081401ede64c8fe30b23c91d7ab8750103acb1a39548a866fb562f2edf4627d6


Quote from: betax on August 19, 2015, 10:44:46 am

I'm there and producing.
Try deleting your witness data directory (i called mine something like testnet_puppies), and restart, that might help.

You are correct, there are not many people at the moment,  consider this as the trial and error setting yourself up before the big testnet.

Got it resynced an am producing blocks again. One of my remaining questions is:
Does a NAT-Situation hinder smooth operation? I can set P2P-Bind in config, is there such thing as a public address? (It's clear that running a node "at home" is not for production use but this was my first try - will prepare a VPS for next version) As I see nodes being connected on my p2p-incoming port I assume it somehow works anyway but I would like to get a better understanding...

Is there a wallet-command for listing Nodes in detail? I did netstat to get an overall status...

Another way to specify incoming port is adding parameter --rpc-endpoint 0.0.0.0:your_port.
Doesn't like bitshares_client, the witness node doesn't enable UPNP by default. I'm currently running at home as well, I have to manually configure NAT port forwarding in my router and it works, I can see incoming connection in p2p.log.

I have this message (now solved but I think I should report)

Code: [Select]

2071000ms th_a       witness.cpp:242               block_production_loo ] Witness 1.6.1446 p
roduction slot has arrived; generating a block now...
2071002ms th_a       db_block.cpp:167              _push_block          ] Failed to push new
 block:
3070000 undo_database_exception: undo database exception
The database does not have enough undo history to support a blockchain with so many missed b
locks. Please add a checkpoint if you would like to continue applying blocks beyond this poi
nt.
    {"recently_missed":1431,"max_undo":1000}
    th_a  db_update.cpp:68 update_global_dynamic_data
    {"next_block.block_num()":150301}
    th_a  db_block.cpp:448 _apply_block
2071003ms th_a       witness.cpp:265               block_production_loo ] Got exception whil
e generating block:
3070000 undo_database_exception: undo database exception
The database does not have enough undo history to support a blockchain with so many missed b
locks. Please add a checkpoint if you would like to continue applying blocks beyond this poi
nt.
    {"recently_missed":1431,"max_undo":1000}
    th_a  db_update.cpp:68 update_global_dynamic_data
    {"next_block.block_num()":150301}
    th_a  db_block.cpp:448 _apply_block
    {"new_block":{"previous":"00024b1c3d09ca1f5830e3e5afa94ad35ef5bd8e","timestamp":"2015-08
-19T03:34:31","witness":"1.6.1446","next_secret_hash":"2618a50760983d98ab43df78cd9a007d2e7ea
ed7","previous_secret":"e93b540ed63a628df6ef29be551fd9354dac7bed","transaction_merkle_root":
"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"20767d9f7cf2
cf5c83d2540b22943e052a5b0d0bb22af5485462a72d330efe82703a5fe6cef4276f0b5b3f76ea419d87901cefef
89c5878ec99add42aa24ab7071","transactions":[]}}
    th_a  db_block.cpp:173 _push_block
    {"witness_id":"1.6.1446"}
    th_a  db_block.cpp:312 _generate_block

The following is what I have done
- Start witness_node and sync blockchain
- Vote for 1.6.1446, close cli_wallet
- Unclean shutdown (Ctrl+C, Ctrl+C)
- Delete object_id(?) and witness_node dir
- Start witness_node
- After syncing, this problem begins to happen

Looks like you're on a fork. Try resync. Don't add parameter '--enable-stale-production'.

//Edit: Ah, just saw you've solved.