My nodes (in.abit and an observer node) are running well. Maybe have been on a fork but switched back successfully.
Updated to latest commit. Switched to another process by command 'update_witness'.

What is the cause of th is fork? IMO it should be dealt seriously..

Quote from: Bhuz on October 07, 2015, 10:40:50 PM
Restarted the witness with --resync, back to normal.

--resync worked great, thanks

Quote from: Spectral on October 08, 2015, 12:24:48 AM

Quote from: liondani on October 08, 2015, 12:05:56 AM
can you post your config.ini entrys here (except your keys of-course  :))

sure:

Code Select Expand


# Endpoint for P2P node to listen on
# p2p-endpoint =

# P2P nodes to connect to on startup (may specify multiple times)
# seed-node =

# Pairs of [BLOCK_NUM,BLOCK_ID] that should be enforced as checkpoints.
# checkpoint =

# Endpoint for websocket RPC to listen on
# rpc-endpoint =

# Endpoint for TLS websocket RPC to listen on
# rpc-tls-endpoint =

# The TLS certificate file for this server
# server-pem =

# Password for this certificate
# server-pem-password =

# File to read Genesis State from
# genesis-json =

# Block signing key to use for init witnesses, overrides genesis file
# dbg-init-key =

# JSON file specifying API permissions
# api-access =

# Enable block production, even if the chain is stale.
enable-stale-production = false

# Percent of witnesses (0-99) that must be participating in order to produce blocks
required-participation = false

# ID of witness controlled by this node (e.g. "1.6.5", quotes are required, may specify multiple times)
witness-id = "1.6.38"

# Tuple of [PublicKey, WIF private key] (may specify multiple times)
private-key = [""]

# Account ID to track history for (may specify multiple times)
# track-account =

# Track market history by grouping orders into buckets of equal size measured in seconds specified as a JSON array of numbers
bucket-size = [15,60,300,3600,86400]

# How far back in time to track history for each bucket size, measured in the number of buckets (default: 1000)
history-per-size = 1000

# declare an appender named "stderr" that writes messages to the console
[log.console_appender.stderr]
stream=std_error

# declare an appender named "p2p" that writes messages to p2p.log
[log.file_appender.p2p]
filename=logs/p2p/p2p.log
# filename can be absolute or relative to this config file

# route any messages logged to the default logger to the "stderr" logger we
# declared above, if they are info level are higher
[logger.default]
level=info
appenders=stderr

# route messages sent to the "p2p" logger to the p2p appender declared above
[logger.p2p]
level=debug
appenders=p2p


To start the witness_node I use the following command:

Code Select Expand


./witness_node --rpc-endpoint "127.0.0.1:8090"  --genesis-json oct5-genesis.json -d oct5 -s "104.236.51.238:2005"


what commands did you use to update graphene today?

Quote from: liondani on October 08, 2015, 12:05:56 AM
can you post your config.ini entrys here (except your keys of-course  :))

sure:


# Endpoint for P2P node to listen on
# p2p-endpoint =

# P2P nodes to connect to on startup (may specify multiple times)
# seed-node =

# Pairs of [BLOCK_NUM,BLOCK_ID] that should be enforced as checkpoints.
# checkpoint =

# Endpoint for websocket RPC to listen on
# rpc-endpoint =

# Endpoint for TLS websocket RPC to listen on
# rpc-tls-endpoint =

# The TLS certificate file for this server
# server-pem =

# Password for this certificate
# server-pem-password =

# File to read Genesis State from
# genesis-json =

# Block signing key to use for init witnesses, overrides genesis file
# dbg-init-key =

# JSON file specifying API permissions
# api-access =

# Enable block production, even if the chain is stale.
enable-stale-production = false

# Percent of witnesses (0-99) that must be participating in order to produce blocks
required-participation = false

# ID of witness controlled by this node (e.g. "1.6.5", quotes are required, may specify multiple times)
witness-id = "1.6.38"

# Tuple of [PublicKey, WIF private key] (may specify multiple times)
private-key = [""]

# Account ID to track history for (may specify multiple times)
# track-account =

# Track market history by grouping orders into buckets of equal size measured in seconds specified as a JSON array of numbers
bucket-size = [15,60,300,3600,86400]

# How far back in time to track history for each bucket size, measured in the number of buckets (default: 1000)
history-per-size = 1000

# declare an appender named "stderr" that writes messages to the console
[log.console_appender.stderr]
stream=std_error

# declare an appender named "p2p" that writes messages to p2p.log
[log.file_appender.p2p]
filename=logs/p2p/p2p.log
# filename can be absolute or relative to this config file

# route any messages logged to the default logger to the "stderr" logger we
# declared above, if they are info level are higher
[logger.default]
level=info
appenders=stderr

# route messages sent to the "p2p" logger to the p2p appender declared above
[logger.p2p]
level=debug
appenders=p2p


To start the witness_node I use the following command:


./witness_node --rpc-endpoint "127.0.0.1:8090"  --genesis-json oct5-genesis.json -d oct5 -s "104.236.51.238:2005"


I'm one of the node out of sync. and both of my two nodes are out of sync from block 63389.
here is something from log.
got block 63389 correct

  2015-10-07T22:08:15 p2p:message read_loop process_block_during ] received a block from peer 108.61.173.111:50696, passing it to client      node.cpp:3290
a>2015-10-07T22:08:15 p2p:message read_loop process_block_during ] Successfully pushed block 63389 (id:0000f79d286062943a634c0fc401d4204fa2187b)      node.cpp:3312     
  2015-10-07T22:08:15 p2p:message read_loop process_block_during ] client validated the block, advertising it to other peers      node.cpp:3336

get block 63390 correct, but failed to  push it.

2015-10-07T22:08:24 p2p:message read_loop process_block_during ] received a block from peer 52.89.36.4:39219, passing it to client      node.cpp:3290
b>2015-10-07T22:08:24 p2p:message read_loop process_block_during ] Failed to push block 63390 (id:0000f79e741dd849af4e69a62a425f11497ce433), client rejected block sent
  by peer     node.cpp:3404                                                                                                                                             
  2015-10-07T22:08:24 p2p:message read_loop process_block_during ] disconnecting client 52.89.36.4:39219 because it offered us the rejected block     node.cpp:3426


2015-10-07T22:08:30 p2p:send_sync_block_to_node_delegate send_sync_block_to_n ] Failed to push sync block 63390 (id:0000f79e741dd849af4e69a62a425f11497ce433): client   rejected sync block sent by peer: {"code":10,"name":"assert_exception","message":"Assert Exception","stack":[{"context":{"level":"error","file":"object_database.cpp",  "line":51,"method":"get_index","hostname":"","thread_name":"th_a","timestamp":"2015-10-07T22:08:30"},"format":"_index[space_id].size() > type_id: ","data":{"space_id"  :0,"type_id":0,"index[space_id].size":0}},{"context":{"level":"warn","file":"db_block.cpp","line":511,"method":"_apply_block","hostname":"","thread_name":"th_a","time  stamp":"2015-10-07T22:08:30"},"format":"","data":{"next_block.block_num()":63390}},{"context":{"level":"warn","file":"db_block.cpp","line":197,"method":"_push_block",  "hostname":"","thread_name":"th_a","timestamp":"2015-10-07T22:08:30"},"format":"","data":{"new_block":{"previous":"0000f79d286062943a634c0fc401d4204fa2187b","timestam  p":"2015-10-07T22:08:24","witness":"1.6.33","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"202007dd6bd3169d  71387c0c9d2f9feaa441ee291eff70d33d02d1d5b2de45f3c41502af9bb41773fd20b7f73aa8daee842cacf9d6501b0bb9bc397a8fad61ecb4","transactions":[]}}},{"context":{"level":"warn","f  ile":"application.cpp","line":434,"method":"handle_block","hostname":"","thread_name":"th_a","timestamp":"2015-10-07T22:08:30"},"format":"","data":{"blk_msg":{"block"  :{"previous":"0000f79d286062943a634c0fc401d4204fa2187b","timestamp":"2015-10-07T22:08:24","witness":"1.6.33","transaction_merkle_root":"000000000000000000000000000000  0000000000","extensions":[],"witness_signature":"202007dd6bd3169d71387c0c9d2f9feaa441ee291eff70d33d02d1d5b2de45f3c41502af9bb41773fd20b7f73aa8daee842cacf9d6501b0bb9bc3  97a8fad61ecb4","transactions":[]},"block_id":"0000f79e741dd849af4e69a62a425f11497ce433"},"sync_mode":true}}]}     node.cpp:3017



It synced after I delete the chain directory and restart node.

Quote from: ElMato on October 07, 2015, 10:37:44 PM
It seems that a lot of witnesses that were working ok during the flood are now out of sync.
https://graphene.bitshares.org/#/explorer/witnesses

Quote from: Spectral on October 07, 2015, 11:47:02 PM
@liondani:
How long have you waited. If I remember correctly, that output is normal at startup...? Also, have you remembered to put your config file back in the oct5 folder after resyncing the blockchain, and then restarting your witness?

more than half hour.
yes I have put back the config.ini file

can you post your config.ini entrys here (except your keys of-course  :))
to compare them with mine

witness spectral updated to latest master


Just to make sure that I'm "cmaking" correctly. I use the following line. Is that what everyone else does?
cmake -DBOOST_ROOT="$BOOST_ROOT" -DCMAKE_BUILD_TYPE=Release .

Quote from: theoretical on October 07, 2015, 02:54:22 PM
For now, use your account history at https://graphene.bitshares.org/#/account/spectral/overview

Thank you for your answers. They were very informative. And I have to say, the graphene web-interface looks really good!

The cli-wallet shows only what my client believes is correct, and cannot be trusted when I'm off on a fork or something. And I still get those memo decrypt errors that I documented a few pages back.


@liondani:
How long have you waited. If I remember correctly, that output is normal at startup...? Also, have you remembered to put your config file back in the oct5 folder after resyncing the blockchain, and then restarting your witness?

Quote from: Bhuz on October 07, 2015, 11:11:41 PM

Quote from: liondani on October 07, 2015, 11:07:31 PM

Code Select Expand


./witness_node -d oct5 --genesis-json oct5-genesis.json -s "104.236.51.238:2005" --resync


try --resync-blockchain

Edit: and look, on the cli_wallet, the sync progression with "info". we are at block #64300 ish

same result  :'(

Quote from: liondani on October 07, 2015, 11:07:31 PM

Code Select Expand


./witness_node -d oct5 --genesis-json oct5-genesis.json -s "104.236.51.238:2005" --resync


try --resync-blockchain

Edit: and look, on the cli_wallet, the sync progression with "info". we are at block #64300 ish

I get this all the time without stopping.
I have deleted folders like "oct5", object_database",
I tried to --resync

./witness_node -d oct5 --genesis-json oct5-genesis.json -s "104.236.51.238:2005" --resync

I reinstalled graphene
but the same output   >:(


312984ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["00001c34043f8cd77a774029c3297b709b9e5055"]
313603ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["00001c6a27a3c5fc9740ac594df500c4d48ebb04"]
314088ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["00001c7e18d092590a343d135490890111fea2c1"]
315198ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["00001cb60b30e0ebfc734ca945560ffd4b280e45"]
316069ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["00001f69c32224c394502ea4ae13e12d1ba91c02"]
316642ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["00002199ee5fb42c3c10a56f80e20aafa4c6236a"]
319179ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["000022bcbf0a696df8916d0477af2df6639c8714","000022c5e914910801e93f9a74d0c3588e247eab","000022c90cb122db11a6fe6b7dc1720af6ef4dca","000022cb5d94abf4b0699cdb50fee429f177cdc5","000022ccff1d23f9b1b9df19e5ae8412ff4d9a0e"]
319184ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["000022bcbf0a696df8916d0477af2df6639c8714","000022c5e914910801e93f9a74d0c3588e247eab","000022c90cb122db11a6fe6b7dc1720af6ef4dca","000022cb5d94abf4b0699cdb50fee429f177cdc5","000022ccff1d23f9b1b9df19e5ae8412ff4d9a0e"]
319189ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["000022bcbf0a696df8916d0477af2df6639c8714","000022c5e914910801e93f9a74d0c3588e247eab","000022c90cb122db11a6fe6b7dc1720af6ef4dca","000022cb5d94abf4b0699cdb50fee429f177cdc5","000022ccff1d23f9b1b9df19e5ae8412ff4d9a0e"]
319209ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["000022bcbf0a696df8916d0477af2df6639c8714","000022c5e914910801e93f9a74d0c3588e247eab","000022c90cb122db11a6fe6b7dc1720af6ef4dca","000022cb5d94abf4b0699cdb50fee429f177cdc5","000022ccff1d23f9b1b9df19e5ae8412ff4d9a0e"]
319250ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["000022bcbf0a696df8916d0477af2df6639c8714","000022c5e914910801e93f9a74d0c3588e247eab","000022c90cb122db11a6fe6b7dc1720af6ef4dca","000022cb5d94abf4b0699cdb50fee429f177cdc5","000022ccff1d23f9b1b9df19e5ae8412ff4d9a0e"]
319314ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["000022bcbf0a696df8916d0477af2df6639c8714","000022c5e914910801e93f9a74d0c3588e247eab","000022c90cb122db11a6fe6b7dc1720af6ef4dca","000022cb5d94abf4b0699cdb50fee429f177cdc5","000022ccff1d23f9b1b9df19e5ae8412ff4d9a0e"]
320023ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["000022bcbf0a696df8916d0477af2df6639c8714"]
320068ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["000022bcbf0a696df8916d0477af2df6639c8714"]
320216ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["000022bcbf0a696df8916d0477af2df6639c8714"]
320346ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["000022bcbf0a696df8916d0477af2df6639c8714","000022c5e914910801e93f9a74d0c3588e247eab","000022c90cb122db11a6fe6b7dc1720af6ef4dca","000022cb5d94abf4b0699cdb50fee429f177cdc5","000022ccff1d23f9b1b9df19e5ae8412ff4d9a0e"]
320390ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["000022bcbf0a696df8916d0477af2df6639c8714"]
320563ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["000022bcbf0a696df8916d0477af2df6639c8714"]
320564ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["000022bcbf0a696df8916d0477af2df6639c8714","000022c5e914910801e93f9a74d0c3588e247eab","000022c90cb122db11a6fe6b7dc1720af6ef4dca","000022cb5d94abf4b0699cdb50fee429f177cdc5","000022ccff1d23f9b1b9df19e5ae8412ff4d9a0e"]
320565ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["000022bcbf0a696df8916d0477af2df6639c8714"]
320579ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["000022bcbf0a696df8916d0477af2df6639c8714","000022c5e914910801e93f9a74d0c3588e247eab","000022c90cb122db11a6fe6b7dc1720af6ef4dca","000022cb5d94abf4b0699cdb50fee429f177cdc5","000022ccff1d23f9b1b9df19e5ae8412ff4d9a0e"]
320929ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["000022bcbf0a696df8916d0477af2df6639c8714"]
321017ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["000022bcbf0a696df8916d0477af2df6639c8714","000022c5e914910801e93f9a74d0c3588e247eab","000022c90cb122db11a6fe6b7dc1720af6ef4dca","000022cb5d94abf4b0699cdb50fee429f177cdc5","000022ccff1d23f9b1b9df19e5ae8412ff4d9a0e"]
321240ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["000022bcbf0a696df8916d0477af2df6639c8714","000022c5e914910801e93f9a74d0c3588e247eab","000022c90cb122db11a6fe6b7dc1720af6ef4dca","000022cb5d94abf4b0699cdb50fee429f177cdc5","000022ccff1d23f9b1b9df19e5ae8412ff4d9a0e"]
321280ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["000022bcbf0a696df8916d0477af2df6639c8714"]
321359ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["000022bcbf0a696df8916d0477af2df6639c8714"]
321364ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["000022bcbf0a696df8916d0477af2df6639c8714"]
321372ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["000022bcbf0a696df8916d0477af2df6639c8714","000022c5e914910801e93f9a74d0c3588e247eab","000022c90cb122db11a6fe6b7dc1720af6ef4dca","000022cb5d94abf4b0699cdb50fee429f177cdc5","000022ccff1d23f9b1b9df19e5ae8412ff4d9a0e"]
321844ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["000022bcbf0a696df8916d0477af2df6639c8714","000022c5e914910801e93f9a74d0c3588e247eab","000022c90cb122db11a6fe6b7dc1720af6ef4dca","000022cb5d94abf4b0699cdb50fee429f177cdc5","000022ccff1d23f9b1b9df19e5ae8412ff4d9a0e"]
321937ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["000022bcbf0a696df8916d0477af2df6639c8714"]
322070ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["000022bcbf0a696df8916d0477af2df6639c8714"]
322718ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["000022bcbf0a696df8916d0477af2df6639c8714"]
322733ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["000022bcbf0a696df8916d0477af2df6639c8714"]
322762ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["000022bcbf0a696df8916d0477af2df6639c8714"]
322839ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["000022bcbf0a696df8916d0477af2df6639c8714"]
323110ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["000022bcbf0a696df8916d0477af2df6639c8714"]
323386ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["000022bcbf0a696df8916d0477af2df6639c8714"]
323405ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["000022bcbf0a696df8916d0477af2df6639c8714"]
323793ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["000022bcbf0a696df8916d0477af2df6639c8714"]
323851ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["000022bcbf0a696df8916d0477af2df6639c8714"]
324085ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["000022d498d16481c794a4643e7f2a7e0acefd64"]
324449ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["0000233f874bacc359f2a54fd47d64f28a6a7401"]
325033ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["000023463acb5ea5775b135e43c5d2e5162e2f69"]
325793ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["0000234903cb2307e8f0496992202f55e9e1373a"]
326656ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["000024f6f5f2b206490a11269f2a1ce0dfaf48f8"]
327512ms th_a       application.cpp:705           get_blockchain_synop ] synopsis: ["000024f6f5f2b206490a11269f2a1ce0dfaf48f8"]


PS I have this problems on my server @home not on the vps located in Germany.... Could it be because of latency problems? Is there some kind of latency limit to the witness node to work?

Restarted the witness with --resync, back to normal.

It seems that a lot of witnesses that were working ok during the flood are now out of sync.
https://graphene.bitshares.org/#/explorer/witnesses

Quote from: ElMato on October 07, 2015, 10:30:47 PM
It seems we both are stuck on the same block.

Quote
info
{
  "head_block_num": 63389,
  "head_block_id": "0000f79d286062943a634c0fc401d4204fa2187b",
  "head_block_age": "22 minutes old",
  "next_maintenance_time": "30 minutes in the future",
  "chain_id": "60e21871125ea9995fe498b7f68a87a85c6583725ea5448f6fd969c59a37df83",
  "participation": "93.75000000000000000",
  "active_witnesses": [

Also spartako_bot stuck on the same block and was not able to notify the missed blocks...now it is restored

What should we do?
Restart the witness with --resync-blockchain?