Looks like the master node with the init witnesses is stuck (probably because Ben hasn't been around to update it with all the latest changes)

Code: [Select]

2931434ms th_a       application.cpp:688           get_blockchain_synop ] synopsis: ["000326d238e7ae19d2c3835531c0ba5ae8024b78"]
2931786ms th_a       db_block.cpp:105              push_block           ] new_block.block_num(): 206547 new_block.id(): 000326d32d03860502c20f80cfba7188a58902cd new_block.witness: 1.6.28 new_block.timestamp: 2015-09-15T12:19:15
2931787ms th_a       db_block.cpp:105              push_block           ] new_block.block_num(): 206548 new_block.id(): 000326d418f9954d1a053be245d67abcc74fb333 new_block.witness: 1.6.42 new_block.timestamp: 2015-09-15T12:19:25
2931787ms th_a       db_block.cpp:105              push_block           ] new_block.block_num(): 206549 new_block.id(): 000326d5b6eeec78e0cf9c0a6f4b7c495150568e new_block.witness: 1.6.24 new_block.timestamp: 2015-09-15T12:19:30
2931788ms th_a       db_block.cpp:105              push_block           ] new_block.block_num(): 206550 new_block.id(): 000326d66e59c5d4d44f3d7e24bc8202b8fc822d new_block.witness: 1.6.13 new_block.timestamp: 2015-09-15T12:19:35
2931788ms th_a       db_block.cpp:105              push_block           ] new_block.block_num(): 206551 new_block.id(): 000326d7acf81a715dc0d1f3ddd037ed54f36ab5 new_block.witness: 1.6.58 new_block.timestamp: 2015-09-15T12:19:40
2931789ms th_a       db_block.cpp:105              push_block           ] new_block.block_num(): 206552 new_block.id(): 000326d8568bc6fa597261862a7186dcd8a98667 new_block.witness: 1.6.69 new_block.timestamp: 2015-09-15T12:19:45
2931790ms th_a       db_block.cpp:105              push_block           ] new_block.block_num(): 206553 new_block.id(): 000326d9a89fbb94f98652b537b6858d3a07269d new_block.witness: 1.6.68 new_block.timestamp: 2015-09-15T12:19:50
2931790ms th_a       db_block.cpp:105              push_block           ] new_block.block_num(): 206554 new_block.id(): 000326da9c0f44e5ff5ea5ee4019a0a1ccdb3f66 new_block.witness: 1.6.39 new_block.timestamp: 2015-09-15T12:19:55
2931791ms th_a       db_block.cpp:105              push_block           ] new_block.block_num(): 206555 new_block.id(): 000326dbba7f985cd09b8b9b2492d14bc7e18ddb new_block.witness: 1.6.51 new_block.timestamp: 2015-09-15T12:20:00
2931847ms th_a       db_block.cpp:105              push_block           ] new_block.block_num(): 206556 new_block.id(): 000326dce6e9ebaf766cae6139a539a731893c9e new_block.witness: 1.6.5252 new_block.timestamp: 2015-09-15T12:20:25
2931847ms th_a       db_block.cpp:105              push_block           ] new_block.block_num(): 206557 new_block.id(): 000326dd6dd791b001061dff9002e71ad2e3a85f new_block.witness: 1.6.1063 new_block.timestamp: 2015-09-15T12:20:30
2931848ms th_a       db_block.cpp:105              push_block           ] new_block.block_num(): 206558 new_block.id(): 000326dee3d9a84169d0bdf5f0b46a459a768795 new_block.witness: 1.6.1538 new_block.timestamp: 2015-09-15T12:27:50
2931902ms th_a       db_block.cpp:105              push_block           ] new_block.block_num(): 206559 new_block.id(): 000326df5f1d8789b1d4be4fcc709f488736b3f0 new_block.witness: 1.6.1527 new_block.timestamp: 2015-09-15T12:30:45
2931958ms th_a       db_block.cpp:105              push_block           ] new_block.block_num(): 206560 new_block.id(): 000326e05ee43c32b52bd4924d9ef846da414ff3 new_block.witness: 1.6.4232 new_block.timestamp: 2015-09-15T12:32:15
2931959ms th_a       db_block.cpp:105              push_block           ] new_block.block_num(): 206561 new_block.id(): 000326e15aad53e16fae3eaeb3f904dccbeea335 new_block.witness: 1.6.3968 new_block.timestamp: 2015-09-15T12:32:25
2931959ms th_a       db_block.cpp:105              push_block           ] new_block.block_num(): 206562 new_block.id(): 000326e27314e948f62035d14f5dbcf9774bb2e7 new_block.witness: 1.6.1527 new_block.timestamp: 2015-09-15T12:36:25
2932021ms th_a       db_block.cpp:105              push_block           ] new_block.block_num(): 206563 new_block.id(): 000326e3b3a2f68bf1004ac6360033b34bdff358 new_block.witness: 1.6.1538 new_block.timestamp: 2015-09-15T12:37:15
2932021ms th_a       db_block.cpp:105              push_block           ] new_block.block_num(): 206564 new_block.id(): 000326e40425207efe4ef960af42e0e918fb77c5 new_block.witness: 1.6.1063 new_block.timestamp: 2015-09-15T12:37:35
2932022ms th_a       db_block.cpp:105              push_block           ] new_block.block_num(): 206565 new_block.id(): 000326e5289b7f016a6903f87bf36ea6adac931c new_block.witness: 1.6.4232 new_block.timestamp: 2015-09-15T12:37:45
2932022ms th_a       db_block.cpp:105              push_block           ] new_block.block_num(): 206566 new_block.id(): 000326e65b9d1cec3c728ac2ee653925da819a1d new_block.witness: 1.6.5247 new_block.timestamp: 2015-09-15T12:39:55
2932022ms th_a       db_block.cpp:105              push_block           ] new_block.block_num(): 206567 new_block.id(): 000326e79d32f42bb6d06efb780a4a15b9b946a4 new_block.witness: 1.6.3968 new_block.timestamp: 2015-09-15T12:40:25
2932081ms th_a       application.cpp:688           get_blockchain_synop ] synopsis: ["000326d238e7ae19d2c3835531c0ba5ae8024b78","000326dd6dd791b001061dff9002e71ad2e3a85f","000326e3b3a2f68bf1004ac6360033b34bdff358","000326e65b9d1cec3c728ac2ee653925da819a1d","000326e79d32f42bb6d06efb780a4a15b9b946a4"]
It looks like everyone who has upgraded is still producing blocks when you get the chance. 

When Ben gets in today I will have him update the main node.

247300ms th_a       db_block.cpp:181              _push_block          ] Failed to push new block:
3070000 undo_database_exception: undo database exception
The database does not have enough undo history to support a blockchain with so many missed blocks. Please add a checkpoint if you would like to continue applying blocks beyond this point.
    {"recently_missed":13547,"max_undo":1000}
    th_a  db_update.cpp:72 update_global_dynamic_data

    {"next_block.block_num()":206568}
    th_a  db_block.cpp:469 _apply_block

We may need a new chain if the master node missed too many blocks.

Crashed on commit f0f12076fa5d80e07758ae5104854efd1dcb748d
745500ms th_a       application.cpp:383           handle_block         ] Got block #199649 from network
745500ms th_a       db_block.cpp:98               push_block           ] new_block.block_num(): 199649 new_block.id(): 00030be11aae3a4891869a7ad9aab215e93aee36747089ms th_a       application.cpp:683           get_blockchain_synop ] synopsis: ["00030b6c048c2452ac552e476f913329d818074a","00030ba7eec13a91d97908a50549d874681ea35a","00030bc587cc15cfdf07f10a72804724cb51217d","00030bd4128e9547c6771b80e5ffff1fdeec37a4","00030bdba291f319d7f0f62ec67394ef613ca8a5","00030bdf303a78916851613360a845e22e0041d8","00030be11aae3a4891869a7ad9aab215e93aee36"]
749560ms th_a       application.cpp:683           get_blockchain_synop ] synopsis: ["00030b6c048c2452ac552e476f913329d818074a","00030ba7eec13a91d97908a50549d874681ea35a","00030bc587cc15cfdf07f10a72804724cb51217d","00030bd4128e9547c6771b80e5ffff1fdeec37a4","00030bdba291f319d7f0f62ec67394ef613ca8a5","00030bdf303a78916851613360a845e22e0041d8","00030be11aae3a4891869a7ad9aab215e93aee36"]
752624ms th_a       application.cpp:683           get_blockchain_synop ] synopsis: ["00030b6c048c2452ac552e476f913329d818074a","00030ba7eec13a91d97908a50549d874681ea35a","00030bc587cc15cfdf07f10a72804724cb51217d","00030bd4128e9547c6771b80e5ffff1fdeec37a4","00030bdba291f319d7f0f62ec67394ef613ca8a5","00030bdf303a78916851613360a845e22e0041d8","00030be11aae3a4891869a7ad9aab215e93aee36"]
752881ms th_a       application.cpp:683           get_blockchain_synop ] synopsis: ["00030b6c048c2452ac552e476f913329d818074a","00030ba7eec13a91d97908a50549d874681ea35a","00030bc587cc15cfdf07f10a72804724cb51217d","00030bd4128e9547c6771b80e5ffff1fdeec37a4","00030bdba291f319d7f0f62ec67394ef613ca8a5","00030bdf303a78916851613360a845e22e0041d8","00030be11aae3a4891869a7ad9aab215e93aee36"]
witness_node: /app/bts/graphene-test2b.7/libraries/app/application.cpp:624: virtual std::vector<fc::ripemd160> graphene::app::detail::application_impl::get_blockchain_synopsis(const item_hash_t&, uint32_t): Assertion `fork_history.back() == reference_point' failed.

Program received signal SIGABRT, Aborted.
0x00007ffff6c01cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56      ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.

(gdb) bt
#0  0x00007ffff6c01cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007ffff6c050d8 in __GI_abort () at abort.c:89
#2  0x00007ffff6bfab86 in __assert_fail_base (fmt=0x7ffff6d4b830 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
    assertion=assertion@entry=0x2aa0530 "fork_history.back() == reference_point",
    file=file@entry=0x2a9fd58 "/app/bts/graphene-test2b.7/libraries/app/application.cpp", line=line@entry=624,
    function=function@entry=0x2aa5f00 <graphene::app::detail::application_impl::get_blockchain_synopsis(fc::ripemd160 const&, unsigned int)::__PRETTY_FUNCTION__> "virtual std::vector<fc::ripemd160> graphene::app::detail::application_impl::get_blockchain_synopsis(const item_hash_t&, uint32_t)") at assert.c:92
#3  0x00007ffff6bfac32 in __GI___assert_fail (assertion=0x2aa0530 "fork_history.back() == reference_point",
    file=0x2a9fd58 "/app/bts/graphene-test2b.7/libraries/app/application.cpp", line=624,
    function=0x2aa5f00 <graphene::app::detail::application_impl::get_blockchain_synopsis(fc::ripemd160 const&, unsigned int)::__PRETTY_FUNCTION__> "virtual std::vector<fc::ripemd160> graphene::app::detail::application_impl::get_blockchain_synopsis(const item_hash_t&, uint32_t)") at assert.c:101
#4  0x0000000001fd088e in graphene::app::detail::application_impl::get_blockchain_synopsis (this=0x3318140, reference_point=...,
    number_of_blocks_after_reference_point=0) at /app/bts/graphene-test2b.7/libraries/app/application.cpp:624
#5  0x000000000284d3a3 in graphene::net::detail::statistics_gathering_node_delegate_wrapper::__lambda59::operator() (
    __closure=0x7fffe411d868) at /app/bts/graphene-test2b.7/libraries/net/node.cpp:5296
#6  0x000000000285d444 in fc::detail::functor_run<graphene::net::detail::statistics_gathering_node_delegate_wrapper::get_blockchain_synopsis(const item_hash_t&, uint32_t)::__lambda59>::run(void *, void *) (functor=0x7fffe411d868, prom=0x7fffe411d950)
    at /app/bts/graphene-test2b.7/libraries/fc/include/fc/thread/task.hpp:77
#7  0x0000000002628a87 in fc::task_base::run_impl (this=0x7fffe411d888)
    at /app/bts/graphene-test2b.7/libraries/fc/src/thread/task.cpp:43
#8  0x0000000002628a18 in fc::task_base::run (this=0x7fffe411d888) at /app/bts/graphene-test2b.7/libraries/fc/src/thread/task.cpp:32
#9  0x000000000261d438 in fc::thread_d::run_next_task (this=0x334c040)
    at /app/bts/graphene-test2b.7/libraries/fc/src/thread/thread_d.hpp:498
#10 0x000000000261d8dc in fc::thread_d::process_tasks (this=0x334c040)
    at /app/bts/graphene-test2b.7/libraries/fc/src/thread/thread_d.hpp:547
#11 0x000000000261cf4f in fc::thread_d::start_process_tasks (my=53788736)
    at /app/bts/graphene-test2b.7/libraries/fc/src/thread/thread_d.hpp:475
#12 0x0000000002975b51 in make_fcontext () at libs/context/src/asm/make_x86_64_sysv_elf_gas.S:64
#13 0x0000000000000000 in ?? ()



witness node stopped with the following error

Code: [Select]

3599870ms th_a       application.cpp:422     
      handle_transaction   ] Got transaction
from network
1ms th_a       db_block.cpp:97               
push_block           ] new_block.block_num():
 172076 new_block.id(): 0002a02c211ed6b4ab4ab
41989d9f0007ad02fef
witness_node: /home/clayop/graphene/libraries
/chain/db_maint.cpp:333: void graphene::chain
::database::process_budget(): Assertion `time
_to_maint > 0' failed.
Aborted (core dumped)

Which commit were you running with? My nodes are unable to go over block 172075, but haven't crash. Looks like your issue is related to block 172076.

Some warnings here. Any thought?
2015-09-12T18:10:33 th_a:invoke handle_transaction   handle_transaction ] Got transaction from network                  application.cp
p:422
2015-09-12T18:10:33 th_a:invoke handle_transaction   handle_transaction ] Got transaction from network                  application.cp
p:422
2015-09-12T18:10:33 th_a:invoke handle_transaction   handle_transaction ] Got transaction from network                  application.cp
p:422
2015-09-12T18:10:36 th_a:invoke handle_block         handle_block ] Got block #171686 from network                      application.cp
p:383
2015-09-12T18:10:36 th_a:invoke handle_block           push_block ] new_block.block_num(): 171686 new_block.id(): 00029ea6553cbe880a9e
0be95c9a0deaef46f1cf                      db_block.cpp:97
2015-09-12T18:10:36 th_a:invoke handle_block ~pending_transaction ] Pending transaction became invalid after switching to block 00029e
a6553cbe880a9e0be95c9a0deaef46f1cf                        db_with.hpp:80
2015-09-12T18:10:36 th_a:invoke handle_block ~pending_transaction ] The invalid pending transaction is {"ref_block_num":40613,"ref_blo
ck_prefix":4073857719,"expiration":"2015-09-12T18:11:00","operations":[[5,{"fee":{"amount":14648,"asset_id":"1.3.0"},"registrar":"1.2.
116","referrer":"1.2.116","referrer_percent":0,"name":"hello-test1","owner":{"weight_threshold":1,"account_auths":[],"key_auths":[["GP
H7PuZQts1QJYkHzXvKA9hbZcww8o9xxmh2DNihXb9onb7jfPXv5",1]],"address_auths":[]},"active":{"weight_threshold":1,"account_auths":[],"key_au
ths":[["GPH5BTF7xh1RzGB9vL8Rvk6kcNr5EC3cRBAC32yyqs5Pas56Gwubz",1]],"address_auths":[]},"options":{"memo_key":"GPH5BTF7xh1RzGB9vL8Rvk6k
cNr5EC3cRBAC32yyqs5Pas56Gwubz","voting_account":"1.2.5","num_witness":0,"num_committee":0,"votes":[],"extensions":[]},"extensions":[]}
]],"extensions":[],"signatures":["1f35d7089c47caa4d3423b9779010495d8058e04ea6b5deeeac8aea9cc4e3d7720698e97dd6eb460ba9f7e8547f18f22173c
fa4ee523bd1cca774f3c68c993d68d"],"operation_results":[[1,"1.2.97867"]]}                       db_with.hpp:81
2015-09-12T18:10:36 th_a:invoke handle_block ~pending_transaction ] The invalid pending transaction caused exception {"code":10,"name"
:"assert_exception","message":"Assert Exception","stack":[{"context":{"level":"error","file":"db_block.cpp","line":507,"method":"_appl
y_transaction","hostname":"","thread_name":"th_a","timestamp":"2015-09-12T18:10:36"},"format":"(skip & skip_transaction_dupe_check) ||
 trx_idx.indices().get<by_trx_id>().find(trx_id) == trx_idx.indices().get<by_trx_id>().end(): ","data":{}},{"context":{"level":"warn",
"file":"db_block.cpp","line":563,"method":"_apply_transaction","hostname":"","thread_name":"th_a","timestamp":"2015-09-12T18:10:36"},"
format":"","data":{"trx":{"ref_block_num":40613,"ref_block_prefix":4073857719,"expiration":"2015-09-12T18:11:00","operations":[[5,{"fe
e":{"amount":14648,"asset_id":"1.3.0"},"registrar":"1.2.116","referrer":"1.2.116","referrer_percent":0,"name":"hello-test1","owner":{"
weight_threshold":1,"account_auths":[],"key_auths":[["GPH7PuZQts1QJYkHzXvKA9hbZcww8o9xxmh2DNihXb9onb7jfPXv5",1]],"address_auths":[]},"
active":{"weight_threshold":1,"account_auths":[],"key_auths":[["GPH5BTF7xh1RzGB9vL8Rvk6kcNr5EC3cRBAC32yyqs5Pas56Gwubz",1]],"address_au
ths":[]},"options":{"memo_key":"GPH5BTF7xh1RzGB9vL8Rvk6kcNr5EC3cRBAC32yyqs5Pas56Gwubz","voting_account":"1.2.5","num_witness":0,"num_c
ommittee":0,"votes":[],"extensions":[]},"extensions":[]}]],"extensions":[],"signatures":["1f35d7089c47caa4d3423b9779010495d8058e04ea6b
5deeeac8aea9cc4e3d7720698e97dd6eb460ba9f7e8547f18f22173cfa4ee523bd1cca774f3c68c993d68d"]}}},{"context":{"level":"warn","file":"db_bloc
k.cpp","line":205,"method":"push_transaction","hostname":"","thread_name":"th_a","timestamp":"2015-09-12T18:10:36"},"format":"","data"
:{"trx":{"ref_block_num":40613,"ref_block_prefix":4073857719,"expiration":"2015-09-12T18:11:00","operations":[[5,{"fee":{"amount":1464
8,"asset_id":"1.3.0"},"registrar":"1.2.116","referrer":"1.2.116","referrer_percent":0,"name":"hello-test1","owner":{"weight_threshold"
:1,"account_auths":[],"key_auths":[["GPH7PuZQts1QJYkHzXvKA9hbZcww8o9xxmh2DNihXb9onb7jfPXv5",1]],"address_auths":[]},"active":{"weight_
threshold":1,"account_auths":[],"key_auths":[["GPH5BTF7xh1RzGB9vL8Rvk6kcNr5EC3cRBAC32yyqs5Pas56Gwubz",1]],"address_auths":[]},"options
":{"memo_key":"GPH5BTF7xh1RzGB9vL8Rvk6kcNr5EC3cRBAC32yyqs5Pas56Gwubz","voting_account":"1.2.5","num_witness":0,"num_committee":0,"votes":[],"extensions":[]},"extensions":[]}]],"extensions":[],"signatures":["1f35d7089c47caa4d3423b9779010495d8058e04ea6b5deeeac8aea9cc4e3d7720698e97dd6eb460ba9f7e8547f18f22173cfa4ee523bd1cca774f3c68c993d68d"]}}}]}                  db_with.hpp:82
2015-09-12T18:10:45 th_a:invoke handle_block         handle_block ] Got block #171687 from network                      application.cpp:383
2015-09-12T18:10:45 th_a:invoke handle_block           push_block ] new_block.block_num(): 171687 new_block.id(): 00029ea7c3d673800f20efc331f42b4235cee25c                      db_block.cpp:97
2015-09-12T18:10:45 th_a:invoke handle_block ~pending_transaction ] Pending transaction became invalid after switching to block 00029ea7c3d673800f20efc331f42b4235cee25c                        db_with.hpp:80
2015-09-12T18:10:45 th_a:invoke handle_block ~pending_transaction ] The invalid pending transaction is {"ref_block_num":40613,"ref_block_prefix":4073857719,"expiration":"2015-09-12T18:11:00","operations":[[0,{"fee":{"amount":2105468,"asset_id":"1.3.0"},"from":"1.2.116","to":"1.2.97867","amount":{"amount":100000000,"asset_id":"1.3.0"},"memo":{"from":"GPH5WCj1mMiiqEE4QRs7xhaFfSaiFroejUp3GuZE9wvfue9nxhPPn","to":"GPH5BTF7xh1RzGB9vL8Rvk6kcNr5EC3cRBAC32yyqs5Pas56Gwubz","nonce":"18015840590915049172","message":"4df7d8d7d3392b82281fd353f4ac8edad7a5ec58328ff795838fe77850c4de69"},"extensions":[]}]],"extensions":[],"signatures":["1f3ee5eefd1a1e93be6ddb75b709a8fa1e90c817fdca7f01133c39a29f172cecae53db1bb034397985b33642839a5e1a088dbb55c635352f30cac1fb137b69497a"],"operation_results":[[0,{}]]}
        db_with.hpp:81
2015-09-12T18:10:45 th_a:invoke handle_block ~pending_transaction ] The invalid pending transaction caused exception {"code":10,"name":"assert_exception","message":"Assert Exception","stack":[{"context":{"level":"error","file":"db_block.cpp","line":507,"method":"_apply_transaction","hostname":"","thread_name":"th_a","timestamp":"2015-09-12T18:10:45"},"format":"(skip & skip_transaction_dupe_check) || trx_idx.indices().get<by_trx_id>().find(trx_id) == trx_idx.indices().get<by_trx_id>().end(): ","data":{}},{"context":{"level":"warn","file":"db_block.cpp","line":563,"method":"_apply_transaction","hostname":"","thread_name":"th_a","timestamp":"2015-09-12T18:10:45"},"format":"","data":{"trx":{"ref_block_num":40613,"ref_block_prefix":4073857719,"expiration":"2015-09-12T18:11:00","operations":[[0,{"fee":{"amount":2105468,"asset_id":"1.3.0"},"from":"1.2.116","to":"1.2.97867","amount":{"amount":100000000,"asset_id":"1.3.0"},"memo":{"from":"GPH5WCj1mMiiqEE4QRs7xhaFfSaiFroejUp3GuZE9wvfue9nxhPPn","to":"GPH5BTF7xh1RzGB9vL8Rvk6kcNr5EC3cRBAC32yyqs5Pas56Gwubz","nonce":"18015840590915049172","message":"4df7d8d7d3392b82281fd353f4ac8edad7a5ec58328ff795838fe77850c4de69"},"extensions":[]}]],"extensions":[],"signatures":["1f3ee5eefd1a1e93be6ddb75b709a8fa1e90c817fdca7f01133c39a29f172cecae53db1bb034397985b33642839a5e1a088dbb55c635352f30cac1fb137b69497a"]}}},{"context":{"level":"warn","file":"db_block.cpp","line":205,"method":"push_transaction","hostname":"","thread_name":"th_a","timestamp":"2015-09-12T18:10:45"},"format":"","data":{"trx":{"ref_block_num":40613,"ref_block_prefix":4073857719,"expiration":"2015-09-12T18:11:00","operations":[[0,{"fee":{"amount":2105468,"asset_id":"1.3.0"},"from":"1.2.116","to":"1.2.97867","amount":{"amount":100000000,"asset_id":"1.3.0"},"memo":{"from":"GPH5WCj1mMiiqEE4QRs7xhaFfSaiFroejUp3GuZE9wvfue9nxhPPn","to":"GPH5BTF7xh1RzGB9vL8Rvk6kcNr5EC3cRBAC32yyqs5Pas56Gwubz","nonce":"18015840590915049172","message":"4df7d8d7d3392b82281fd353f4ac8edad7a5ec58328ff795838fe77850c4de69"},"extensions":[]}]],"extensions":[],"signatures":["1f3ee5eefd1a1e93be6ddb75b709a8fa1e90c817fdca7f01133c39a29f172cecae53db1bb034397985b33642839a5e1a088dbb55c635352f30cac1fb137b69497a"]}}}]}                        db_with.hpp:82

2015-09-12T18:17:21 th_a:invoke handle_transaction   handle_transaction ] Got transaction from network                  application.cp
p:422
2015-09-12T18:17:25 th_a:invoke handle_block         handle_block ] Got block #171750 from network                      application.cp
p:383
2015-09-12T18:17:25 th_a:invoke handle_block           push_block ] new_block.block_num(): 171750 new_block.id(): 00029ee60757de7e83bd
50d871de89f73ae312fe                      db_block.cpp:97
2015-09-12T18:17:25 th_a:invoke handle_block ~pending_transaction ] Pending transaction became invalid after switching to block 00029ee60757de7e83bd50d871de89f73ae312fe                        db_with.hpp:80
2015-09-12T18:17:25 th_a:invoke handle_block ~pending_transaction ] The invalid pending transaction is {"ref_block_num":40676,"ref_block_prefix":3655628590,"expiration":"2015-09-12T18:29:50","operations":[[1,{"fee":{"amount":500000,"asset_id":"1.3.0"},"seller":"1.2.97867","amount_to_sell":{"amount":100000,"asset_id":"1.3.0"},"min_to_receive":{"amount":2900000,"asset_id":"1.3.325"},"expiration":"2020-09-12T18:19:00","fill_or_kill":false,"extensions":[]}]],"extensions":[],"signatures":["1f1af6117c8f0857b1be1904b66c594e8ef29facc0307417c0e9c6bbb0bde829ad10d3642c14ff60af3cb222d1d3a89a9f12471d160900992f671509f04d928b4d"],"operation_results":[[1,"1.7.53"]]}
                db_with.hpp:81
2015-09-12T18:17:25 th_a:invoke handle_block ~pending_transaction ] The invalid pending transaction caused exception {"code":10,"name":"assert_exception","message":"Assert Exception","stack":[{"context":{"level":"error","file":"db_block.cpp","line":507,"method":"_apply_transaction","hostname":"","thread_name":"th_a","timestamp":"2015-09-12T18:17:25"},"format":"(skip & skip_transaction_dupe_check) || trx_idx.indices().get<by_trx_id>().find(trx_id) == trx_idx.indices().get<by_trx_id>().end(): ","data":{}},{"context":{"level":"warn","file":"db_block.cpp","line":563,"method":"_apply_transaction","hostname":"","thread_name":"th_a","timestamp":"2015-09-12T18:17:25"},"format":"","data":{"trx":{"ref_block_num":40676,"ref_block_prefix":3655628590,"expiration":"2015-09-12T18:29:50","operations":[[1,{"fee":{"amount":500000,"asset_id":"1.3.0"},"seller":"1.2.97867","amount_to_sell":{"amount":100000,"asset_id":"1.3.0"},"min_to_receive":{"amount":2900000,"asset_id":"1.3.325"},"expiration":"2020-09-12T18:19:00","fill_or_kill":false,"extensions":[]}]],"extensions":[],"signatures":["1f1af6117c8f0857b1be1904b66c594e8ef29facc0307417c0e9c6bbb0bde829ad10d3642c14ff60af3cb222d1d3a89a9f12471d160900992f671509f04d928b4d"]}}},{"context":{"level":"warn","file":"db_block.cpp","line":205,"method":"push_transaction","hostname":"","thread_name":"th_a","timestamp":"2015-09-12T18:17:25"},"format":"","data":{"trx":{"ref_block_num":40676,"ref_block_prefix":3655628590,"expiration":"2015-09-12T18:29:50","operations":[[1,{"fee":{"amount":500000,"asset_id":"1.3.0"},"seller":"1.2.97867","amount_to_sell":{"amount":100000,"asset_id":"1.3.0"},"min_to_receive":{"amount":2900000,"asset_id":"1.3.325"},"expiration":"2020-09-12T18:19:00","fill_or_kill":false,"extensions":[]}]],"extensions":[],"signatures":["1f1af6117c8f0857b1be1904b66c594e8ef29facc0307417c0e9c6bbb0bde829ad10d3642c14ff60af3cb222d1d3a89a9f12471d160900992f671509f04d928b4d"]}}}]}                  db_with.hpp:82


Is it working now? I cannot sync using this thing.

Not working for me.. I switched to the original p2p protocol.

It's stuck now.

UNABLE TO FIND BLOCK:  123657

From the logs I can see I generated a block but it seems didn't get included in the main chain.
//Edit: looks like it's because the correct block came later.
2015-09-09T04:28:03 th_a:Witness Block Production schedule_production_ ] now.time_since_epoch().count(): 1441772882997063 next_second:
 2015-09-09T04:28:04                      witness.cpp:146
2015-09-09T04:28:04 th_a:Witness Block Production block_production_loo ] Not producing block because slot has not yet arrived
                witness.cpp:182
2015-09-09T04:28:04 th_a:Witness Block Production schedule_production_ ] now.time_since_epoch().count(): 1441772883997067 next_second:
 2015-09-09T04:28:05                      witness.cpp:146
2015-09-09T04:28:05 th_a:Witness Block Production block_production_loo ] Not producing block because it isn't my turn
        witness.cpp:179
2015-09-09T04:28:05 th_a:Witness Block Production schedule_production_ ] now.time_since_epoch().count(): 1441772884997090 next_second:
 2015-09-09T04:28:06                      witness.cpp:146
2015-09-09T04:28:06 th_a:Witness Block Production block_production_loo ] Not producing block because it isn't my turn
        witness.cpp:179
2015-09-09T04:28:06 th_a:Witness Block Production schedule_production_ ] now.time_since_epoch().count(): 1441772885997094 next_second:
 2015-09-09T04:28:07                      witness.cpp:146
2015-09-09T04:28:07 th_a:Witness Block Production block_production_loo ] Not producing block because it isn't my turn
        witness.cpp:179
2015-09-09T04:28:07 th_a:Witness Block Production schedule_production_ ] now.time_since_epoch().count(): 1441772886997099 next_second:
 2015-09-09T04:28:08                      witness.cpp:146
2015-09-09T04:28:08 th_a:Witness Block Production block_production_loo ] Not producing block because it isn't my turn
        witness.cpp:179
2015-09-09T04:28:08 th_a:Witness Block Production schedule_production_ ] now.time_since_epoch().count(): 1441772887997102 next_second:
 2015-09-09T04:28:09                      witness.cpp:146
2015-09-09T04:28:09 th_a:Witness Block Production block_production_loo ] Not producing block because it isn't my turn
        witness.cpp:179
2015-09-09T04:28:09 th_a:Witness Block Production schedule_production_ ] now.time_since_epoch().count(): 1441772888997099 next_second:
 2015-09-09T04:28:10                      witness.cpp:146
2015-09-09T04:28:10 th_a:Witness Block Production           push_block ] new_block.block_num(): 123451 new_block.id(): 0001e23b1e939a70bd340efc1e5c5dcf484f8975                         db_block.cpp:86
2015-09-09T04:28:10 th_a:Witness Block Production block_production_loo ] Generated block #123451 with timestamp 2015-09-09T04:28:10 at
 time 2015-09-09T04:28:10                 witness.cpp:173
2015-09-09T04:28:10 th_a:Witness Block Production schedule_production_ ] now.time_since_epoch().count(): 1441772889998632 next_second:
 2015-09-09T04:28:11                      witness.cpp:146
2015-09-09T04:28:10                th_a:?           push_block ] new_block.block_num(): 123451 new_block.id(): 0001e23b1e939a70bd340ef
c1e5c5dcf484f8975                         db_block.cpp:86
2015-09-09T04:28:11 th_a:Witness Block Production block_production_loo ] Not producing block because slot has not yet arrived
                witness.cpp:182
2015-09-09T04:28:11 th_a:Witness Block Production schedule_production_ ] now.time_since_epoch().count(): 1441772890997069 next_second:
 2015-09-09T04:28:12                      witness.cpp:146
2015-09-09T04:28:12 th_a:Witness Block Production block_production_loo ] Not producing block because slot has not yet arrived
                witness.cpp:182
2015-09-09T04:28:12 th_a:Witness Block Production schedule_production_ ] now.time_since_epoch().count(): 1441772891997068 next_second:
 2015-09-09T04:28:13                      witness.cpp:146
2015-09-09T04:28:13 th_a:Witness Block Production block_production_loo ] Not producing block because slot has not yet arrived
                witness.cpp:182
2015-09-09T04:28:13 th_a:Witness Block Production schedule_production_ ] now.time_since_epoch().count(): 1441772892997044 next_second:
 2015-09-09T04:28:14                      witness.cpp:146
2015-09-09T04:28:14 th_a:Witness Block Production block_production_loo ] Not producing block because slot has not yet arrived
                witness.cpp:182
2015-09-09T04:28:14 th_a:Witness Block Production schedule_production_ ] now.time_since_epoch().count(): 1441772893997052 next_second:
 2015-09-09T04:28:15                      witness.cpp:146
2015-09-09T04:28:15 th_a:Witness Block Production block_production_loo ] Not producing block because it isn't my turn
        witness.cpp:179
2015-09-09T04:28:15 th_a:Witness Block Production schedule_production_ ] now.time_since_epoch().count(): 1441772894997099 next_second:
 2015-09-09T04:28:16                      witness.cpp:146
2015-09-09T04:28:15                th_a:?           push_block ] new_block.block_num(): 123451 new_block.id(): 0001e23b7b7768510491e30c156fe168e9bcc160                         db_block.cpp:86


get_block 123451
{
  "previous": "0001e23aefb943aced06ea1055b06b4520f09751",
  "timestamp": "2015-09-09T04:28:15",
  "witness": "1.6.65",
  "transaction_merkle_root": "0000000000000000000000000000000000000000",
  "extensions": [],
  "witness_signature": "1f4eef9dda295a5eeaf0f3f2feb11f00ee65cbbb8e507002d6fcd12527cb23a1142049714b5a9ebdd24fc59fca4c3008c857a3fd90197bb8ea5dae39ff170f5eaa",
  "transactions": [],
  "block_id": "0001e23b7b7768510491e30c156fe168e9bcc160",
  "signing_key": "GPH7eYmrY2wJq8fn8gRSN3eXWSgMoyZYFPYRD6F5uhwNuwzDuFCYE"
}


1. Current version of Graphene has sync issues, especially when the network is under pressure (when there is transaction flooding). I think it's better to do this wider test after the sync issues fixed.
2. Current test network looks like stable, because it is highly centralized -- 80% of witnesses are running on one or two VPS(es) which controlled by BM/CryptoNomex, every forks made by other witnesses are simply discarded. In a decentralized network it's harder to decide which fork is effective (or say longest?). We'd better prove that it's stable on a decentralized network first.