My node got stuck during the night, too (didn't crash) on block 14541 with an unlinkable block error. I relaunched it (with --enable-stale-production, wouldn't do anything otherwise) and it seems like I'm producing blocks again (seems like I'm the only one, I did 14542, 14543, 14544 and 14545 as of now). Not sure if I am on my own fork and nobody sees it or if I'm the only one producing blocks (feeling a bit lonely there )

witness wackou running on latest master, proxy voting account set to dan, waiting for votes

get_witness wackou
{
  "id": "1.6.30",
  "witness_account": "1.2.83349",
  "last_aslot": 0,
  "signing_key": "GPH8C1Cz3LDu732VT74bYvNE2G25NLghV96zcMnFwLd4Z6aXWup9i",
  "vote_id": "1:50",
  "total_votes": 0,
  "url": "http://digitalgaia.io",
  "total_missed": 0
}

had this crash too:

1294000ms th_a       witness.cpp:182               block_production_loo ] Not producing block because it isn't my turn
1295000ms th_a       witness.cpp:182               block_production_loo ] Not producing block because it isn't my turn
witness_node: /home/admin/.BitShares2_build/libraries/chain/db_market.cpp:290: bool graphene::chain::database::fill_order(const graphene::chain::call_order_object&, const graphene::chain::asset&, const graphene::chain::asset&): Assertion `order.get_collateral() >= pays' failed.

Program received signal SIGABRT, Aborted.
0x00007ffff6516107 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0  0x00007ffff6516107 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007ffff65174e8 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007ffff650f226 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007ffff650f2d2 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#4  0x00000000024ec8a3 in graphene::chain::database::fill_order (this=0x360f160, order=..., pays=..., receives=...)
    at /home/admin/.BitShares2_build/libraries/chain/db_market.cpp:290
#5  0x00000000024ec173 in graphene::chain::database::match (this=0x360f160, call=..., settle=..., match_price=..., max_settlement=...)
    at /home/admin/.BitShares2_build/libraries/chain/db_market.cpp:239
#6  0x00000000024f1f82 in graphene::chain::database::clear_expired_orders (this=0x360f160) at /home/admin/.BitShares2_build/libraries/chain/db_update.cpp:237
#7  0x00000000024d4b90 in graphene::chain::database::_apply_block (this=0x360f160, next_block=...)
    at /home/admin/.BitShares2_build/libraries/chain/db_block.cpp:478
#8  0x00000000024d4088 in graphene::chain::database::<lambda()>::operator()(void) const (__closure=0x7fffe7df1330)
    at /home/admin/.BitShares2_build/libraries/chain/db_block.cpp:436
#9  0x00000000024f2f55 in graphene::chain::detail::with_skip_flags<graphene::chain::database::apply_block(const graphene::chain::signed_block&, uint32_t)::<lambda()> >(graphene::chain::database &, uint32_t, graphene::chain::database::<lambda()>) (db=..., skip_flags=0, callback=...)
    at /home/admin/.BitShares2_build/libraries/chain/include/graphene/chain/db_with.hpp:123
#10 0x00000000024d44ad in graphene::chain::database::apply_block (this=0x360f160, next_block=..., skip=0)
    at /home/admin/.BitShares2_build/libraries/chain/db_block.cpp:437
#11 0x00000000024ce648 in graphene::chain::database::_push_block (this=0x360f160, new_block=...)
    at /home/admin/.BitShares2_build/libraries/chain/db_block.cpp:185
#12 0x00000000024cdc21 in graphene::chain::database::<lambda()>::<lambda()>::operator()(void) const (__closure=0x7fffe7df1e20)
    at /home/admin/.BitShares2_build/libraries/chain/db_block.cpp:112
#13 0x00000000024f2d5e in graphene::chain::detail::without_pending_transactions<graphene::chain::database::push_block(const graphene::chain::signed_block&, uint32_t)::<lambda()>::<lambda()> >(graphene::chain::database &, <unknown type in /home/admin/.BitShares2_bin/witness_node_2015-09-24_test3c-1-gd01fc0a, CU 0xfa3feb, DIE 0x11f4221>, graphene::chain::database::<lambda()>::<lambda()>) (db=...,
    pending_transactions=<unknown type in /home/admin/.BitShares2_bin/witness_node_2015-09-24_test3c-1-gd01fc0a, CU 0xfa3feb, DIE 0x11f4221>, callback=...)
    at /home/admin/.BitShares2_build/libraries/chain/include/graphene/chain/db_with.hpp:140
#14 0x00000000024cdc92 in graphene::chain::database::<lambda()>::operator()(void) const (__closure=0x7fffe7df1ed0)
    at /home/admin/.BitShares2_build/libraries/chain/db_block.cpp:113
#15 0x00000000024f2dd9 in graphene::chain::detail::with_skip_flags<graphene::chain::database::push_block(const graphene::chain::signed_block&, uint32_t)::<lambda()> >(graphene::chain::database &, uint32_t, graphene::chain::database::<lambda()>) (db=..., skip_flags=0, callback=...)
    at /home/admin/.BitShares2_build/libraries/chain/include/graphene/chain/db_with.hpp:123
#16 0x00000000024cdce3 in graphene::chain::database::push_block (this=0x360f160, new_block=..., skip=0)
    at /home/admin/.BitShares2_build/libraries/chain/db_block.cpp:114
---Type <return> to continue, or q <return> to quit---
#17 0x00000000024d158b in graphene::chain::database::_generate_block (this=0x360f160, when=..., witness_id=..., block_signing_private_key=..., retry_on_failure=true) at /home/admin/.BitShares2_build/libraries/chain/db_block.cpp:368
#18 0x00000000024d07d3 in graphene::chain::database::<lambda()>::operator()(void) const (__closure=0x7fffe7df2c40) at /home/admin/.BitShares2_build/libraries/chain/db_block.cpp:276
#19 0x00000000024f2ecf in graphene::chain::detail::with_skip_flags<graphene::chain::database::generate_block(fc::time_point_sec, graphene::chain::witness_id_type, const fc::ecc::private_key&, uint32_t)::<lambda()> >(graphene::chain::database &, uint32_t, graphene::chain::database::<lambda()>) (db=..., skip_flags=0, callback=...) at /home/admin/.BitShares2_build/libraries/chain/include/graphene/chain/db_with.hpp:123
#20 0x00000000024d089f in graphene::chain::database::generate_block (this=0x360f160, when=..., witness_id=..., block_signing_private_key=..., skip=0) at /home/admin/.BitShares2_build/libraries/chain/db_block.cpp:277
#21 0x00000000024c218f in graphene::witness_plugin::witness_plugin::maybe_produce_block (this=0x3618410, capture=...) at /home/admin/.BitShares2_build/libraries/plugins/witness/witness.cpp:276
#22 0x00000000024bff51 in graphene::witness_plugin::witness_plugin::block_production_loop (this=0x3618410) at /home/admin/.BitShares2_build/libraries/plugins/witness/witness.cpp:160
#23 0x00000000024bfe19 in graphene::witness_plugin::witness_plugin::<lambda()>::operator()(void) const (__closure=0x1639d188) at /home/admin/.BitShares2_build/libraries/plugins/witness/witness.cpp:150
#24 0x00000000024c3288 in fc::detail::void_functor_run<graphene::witness_plugin::witness_plugin::schedule_production_loop()::<lambda()> >::run(void *, void *) (functor=0x1639d188, prom=0x1639d180)
    at /home/admin/.BitShares2_build/libraries/fc/include/fc/thread/task.hpp:83
#25 0x00000000027e209d in fc::task_base::run_impl (this=0x1639d190) at /home/admin/.BitShares2_build/libraries/fc/src/thread/task.cpp:43
#26 0x00000000027e202c in fc::task_base::run (this=0x1639d190) at /home/admin/.BitShares2_build/libraries/fc/src/thread/task.cpp:32
#27 0x00000000027d608a in fc::thread_d::run_next_task (this=0x364d2f0) at /home/admin/.BitShares2_build/libraries/fc/src/thread/thread_d.hpp:498
#28 0x00000000027d6554 in fc::thread_d::process_tasks (this=0x364d2f0) at /home/admin/.BitShares2_build/libraries/fc/src/thread/thread_d.hpp:547
#29 0x00000000027d5b7e in fc::thread_d::start_process_tasks (my=56939248) at /home/admin/.BitShares2_build/libraries/fc/src/thread/thread_d.hpp:475
#30 0x0000000002b50711 in make_fcontext ()
#31 0x00010102464c457f in ?? ()
#32 0x0000000000000000 in ?? ()
(gdb)
Note that I hadn't upgraded to the latest master yet, will do so now.

Quote from: spartako on September 22, 2015, 09:14:11 am

Quote from: theoretical on September 22, 2015, 05:19:45 am

Quote from: spartako on September 21, 2015, 11:10:52 pm

What is the best practice for not missing any block during an update?

The update_witness command in CLI wallet allows you to change your witness's block signing key.  This architecture allows several different downtime-free update procedures according to your specific hosting situation:
...

Thanks theoretical! Great work!

  indeed, very useful and sounds like a fool-proof way to do it with no risk of signing double blocks.

One question: how long does it take for the signing key to be updated after you call the update_witness command? As soon as the transaction is processed by the network? After the next maintenance interval? Sth else?

I seem to recall that in BitShares 0.9.x changing the signing key did take a bit of time to go into effect, so that set_block_production true/false was the preferred way. If in graphene the call to changing the signing key takes effect immediately, then that's really nice as it's a much better way of doing it.

bump question @theoretical @bytemaster

Quote from: theoretical on September 22, 2015, 05:19:45 am

Quote from: spartako on September 21, 2015, 11:10:52 pm

What is the best practice for not missing any block during an update?

The update_witness command in CLI wallet allows you to change your witness's block signing key.  This architecture allows several different downtime-free update procedures according to your specific hosting situation:
...

Thanks theoretical! Great work!

  indeed, very useful and sounds like a fool-proof way to do it with no risk of signing double blocks.

One question: how long does it take for the signing key to be updated after you call the update_witness command? As soon as the transaction is processed by the network? After the next maintenance interval? Sth else?

I seem to recall that in BitShares 0.9.x changing the signing key did take a bit of time to go into effect, so that set_block_production true/false was the preferred way. If in graphene the call to changing the signing key takes effect immediately, then that's really nice as it's a much better way of doing it.

wackou now up and running

get_witness wackou
{
  "id": "1.6.5248",
  "witness_account": "1.2.83349",
  "last_aslot": 0,
  "signing_key": "GPH8C1Cz3LDu732VT74bYvNE2G25NLghV96zcMnFwLd4Z6aXWup9i",
  "vote_id": "1:5268",
  "total_votes": 0,
  "url": "http://digitalgaia.io",
  "total_missed": 0
}

that means that they are confident that everything will go just fine and that's a very  good sign...

hmmm no, that's not really what it means... I fully agree with Monsterer here, this release seems a bit premature and setting up a time-bomb like this feels *very* dangerous. I have the impression that what this means is that it was done to please investors and people who keep complaining about shifts in the release date, to show them that it will happen on that date for sure. But what if the network isn't stable by then? You have to have a plan B, and "release 0.9.4 a couple of days before to fix it" doesn't seem like a very good one to me.

This also puts an enormous pressure on the devs, which is not necessarily what you want to have to be able to focus clearly on coding (it's effectively a sword of Damocles hanging above their head...) I'd rather have an undefined release date that slips by a month (heck, even 3 months is very little compared to what this sets up to be) and have a rock-stable release then.

But let's see how the next testnet goes, if it runs really smooth then maybe the release can still be made on time. I trust that Stan will find some magic potion that he will feed to all core devs so they write bug-free code during the next 2 weeks

Here comes the current update on the development of the backbone:

I have set up a backbone of 3 nodes already, you can see the details here: http://digitalgaia.io/backbone.html, or can check http://backbone03.digitalgaia.io/backbone to see the current status of the 3rd backbone node and how it is connected to the other ones.

I was planning to set up more with Thom, but I hit kind of a roadblock when implementing the monitoring plugin that allows a delegate to connect only to the backbone, thereby hiding its IP (1). I have a workaround for it, however this requires a lot more setup on the nodes and is only valid until graphene comes out (network api will change and this might not even apply any longer).

So I decided to work on adapting the tools for graphene instead as it is now coming very soon (fingers crossed!) and started runnning a witness in the testnet to be able to do so.

I won't set up more backbone nodes for now, but will leave those already running anyway if people want to use them, or the chain servers on them, for instance.

----

(1) The issue is that the call to "network_set_allowed_peers" expects to be given the peer ids, which is not public info and needs to be obtained from "network_get_info" on the node itself. I could solve it by only exposing access to it (by using my bts_proxy project) but that would involve quite a bit of setup on the nodes for a workaround that won't likely be applicable for graphene, so that seemed quite a waste of time. If this functionality is also missing in graphene and core devs don't have the time or need to do it themselves I might also be able to implement it but let's wait until then.