Hi all, so I'm (finally) on the test net, witness wackou (already voted in). And I got a few crashes to report, too 

one I had 2 days ago, just checked on github and it's fixed already (ahhh I missed the days of the first testnets when you had to update everyday to see the bugs from the day before fixed, and a whole new slew of fresh ones for today)

One I had today, same as abit in reply 876 of this thread, started a witness with a fresh download of the chain, log + stack trace follow (on commit 640d69aa05cdc629755edef9884444975dc41a82):

2735099ms th_a       db_block.cpp:98               push_block           ] new_block.block_num(): 198068 new_block.id(): 000305b4cedbe47143d0ff393eef37113933ff6c
2735100ms th_a       db_block.cpp:98               push_block           ] new_block.block_num(): 198069 new_block.id(): 000305b5b4a47445460fbe4f570f6fcdd5640085
2735100ms th_a       db_block.cpp:98               push_block           ] new_block.block_num(): 198070 new_block.id(): 000305b6b7a20c14f5e1827af055a7bc4da82eb0
2735101ms th_a       db_block.cpp:98               push_block           ] new_block.block_num(): 198071 new_block.id(): 000305b748128bac176f75273a0023e892c52928
2735101ms th_a       db_block.cpp:98               push_block           ] new_block.block_num(): 198072 new_block.id(): 000305b84a7b19f36c111e13ab8f4617f111e812
2735102ms th_a       db_block.cpp:98               push_block           ] new_block.block_num(): 198073 new_block.id(): 000305b904de6654a5450b597ecb0d2ec6217599
2735102ms th_a       db_block.cpp:98               push_block           ] new_block.block_num(): 198074 new_block.id(): 000305ba363cb379bdde904800269ed0e60a88b4
2735103ms th_a       db_block.cpp:98               push_block           ] new_block.block_num(): 198075 new_block.id(): 000305bb716d1936a8294d55215c5290f3254174
2735103ms th_a       db_block.cpp:98               push_block           ] new_block.block_num(): 198076 new_block.id(): 000305bc3d4eba5a0e48b33c26b846070aa90161
witness_node: /home/admin/.BitShares2_build/libraries/app/application.cpp:624: virtual std::vector<fc::ripemd160> graphene::app::detail::application_impl::get_blockchain_synopsis(const item_hash_t&, uint32_t): Assertion `fork_history.back() == reference_point' failed.

Program received signal SIGABRT, Aborted.
0x00007ffff6516107 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0  0x00007ffff6516107 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007ffff65174e8 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007ffff650f226 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007ffff650f2d2 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#4  0x00000000020bd3c5 in graphene::app::detail::application_impl::get_blockchain_synopsis (this=0x356b030, reference_point=..., number_of_blocks_after_reference_point=0) at /home/admin/.BitShares2_build/libraries/app/application.cpp:624
#5  0x000000000297a705 in graphene::net::detail::statistics_gathering_node_delegate_wrapper::<lambda()>::operator()(void) const (__closure=0x7fffe0649698) at /home/admin/.BitShares2_build/libraries/net/node.cpp:5296
#6  0x000000000298a97b in fc::detail::functor_run<graphene::net::detail::statistics_gathering_node_delegate_wrapper::get_blockchain_synopsis(const item_hash_t&, uint32_t)::<lambda()> >::run(void *, void *) (functor=0x7fffe0649698, prom=0x7fffe0649780)
    at /home/admin/.BitShares2_build/libraries/fc/include/fc/thread/task.hpp:77
#7  0x00000000027519cd in fc::task_base::run_impl (this=0x7fffe06496b8) at /home/admin/.BitShares2_build/libraries/fc/src/thread/task.cpp:43
#8  0x000000000275195c in fc::task_base::run (this=0x7fffe06496b8) at /home/admin/.BitShares2_build/libraries/fc/src/thread/task.cpp:32
#9  0x00000000027459ba in fc::thread_d::run_next_task (this=0x359eab0) at /home/admin/.BitShares2_build/libraries/fc/src/thread/thread_d.hpp:498
#10 0x0000000002745e84 in fc::thread_d::process_tasks (this=0x359eab0) at /home/admin/.BitShares2_build/libraries/fc/src/thread/thread_d.hpp:547
#11 0x00000000027454ae in fc::thread_d::start_process_tasks (my=56224432) at /home/admin/.BitShares2_build/libraries/fc/src/thread/thread_d.hpp:475
#12 0x0000000002abf931 in make_fcontext ()
#13 0x00007fffe8000020 in ?? ()
#14 0x0000000000000000 in ?? ()

Just to be clear about something. If we update to the new version, then the delegate using bts_tools automatically will start running through the backbone system that you have running now?

No, at the moment this is only meant for operators of the backbone (Thom and me for now) to allow to set up backbone nodes that behave properly. The next release (likely 0.2.11, I hope to get this in about 2 weeks, but don't take my word for it) will add the possibility for delegates to connect to the backbone automatically. This will be accompanied by a more prominent announcement so that delegates don't miss it. This post was just meant to be an update to the delegate proposal, now that we are voted in.

Do you believe with he two delegates providing 130% total now with the market cap that the funds will be enough to provide enough of a distribution network.. or will there be some kind of option for some delegates to become a part of this backbone where it makes geographic sense?

At the moment, I propose that Thom and I manage the backbone so that it's easier to coordinate. My initial plan was to have 2 nodes in the US, 2 nodes in Europe and 2 nodes in China/Asia. Once it's up and running for some time without glitches, we should open it up to anyone that wants to participate and contribute a node (similar to how people contribute seed nodes).

There is a question of trust involved, though, because the timing attack that allows a peer to detect the IP of delegates--that the backbone is designed to thwart--can still be performed by the operator of a backbone node. So we should make sure that these are trusted community members.

This is the long-term plan anyway, because if people contribute backbone nodes to the network then this doesn't need to be a recurring cost for the blockchain and everyone wins: blockchain doesn't pay for the backbone but everyone benefits from it, and backbone operators get fame and gratitude for the service they give to the network  And if we can find financial incentives to run a backbone node that isn't supported directly by the blockchain, then that would be even better!

Should we no longer vote or the delegate  btstools.digitalgaia and put all our support into the verbaltech delegate to continue supporting you?

At this point, I don't recommend to vote for btstools.digitalgaia anymore, as the pay from verbaltech is more than enough to pay for the servers. That shouldn't change anything, though, because I didn't expect my delegate to be voted in before the release of BitShares 2.0 anyway.

In the (unlikely) event it gets voted in, I think I would do a poll like Thom did to ask the community for their opinions, and burn it or redirect it to some other deserving delegates (I can think of a few that are still not voted in but have contributed a lot).

In summary:
 - Thom and I work as a team and are funded by the pay of delegate.verbaltech
 - btstools.digitalgaia is deprecated and people wanting to support me should support delegate.verbaltech.
 - when BTS2.0 is released, we will come up with a unified worker proposal

An update on the situation:

I am now back from holidays, and ready to kick into gear for another year!

I have talked with Thom yesterday, and as previously mentioned here, we will now work together with the aim to enhance and improve the BitShares network security. Mostly what this means at the moment is that we will start implementing the backbone proposal as described here: http://digitalgaia.io/backbone.html  and set up various VPS instances spread all over the world, with the aim of getting global coverage, low-latency, strong connectivity and decentralization of the providers to ensure robustness. Thom has already started setting up some nodes, and I will follow suit soon.

The first few things I intend to work on are:
 - adapt the bts_tools to be able to manage backbone nodes
 - setup and configure an actual backbone for BitShares 0.9.x
 - adapt the bts_tools so that delegates managed by it connect exclusively to the backbone, if online (this is the part that provides DDoS protection to the delegates)
 - setup a webpage that show the real-time online status of the backbone nodes

I will be leading the development for the bts_tools, but Thom will help me when/where needed, and will be testing also. Eventually, we will be managing the backbone between the 2 of us, so given that we are in 2 different timezones, this means we can ensure close to 24/7 monitoring of the backbone nodes.

Another task will be to look into graphene, port the bts_tools to be able to work with graphene clients and have them ready for when the official BitShares 2.0 chain launches.

As for the delegate pay, we agreed that Thom will be sending me 70% of the pay from his delegate, and will thus keep 30%. I will use the funds to pay for VPS instances and time devoted, and leftovers (if any) will be kept to run seed nodes / backbones nodes for when graphene is launched.

All in all, this is a rehash of my previous proposal + Thom's previous posts, which shouldn't surprise lots of people. We plan to submit a new, revised joint proposal for when BitShares 2.0 launches, but at the moment we feel like we want to spend more time coding than campaigning for funds, especially given that bts2.0 is not that far away and lots of things are going to change to the delegate/worker structure when it is released. We also have some more ideas that we want to flesh out nicely in a new proposal rather than rush them now. I will post further updates in this thread now that delegate.verbaltech is the delegate funding development, but will still post relevant updates to the bts_tools in my original thread here: https://bitsharestalk.org/index.php/topic,12534

Thank you everyone for the support!

well, if I had had to choose one delegate that did not have a full pay before, that would have been you without a doubt, and I'm not the only one here that would have done so. So no need to apologize, such is life hoping that I'll get back in when tradebts.gulu gets voted out (or some whale votes for me!)

Quote from: wackou on July 30, 2015, 09:38:07 am

Do you have your email properly configured in the "notification" section of your config.yaml file? Is there something in the logs that a notification cannot be sent or does everything seem normal? Also, your client need to be synchronized and on the main fork to be able to detect the missed blocks, if you end up on a minority fork then the detection cannot work.

Yes, I get email notifications. I tested that with network connections limits by adjusting the minimum number to be high. The tools notify if you drop below that minimum.

There is far less than 1 second deviation of the system time, NTP sync is working and the avg delegate participation is > 95% . No indication when this "locking" is happening that the node is on a minority fork.

hmmm that is weird, then, not sure what can cause blocks to be missed without being noticed. I'll have another look at the relevant code, see if I can spot something. Another thing I will most certainly do is add another monitoring plugin that checks whether the wallet is locked / unlocked and send a notification upon change of state. That way, you'll get a notification when the wallets gets locked.

  You get full support from me too! Will also add you to my slate as soon as I get around to it.

oh, and Ken, welcome to the top 101! It has been a long time coming