Quote from: clayop on October 09, 2015, 06:19:51 PM
Okay. Here's how I build. Can you compare it with yours?

Code Select Expand

git clone https://github.com/cryptonomex/graphene.git
cd graphene
git submodule update --init --recursive
cmake -DCMAKE_BUILD_TYPE=Release
make witness_node cli_wallet

Yep, the std way like everybody else that doesn't use some type of automation.

But like I said, irrelevant for the reasons stated.

I got this on my cli_wallet, in red:


7364ms th_a       wallet.cpp:1717               sign_transaction     ] Caught exception while broadcasting transaction with id 0be4839dfa602f3be290b7d2a5524a318f889e50


Still, it is not crashed...

Quote from: Thom on October 09, 2015, 05:44:27 PM
Can't seem to get another instance up and running. I've verified all of the wallets, config and binaries with md5sum to working versions. I've deleted the object_database and blockchain. I've tried --resync-blockchain, --replay-blockchain options. Everytime I start the witness I get this output:

Code Select Expand


2114494ms th_a       application.cpp:712           get_blockchain_synop ] synopsis: ["0000019a905fb92
2c6c7c8e72d1b09c04425e0c1"]
2114531ms th_a       application.cpp:712           get_blockchain_synop ] synopsis: ["0000019a905fb92
2c6c7c8e72d1b09c04425e0c1"]
2118682ms th_a       application.cpp:712           get_blockchain_synop ] synopsis: ["0000019a905fb92
2c6c7c8e72d1b09c04425e0c1","000001a1895e7bb499ff3f12bd5eda0a050488de","000001a4236eb3795a2cfa9c5d9f00177fd0b14c","000001a64a8f85ec0c46910dc9eb8ad1ad6a5d10"]
2118852ms th_a       application.cpp:712           get_blockchain_synop ] synopsis: ["0000019a905fb922c6c7c8e72d1b09c04425e0c1","000001a1895e7bb499ff3f12bd5eda0a050488de","000001a4236eb3795a2cfa9c5d9f00
177fd0b14c","000001a64a8f85ec0c46910dc9eb8ad1ad6a5d10"]


cli_wallet does not show correct "info" at all. Using the same seed as working VPS (104.236.51.238:2005)

How does the info command output look incorrect?  If you are doing --resync-blockchain then it nukes all the downloaded blocks and starts at the beginning, so you will see block times / block numbers in the past until you've finished downloading the chain.  The get_blockchain_synop messages are normal, they show your node exchanging block ID's with peers.

Quote from: Thom on October 09, 2015, 05:44:27 PM
BTW - I still haven't found a way to stop the witness cleanly. The only way I know to stop it at all is with ^C. Sometimes it reports "Aborted ..." in the output but most of the time when it is restarted it complains it was shutdown uncleanly.

Yeah, our shutdown code isn't that robust.  I filed a ticket.

Quote from: Thom on October 09, 2015, 05:44:27 PM
Is there a signal that will do it properly? St least in the old code there was a "quit" command, although it often resulted in a corrupted blockchain database. Can this fundamental issue not be fixed?

SIGINT is your best bet.

Quote from: Thom on October 09, 2015, 06:17:00 PM

Quote from: clayop on October 09, 2015, 05:47:37 PM
Can you post your build commands?

Hey clayop, thanks for the response. To answere your question, no, not easily. I'm using a python script (bts_tools) and some of it is buried in the config files.

Besides, I don't think that's relevant, given that the binaries are in operation now and I haven't missed a block in several hours on VPS seed07 in Singnapore.

Also, as I mentioned, I verified the binaries, config etc were all transfered from seed07 to seed06 in LA without changes using md5sum.

I just tried to start it as a seed node on seed06 (no block production) and it fails in the same way, but I expected that.

I will try to start a seed on another VPS in Germany and see if I can get it to sync. If I can it will be some issue on the VPS, in which case I'll rebuild it from scratch. All 4 of the VPSs should be the same, but I have noticed subtle differences in the images each host uses, tho they are all Ubuntu 14.04, and I used the same setup script on each one.

It's been fast pace work, and I may have overlooked something tho. It only goes to show if it's this difficult to determine why I can't bring up a new witness instance with all of these checks and verifications it's not only a suble difference but also that the voluminous withness_node output is not easy to distill information from.

Okay. Here's how I build. Can you compare it with yours?

git clone https://github.com/cryptonomex/graphene.git
cd graphene
git submodule update --init --recursive
cmake -DCMAKE_BUILD_TYPE=Release
make witness_node cli_wallet

Quote from: clayop on October 09, 2015, 05:47:37 PM
Can you post your build commands?

Hey clayop, thanks for the response. To answere your question, no, not easily. I'm using a python script (bts_tools) and some of it is buried in the config files.

Besides, I don't think that's relevant, given that the binaries are in operation now and I haven't missed a block in several hours on VPS seed07 in Singnapore.

Also, as I mentioned, I verified the binaries, config etc were all transfered from seed07 to seed06 in LA without changes using md5sum.

I just tried to start it as a seed node on seed06 (no block production) and it fails in the same way, but I expected that.

I will try to start a seed on another VPS in Germany and see if I can get it to sync. If I can it will be some issue on the VPS, in which case I'll rebuild it from scratch. All 4 of the VPSs should be the same, but I have noticed subtle differences in the images each host uses, tho they are all Ubuntu 14.04, and I used the same setup script on each one.

It's been fast pace work, and I may have overlooked something tho. It only goes to show if it's this difficult to determine why I can't bring up a new witness instance with all of these checks and verifications it's not only a suble difference but also that the voluminous withness_node output is not easy to distill information from.

I am on commit: e68e99ed3ae11ddac607e983e7549e8278fdecc4 by BM, I miss the last two from emfrias... are they important for the hardfork?

Quote from: Ander on October 09, 2015, 05:47:41 PM

Quote from: clayop on October 09, 2015, 05:08:44 PM
Stress test will begin at 19:00 UTC (in 2 hours). I expect to see the performance of the new code!

Hopefully, the network won't be broken  :P

Quote from: clayop on October 09, 2015, 05:08:44 PM
Stress test will begin at 19:00 UTC (in 2 hours). I expect to see the performance of the new code!

Quote from: Thom on October 09, 2015, 05:44:27 PM
Can't seem to get another instance up and running. I've verified all of the wallets, config and binaries with md5sum to working versions. I've deleted the object_database and blockchain. I've tried --resync-blockchain, --replay-blockchain options. Everytime I start the witness I get this output:

Code Select Expand


2114494ms th_a       application.cpp:712           get_blockchain_synop ] synopsis: ["0000019a905fb92
2c6c7c8e72d1b09c04425e0c1"]
2114531ms th_a       application.cpp:712           get_blockchain_synop ] synopsis: ["0000019a905fb92
2c6c7c8e72d1b09c04425e0c1"]
2118682ms th_a       application.cpp:712           get_blockchain_synop ] synopsis: ["0000019a905fb92
2c6c7c8e72d1b09c04425e0c1","000001a1895e7bb499ff3f12bd5eda0a050488de","000001a4236eb3795a2cfa9c5d9f00177fd0b14c","000001a64a8f85ec0c46910dc9eb8ad1ad6a5d10"]
2118852ms th_a       application.cpp:712           get_blockchain_synop ] synopsis: ["0000019a905fb922c6c7c8e72d1b09c04425e0c1","000001a1895e7bb499ff3f12bd5eda0a050488de","000001a4236eb3795a2cfa9c5d9f00
177fd0b14c","000001a64a8f85ec0c46910dc9eb8ad1ad6a5d10"]


cli_wallet does not show correct "info" at all. Using the same seed as working VPS (104.236.51.238:2005)

I will repeat the attempt on another VPS with the same exact files and see what happens. I will make one last attempt to restart this VPS as a seed.

Can you post your build commands?

Can't seem to get another instance up and running. I've verified all of the wallets, config and binaries with md5sum to working versions. I've deleted the object_database and blockchain. I've tried --resync-blockchain, --replay-blockchain options. Everytime I start the witness I get this output:


2114494ms th_a       application.cpp:712           get_blockchain_synop ] synopsis: ["0000019a905fb92
2c6c7c8e72d1b09c04425e0c1"]
2114531ms th_a       application.cpp:712           get_blockchain_synop ] synopsis: ["0000019a905fb92
2c6c7c8e72d1b09c04425e0c1"]
2118682ms th_a       application.cpp:712           get_blockchain_synop ] synopsis: ["0000019a905fb92
2c6c7c8e72d1b09c04425e0c1","000001a1895e7bb499ff3f12bd5eda0a050488de","000001a4236eb3795a2cfa9c5d9f00177fd0b14c","000001a64a8f85ec0c46910dc9eb8ad1ad6a5d10"]
2118852ms th_a       application.cpp:712           get_blockchain_synop ] synopsis: ["0000019a905fb922c6c7c8e72d1b09c04425e0c1","000001a1895e7bb499ff3f12bd5eda0a050488de","000001a4236eb3795a2cfa9c5d9f00
177fd0b14c","000001a64a8f85ec0c46910dc9eb8ad1ad6a5d10"]


cli_wallet does not show correct "info" at all. Using the same seed as working VPS (104.236.51.238:2005)

I will repeat the attempt on another VPS with the same exact files and see what happens. I will make one last attempt to restart this VPS as a seed.


BTW - I still haven't found a way to stop the witness cleanly. The only way I know to stop it at all is with ^C. Sometimes it reports "Aborted ..." in the output but most of the time when it is restarted it complains it was shutdown uncleanly.

Is there a signal that will do it properly? St least in the old code there was a "quit" command, although it often resulted in a corrupted blockchain database. Can this fundamental issue not be fixed?

Quote from: theoretical on October 09, 2015, 05:35:38 PM

Quote from: clayop on October 09, 2015, 05:08:44 PM
Stress test will begin at 19:00 UTC (in 2 hours).

Can you postpone it to 19:10 UTC or so?  Some new code goes into effect at 18:45 UTC, and 19:00 UTC will be the first maintenance interval after the new code goes into effect.  It might be helpful for debugging to have a way to tell the difference between witnesses going down due to the new code (18:45-19:10) and witnesses going down due to the stress (19:10 or later).

Sure. What about 19:30 UTC?

Quote from: clayop on October 09, 2015, 05:08:44 PM
Stress test will begin at 19:00 UTC (in 2 hours).

Can you postpone it to 19:10 UTC or so?  Some new code goes into effect at 18:45 UTC, and 19:00 UTC will be the first maintenance interval after the new code goes into effect.  It might be helpful for debugging to have a way to tell the difference between witnesses going down due to the new code (18:45-19:10) and witnesses going down due to the stress (19:10 or later).

Stress test will begin at 19:00 19:30 UTC (in 2 hours). I expect to see the performance of the new code!

Quote from: bytemaster on October 09, 2015, 03:25:51 PM
I would not assume that all missed blocks are the fault of your witness or an indication that you need to switch nodes.    I would probably use the "last_confirmed_block_num" property on the witness and only switch if it is more than 3 rounds behind the head block number.

Wouldn't waiting for 3 missed blocks before switching have the same effect?  Either way the number of missed blocks before you switch could pretty easily be configured in the config.

***edit*** I do get that 3 missed blocks will not always be equal to 3x active witnesses since last generated block.  I was just curious if there is a reason why measuring missed blocks by number of rounds completed since last block is a better way of measuring.

Quote from: Spectral on October 09, 2015, 03:33:23 PM
Witness spectral stopped producing blocks and started producing error messages at the witness output. Here is a log from the screen buffer of the witness output:

https://drive.google.com/file/d/0BzEEZBS7b6xPUnJvQXFZUHZQZW8/view?usp=sharing

Did this also happen to the others?

I deleted oct5 and object_database, restarted witness, and it's back to producing blocks and price feed.

Looks like we missed the real error.