Test Net for Advanced Users

[Thom]

iHashFury, how are you determining there are no vote in witnesses in place? Look at my last couple of posts. I noticed I had a non-zero "total_votes" earlier for example.

Code: [Select]

get_global_properties in the cli_wallet and watching the output of witness_node in tmux (screen alternative)

I C.

I've been using info. I use tmux as well, never used screen. However ATM I'm just using different shell windows b/c I don't know how to scroll a tmux terminal beyond the visible buffer window; pg up & pg dn keys just recall the previous cmd line options.

While the code is rebuilding on the vps I'll try the witness once more to see what get_global_props returns for me and if it shows my witness being voted in.

[iHashFury]

info also works in the cli_wallet

Code: [Select]

info
info
{
  "head_block_num": 0,
  "head_block_id": "0000000000000000000000000000000000000000",
  "head_block_age": "7 days old",
  "next_maintenance_time": "45 years  ago",
  "chain_id": "d011922587473757011118587f93afcc314fbaea094fc1055574721b27975083",
  "active_witnesses": [
    "1.6.0",
    "1.6.1",
    "1.6.2",
    "1.6.3",
    "1.6.4",
    "1.6.5",
    "1.6.6",
    "1.6.7",
    "1.6.8",
    "1.6.9",
    "1.6.10",
    "1.6.11",
    "1.6.12",
    "1.6.13",
    "1.6.14",
    "1.6.15",
    "1.6.16",
    "1.6.17",
    "1.6.18",
    "1.6.19",
    "1.6.20",
    "1.6.21",
    "1.6.22",
    "1.6.23",
    "1.6.24",
    "1.6.25",
    "1.6.26",
    "1.6.27",
    "1.6.28",
    "1.6.29",
    "1.6.30",
    "1.6.31",
    "1.6.32",
    "1.6.33",
    "1.6.34",
    "1.6.35",
    "1.6.36",
    "1.6.37",
    "1.6.38",
    "1.6.39",
    "1.6.40",
    "1.6.41",
    "1.6.42",
    "1.6.43",
    "1.6.44",
    "1.6.45",
    "1.6.46",
    "1.6.47",
    "1.6.48",
    "1.6.49",
    "1.6.50",
    "1.6.51",
    "1.6.52",
    "1.6.53",
    "1.6.54",
    "1.6.55",
    "1.6.56",
    "1.6.57",
    "1.6.58",
    "1.6.59",
    "1.6.60",
    "1.6.61",
    "1.6.62",
    "1.6.63",
    "1.6.64",
    "1.6.65",
    "1.6.66",
    "1.6.67",
    "1.6.68",
    "1.6.69",
    "1.6.70",
    "1.6.71",
    "1.6.72",
    "1.6.73",
    "1.6.74",
    "1.6.75",
    "1.6.76",
    "1.6.77",
    "1.6.78",
    "1.6.79",
    "1.6.80",
    "1.6.81",
    "1.6.82",
    "1.6.83",
    "1.6.84",
    "1.6.85",
    "1.6.86",
    "1.6.87",
    "1.6.88",
    "1.6.89",
    "1.6.90",
    "1.6.91",
    "1.6.92",
    "1.6.93",
    "1.6.94",
    "1.6.95",
    "1.6.96",
    "1.6.97",
    "1.6.98",
    "1.6.99",
    "1.6.100"
  ],
  "active_committee_members": [],
  "entropy": "0000000000000000000000000000000000000000"
}



[iHashFury]

iHashFury, how are you determining there are no vote in witnesses in place? Look at my last couple of posts. I noticed I had a non-zero "total_votes" earlier for example.

Code: [Select]

get_global_properties in the cli_wallet and watching the output of witness_node in tmux (screen alternative)

[Thom]

My witness must be voted it or it wouldn't be attempting to produce blocks, which it consistently is trying to do after the blk chn gets synced.

This time when it crashed memory was still only 9%, and cpu was at 100%.

My next attempt is to try this on a vps node. Now that I seem to have all the ingredients in place to produce blocks I don't understand what is causing the witness to crash in doing so. The vps will eliminate anything concerning my network here at home as well as the system itself. If I see the same crash symptom I am not sure what the next step should be.

iHashFury, how are you determining there are no vote in witnesses in place? Look at my last couple of posts. I noticed I had a non-zero "total_votes" earlier for example.

[iHashFury]

I have been producing blocks over the weekend both at home and on a VPS.

Now I can only see the 100 init witnesses on the witness_node. No voted in witnesses.

Code: [Select]

1458062ms th_a       witness.cpp:240               block_production_loo ] slot: 593448 scheduled_witness: 1.6.3 scheduled_time: 2015-08-24T17:24:18 now: 2015-08-24T17:24:18
1459048ms th_a       witness.cpp:240               block_production_loo ] slot: 593449 scheduled_witness: 1.6.83 scheduled_time: 2015-08-24T17:24:19 now: 2015-08-24T17:24:19
1460043ms th_a       witness.cpp:240               block_production_loo ] slot: 593450 scheduled_witness: 1.6.73 scheduled_time: 2015-08-24T17:24:20 now: 2015-08-24T17:24:20
1461044ms th_a       witness.cpp:240               block_production_loo ] slot: 593451 scheduled_witness: 1.6.23 scheduled_time: 2015-08-24T17:24:21 now: 2015-08-24T17:24:21
1462044ms th_a       witness.cpp:240               block_production_loo ] slot: 593452 scheduled_witness: 1.6.4 scheduled_time: 2015-08-24T17:24:22 now: 2015-08-24T17:24:22
1463052ms th_a       witness.cpp:240               block_production_loo ] slot: 593453 scheduled_witness: 1.6.8 scheduled_time: 2015-08-24T17:24:23 now: 2015-08-24T17:24:23
1464049ms th_a       witness.cpp:240               block_production_loo ] slot: 593454 scheduled_witness: 1.6.25 scheduled_time: 2015-08-24T17:24:24 now: 2015-08-24T17:24:24
1465055ms th_a       witness.cpp:240               block_production_loo ] slot: 593455 scheduled_witness: 1.6.82 scheduled_time: 2015-08-24T17:24:25 now: 2015-08-24T17:24:25
1466072ms th_a       witness.cpp:240               block_production_loo ] slot: 593456 scheduled_witness: 1.6.91 scheduled_time: 2015-08-24T17:24:26 now: 2015-08-24T17:24:26
1467044ms th_a       witness.cpp:240               block_production_loo ] slot: 593457 scheduled_witness: 1.6.9 scheduled_time: 2015-08-24T17:24:27 now: 2015-08-24T17:24:27
1468045ms th_a       witness.cpp:240               block_production_loo ] slot: 593458 scheduled_witness: 1.6.100 scheduled_time: 2015-08-24T17:24:28 now: 2015-08-24T17:24:28
1469042ms th_a       witness.cpp:240               block_production_loo ] slot: 593459 scheduled_witness: 1.6.51 scheduled_time: 2015-08-24T17:24:29 now: 2015-08-24T17:24:29 

Code: [Select]

Chain ID is d011922587473757011118587f93afcc314fbaea094fc1055574721b27975083
I have tried

Code: [Select]

--resync-blockchain -s "176.221.43.130:33323" -s "45.55.6.216:1776" -s "114.92.254.159:62015" and deleting blockchain, p2p folders.

I also lost imported CORE - but it could be not detaching the wallet correctly ctrl + D before a witness_node crash

Is there any thing else I could have missed?

[Thom]

Trying again, but forgot to remove --resync-blockchain from the cmd line. Next time it will not be there should the witness crash again.

It didn't take too long to get resynced, so in a few minutes I'll see if I produce blocks or crash...

I'd say it crashed. This time it resulted in the witness and cli locking up, then finally seg faulting after a long delay.

Trying again, this time with no resync cmd line param...

It resynced AND ignored my config.ini, witness output looks much different (not good):

Code: [Select]

deletech@Jessie:~/bts2.0/aug20$ ./witness
1247707ms th_a       main.cpp:112                  main                 ] Error parsing logging config from config file /home/deletech/bts2.0/aug20/graphene/programs/witness_node/test_net/config.ini, using default config
1247707ms th_a       witness.cpp:70                plugin_initialize    ] key_id_to_wif_pair: ["GPH........................","5.................................."]
1247707ms th_a       application.cpp:265           startup              ] Detected unclean shutdown. Replaying blockchain...
1247707ms th_a       application.cpp:228           operator()           ] Initializing database...
1268250ms th_a       db_management.cpp:67          wipe                 ] Wiping database
1268295ms th_a       object_database.cpp:82        wipe                 ] Wiping object_database.
1283059ms th_a       market_history_plugin.cpp:77  operator()           ] processing {"fee":{"amount":0,"asset_id":"1.3.527"},"order_id":"1.7.3","account_id":"1.2.2994","pays":{"amount":10000000,"asset_id":"1.3.0"},"receives":{"amount":4000,"asset_id":"1.3.527"}}
1283059ms th_a       market_history_plugin.cpp:123 operator()           ]     creating bucket {"id":"5.1.0","key":{"base":"1.3.0","quote":"1.3.527","seconds":15,"open":"2015-08-20T18:28:00"},"high_base":10000000,"high_quote":4000,"low_base":10000000,"low_quote":4000,"open_base":10000000,"open_quote":4000,"close_base":10000000,"close_quote":4000,"base_volume":10000000,"quote_volume":4000}
1283059ms th_a       market_history_plugin.cpp:127 operator()           ]     before updating bucket {"id":"5.1.0","key":{"base":"1.3.0","quote":"1.3.527","seconds":15,"open":"2015-08-20T18:28:00"},"high_base":10000000,"high_quote":4000,"low_base":10000000,"low_quote":4000,"open_base":10000000,"open_quote":4000,"close_base":10000000,"close_quote":4000,"base_volume":10000000,"quote_volume":4000}
1283059ms th_a       market_history_plugin.cpp:144 operator()           ]     after bucket bucket {"id":"5.1.0","key":{"base":"1.3.0","
I wiped the blockchain, logs and p2p folders of the wallet and attemped to restart (without the --resync-blockchain cmd line option) and I see this in the log:

Code: [Select]

1621853ms th_a       main.cpp:112                  main                 ] Error parsing logging config from config file /home/deletech/bts2.0/aug20/graphene/programs/witness_node/test_net/config.ini, using default config

I have no idea why this started showing up. Was it always there & I missed it? anyway trying once again. If it crashes this time I'll be looking to see what top says mem or cpu is doing. Tried using --checkpoint '[40902,"00009fc6d4e24db66e13af461b94063165999bfa"]' but decided to remove it on this trial and see what top says when it comes time for my witness to generate a block.

I'm running on a 4GB RAM quad core CPU system, and top says memory is only 8.5% and cpu is at 170% with 45 hours to go to resync.

[Thom]

I resumed using the exact same parameters as I used late Saturday. I was told by abit & puppies my being voted in status wouldn't change unless a new chain was started (if this occurred yesterday for example).

It looks like I'll need to wait until my blockchain is synced. It is old b/c I start with the --resync-blockchain parameter. When it is synced I expect I should start to produce blocks unless my voted in status has changed. I'm using the get_witness delegate.verbaltech API call and looking for a change in the next_secret_hash to confirm I'm producing blocks. Not seeing any change, and age of blockchain is still 7 days old.

I'm going to restart after clearing both the blockchain & p2p folders. My cmd line invocation is: ./witness_node --resync-blockchain -d test_net --enable-stale-production, all other parameters are in the config.ini.  After restarting the blockchain age reported by info started at 4 days and is now down to 57 hours. The witness output is much different than before I wiped the blk chn & p2p folders. It looks like I may finally be on the verge of producing blocks! I now see my total_votes is no longer zero.

Shit! The witness crashed as it was time to produce my first block:

Code: [Select]

77061ms th_a       witness.cpp:243               block_production_loo ] Witness 1.6.1621 production slot has arrived; generating a block now...
./witness: line 5: 23013 Segmentation fault      ./witness_node --resync-blockchain -d test_net --enable-stale-production

Is there a checkpoint I can use to resync faster, should I need to restart?

[spartako]

I have another problem, when I restart witness_node, usually it takes a long time to get connected to network, and sometimes won't get connected at all.

When I have connection problems I remove the p2p folder, it worked for me sometimes.

[abit]

Code: [Select]

3508896ms th_a       application.cpp:265           startup              ] Detected unclean shutdown. Replaying blockchain...
It takes too much time to replay blockchain.. How to "cleanly" shutdown the witness node?

Question bump.

I use this strategy:
When I am synced, I shutdown with C-c (it should shutdown in a clean way) and I copy the blockchain folder as backup (if it shutdown without errors).
Everytime my blockchain is corrupted, I remove the blockchain folder and I copy with the backup one and restart the witness.
Finally I backup the blockchain folder every day.

Thanks, it works. 

If witness_node was launched in gdb, we need to press ctrl+c, then from the (gdb) prompt, type 'signal SIGINT'. This will send SIGINT to the program being debugged. I always type 'kill' before I know this, so the program always shutdown uncleanly.

I have another problem, when I restart witness_node, usually it takes a long time to get connected to network, and sometimes won't get connected at all.

[Thom]

In the interest of not slipping the release date we are going to fall back to 3 or 5 second blocks using the current P2P code.   Then after we update the P2P code we can increase the block rate to 2 and ultimately 1 second block times.

Ben is in the process of preparing instructions for committee members on how to change the block interval and we plan to dynamically update the test network to prove that we can do this on a live network.

I previously asked about changes to terminology I've seen in this thread. I just did a search and found the only clue about "committee members" here: https://bitsharestalk.org/index.php/topic,17960.msg229223.html#msg229223  Is there a doc where this was discussed previously that I missed? If I understand correctly "committee members" are what was previously referred to as delegates in the trinity of witness / worker / delegate. Is that correct?

FYI if that is true I think it makes good sense, tho it would have been better had that change been considered when 2.0 was initially announced. Such is the nature of evolution in a fast past project. Nothing unusual or sinister going on.

[abit]

In the interest of not slipping the release date we are going to fall back to 3 or 5 second blocks using the current P2P code.   Then after we update the P2P code we can increase the block rate to 2 and ultimately 1 second block times.

Ben is in the process of preparing instructions for committee members on how to change the block interval and we plan to dynamically update the test network to prove that we can do this on a live network.

Sounds interesting.

[bytemaster]

In the interest of not slipping the release date we are going to fall back to 3 or 5 second blocks using the current P2P code.   Then after we update the P2P code we can increase the block rate to 2 and ultimately 1 second block times.

Ben is in the process of preparing instructions for committee members on how to change the block interval and we plan to dynamically update the test network to prove that we can do this on a live network.

[spartako]

I have added a new section Tips on the wiki.

https://github.com/cryptonomex/graphene/wiki/How-to-setup-your-witness-for-test-net-(Ubuntu-14.04)

So if you have any tips, you can add them there if is not general setup. I have just added spartako's from the previous post.

Also I have put some credits to list everyone

Thanks betax, good idea!

[betax]

I have added a new section Tips on the wiki.

https://github.com/cryptonomex/graphene/wiki/How-to-setup-your-witness-for-test-net-(Ubuntu-14.04)

So if you have any tips, you can add them there if is not general setup. I have just added spartako's from the previous post.

Also I have put some credits to list everyone

[betax]

Code: [Select]

3508896ms th_a       application.cpp:265           startup              ] Detected unclean shutdown. Replaying blockchain...
It takes too much time to replay blockchain.. How to "cleanly" shutdown the witness node?

Question bump.

I use this strategy:
When I am synced, I shutdown with C-c (it should shutdown in a clean way) and I copy the blockchain folder as backup (if it shutdown without errors).
Everytime my blockchain is corrupted, I remove the blockchain folder and I copy with the backup one and restart the witness.
Finally I backup the blockchain folder every day.

Good idea !