Test Net for Advanced Users

[bytemaster]

Latest master should still work with the test net.

[ebit]

 

[Thom]

once you sync, do a backup of your blockchain it will help you later on. See spartako's comment.

Code: [Select]

3508896ms th_a       application.cpp:265           startup              ] Detected unclean shutdown. Replaying blockchain...
It takes too much time to replay blockchain.. How to "cleanly" shutdown the witness node?

Question bump.

I use this strategy:
When I am synced, I shutdown with C-c (it should shutdown in a clean way) and I copy the blockchain folder as backup (if it shutdown without errors).
Everytime my blockchain is corrupted, I remove the blockchain folder and I copy with the backup one and restart the witness.
Finally I backup the blockchain folder every day.

Thx betax.

Now my vps node is behaving like my home system. As soon as my witness is called on to produce a block it seg faults & dies. As soon as it gets resynced it is scheduled so I can't quit to save the chain.

The last time it died I cleared the blockchain (+ p2p and logs folders) with rm -rf graphene/programs/witness_node/test_net/blockchain/*

That has always allowed me to restart, but now I can't without this crashing. Here's the output:

Code: [Select]

./witness: line 7:  32471 Segmentation fault      ./witness_node --resync-blockchain -d test_net
I almost seems like the executable gets corrupted somehow, but that seems highly unlikely. Yet 2 systems are now behaving the same way.

This time I'm also deleting the object_database folder.

@BM: are you planning on publishing that patch or providing a new tag we can build those performance improvements with?

[bytemaster]

The call that is consuming over 40% of the CPU time while reindexing only takes .00018 seconds to run on average and does so once per block.   

After recent optimizations I am down to 146 seconds to reindex the past 3 days.

[bytemaster]

It takes about 10 minutes to resync current chain. But need at least 10 minutes to replay if crash. Backup is a good idea.

Code: [Select]

{
  "head_block_num": 319236,
  "head_block_id": "0004df042bd2fc6c4bf3168d7116de8d0e55842f",
  "head_block_age": "1 second old",
}


I did some looking into why it takes so long to reindex because it seemed to me that 10 minutes was entirely too long for 3 days worth of blocks.   I found two spots in the code that are dramatically slowing down the reindex process:

1. There are over 2 years worth of daily maintenance intervals being performed because we have 5 minute maintenance intervals on the test network.   If this were a real network then there would have only been .3% of the maintenance intervals and reindexing time would have been reduced by about 40%.   Maintenance intervals are where we tally all votes and is a relatively expensive calculation, especially when the test network has all of the same accounts as BTS after 1 year.

2. The rest of the time is being spent updating the witness schedule which involves some expensive hashing and set containers.      There is no reason to perform these operations during a reindex because we assume a valid blockchain while reindexing. This accounts for about 36% of the computational time.   

3. We were maintaining undo history while reindexing which is unnecessary, I have just checked in a patch to resolve this.   After doing this simple (and planned) optimization, reindexing takes 152 seconds on my computer.

If we remove first two factors then reindexing could be performed in 25% of the time which would be 38 seconds to reindex or about 10 seconds per day, or 1 hour per year with 1 second blocks.   

There is still a lot of low-hanging fruit that can be used to accelerate this reindexing performance. 

[abit]

It takes about 10 minutes to resync current chain. But need at least 10 minutes to replay if crash. Backup is a good idea.

Code: [Select]

{
  "head_block_num": 319236,
  "head_block_id": "0004df042bd2fc6c4bf3168d7116de8d0e55842f",
  "head_block_age": "1 second old",
}


[iHashFury]

back in the game

seed nodes and a new -d (chain folder) did the job

Thanks everyone for the list of seed nodes

[Thom]

The vps node is up and running, the blk chn appears to be synced, and I appear to be producing blocks!!! Whew who!!!!!!!!!! Finally!!!

I haven't started with the other seed nodes, but my config.ini now lists them for the next restart:

Code: [Select]

seed-node = 45.55.6.216:1776
seed-node = 45.115.36.171:57281
seed-node = 45.55.6.216:37308
seed-node = 104.200.28.117:61705
seed-node = 104.236.51.238:1776
seed-node = 104.156.226.183:60715
seed-node = 104.156.226.183:40479
seed-node = 104.236.255.53:52995
seed-node = 114.92.254.159:62015
seed-node = 114.92.254.159:62015
seed-node = 176.221.43.130:33323
seed-node = 176.9.234.167:34858
seed-node = 176.9.234.167:57727
seed-node = 178.62.88.151:59148
seed-node = 178.62.88.151:41574
seed-node = 188.226.252.109:58843

Witness created: "1.6.1621" ("1.6.1530 if you count my previous non-producing attempts)
Please send my brownies reward here: BTS68hiHJQ1vBhsJYr7Bg61udi1fgmQYDr2MvGF8EkhPsaH1AAKus

[Thom]

I'm going to restart after clearing both the blockchain & p2p folders. My cmd line invocation is: ./witness_node --resync-blockchain -d test_net --enable-stale-production, all other parameters are in the config.ini.  After restarting the blockchain age reported by info started at 4 days and is now down to 57 hours. The witness output is much different than before I wiped the blk chn & p2p folders. It looks like I may finally be on the verge of producing blocks! I now see my total_votes is no longer zero.

Don't use '--enable-stale-production' parameter. It's for the init node.

Is there a checkpoint I can use to resync faster, should I need to restart?

Try start with more seed nodes:

Code: [Select]

-s "104.236.51.238:1776" -s "176.221.43.130:33323" -s "45.55.6.216:1776" -s "114.92.254.159:62015" -s 104.156.226.183:60715 -s 104.156.226.183:40479 -s 104.236.255.53:52995 -s 176.9.234.167:34858 -s 176.9.234.167:57727 -s 178.62.88.151:59148 -s 178.62.88.151:41574 -s 188.226.252.109:58843 -s 45.115.36.171:57281 -s 45.55.6.216:37308


I'm using these seed nodes in the config.ini:

Code: [Select]

# P2P nodes to connect to on startup (may specify multiple times)
seed-node = 114.92.254.159:62015
seed-node = 104.200.28.117:61705
seed-node = 104.236.51.238:1776
seed-node = 176.221.43.130:33323
seed-node = 45.55.6.216:1776

I'll add the others you listed and remove the enable-stale-production if it crashes on the vps.

[betax]

once you sync, do a backup of your blockchain it will help you later on. See spartako's comment.

Code: [Select]

3508896ms th_a       application.cpp:265           startup              ] Detected unclean shutdown. Replaying blockchain...
It takes too much time to replay blockchain.. How to "cleanly" shutdown the witness node?

Question bump.

I use this strategy:
When I am synced, I shutdown with C-c (it should shutdown in a clean way) and I copy the blockchain folder as backup (if it shutdown without errors).
Everytime my blockchain is corrupted, I remove the blockchain folder and I copy with the backup one and restart the witness.
Finally I backup the blockchain folder every day.

[abit]

I'm going to restart after clearing both the blockchain & p2p folders. My cmd line invocation is: ./witness_node --resync-blockchain -d test_net --enable-stale-production, all other parameters are in the config.ini.  After restarting the blockchain age reported by info started at 4 days and is now down to 57 hours. The witness output is much different than before I wiped the blk chn & p2p folders. It looks like I may finally be on the verge of producing blocks! I now see my total_votes is no longer zero.

Don't use '--enable-stale-production' parameter. It's for the init node.

Is there a checkpoint I can use to resync faster, should I need to restart?

Try start with more seed nodes:

Code: [Select]

-s "104.236.51.238:1776" -s "176.221.43.130:33323" -s "45.55.6.216:1776" -s "114.92.254.159:62015" -s 104.156.226.183:60715 -s 104.156.226.183:40479 -s 104.236.255.53:52995 -s 176.9.234.167:34858 -s 176.9.234.167:57727 -s 178.62.88.151:59148 -s 178.62.88.151:41574 -s 188.226.252.109:58843 -s 45.115.36.171:57281 -s 45.55.6.216:37308


[abit]

info also works in the cli_wallet

Code: [Select]

info
info
{
  "head_block_num": 0,
  "head_block_id": "0000000000000000000000000000000000000000",
  "head_block_age": "7 days old",
  "next_maintenance_time": "45 years  ago",
  "chain_id": "d011922587473757011118587f93afcc314fbaea094fc1055574721b27975083",
  "active_witnesses": [
    "1.6.0",
    "1.6.1",
    "1.6.2",
    "1.6.3",
    "1.6.4",
    "1.6.5",
    "1.6.6",
    "1.6.7",
    "1.6.8",
    "1.6.9",
    "1.6.10",
    "1.6.11",
    "1.6.12",
    "1.6.13",
    "1.6.14",
    "1.6.15",
    "1.6.16",
    "1.6.17",
    "1.6.18",
    "1.6.19",
    "1.6.20",
    "1.6.21",
    "1.6.22",
    "1.6.23",
    "1.6.24",
    "1.6.25",
    "1.6.26",
    "1.6.27",
    "1.6.28",
    "1.6.29",
    "1.6.30",
    "1.6.31",
    "1.6.32",
    "1.6.33",
    "1.6.34",
    "1.6.35",
    "1.6.36",
    "1.6.37",
    "1.6.38",
    "1.6.39",
    "1.6.40",
    "1.6.41",
    "1.6.42",
    "1.6.43",
    "1.6.44",
    "1.6.45",
    "1.6.46",
    "1.6.47",
    "1.6.48",
    "1.6.49",
    "1.6.50",
    "1.6.51",
    "1.6.52",
    "1.6.53",
    "1.6.54",
    "1.6.55",
    "1.6.56",
    "1.6.57",
    "1.6.58",
    "1.6.59",
    "1.6.60",
    "1.6.61",
    "1.6.62",
    "1.6.63",
    "1.6.64",
    "1.6.65",
    "1.6.66",
    "1.6.67",
    "1.6.68",
    "1.6.69",
    "1.6.70",
    "1.6.71",
    "1.6.72",
    "1.6.73",
    "1.6.74",
    "1.6.75",
    "1.6.76",
    "1.6.77",
    "1.6.78",
    "1.6.79",
    "1.6.80",
    "1.6.81",
    "1.6.82",
    "1.6.83",
    "1.6.84",
    "1.6.85",
    "1.6.86",
    "1.6.87",
    "1.6.88",
    "1.6.89",
    "1.6.90",
    "1.6.91",
    "1.6.92",
    "1.6.93",
    "1.6.94",
    "1.6.95",
    "1.6.96",
    "1.6.97",
    "1.6.98",
    "1.6.99",
    "1.6.100"
  ],
  "active_committee_members": [],
  "entropy": "0000000000000000000000000000000000000000"
}



You're out of sync. Try add more seed nods and resync.

Code: [Select]

-s "104.236.51.238:1776" -s "176.221.43.130:33323" -s "45.55.6.216:1776" -s "114.92.254.159:62015" -s 104.156.226.183:60715 -s 104.156.226.183:40479 -s 104.236.255.53:52995 -s 176.9.234.167:34858 -s 176.9.234.167:57727 -s 178.62.88.151:59148 -s 178.62.88.151:41574 -s 188.226.252.109:58843 -s 45.115.36.171:57281 -s 45.55.6.216:37308


[Thom]

at 36 hrs to go for resync, my delegate info shows:

Code: [Select]

get_witness delegate.verbaltech
{
  "id": "1.6.1621",
  "witness_account": "1.2.22408",
  "signing_key": "GPH52ms1dYJko2v5vS3rCdVLzQBogjeDRc1CpkaZ4seC4J4H7Uc71",
  "next_secret_hash": "5c..........................................65",
  "previous_secret": "0000000000000000000000000000000000000000",
  "vote_id": "1:1621",
  "total_votes": "4389187579",
  "url": ""
}

Still can't scroll in tmux - c-b pgUp or c-b pgDn does nothing. Scroll wheel does the same as pgUp / dn, it just scrolls the last cmd line like up or dn arrow does.

Build on vps is 95% done. Hoping for different results than on my local machine. We'll see soon...

[iHashFury]

ctrl +b then press page up or down

to scroll in tmux - you can then use the mouse wheel

Esc to cancel

[Thom]

You seem to be on a fork, based on the 7 day old chain. Your chain-id is correct, that's what I'm using.

Review my posts and you will see I too had the 7 day old thing going on, then I changed someting and now starting from scratch starts at 4 days old.

Make sure you remove p2p/* and logs/* too.

Restarted. Using your global API I see 3 active witnesses:

    "1.6.5246",
    "1.6.5247",
    "1.6.5248"

Mine isn't active yet b/c the blk chn isn't yet synced. As soon as it does and my witness tries to produce a block it crashes.

I see many more active as the blk chn gets closer to being synced:

    "1.6.1526",
    "1.6.1530",
    "1.6.1537",
    "1.6.1625",
    "1.6.5246",
    "1.6.5247",
    "1.6.5248",
    "1.6.5249",
    "1.6.5250",
    "1.6.5251",
    "1.6.5252"