During the stress test, all inits are forked. Are they on the same machine?

It seems that the stress testing is being handled by the witnesses, it is just the UI that has some random freezes.

Quote from: bytemaster on October 07, 2015, 07:29:22 PM

Quote from: rnglab on October 07, 2015, 06:58:34 PM

Quote from: bytemaster on October 07, 2015, 01:11:40 PM
bitshares-argentina  can you please let me know if you have changed your software in any way because it appears your node is consistently producing blocks even though you are not in the set of active witnesses.

Everyone else, please update your witness to the latest master before block 58500  (about 4 hours).

Hi dan, I didn't change anything. Neither played with uncommon settings as far as I remember . 

Running from master branch with last commit db84a492b9a2cf290f92a3987b59c11808138d56

How can I help debugging?

Can you post your config file and command line arguments.

It seems to be the same issue with the time:


1272283ms th_a       db_block.cpp:189              _push_block          ] Failed to push new block:
10 assert_exception: Assert Exception
next_block.witness == scheduled_witness: Witness produced block at wrong time
    {"block witness":"1.6.37","scheduled":"1.6.39","slot_num":1}
    th_a  db_block.cpp:644 validate_block_header

    {"next_block.block_num()":62650}
    th_a  db_block.cpp:509 _apply_block
1272290ms th_a       application.cpp:425           handle_block         ] Error when pushing block:
10 assert_exception: Assert Exception
next_block.witness == scheduled_witness: Witness produced block at wrong time
    {"block witness":"1.6.37","scheduled":"1.6.39","slot_num":1}
    th_a  db_block.cpp:644 validate_block_header

    {"next_block.block_num()":62650}
    th_a  db_block.cpp:509 _apply_block

    {"new_block":{"previous":"0000f4b90fc8a15cc975aa571539d3e7f9053b6f","timestamp":"2015-10-07T21:21:12","witness":"1.6.37","transaction_merkle_root":"aa42913b2a226f9132200a3cb19aa076e3c0d7b4","extensions":[],"witness_signature":"20449d3e22302d92dd18f54cd2406de08678774e6648073db27409ea25bdcd4425561a3a7e75a484ee161eab4e02ba806147824747553537ed57c964af3261e330","transactions":[{"ref_block_num":62648,"ref_block_prefix":1109897537,"expiration":"2015-10-07T21:21:36","operations":[[19,{"fee":{"amount":100000,"asset_id":"1.3.0"},"publisher":"1.2.67704","asset_id":"1.3.285","feed":{"settlement_price":{"base":{"amount":3327214,"asset_id":"1.3.285"},"quote":{"amount":1000000000,"asset_id":"1.3.0"}},"maintenance_collateral_ratio":1750,"maximum_short_squeeze_ratio":1500,"core_exchange_rate":{"base":{"amount":1000000000,"asset_id":"1.3.0"},"quote":{"amount":3327214,"asset_id":"1.3.285"}}},"extensions":[]}]],"extensions":[],"signatures":["1f3a600a648a94885229e3b4cc91a8f8ea3d6cb368f555bf52ba233a44e648e08170d12c36bca47097846b6cd334c69392bc15a2796d87b620172235bd5bff705c"],"operation_results":[[0,{}]]},{"ref_block_num":62648,"ref_block_prefix":1109897537,"expiration":"2015-10-07T21:21:36","operations":[[19,{"fee":{"amount":100000,"asset_id":"1.3.0"},"publisher":"1.2.67704","asset_id":"1.3.218","feed":{"settlement_price":{"base":{"amount":21452528,"asset_id":"1.3.218"},"quote":{"amount":1000000000,"asset_id":"1.3.0"}},"maintenance_collateral_ratio":1750,"maximum_short_squeeze_ratio":1500,"core_exchange_rate":{"base":{"amount":1000000000,"asset_id":"1.3.0"},"quote":{"amount":21452528,"asset_id":"1.3.218"}}},"extensions":[]}]],"extensions":[],"signatures":["1f7e6481ccef5712d331e5613cfa706e4f6821fffd58ab88a3b05a7c1d18b2bda80487e0b2007369d6c10eeb8453f07573717c7cf15f3e5b4d17db45c6347cc49e"],"operation_results":[[0,{}]]},{"ref_block_num":62648,"ref_block_prefix":1109897537,"expiration":"2015-10-07T21:21:36","operations":[[19,{"fee":{"amount":100000,"asset_id":"1.3.0"},"publisher":"1.2.67704","asset_id":"1.3.536","feed":{"settlement_price":{"base":{"amount":523393,"asset_id":"1.3.536"},"quote":{"amount":1000000000,"asset_id":"1.3.0"}},"maintenance_collateral_ratio":1750,"maximum_short_squeeze_ratio":1500,"core_exchange_rate":{"base":{"amount":1000000000,"asset_id":"1.3.0"},"quote":{"amount":523393,"asset_id":"1.3.536"}}},"extensions":[]}]],"extensions":[],"signatures":["203cb740ed1f1706d0f0e0b273234fb3902ef987a28271d3005d613a3e9582075a4f1f2010d7ef27af1e2dbc2e78ecded4a66d1f5afb110333fc83eb84b52bccb3"],"operation_results":[[0,{}]]},{"ref_block_num":62648,"ref_block_prefix":1109897537,"expiration":"2015-10-07T21:21:36","operations":[[19,{"fee":{"amount":100000,"asset_id":"1.3.0"},"publisher":"1.2.67704","asset_id":"1.3.330","feed":{"settlement_price":{"base":{"amount":465809,"asset_id":"1.3.330"},"quote":{"amount":1000000000,"asset_id":"1.3.0"}},"maintenance_collateral_ratio":1750,"maximum_short_squeeze_ratio":1500,"core_exchange_rate":{"base":{"amount":1000000000,"asset_id":"1.3.0"},"quote":{"amount":465809,"asset_id":"1.3.330"}}},"extensions":[]}]],"extensions":[],"signatures":["1f0e44bd90670d6d35ad25202d54ec45968422d6315db6eb611c785f6dbc9c239d03b1da1049e9f429fce90e6ba6cc3c0e034e2803b04dd76ab72b3d53ad872082"],"operation_results":[[0,{}]]}]}}
    th_a  db_block.cpp:195 _push_block


plus, mine seem to needed a wipe and resync of the database...

Quote from: clayop on October 07, 2015, 08:00:36 PM
60 TPS / 20 min test will begin at 21:00 UTC

Add: CORE donation to 'clayop' is more than welcome :D

Oops.... made a mistake. The test will begin at 21:25 UTC. Sorry about that.

Thanks Thom!

Quote from: clayop on October 07, 2015, 08:00:36 PM
60 TPS / 20 min test will begin at 21:00 UTC

Add: CORE donation to 'clayop' is more than welcome :D

Sent you 4000 CORE

60 TPS / 20 min test will begin at 21:00 UTC

Add: CORE donation to 'clayop' is more than welcome :D

Quote from: bytemaster on October 07, 2015, 06:52:21 PM
It isn't just a matter of getting high end servers, we need to actually change the P2P protocol before we can hope to achieve those speeds.

When you are not busy, could you read my post about witnesses signing blocks in General Discussion?

Quote from: bytemaster on October 07, 2015, 06:44:33 PM

Quote from: jakub on October 06, 2015, 11:03:31 PM
BM, I think there are now 3 witnesses waiting to be voted in:

bitshares-argentina
triox-delegate
pmc

Voted.   Any others?

yep, I think I got voted out by this (had only my votes) 1.6.31

Quote from: bytemaster on October 07, 2015, 07:29:22 PM

Quote from: rnglab on October 07, 2015, 06:58:34 PM

Quote from: bytemaster on October 07, 2015, 01:11:40 PM
bitshares-argentina  can you please let me know if you have changed your software in any way because it appears your node is consistently producing blocks even though you are not in the set of active witnesses.

Everyone else, please update your witness to the latest master before block 58500  (about 4 hours).

Hi dan, I didn't change anything. Neither played with uncommon settings as far as I remember . 

Running from master branch with last commit db84a492b9a2cf290f92a3987b59c11808138d56

How can I help debugging?

Can you post your config file and command line arguments.

./witness_node -d oct5-2 --genesis-json oct5-genesis.json -s "104.236.51.238:2005"


# Enable block production, even if the chain is stale.
enable-stale-production = false

# Percent of witnesses (0-99) that must be participating in order to produce blocks
required-participation = false

# ID of witness controlled by this node (e.g. "1.6.5", quotes are required, may specify multiple times)
witness-id = "1.6.37"

# Tuple of [PublicKey, WIF private key] (may specify multiple times)
private-key = ["GPH7vyDvX1dxcPvXs63A4cctf8ZFRcb7a3QGWoQYdX6ckuKJonbyx","5Kxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"]


updated but not producing blocks. Now when I dump_private_keys there's no priv key for the signing key. I'll try on a new wallet.

Quote from: rnglab on October 07, 2015, 06:58:34 PM

Quote from: bytemaster on October 07, 2015, 01:11:40 PM
bitshares-argentina  can you please let me know if you have changed your software in any way because it appears your node is consistently producing blocks even though you are not in the set of active witnesses.

Everyone else, please update your witness to the latest master before block 58500  (about 4 hours).

Hi dan, I didn't change anything. Neither played with uncommon settings as far as I remember . 

Running from master branch with last commit db84a492b9a2cf290f92a3987b59c11808138d56

How can I help debugging?

Can you post your config file and command line arguments. 

Would PTP be a viable option?

I see. Thanks for the thorough explanation.

Quote from: bytemaster on October 07, 2015, 07:21:13 PM

Quote from: Thom on October 07, 2015, 06:59:46 PM

Quote from: bytemaster on October 07, 2015, 06:46:37 PM
The accuracy of each individual computers clock can be off by up to 300 ms.   It just means that your local clock is ahead of the other peer by 23 ms plus the network latency in fetching the block.

So there is a margin of error of 300ms in latency numbers?

That makes it pretty tough to get an accurate picture. I thought ntp would take care of that, not so?

The built in NTP client pings an NTP server and updates an internal offset if the delta time between the request and response is less than 300 ms.   So if you can ping the NTP server in 0 ms then you can have up to 300 ms in error.   If your ping time to the NTP server is 100 ms, then you will only have 200 ms of error.

Technically it is possible to get a more accurate time sync than this.

Quote
NTP v4 with kernel mods to support it, is capable of much better than 1ms accuracy, possibly as good as 1ns. According to [Dave Mills] article, NTP v3 is accurate to 1-2ms in a LAN and 10s of ms in WAN nets. http://www.cis.udel.edu/~mills/ntp.html

Other articles suggest that with an accurate time source, such as a GPS time source, NTP is accurate to 50us, but the links on the Linux kernel support say that accuracy of a few ms are possible. http://www.atomic-clock.galleon.eu.com/support/ntp-time-server-accuracy.html

Another article says that it is dependent on the predictability of network delays (i.e. a low jitter network). http://www.postel.org/pipermail/end2end-interest/2003-April/002925.html

In other words, highly paid witnesses could configure their local clocks with higher quality NTP solutions and get down to under 100 ms.    It would be a lot of work to get the internal NTP to be more accurate.

Quote from: Thom on October 07, 2015, 06:59:46 PM

Quote from: bytemaster on October 07, 2015, 06:46:37 PM
The accuracy of each individual computers clock can be off by up to 300 ms.   It just means that your local clock is ahead of the other peer by 23 ms plus the network latency in fetching the block.

So there is a margin of error of 300ms in latency numbers?

That makes it pretty tough to get an accurate picture. I thought ntp would take care of that, not so?

The built in NTP client pings an NTP server and updates an internal offset if the delta time between the request and response is less than 300 ms.   So if you can ping the NTP server in 0 ms then you can have up to 300 ms in error.   If your ping time to the NTP server is 100 ms, then you will only have 200 ms of error.

Technically it is possible to get a more accurate time sync than this.

Quote
NTP v4 with kernel mods to support it, is capable of much better than 1ms accuracy, possibly as good as 1ns. According to [Dave Mills] article, NTP v3 is accurate to 1-2ms in a LAN and 10s of ms in WAN nets. http://www.cis.udel.edu/~mills/ntp.html

Other articles suggest that with an accurate time source, such as a GPS time source, NTP is accurate to 50us, but the links on the Linux kernel support say that accuracy of a few ms are possible. http://www.atomic-clock.galleon.eu.com/support/ntp-time-server-accuracy.html

Another article says that it is dependent on the predictability of network delays (i.e. a low jitter network). http://www.postel.org/pipermail/end2end-interest/2003-April/002925.html

In other words, highly paid witnesses could configure their local clocks with higher quality NTP solutions and get down to under 100 ms.    It would be a lot of work to get the internal NTP to be more accurate.   

Quote from: bytemaster on October 07, 2015, 06:44:33 PM

Quote from: jakub on October 06, 2015, 11:03:31 PM
BM, I think there are now 3 witnesses waiting to be voted in:

bitshares-argentina
triox-delegate
pmc

Voted.   Any others?

betaxtrade (please)

Quote from: rnglab on October 07, 2015, 07:03:11 PM

Quote from: bytemaster on October 07, 2015, 06:32:32 PM
This particular witness has been producing blocks at the wrong time this whole time and fortunately helped us identify a bug that we fixed with a hard fork that took effect and is now generating these error messages.

@bytemaster let me know if I should update 1.6.37 now or if you need  it for debugging

You should upgrade to the latest master and reindex.