Test Net for Advanced Users

[bytemaster]

https://bitsharestalk.org/index.php/topic,18699.msg240322.html#msg240322

[bytemaster]

... it resulted in a hard fork several days ago because BitUSD technically had a black swan event back then.

and why have we not identified the event just after it occurred and only a couple of days later?

Because the market was thin


Sent from my iPhone using Tapatalk

Black Swan detection was only kicking in if there was a margin call, but margin calls were not triggering because there were no orders above the short-squeeze protection threshold.   
Black Swan should only happen when the MAX( Highest Bid, Settlement Price ) <  LeastCollateralizedPosition.   I moved the check to catch sooner in the evaluation which means that
the BlackSwan occurred as soon as the price feed was updated rather than delayed until the first forced settlement.

[sittingduck]

... it resulted in a hard fork several days ago because BitUSD technically had a black swan event back then.

and why have we not identified the event just after it occurred and only a couple of days later?

Because the market was thin


Sent from my iPhone using Tapatalk

[CalabiYau]

@abit: Congratulations ... it seems the both of us were the very reason for the first every blockchain based black swan event :-P

  that`s what legends are made off.....

[liondani]

... it resulted in a hard fork several days ago because BitUSD technically had a black swan event back then.

and why have we not identified the event just after it occurred and only a couple of days later?

[xeroc]

@abit: Congratulations ... it seems the both of us were the very reason for the first every blockchain based black swan event :-P

[abit]

I have just checked in a fix for this issue, but it resulted in a hard fork several days ago because BitUSD technically had a black swan event back then.

Theoretical is working on unit tests for the edge cases we have identified so that we can verify the fix works in all cases.

We will launch a new test network tomorrow based upon this fix and the party can continue.

I'm unable to access my server right now. Wish I could be back when the new test network is ready.

[abit]

It took 75 seconds to replay the 280k-blocks chain.

Maybe on a SLOW vps, but on my machine it takes a mere 11 8.79 seconds.

In Debug build it took me 59 seconds to reindex.
In Release build it took me 8.79 seconds to reindex.

There were a total of 444758 operations in the blockchain which means we are averaging over 50K TPS.
The average maintenance calculation time is:  7 ms 
1.5 seconds were spent doing maintenance calculations (on the hour, instead of every day) leaving us at 62K TPS average
The vast majority of blocks have been empty which means our TPS average is being diluted by block processing overhead.

It looks like in Debug mode we process closer to 10K TPS.

Yes I'm running with debug build. I remember the log said replay took about 60 seconds, but CPU kept at 100% for 75 seconds. Maybe the 15 more seconds are before or after replay.

[Thom]

I have just checked in a fix for this issue, but it resulted in a hard fork several days ago because BitUSD technically had a black swan event back then.

Theoretical is working on unit tests for the edge cases we have identified so that we can verify the fix works in all cases.

We will launch a new test network tomorrow based upon this fix and the party can continue.

Awesome, I'll be ready to join back in with several VPS seed nodes to help.

[bytemaster]

I have just checked in a fix for this issue, but it resulted in a hard fork several days ago because BitUSD technically had a black swan event back then.

Theoretical is working on unit tests for the edge cases we have identified so that we can verify the fix works in all cases.

We will launch a new test network tomorrow based upon this fix and the party can continue.

[bytemaster]

4.  Attempt to force settle resulted in a condition that was previously believed to be impossible (a black swan)

  Very good that this happened, so this case can be properly reviewed and re-tested.

 

Yep I am glad it occurred on the test-network too !

We have identified the problem and have a solution in the works.    A big thank you to all of the testers who are pushing the test network through all kinds of edge cases.

[liondani]

4.  Attempt to force settle resulted in a condition that was previously believed to be impossible (a black swan)

  Very good that this happened, so this case can be properly reviewed and re-tested.

 

Yep I am glad it occurred on the test-network too !

[testz]

4.  Attempt to force settle resulted in a condition that was previously believed to be impossible (a black swan)

  Very good that this happened, so this case can be properly reviewed and re-tested.

[bytemaster]

Quick Update:

We have identified the corner case that caused the problem and are preparing fixes for it now.   A quick summary:

1.  Early feed producers were off by a factor of 10
2.  Early margin testors created margin positions based upon the old feed prices
3.  Feed producers fixed their feed which left the early margin testors in black-swan territory
4.  Attempt to force settle resulted in a condition that was previously believed to be impossible (a black swan)

The fix we are adding will cause BitUSD to black swan on the test network. 

We are in the process of auditing the transaction history to answer a couple of questions:

1. Should the black swan have been triggered much earlier as a result of market orders being placed, canceled, etc?
2. If it should have, why wasn't it?

We are also evaluating if we should take any precautions to protect the network from future unexpected bugs.   Right now there exist
a slew of operations that are applied outside the context of a transaction.  If any of those operations fail then the network will be unable
to produce blocks.   In effect, these operations should always succeed if the transaction evaluation maintains the proper invariants.

Will be back with more updates when we have them.

[bytemaster]

"head_block_num": 314814,
  "head_block_id": "0004cdbe86f24753a0133e23c6e26ebd3c05bf9e",
  "head_block_age": "6 hours old",
  "next_maintenance_time": "6 hours ago",
  "chain_id": "0f8b631d7a9dfebf16d6776fab96b629a14429762bf9c3eb95db1e4e4af637a4",


While replaying the blockchain

Code: [Select]


   90.212%   284000 of 314814                                                                                                                                       

1858849ms th_a       db_block.cpp:263              push_proposal        ] e
1858850ms th_a       db_update.cpp:140             clear_expired_propos ] Failed to apply proposed transaction on its expiration. Deleting it.
{"id":"1.10.1","expiration_time":"2015-09-29T05:15:00","review_period_time":"2015-09-29T04:15:00","proposed_transaction":{"ref_block_num":0,"ref_block_prefix":0,"expiration":"
2015-09-29T05:15:00","operations":[[31,{"fee":{"amount":1000000,"asset_id":"1.3.0"},"new_parameters":{"current_fees":{"parameters":[[0,{"fee":2000000,"price_per_kbyte":1000000
}],[1,{"fee":500000}],[2,{"fee":0}],[3,{"fee":2000000}],[4,{}],[5,{"basic_fee":0,"premium_fee":200000000,"price_per_kbyte":100000}],[6,{"fee":2000000,"price_per_kbyte":100000}
],[7,{"fee":300000}],[8,{"membership_annual_fee":200000000,"membership_lifetime_fee":1000000000}],[9,{"fee":50000000}],[10,{"symbol3":"50000000000","symbol4":"30000000000","lo
ng_symbol":500000000,"price_per_kbyte":10}],[11,{"fee":50000000,"price_per_kbyte":10}],[12,{"fee":50000000}],[13,{"fee":50000000}],[14,{"fee":2000000,"price_per_kbyte":100000}
],[15,{"fee":2000000}],[16,{"fee":100000}],[17,{"fee":10000000}],[18,{"fee":50000000}],[19,{"fee":100000}],[20,{"fee":500000000}],[21,{"fee":2000000}],[22,{"fee":2000000,"pric
e_per_kbyte":10}],[23,{"fee":100000,"price_per_kbyte":10}],[24,{"fee":100000}],[25,{"fee":100000}],[26,{"fee":2000000}],[27,{"fee":0,"price_per_kbyte":10}],[28,{"fee":50000000
0}],[29,{"fee":100000}],[30,{"fee":100000}],[31,{"fee":2000000}],[32,{"fee":500000000}],[33,{"fee":100000}],[34,{"fee":100000}],[35,{"fee":100000,"price_per_kbyte":10}],[36,{"
fee":2000000}],[37,{}],[38,{"fee":500000,"price_per_kbyte":10}],[39,{"fee":500000,"price_per_output":500000}]],"scale":5000},"block_interval":3,"maintenance_interval":5400,"ma
intenance_skip_slots":4,"committee_proposal_review_period":300,"maximum_transaction_size":65536,"maximum_block_size":10485760,"maximum_time_until_expiration":86400,"maximum_pr
oposal_lifetime":2419200,"maximum_asset_whitelist_authorities":10,"maximum_asset_feed_publishers":10,"maximum_witness_count":1001,"maximum_committee_count":1001,"maximum_autho
rity_membership":10,"reserve_percent_of_fee":2000,"network_percent_of_fee":2000,"lifetime_referrer_percent_of_fee":3000,"cashback_vesting_period_seconds":7776000,"cashback_ves
ting_threshold":10000000,"count_non_member_votes":true,"allow_non_member_whitelists":false,"witness_pay_per_block":500000,"worker_budget_per_day":"50000000000","max_predicate_
opcode":1,"fee_liquidation_threshold":10000000,"accounts_per_fee_scale":1000,"account_fee_scale_bitshifts":4,"max_authority_depth":2,"extensions":[]}}]],"extensions":[]},"requ
ired_active_approvals":["1.2.0"],"available_active_approvals":["1.2.102","1.2.103","1.2.104","1.2.105","1.2.106","1.2.107","1.2.108","1.2.109","1.2.110"],"required_owner_appro
vals":[],"available_owner_approvals":[],"available_key_approvals":[]}
10 assert_exception: Assert Exception
itr->get_balance() >= -delta: Insufficient Balance: committee-account's balance of 0 CORE is less than required 10 CORE
    {"a":"committee-account","b":"0 CORE","r":"10 CORE"}
    th_a  db_balance.cpp:67 adjust_balance

    {"account":"1.2.0","delta":{"amount":-1000000,"asset_id":"1.3.0"}}
    th_a  db_balance.cpp:73 adjust_balance

    {}
    th_a  evaluator.cpp:42 start_evaluate

intenance_skip_slots":4,"committee_proposal_review_period":300,"maximum_transaction_size":65536,"maximum_block_size":10485760,"maximum_time_until_expiration":86400,"[540/1909]
oposal_lifetime":2419200,"maximum_asset_whitelist_authorities":10,"maximum_asset_feed_publishers":10,"maximum_witness_count":1001,"maximum_committee_count":1001,"maximum_autho
rity_membership":10,"reserve_percent_of_fee":2000,"network_percent_of_fee":2000,"lifetime_referrer_percent_of_fee":3000,"cashback_vesting_period_seconds":7776000,"cashback_ves
ting_threshold":10000000,"count_non_member_votes":true,"allow_non_member_whitelists":false,"witness_pay_per_block":500000,"worker_budget_per_day":"50000000000","max_predicate_
opcode":1,"fee_liquidation_threshold":10000000,"accounts_per_fee_scale":1000,"account_fee_scale_bitshifts":4,"max_authority_depth":2,"extensions":[]}}]],"extensions":[]},"requ
ired_active_approvals":["1.2.0"],"available_active_approvals":["1.2.102","1.2.103","1.2.104","1.2.105","1.2.106","1.2.107","1.2.108","1.2.109","1.2.110"],"required_owner_appro
vals":[],"available_owner_approvals":[],"available_key_approvals":[]}
10 assert_exception: Assert Exception
itr->get_balance() >= -delta: Insufficient Balance: committee-account's balance of 0 CORE is less than required 10 CORE
    {"a":"committee-account","b":"0 CORE","r":"10 CORE"}
    th_a  db_balance.cpp:67 adjust_balance

    {"account":"1.2.0","delta":{"amount":-1000000,"asset_id":"1.3.0"}}
    th_a  db_balance.cpp:73 adjust_balance

    {}
    th_a  evaluator.cpp:42 start_evaluate

    {}
    th_a  db_block.cpp:621 apply_operation

    {"proposal":{"id":"1.10.1","expiration_time":"2015-09-29T05:15:00","review_period_time":"2015-09-29T04:15:00","proposed_transaction":{"ref_block_num":0,"ref_block_prefix":
0,"expiration":"2015-09-29T05:15:00","operations":[[31,{"fee":{"amount":1000000,"asset_id":"1.3.0"},"new_parameters":{"current_fees":{"parameters":[[0,{"fee":2000000,"price_pe
r_kbyte":1000000}],[1,{"fee":500000}],[2,{"fee":0}],[3,{"fee":2000000}],[4,{}],[5,{"basic_fee":0,"premium_fee":200000000,"price_per_kbyte":100000}],[6,{"fee":2000000,"price_pe
r_kbyte":100000}],[7,{"fee":300000}],[8,{"membership_annual_fee":200000000,"membership_lifetime_fee":1000000000}],[9,{"fee":50000000}],[10,{"symbol3":"50000000000","symbol4":"
30000000000","long_symbol":500000000,"price_per_kbyte":10}],[11,{"fee":50000000,"price_per_kbyte":10}],[12,{"fee":50000000}],[13,{"fee":50000000}],[14,{"fee":2000000,"price_pe
r_kbyte":100000}],[15,{"fee":2000000}],[16,{"fee":100000}],[17,{"fee":10000000}],[18,{"fee":50000000}],[19,{"fee":100000}],[20,{"fee":500000000}],[21,{"fee":2000000}],[22,{"fe
e":2000000,"price_per_kbyte":10}],[23,{"fee":100000,"price_per_kbyte":10}],[24,{"fee":100000}],[25,{"fee":100000}],[26,{"fee":2000000}],[27,{"fee":0,"price_per_kbyte":10}],[28
,{"fee":500000000}],[29,{"fee":100000}],[30,{"fee":100000}],[31,{"fee":2000000}],[32,{"fee":500000000}],[33,{"fee":100000}],[34,{"fee":100000}],[35,{"fee":100000,"price_per_kb
yte":10}],[36,{"fee":2000000}],[37,{}],[38,{"fee":500000,"price_per_kbyte":10}],[39,{"fee":500000,"price_per_output":500000}]],"scale":5000},"block_interval":3,"maintenance_in
terval":5400,"maintenance_skip_slots":4,"committee_proposal_review_period":300,"maximum_transaction_size":65536,"maximum_block_size":10485760,"maximum_time_until_expiration":8
6400,"maximum_proposal_lifetime":2419200,"maximum_asset_whitelist_authorities":10,"maximum_asset_feed_publishers":10,"maximum_witness_count":1001,"maximum_committee_count":100
1,"maximum_authority_membership":10,"reserve_percent_of_fee":2000,"network_percent_of_fee":2000,"lifetime_referrer_percent_of_fee":3000,"cashback_vesting_period_seconds":77760
00,"cashback_vesting_threshold":10000000,"count_non_member_votes":true,"allow_non_member_whitelists":false,"witness_pay_per_block":500000,"worker_budget_per_day":"50000000000"
,"max_predicate_opcode":1,"fee_liquidation_threshold":10000000,"accounts_per_fee_scale":1000,"account_fee_scale_bitshifts":4,"max_authority_depth":2,"extensions":[]}}]],"exten
sions":[]},"required_active_approvals":["1.2.0"],"available_active_approvals":["1.2.102","1.2.103","1.2.104","1.2.105","1.2.106","1.2.107","1.2.108","1.2.109","1.2.110"],"requ
ired_owner_approvals":[],"available_owner_approvals":[],"available_key_approvals":[]}}
    th_a  db_block.cpp:269 push_proposal


That is more of a status message than an error message and is OK to ignore.