Author Topic: Test Net for Advanced Users  (Read 263659 times)

0 Members and 1 Guest are viewing this topic.

Offline bytemaster

For the latest updates checkout my blog: http://bytemaster.bitshares.org
Anything said on these forums does not constitute an intent to create a legal obligation or contract between myself and anyone else.   These are merely my opinions and I reserve the right to change them at any time.

Offline bytemaster


... it resulted in a hard fork several days ago because BitUSD technically had a black swan event back then.

and why have we not identified the event just after it occurred and only a couple of days later?

Because the market was thin


Sent from my iPhone using Tapatalk

Black Swan detection was only kicking in if there was a margin call, but margin calls were not triggering because there were no orders above the short-squeeze protection threshold.   
Black Swan should only happen when the MAX( Highest Bid, Settlement Price ) <  LeastCollateralizedPosition.   I moved the check to catch sooner in the evaluation which means that
the BlackSwan occurred as soon as the price feed was updated rather than delayed until the first forced settlement.
For the latest updates checkout my blog: http://bytemaster.bitshares.org
Anything said on these forums does not constitute an intent to create a legal obligation or contract between myself and anyone else.   These are merely my opinions and I reserve the right to change them at any time.

Offline sittingduck

  • Sr. Member
  • ****
  • Posts: 246
    • View Profile

... it resulted in a hard fork several days ago because BitUSD technically had a black swan event back then.

and why have we not identified the event just after it occurred and only a couple of days later?

Because the market was thin


Sent from my iPhone using Tapatalk

Offline CalabiYau

@abit: Congratulations ... it seems the both of us were the very reason for the first every blockchain based black swan event :-P

 :) :) that`s what legends are made off.....

Offline liondani

  • Hero Member
  • *****
  • Posts: 3737
  • Inch by inch, play by play
    • View Profile
    • My detailed info
  • BitShares: liondani
  • GitHub: liondani
... it resulted in a hard fork several days ago because BitUSD technically had a black swan event back then.

and why have we not identified the event just after it occurred and only a couple of days later?

Offline xeroc

  • Board Moderator
  • Hero Member
  • *****
  • Posts: 12922
  • ChainSquad GmbH
    • View Profile
    • ChainSquad GmbH
  • BitShares: xeroc
  • GitHub: xeroc
@abit: Congratulations ... it seems the both of us were the very reason for the first every blockchain based black swan event :-P

Offline abit

  • Committee member
  • Hero Member
  • *
  • Posts: 4664
    • View Profile
    • Abit's Hive Blog
  • BitShares: abit
  • GitHub: abitmore
I have just checked in a fix for this issue, but it resulted in a hard fork several days ago because BitUSD technically had a black swan event back then.

Theoretical is working on unit tests for the edge cases we have identified so that we can verify the fix works in all cases.

We will launch a new test network tomorrow based upon this fix and the party can continue.
I'm unable to access my server right now. Wish I could be back when the new test network is ready.
BitShares committee member: abit
BitShares witness: in.abit

Offline abit

  • Committee member
  • Hero Member
  • *
  • Posts: 4664
    • View Profile
    • Abit's Hive Blog
  • BitShares: abit
  • GitHub: abitmore
It took 75 seconds to replay the 280k-blocks chain.

Maybe on a SLOW vps, but on my machine it takes a mere 11 8.79 seconds.

In Debug build it took me 59 seconds to reindex.
In Release build it took me 8.79 seconds to reindex.

There were a total of 444758 operations in the blockchain which means we are averaging over 50K TPS.
The average maintenance calculation time is:  7 ms 
1.5 seconds were spent doing maintenance calculations (on the hour, instead of every day) leaving us at 62K TPS average
The vast majority of blocks have been empty which means our TPS average is being diluted by block processing overhead.

It looks like in Debug mode we process closer to 10K TPS.
Yes I'm running with debug build. I remember the log said replay took about 60 seconds, but CPU kept at 100% for 75 seconds. Maybe the 15 more seconds are before or after replay.
BitShares committee member: abit
BitShares witness: in.abit

Offline Thom

I have just checked in a fix for this issue, but it resulted in a hard fork several days ago because BitUSD technically had a black swan event back then.

Theoretical is working on unit tests for the edge cases we have identified so that we can verify the fix works in all cases.

We will launch a new test network tomorrow based upon this fix and the party can continue.

Awesome, I'll be ready to join back in with several VPS seed nodes to help.
Injustice anywhere is a threat to justice everywhere - MLK |  Verbaltech2 Witness Reports: https://bitsharestalk.org/index.php/topic,23902.0.html

Offline bytemaster

I have just checked in a fix for this issue, but it resulted in a hard fork several days ago because BitUSD technically had a black swan event back then.

Theoretical is working on unit tests for the edge cases we have identified so that we can verify the fix works in all cases.

We will launch a new test network tomorrow based upon this fix and the party can continue.
For the latest updates checkout my blog: http://bytemaster.bitshares.org
Anything said on these forums does not constitute an intent to create a legal obligation or contract between myself and anyone else.   These are merely my opinions and I reserve the right to change them at any time.

Offline bytemaster

4.  Attempt to force settle resulted in a condition that was previously believed to be impossible (a black swan)

 +5% Very good that this happened, so this case can be properly reviewed and re-tested.

 +5%

Yep I am glad it occurred on the test-network too !

We have identified the problem and have a solution in the works.    A big thank you to all of the testers who are pushing the test network through all kinds of edge cases.
For the latest updates checkout my blog: http://bytemaster.bitshares.org
Anything said on these forums does not constitute an intent to create a legal obligation or contract between myself and anyone else.   These are merely my opinions and I reserve the right to change them at any time.

Offline liondani

  • Hero Member
  • *****
  • Posts: 3737
  • Inch by inch, play by play
    • View Profile
    • My detailed info
  • BitShares: liondani
  • GitHub: liondani
4.  Attempt to force settle resulted in a condition that was previously believed to be impossible (a black swan)

 +5% Very good that this happened, so this case can be properly reviewed and re-tested.

 +5%

Yep I am glad it occurred on the test-network too !

Offline testz

4.  Attempt to force settle resulted in a condition that was previously believed to be impossible (a black swan)

 +5% Very good that this happened, so this case can be properly reviewed and re-tested.

Offline bytemaster

Quick Update:

We have identified the corner case that caused the problem and are preparing fixes for it now.   A quick summary:

1.  Early feed producers were off by a factor of 10
2.  Early margin testors created margin positions based upon the old feed prices
3.  Feed producers fixed their feed which left the early margin testors in black-swan territory
4.  Attempt to force settle resulted in a condition that was previously believed to be impossible (a black swan)

The fix we are adding will cause BitUSD to black swan on the test network. 

We are in the process of auditing the transaction history to answer a couple of questions:

1. Should the black swan have been triggered much earlier as a result of market orders being placed, canceled, etc?
2. If it should have, why wasn't it?

We are also evaluating if we should take any precautions to protect the network from future unexpected bugs.   Right now there exist
a slew of operations that are applied outside the context of a transaction.  If any of those operations fail then the network will be unable
to produce blocks.   In effect, these operations should always succeed if the transaction evaluation maintains the proper invariants.

Will be back with more updates when we have them.

For the latest updates checkout my blog: http://bytemaster.bitshares.org
Anything said on these forums does not constitute an intent to create a legal obligation or contract between myself and anyone else.   These are merely my opinions and I reserve the right to change them at any time.

Offline bytemaster

"head_block_num": 314814,
  "head_block_id": "0004cdbe86f24753a0133e23c6e26ebd3c05bf9e",
  "head_block_age": "6 hours old",
  "next_maintenance_time": "6 hours ago",
  "chain_id": "0f8b631d7a9dfebf16d6776fab96b629a14429762bf9c3eb95db1e4e4af637a4",


While replaying the blockchain
Code: [Select]

   90.212%   284000 of 314814                                                                                                                                       

1858849ms th_a       db_block.cpp:263              push_proposal        ] e
1858850ms th_a       db_update.cpp:140             clear_expired_propos ] Failed to apply proposed transaction on its expiration. Deleting it.
{"id":"1.10.1","expiration_time":"2015-09-29T05:15:00","review_period_time":"2015-09-29T04:15:00","proposed_transaction":{"ref_block_num":0,"ref_block_prefix":0,"expiration":"
2015-09-29T05:15:00","operations":[[31,{"fee":{"amount":1000000,"asset_id":"1.3.0"},"new_parameters":{"current_fees":{"parameters":[[0,{"fee":2000000,"price_per_kbyte":1000000
}],[1,{"fee":500000}],[2,{"fee":0}],[3,{"fee":2000000}],[4,{}],[5,{"basic_fee":0,"premium_fee":200000000,"price_per_kbyte":100000}],[6,{"fee":2000000,"price_per_kbyte":100000}
],[7,{"fee":300000}],[8,{"membership_annual_fee":200000000,"membership_lifetime_fee":1000000000}],[9,{"fee":50000000}],[10,{"symbol3":"50000000000","symbol4":"30000000000","lo
ng_symbol":500000000,"price_per_kbyte":10}],[11,{"fee":50000000,"price_per_kbyte":10}],[12,{"fee":50000000}],[13,{"fee":50000000}],[14,{"fee":2000000,"price_per_kbyte":100000}
],[15,{"fee":2000000}],[16,{"fee":100000}],[17,{"fee":10000000}],[18,{"fee":50000000}],[19,{"fee":100000}],[20,{"fee":500000000}],[21,{"fee":2000000}],[22,{"fee":2000000,"pric
e_per_kbyte":10}],[23,{"fee":100000,"price_per_kbyte":10}],[24,{"fee":100000}],[25,{"fee":100000}],[26,{"fee":2000000}],[27,{"fee":0,"price_per_kbyte":10}],[28,{"fee":50000000
0}],[29,{"fee":100000}],[30,{"fee":100000}],[31,{"fee":2000000}],[32,{"fee":500000000}],[33,{"fee":100000}],[34,{"fee":100000}],[35,{"fee":100000,"price_per_kbyte":10}],[36,{"
fee":2000000}],[37,{}],[38,{"fee":500000,"price_per_kbyte":10}],[39,{"fee":500000,"price_per_output":500000}]],"scale":5000},"block_interval":3,"maintenance_interval":5400,"ma
intenance_skip_slots":4,"committee_proposal_review_period":300,"maximum_transaction_size":65536,"maximum_block_size":10485760,"maximum_time_until_expiration":86400,"maximum_pr
oposal_lifetime":2419200,"maximum_asset_whitelist_authorities":10,"maximum_asset_feed_publishers":10,"maximum_witness_count":1001,"maximum_committee_count":1001,"maximum_autho
rity_membership":10,"reserve_percent_of_fee":2000,"network_percent_of_fee":2000,"lifetime_referrer_percent_of_fee":3000,"cashback_vesting_period_seconds":7776000,"cashback_ves
ting_threshold":10000000,"count_non_member_votes":true,"allow_non_member_whitelists":false,"witness_pay_per_block":500000,"worker_budget_per_day":"50000000000","max_predicate_
opcode":1,"fee_liquidation_threshold":10000000,"accounts_per_fee_scale":1000,"account_fee_scale_bitshifts":4,"max_authority_depth":2,"extensions":[]}}]],"extensions":[]},"requ
ired_active_approvals":["1.2.0"],"available_active_approvals":["1.2.102","1.2.103","1.2.104","1.2.105","1.2.106","1.2.107","1.2.108","1.2.109","1.2.110"],"required_owner_appro
vals":[],"available_owner_approvals":[],"available_key_approvals":[]}
10 assert_exception: Assert Exception
itr->get_balance() >= -delta: Insufficient Balance: committee-account's balance of 0 CORE is less than required 10 CORE
    {"a":"committee-account","b":"0 CORE","r":"10 CORE"}
    th_a  db_balance.cpp:67 adjust_balance

    {"account":"1.2.0","delta":{"amount":-1000000,"asset_id":"1.3.0"}}
    th_a  db_balance.cpp:73 adjust_balance

    {}
    th_a  evaluator.cpp:42 start_evaluate

intenance_skip_slots":4,"committee_proposal_review_period":300,"maximum_transaction_size":65536,"maximum_block_size":10485760,"maximum_time_until_expiration":86400,"[540/1909]
oposal_lifetime":2419200,"maximum_asset_whitelist_authorities":10,"maximum_asset_feed_publishers":10,"maximum_witness_count":1001,"maximum_committee_count":1001,"maximum_autho
rity_membership":10,"reserve_percent_of_fee":2000,"network_percent_of_fee":2000,"lifetime_referrer_percent_of_fee":3000,"cashback_vesting_period_seconds":7776000,"cashback_ves
ting_threshold":10000000,"count_non_member_votes":true,"allow_non_member_whitelists":false,"witness_pay_per_block":500000,"worker_budget_per_day":"50000000000","max_predicate_
opcode":1,"fee_liquidation_threshold":10000000,"accounts_per_fee_scale":1000,"account_fee_scale_bitshifts":4,"max_authority_depth":2,"extensions":[]}}]],"extensions":[]},"requ
ired_active_approvals":["1.2.0"],"available_active_approvals":["1.2.102","1.2.103","1.2.104","1.2.105","1.2.106","1.2.107","1.2.108","1.2.109","1.2.110"],"required_owner_appro
vals":[],"available_owner_approvals":[],"available_key_approvals":[]}
10 assert_exception: Assert Exception
itr->get_balance() >= -delta: Insufficient Balance: committee-account's balance of 0 CORE is less than required 10 CORE
    {"a":"committee-account","b":"0 CORE","r":"10 CORE"}
    th_a  db_balance.cpp:67 adjust_balance

    {"account":"1.2.0","delta":{"amount":-1000000,"asset_id":"1.3.0"}}
    th_a  db_balance.cpp:73 adjust_balance

    {}
    th_a  evaluator.cpp:42 start_evaluate

    {}
    th_a  db_block.cpp:621 apply_operation

    {"proposal":{"id":"1.10.1","expiration_time":"2015-09-29T05:15:00","review_period_time":"2015-09-29T04:15:00","proposed_transaction":{"ref_block_num":0,"ref_block_prefix":
0,"expiration":"2015-09-29T05:15:00","operations":[[31,{"fee":{"amount":1000000,"asset_id":"1.3.0"},"new_parameters":{"current_fees":{"parameters":[[0,{"fee":2000000,"price_pe
r_kbyte":1000000}],[1,{"fee":500000}],[2,{"fee":0}],[3,{"fee":2000000}],[4,{}],[5,{"basic_fee":0,"premium_fee":200000000,"price_per_kbyte":100000}],[6,{"fee":2000000,"price_pe
r_kbyte":100000}],[7,{"fee":300000}],[8,{"membership_annual_fee":200000000,"membership_lifetime_fee":1000000000}],[9,{"fee":50000000}],[10,{"symbol3":"50000000000","symbol4":"
30000000000","long_symbol":500000000,"price_per_kbyte":10}],[11,{"fee":50000000,"price_per_kbyte":10}],[12,{"fee":50000000}],[13,{"fee":50000000}],[14,{"fee":2000000,"price_pe
r_kbyte":100000}],[15,{"fee":2000000}],[16,{"fee":100000}],[17,{"fee":10000000}],[18,{"fee":50000000}],[19,{"fee":100000}],[20,{"fee":500000000}],[21,{"fee":2000000}],[22,{"fe
e":2000000,"price_per_kbyte":10}],[23,{"fee":100000,"price_per_kbyte":10}],[24,{"fee":100000}],[25,{"fee":100000}],[26,{"fee":2000000}],[27,{"fee":0,"price_per_kbyte":10}],[28
,{"fee":500000000}],[29,{"fee":100000}],[30,{"fee":100000}],[31,{"fee":2000000}],[32,{"fee":500000000}],[33,{"fee":100000}],[34,{"fee":100000}],[35,{"fee":100000,"price_per_kb
yte":10}],[36,{"fee":2000000}],[37,{}],[38,{"fee":500000,"price_per_kbyte":10}],[39,{"fee":500000,"price_per_output":500000}]],"scale":5000},"block_interval":3,"maintenance_in
terval":5400,"maintenance_skip_slots":4,"committee_proposal_review_period":300,"maximum_transaction_size":65536,"maximum_block_size":10485760,"maximum_time_until_expiration":8
6400,"maximum_proposal_lifetime":2419200,"maximum_asset_whitelist_authorities":10,"maximum_asset_feed_publishers":10,"maximum_witness_count":1001,"maximum_committee_count":100
1,"maximum_authority_membership":10,"reserve_percent_of_fee":2000,"network_percent_of_fee":2000,"lifetime_referrer_percent_of_fee":3000,"cashback_vesting_period_seconds":77760
00,"cashback_vesting_threshold":10000000,"count_non_member_votes":true,"allow_non_member_whitelists":false,"witness_pay_per_block":500000,"worker_budget_per_day":"50000000000"
,"max_predicate_opcode":1,"fee_liquidation_threshold":10000000,"accounts_per_fee_scale":1000,"account_fee_scale_bitshifts":4,"max_authority_depth":2,"extensions":[]}}]],"exten
sions":[]},"required_active_approvals":["1.2.0"],"available_active_approvals":["1.2.102","1.2.103","1.2.104","1.2.105","1.2.106","1.2.107","1.2.108","1.2.109","1.2.110"],"requ
ired_owner_approvals":[],"available_owner_approvals":[],"available_key_approvals":[]}}
    th_a  db_block.cpp:269 push_proposal

That is more of a status message than an error message and is OK to ignore.
For the latest updates checkout my blog: http://bytemaster.bitshares.org
Anything said on these forums does not constitute an intent to create a legal obligation or contract between myself and anyone else.   These are merely my opinions and I reserve the right to change them at any time.