October 5 Test Network

[bytemaster]

Participation has dropped to 50% and all init witnesses stopped producing blocks.

The init witnesses along with btsnow and a few others went off on a minority fork until I restarted the init witnesses and it rejoined the proper fork.

Upon looking into the issue with Eric  (btsnow)  we discovered his chain was in an inconsistent state where get_block failed to find random blocks between the last_irreversible_block and the head_block.

The fact that his blockchain was missing blocks in its history meant it was unable to switch back to the proper fork. 

It would have been far less dramatic if the init witnesses were not on a single node.

[clayop]

delegate.verbaltech
jtm1
delegate.baozi
roadscape

They showed low performance in generating blocks. (under 20 txs per block).

[triox]

2015-10-06T20:51:09 Transfer 20000 CORE from spectral to triox-delegate -- could not decrypt memo   (Fee: 20.89843 CORE)

It would be interesting to know if the transfer went through, and the account history from triox-delegate. Maybe I've messed up something...

Code: [Select]

2015-10-06T20:51:09 Transfer 20000 CORE from spectral to triox-delegate -- Memo: nice specs   (Fee: 20.89843 CORE)

Thanks!
and ready (I believe) to be voted in
"id": "1.6.41",
 "witness_account": "1.2.79789",

[Spectral]

Is your wallet unlocked when you make those calls?

Good question. I think it must have been unlocked, because I made the call right after the transfer, otherwise I would get an error message on the transfer call?

@bytemaster. Another piece of info: I just remembered that recently, before this whole mess, I made a transfer call where I reversed the parameters, so that I did

Code: [Select]

transfer  20000  CORE  <from>  <to>  "memo"
and it returned an ugly error message that I didn't log. I don't know if this might have influenced the wallet state. Is there a way to check if the wallet or keys have become corrupted somehow?

[ElMato]

I have been borrowing assets starting with 1000 CORE as collateral but then lowering till 0 and finally a blackswan event triggered.

borrow_asset delmoro 1 CNY 0.0000 true
(This actually works)

787848ms th_a       db_update.cpp:241             check_for_blackswan  ] Black Swan detected:
   Least collateralized call: 0.10109393536387817  9.89179020878545856
   Settle Price:              0.00316518505364101  315.93729372937292510
   Max:                       0.00316518505364101   315.93729372937292510

@bytemaster

[clayop]

Next small test in 15 min (22:25 UTC). 30 TPS for 1 hour.

[Bhuz]

Code: [Select]

3456827ms th_a       application.cpp:419           handle_block         ] Error when pushing block:
10 assert_exception: Assert Exception
item->num > std::max<int64_t>( 0, int64_t(_head->num) - (_max_size) ): attempting to push a block that is too old
    {"item->num":36717,"head":36876,"max_size":33}
    th_a  fork_database.cpp:71 _push_block

    {"new_block":{"previous":"00008f6c7b1e63e5645dc3ea16cdc146c87a53ae","timestamp":"2015-10-06T21:57:36","witness":"1.6.12","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"204d32a97bff59567fcc5b585ae7ef777115f245fbb6fa9e28e8a475c705cf0e8b17d942f9065ff9d262afead638e8bc77f7bc732f0e3fc1f618193ce16928ea66","transactions":[]}}
    th_a  db_block.cpp:195 _push_block



Got that error too

Getting also some Switching to fork like
546364ms th_a       db_block.cpp:134              _push_block          ] Switching to fork: 000090d3acad2d699b38e6f161be0ad42c3c4a99

[spartako]

I'm in sync now even if I see this error:

Code: [Select]

3456827ms th_a       application.cpp:419           handle_block         ] Error when pushing block:
10 assert_exception: Assert Exception
item->num > std::max<int64_t>( 0, int64_t(_head->num) - (_max_size) ): attempting to push a block that is too old
    {"item->num":36717,"head":36876,"max_size":33}
    th_a  fork_database.cpp:71 _push_block

    {"new_block":{"previous":"00008f6c7b1e63e5645dc3ea16cdc146c87a53ae","timestamp":"2015-10-06T21:57:36","witness":"1.6.12","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"204d32a97bff59567fcc5b585ae7ef777115f245fbb6fa9e28e8a475c705cf0e8b17d942f9065ff9d262afead638e8bc77f7bc732f0e3fc1f618193ce16928ea66","transactions":[]}}
    th_a  db_block.cpp:195 _push_block



[spartako]

My witness started to missing blocks and give this errors....resyncing

Code: [Select]

3291849ms th_a       fork_database.cpp:57          push_block           ] Pushing block to fork database that failed to link: 00008eb6eadbeb4a8ddf3f411786e9e7750c9d0b,
36534
3291850ms th_a       fork_database.cpp:58          push_block           ] Head: 36713, 00008f69e3262839ea3e10cdda6e40e0f2286dcc
3291850ms th_a       application.cpp:416           handle_block         ] Error when pushing block:
3080000 unlinkable_block_exception: unlinkable block
block does not link to known chain
    {}
    th_a  fork_database.cpp:79 _push_block

    {"new_block":{"previous":"00008eb5f524cc47c58abeb93f70252edf73e288","timestamp":"2015-10-06T21:28:09","witness":"1.6.33","transaction_merkle_root":"0f95eeb9a1b32e56
1ca3b3212f438f0e9b0377cc","extensions":[],"witness_signature":"1f27a9b5176289037cd61d42f071b7e35a0a89abf8e5e524c876dd7e22e4486be73ec05247429c38578b4907f8fd4d5695d1590e2
1fb9aaeea0470b6a1cc9ba841","transactions":[{"ref_block_num":36528,"ref_block_prefix":3755663456,"expiration":"2015-10-06T21:28:12","operations":[[33,{"fee":{"amount":10
0000,"asset_id":"1.3.0"},"vesting_balance":"1.13.13","owner":"1.2.17269","amount":{"amount":"40000000000","asset_id":"1.3.0"}}]],"extensions":[],"signatures":["1f2c7956
9ddd9f2c819fef84d04722eb59e958ad5e3f7d91fdcdbe8ec2329a0eb52896e1817deb0b414e79bd83f33a17d7afdc9e45da89d5f04a1f365cfeb33517"],"operation_results":[[0,{}]]}]}}
    th_a  db_block.cpp:195 _push_block



[Spectral]

Is your wallet unlocked when you make those calls?

Good question. I think it must have been unlocked, because I made the call right after the transfer, otherwise I would get an error message on the transfer call?

It seems that the issue is repeatable, I am making the call right now and getting the same output:

Code: [Select]

unlocked >>> get_account_history spectral 100
get_account_history spectral 100
2174484ms th_a       wallet.cpp:2321               operator()           ] Error when decrypting memo: 10 assert_exception: Assert Exception
wallet._keys.count(op.memo->to): Memo is encrypted to a key GPH8jnprUyAGGJ4M2HGD1kKNN5NgE85anQEdsvrFpLX4nvQ4CxA2k not in this wallet.
    {"k":"GPH8jnprUyAGGJ4M2HGD1kKNN5NgE85anQEdsvrFpLX4nvQ4CxA2k"}
    th_a  wallet.cpp:2314 operator()
2174502ms th_a       wallet.cpp:2321               operator()           ] Error when decrypting memo: 10 assert_exception: Assert Exception
wallet._keys.count(op.memo->to): Memo is encrypted to a key GPH7B4bszRYW5SKGFUKuM6ta95MUR81ZsjUTyxLAn4Pezu5Ck9xsw not in this wallet.
    {"k":"GPH7B4bszRYW5SKGFUKuM6ta95MUR81ZsjUTyxLAn4Pezu5Ck9xsw"}
    th_a  wallet.cpp:2314 operator()
2174515ms th_a       wallet.cpp:2321               operator()           ] Error when decrypting memo: 10 assert_exception: Assert Exception
wallet._keys.count(op.memo->to): Memo is encrypted to a key GPH8jnprUyAGGJ4M2HGD1kKNN5NgE85anQEdsvrFpLX4nvQ4CxA2k not in this wallet.
    {"k":"GPH8jnprUyAGGJ4M2HGD1kKNN5NgE85anQEdsvrFpLX4nvQ4CxA2k"}
    th_a  wallet.cpp:2314 operator()
2174545ms th_a       wallet.cpp:2321               operator()           ] Error when decrypting memo: 10 assert_exception: Assert Exception
wallet._keys.count(op.memo->to): Memo is encrypted to a key GPH7B4bszRYW5SKGFUKuM6ta95MUR81ZsjUTyxLAn4Pezu5Ck9xsw not in this wallet.
    {"k":"GPH7B4bszRYW5SKGFUKuM6ta95MUR81ZsjUTyxLAn4Pezu5Ck9xsw"}
    th_a  wallet.cpp:2314 operator()
2015-10-06T21:24:00 asset_publish_feed_operation spectral fee: 1 CORE
2015-10-06T21:24:00 asset_publish_feed_operation spectral fee: 1 CORE
2015-10-06T21:24:00 asset_publish_feed_operation spectral fee: 1 CORE
2015-10-06T21:24:00 asset_publish_feed_operation spectral fee: 1 CORE
2015-10-06T20:51:09 Transfer 20000 CORE from spectral to triox-delegate -- could not decrypt memo   (Fee: 20.89843 CORE)

It would be interesting to know if the transfer went through, and the account history from triox-delegate. Maybe I've messed up something...

I made a test transfer to you Dan, with the following output:

Code: [Select]

transfer spectral dan 10 CORE "testmessage" true
{
  "ref_block_num": 36654,
  "ref_block_prefix": 724632219,

and the same problem in the account history, I can't decrypt the memo.

Code: [Select]

get_account_history spectral 100
2535952ms th_a       wallet.cpp:2321               operator()           ] Error when decrypting memo: 10 assert_exception: Assert Exception
wallet._keys.count(op.memo->to): Memo is encrypted to a key GPH8JvEXv853AzosCgJEpHLDreNG3pDwr9uxYeA86cL587ooKLmJG not in this wallet.
    {"k":"GPH8JvEXv853AzosCgJEpHLDreNG3pDwr9uxYeA86cL587ooKLmJG"}
    th_a  wallet.cpp:2314 operator()
2535956ms th_a       wallet.cpp:2321               operator()           ] Error when decrypting memo: 10 assert_exception: Assert Exception
wallet._keys.count(op.memo->to): Memo is encrypted to a key GPH8jnprUyAGGJ4M2HGD1kKNN5NgE85anQEdsvrFpLX4nvQ4CxA2k not in this wallet.
    {"k":"GPH8jnprUyAGGJ4M2HGD1kKNN5NgE85anQEdsvrFpLX4nvQ4CxA2k"}
    th_a  wallet.cpp:2314 operator()
2535972ms th_a       wallet.cpp:2321               operator()           ] Error when decrypting memo: 10 assert_exception: Assert Exception
wallet._keys.count(op.memo->to): Memo is encrypted to a key GPH7B4bszRYW5SKGFUKuM6ta95MUR81ZsjUTyxLAn4Pezu5Ck9xsw not in this wallet.
    {"k":"GPH7B4bszRYW5SKGFUKuM6ta95MUR81ZsjUTyxLAn4Pezu5Ck9xsw"}
    th_a  wallet.cpp:2314 operator()
2535981ms th_a       wallet.cpp:2321               operator()           ] Error when decrypting memo: 10 assert_exception: Assert Exception
wallet._keys.count(op.memo->to): Memo is encrypted to a key GPH8JvEXv853AzosCgJEpHLDreNG3pDwr9uxYeA86cL587ooKLmJG not in this wallet.
    {"k":"GPH8JvEXv853AzosCgJEpHLDreNG3pDwr9uxYeA86cL587ooKLmJG"}
    th_a  wallet.cpp:2314 operator()
2535987ms th_a       wallet.cpp:2321               operator()           ] Error when decrypting memo: 10 assert_exception: Assert Exception
wallet._keys.count(op.memo->to): Memo is encrypted to a key GPH8jnprUyAGGJ4M2HGD1kKNN5NgE85anQEdsvrFpLX4nvQ4CxA2k not in this wallet.
    {"k":"GPH8jnprUyAGGJ4M2HGD1kKNN5NgE85anQEdsvrFpLX4nvQ4CxA2k"}
    th_a  wallet.cpp:2314 operator()
2536015ms th_a       wallet.cpp:2321               operator()           ] Error when decrypting memo: 10 assert_exception: Assert Exception
wallet._keys.count(op.memo->to): Memo is encrypted to a key GPH7B4bszRYW5SKGFUKuM6ta95MUR81ZsjUTyxLAn4Pezu5Ck9xsw not in this wallet.
    {"k":"GPH7B4bszRYW5SKGFUKuM6ta95MUR81ZsjUTyxLAn4Pezu5Ck9xsw"}
    th_a  wallet.cpp:2314 operator()
2015-10-06T21:41:36 Transfer 10 CORE from spectral to dan -- could not decrypt memo   (Fee: 20.89843 CORE)
2015-10-06T21:24:00 asset_publish_feed_operation spectral fee: 1 CORE
2015-10-06T21:24:00 asset_publish_feed_operation spectral fee: 1 CORE
2015-10-06T21:24:00 asset_publish_feed_operation spectral fee: 1 CORE
2015-10-06T21:24:00 asset_publish_feed_operation spectral fee: 1 CORE
2015-10-06T20:51:09 Transfer 20000 CORE from spectral to triox-delegate -- could not decrypt memo   (Fee: 20.89843 CORE)

Additional information: I also tried publishing a price feed through xeroc's script and that returned no error message.

And witness shows no missed blocks.

[Bhuz]

Is normal that publishing feeds consist in 4 asset_publish_feed_operation ?
Why 4?

[jakub]

Participation has dropped to 50% and all init witnesses stopped producing blocks.

[bytemaster]

Is your wallet unlocked when you make those calls?

[Spectral]

triox-delegate requesting core asset for registration.

4core/16GB/120GB SSD physical rig.

Code: [Select]

transfer spectral triox-delegate 20000 CORE "nice specs" true
{
  "ref_block_num": 35854,


For some reason a subsequent call to the 'get_account_history' function returns some errors concerning the memo... is this normal?

Code: [Select]

get_account_history spectral 100
3353392ms th_a       wallet.cpp:2321               operator()           ] Error when decrypting memo: 10 assert_exception: Assert Exception
wallet._keys.count(op.memo->to): Memo is encrypted to a key GPH8jnprUyAGGJ4M2HGD1kKNN5NgE85anQEdsvrFpLX4nvQ4CxA2k not in this wallet.
    {"k":"GPH8jnprUyAGGJ4M2HGD1kKNN5NgE85anQEdsvrFpLX4nvQ4CxA2k"}
    th_a  wallet.cpp:2314 operator()
3353410ms th_a       wallet.cpp:2321               operator()           ] Error when decrypting memo: 10 assert_exception: Assert Exception
wallet._keys.count(op.memo->to): Memo is encrypted to a key GPH7B4bszRYW5SKGFUKuM6ta95MUR81ZsjUTyxLAn4Pezu5Ck9xsw not in this wallet.
    {"k":"GPH7B4bszRYW5SKGFUKuM6ta95MUR81ZsjUTyxLAn4Pezu5Ck9xsw"}
    th_a  wallet.cpp:2314 operator()
3353416ms th_a       wallet.cpp:2321               operator()           ] Error when decrypting memo: 10 assert_exception: Assert Exception
wallet._keys.count(op.memo->to): Memo is encrypted to a key GPH8jnprUyAGGJ4M2HGD1kKNN5NgE85anQEdsvrFpLX4nvQ4CxA2k not in this wallet.
    {"k":"GPH8jnprUyAGGJ4M2HGD1kKNN5NgE85anQEdsvrFpLX4nvQ4CxA2k"}
    th_a  wallet.cpp:2314 operator()
3353443ms th_a       wallet.cpp:2321               operator()           ] Error when decrypting memo: 10 assert_exception: Assert Exception
wallet._keys.count(op.memo->to): Memo is encrypted to a key GPH7B4bszRYW5SKGFUKuM6ta95MUR81ZsjUTyxLAn4Pezu5Ck9xsw not in this wallet.
    {"k":"GPH7B4bszRYW5SKGFUKuM6ta95MUR81ZsjUTyxLAn4Pezu5Ck9xsw"}
    th_a  wallet.cpp:2314 operator()


[Bhuz]

  The witness_node apparently doesn't need a chainID, only the cli_wallet?

witness_node needs --jenesis-json
cli_wallet needs --chain-id