October 5 Test Network

[Thom]

Until now I didn't realize the object_database and genesis.json file location (if it is provided in the config.ini and not the --genesis-json cmd line arg) are relative to the folder from which the witness_node binary is executed.

I made the assumption that everything is relative to the --data-dir folder where the config.ini is found. Although I noticed the genesis.json and object_database were in a different location than the blockchain, config.ini etc, it never occured to me why, since I always launched from the same place each time.

I realize this is a very low priority request, but may I suggest the code be changed to "chroot" (so to speak) the location of all files to the value provided by the --data-dir (-d) command line argument. What would the reason be for not doing that, or not putting the object_database in the same place as the blockchain? Perhaps there is a good reason, but I can't think of any.

[Thom]

+5% Thanks Troglodactyl, greatly appreciated!

[Troglodactyl]

When you start the witness, what determines the chainID it runs on?

I don't see any specific cmd line options for a witness to act as a seed node, is that just a matter of not enabling block production and changing the --p2p-endpoint arg it will listen on?

When I do that it provides some (arbitrary, self-generated?) chainID, could each seed node theoretically represent it's own chain that other witness nodes could join to form their own P2P network?

I can easily see why the cli_wallet needs a chainID parameter, just not sure how it is determined for each witness.

Can anyone help clarify this?

Isn't the chain ID deterministic based on the genesis state?

EDIT:  Confirmed.

Code Select Expand

   /**
    * Get the chain_id corresponding to this genesis state.
    *
    * This is the SHA256 serialization of the genesis_state.
    */
   chain_id_type compute_chain_id() const;

Code Select Expand

chain_id_type genesis_state_type::compute_chain_id() const
{
   return fc::sha256::hash( *this );
}

The chain ID is just the hash of the genesis state.

[Spectral]

-s 188.165.233.53:1777

Thanks. That worked.

Is the other seed node down? Is there a list of seed nodes somewhere? How could i have known about this if clayop didn't post it here?

[Thom]

When you start the witness, what determines the chainID it runs on?

I don't see any specific cmd line options for a witness to act as a seed node, is that just a matter of not enabling block production and changing the --p2p-endpoint arg it will listen on?

When I do that it provides some (arbitrary, self-generated?) chainID, could each seed node theoretically represent it's own chain that other witness nodes could join to form their own P2P network?

I can easily see why the cli_wallet needs a chainID parameter, just not sure how it is determined for each witness.

Can anyone help clarify this?

[clayop]

Witness spectral crashed (version test6-41-g837e4f2). Log:
https://drive.google.com/file/d/0BzEEZBS7b6xPRWhHeU9pZDROd0E/view?usp=sharing


And I can't seem to reconnect to the network (version test6-42-g9c650bd). Log:
https://drive.google.com/file/d/0BzEEZBS7b6xPb2pQZXNzaFktU1E/view?usp=sharing

-s 188.165.233.53:1777

[Spectral]

Witness spectral crashed (version test6-41-g837e4f2). Log:
https://drive.google.com/file/d/0BzEEZBS7b6xPRWhHeU9pZDROd0E/view?usp=sharing


And I can't seem to reconnect to the network (version test6-42-g9c650bd). Log:
https://drive.google.com/file/d/0BzEEZBS7b6xPb2pQZXNzaFktU1E/view?usp=sharing

[wuyanren]

The witness that lost contact today is at the same time.I think it's not a question of the witness.

[spartako]

spartako_bot went in a minor fork and started to notify false alarm to witnesses (spartako witness producer was in the main fork and didn't missed any blocks), this is the error that I found:

Code Select Expand


2606729ms th_a       fork_database.cpp:57          push_block           ] Pushing block to fork database that failed to link: 0002315c15ad904d17192c8080121ba774c1a962, 143708
2606729ms th_a       fork_database.cpp:58          push_block           ] Head: 143799, 000231b77729fe84f473d51aabdbf31e530d1b82
2606730ms th_a       application.cpp:429           handle_block         ] Error when pushing block:
3080000 unlinkable_block_exception: unlinkable block
block does not link to known chain
    {}
    th_a  fork_database.cpp:78 _push_block

    {"new_block":{"previous":"0002315b615b561c0ea5eab6aa74aee790f5de0c","timestamp":"2015-10-11T00:28:30","witness":"1.6.11","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"1f6551667be1ad32fe674e9f4df852f0afe69450ccb396036026e97f468de4af7d1d0613c69ef2d9d35125e7a2bd7098dbc2fa1d44dff9aaa20a6a2cfa04fa5d2f","transactions":[]}}
    th_a  db_block.cpp:197 _push_block



Tomorrow I will change the code of spartako_bot adding a backup nodes list for prevent this type of problems.

[abit]

Every day so many witnesses are not online, what is the reason?

We've identified an issue where the assignment of new object ID's is inconsistent (we use a hashed index with an undefined iteration order).  Fortunately the testnet's main fork has them in increasing order (I suspect it has to do with the memory allocator tending to put things allocated later at higher addresses), in the next hardfork this condition will be enforced (by replacing most of our hashed indexes with ordered indexes).  We suspect some witnesses are getting their state wrong when they assign an ID differently from the main network, if someone publishes a transaction using that ID then the wrong state turns into a desync (they'll no longer sign or pay attention to blocks on the main fork because they think the main fork has an illegal transaction.)

If you get stuck and the first error message mentions a vesting balance object, this is almost certainly what's happening to you.

The next hardfork will fix this issue, and in addition, I'm working on a tool to help us find if there are any similar issues.

The transactions being executed by the community testers have been incredibly helpful to us.

@bytemaster @theoretical The 'vesting balance' issue happened again on my node. Is it normal? So, in order to prevent this from happening again, I have to manually replay before the hard fork? What block number is the hard fork?

Code Select Expand


2015-10-09T10:09:12 th_a:invoke handle_block         handle_block ] Got block: #101080 time: 2015-10-09T10:09:12 latency: 299 ms from: jtm1  irreversible: 101055 (-25)                 application.cpp:401
2015-10-09T10:09:12 th_a:invoke handle_block          _push_block ] Failed to push new block:
10 assert_exception: Assert Exception
vbo.is_withdraw_allowed( now, op.amount ):
    {"now":"2015-10-09T10:09:09","op":{"fee":{"amount":100000,"asset_id":"1.3.0"},"vesting_balance":"1.13.60","owner":"1.2.8517","amount":{"amount":800000000,"asset_id":"1.3.0"}},"vbo":{"id":"1.13.60","owner":"1.2.8517","balance":{"amount":1128200000,"asset_id":"1.3.0"},"policy":[1,{"vesting_seconds":86400,"start_claim":"1970-01-01T00:00:00","coin_seconds_earned":"68174256300000","coin_seconds_earned_last_update":"2015-10-09T10:08:03"}]}}
    th_a  vesting_balance_evaluator.cpp:103 do_evaluate

    {"op":{"fee":{"amount":100000,"asset_id":"1.3.0"},"vesting_balance":"1.13.60","owner":"1.2.8517","amount":{"amount":800000000,"asset_id":"1.3.0"}}}
    th_a  vesting_balance_evaluator.cpp:109 do_evaluate

    {}
    th_a  evaluator.cpp:42 start_evaluate

    {}
    th_a  db_block.cpp:628 apply_operation

    {"trx":{"ref_block_num":35543,"ref_block_prefix":1295075313,"expiration":"2015-10-09T10:09:39","operations":[[33,{"fee":{"amount":100000,"asset_id":"1.3.0"},"vesting_balance":"1.13.60","owner":"1.2.8517","amount":{"amount":800000000,"asset_id":"1.3.0"}}]],"extensions":[],"signatures":["2010951608daae48abcc6a9c9c6c43fe449803bf4af875602c02556915d9c680e32d808c391b128858f7e9f72b6d0dfd54036abe2852eabba55f3d479ebd0ee25f"]}}
    th_a  db_block.cpp:611 _apply_transaction

    {"next_block.block_num()":101080}
    th_a  db_block.cpp:514 _apply_block                 db_block.cpp:191


[abit]

Crashed again with error: Assertion `_closing_connections.find(peer_to_delete) == _closing_connections.end()' failed..
Commit e68e99ed3ae11ddac607e983e7549e8278fdecc4. Any changes after that?
I've encountered same error several times already. Due to my high latency?

Code Select Expand


2530767ms th_a       fork_database.cpp:57          push_block           ] Pushing block to fork database that failed to link: 00018ad9d0532f33aca1e0d35e97e2d8c7503630, 101081
2530767ms th_a       fork_database.cpp:58          push_block           ] Head: 102059, 00018eab29598c6da18be3a60c9ee321861f180c
2530862ms th_a       application.cpp:429           handle_block         ] Error when pushing block:
3080000 unlinkable_block_exception: unlinkable block
block does not link to known chain
    {}
    th_a  fork_database.cpp:78 _push_block

    {"new_block":{"previous":"00018ad8150b29e0a9278acb90fe7c592f4ec15c","timestamp":"2015-10-09T10:09:15","witness":"1.6.35","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"202d638e8938d8fde6e107105b4e95edc26068af86d68894199480ae6dbb7d16052e8ec457759e5b70f3866bfd6afcb1d8f2e5526bb17af292fbb56d1542058ce8","transactions":[]}}
    th_a  db_block.cpp:197 _push_block
witness_node: /app/bts/graphene-test6.2/libraries/net/node.cpp:1611: void graphene::net::detail::node_impl::schedule_peer_for_deletion(const peer_connection_ptr&): Assertion `_closing_connections.find(peer_to_delete) == _closing_connections.end()' failed.

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffff3532700 (LWP 14836)]
0x00007ffff6c01cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56      ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.



[Thom]

Did anything change again with the witness command line arguments,  like the location of the genesis json file?
The md5 of the json file matches my active witness so I know that's OK. Permissions are fine too.

Code Select Expand


/home/admin/BitShares2_bin/witness_node --data-dir /home/admin/BitShares2 --genesis-json /home/admin/oct5-genesis.json


That cmd line produces this error, looks to me like it can't read the genesis.json file:

admin@seed06:~$ /home/admin/BitShares2_bin/witness_node --data-dir /home/admin/BitShares2 --genesis-json /home/admin/oct5-genesis.json
3157079ms th_a       witness.cpp:83                plugin_initialize    ] witness plugin:  plugin_initialize() begin
3157093ms th_a       witness.cpp:112               plugin_initialize    ] 11 eof_exception: End Of File
stringstream
    {}
    th_a  sstream.cpp:109 peek

    {"str":""}
    th_a  json.cpp:478 from_string
3157093ms th_a       main.cpp:183                  main                 ] Exiting with error:
11 eof_exception: End Of File
stringstream
    {}
    th_a  sstream.cpp:109 peek

    {"str":""}
    th_a  json.cpp:478 from_string
rethrow
    {}
    th_a  witness.cpp:112 plugin_initialize

Edit: grabbed the wrong cmd history

[tonyk]

Can somebody running the real thing check the transaction history for core/USD. From the light wallet GUI it seems the transactions are matched but the 'sell Core' order remains on the order book.

Thanks

[puppies]

With -s you can specify another witness to connect to .. is has to be prepended with ws:// or wss:// (need to fix the docs) .. if no -s is present then default is localhost:8090

Gotcha, thanks.

[xeroc]

With -s you can specify another witness to connect to .. is has to be prepended with ws:// or wss:// (need to fix the docs) .. if no -s is present then default is localhost:8090