Author Topic: Database not reindexed/replayed at witness node startup  (Read 228 times)

0 Members and 1 Guest are viewing this topic.

Offline FrankBlabu

Database not reindexed/replayed at witness node startup
« on: March 16, 2019, 07:18:07 pm »
I'm trying to develop a plugin extracting some statistical information from the blockchain. I kept the code similar to the market history plugin and it works well - except that the in-memory database is not rebuild when the witness node has been terminated and then started again - neither for the market history nor for my own plugin. Instead, the blockchain state is rewound for some blocks and blocks will start coming in from that point of time on. All in-memory information from the blocks before is lost.

In detail, the 'applied_block' hook is used to connect to the database during plugin initialization:

Code: [Select]
database ().applied_block.connect ([this] (const signed_block& b) { my->on_block_applied (b); });

When a block is applied, I will consider both applied operations and the current block transactions:

Code: [Select]
void extractor_plugin_impl::on_block_applied( const signed_block& b )
{
        graphene::chain::database& db = database ();

        // Applied operations
        for( const optional< operation_history_object >& o_op : db.get_applied_operations () )
          ...

        // Transactions of incoming block
        operation_handler handler (_plugin, b.block_num (), b.timestamp);

        for (const auto& t : b.transactions)
          for (const auto& op: t.operations)
            ...

This is the output when the witness node is started:

Code: [Select]
...
927579ms th_a       object_database.cpp:115       open                 ] Opening object database from /data/blockchain/bitshares/blockchain ...
928582ms th_a       object_database.cpp:124       open                 ] Done opening object database.
928599ms th_a       db_management.cpp:64          reindex              ] reindexing blockchain
928599ms th_a       db_management.cpp:70          reindex              ] Replaying blocks, starting at 350003...
...

* Is this the right way to ensure that, at witness node restart, the already processed blocks are replayed ? Or is some hook missing ?
* Or do I need to care for storing the already gathered information in a local database myself because there is no replay at all ?
* Why does the official market history plugin fail in this case, too ?

Thank you in advance !

Offline pc

  • Hero Member
  • *****
  • Posts: 1463
    • View Profile
    • Bitcoin - Perspektive oder Risiko?
  • BitShares: cyrano
Re: Database not reindexed/replayed at witness node startup
« Reply #1 on: March 17, 2019, 08:46:45 am »
witness_node does not replay automatically on every restart because replay takes a long time.
Instead, on shutdown it rewinds its internal database for a couple of blocks and then dumps it to disk. On restart it checks if such a dump is present, and if so it loads the dump and resumes from that point.

If your plugin also uses the internal database, like the market plugin, then your plugin state will also be dumped and reloaded automatically.

Note that you *should* use the internal database for storing your plugin state, because otherwise it will be difficult to handle chain forking correctly.
Bitcoin - Perspektive oder Risiko? ISBN 978-3-8442-6568-2 http://bitcoin.quisquis.de

Offline FrankBlabu

Re: Database not reindexed/replayed at witness node startup
« Reply #2 on: March 17, 2019, 02:04:51 pm »
If your plugin also uses the internal database, like the market plugin, then your plugin state will also be dumped and reloaded automatically.

Then I must have missed something there and will start analyzing the dump mechanism. Thank you very much for the quick and detailed answer !

UPDATE: Got it ! I simply did not list the database object fields to be stored permanently in the FC_REFLECT_DERIVED clause of the database object. So they never got saved permanently. Case closed, thanks again !

BTW: The bitshares core repo is a really nice piece of code, I already learned a lot !


« Last Edit: March 17, 2019, 03:46:56 pm by FrankBlabu »

Offline benniehag

  • Newbie
  • *
  • Posts: 10
    • View Profile
Database not reindexed/replayed at witness node startup
« Reply #3 on: April 24, 2019, 10:27:51 am »
Thank you for the information. I havent been able to replicate the problem, so Im not sure what is causing it. You might be able to fix it by deleting the series and re-creating it.