Author Topic: New API for witness_node  (Read 1962 times)

0 Members and 1 Guest are viewing this topic.

Offline ElMato

  • Sr. Member
  • ****
  • Posts: 288
    • View Profile
New API for witness_node
« on: December 26, 2016, 07:50:44 pm »
Continuing  the discussion on changing / updating the API of witness_node.

The BitShares blockchain has a more diverse STATE than the Bitcoin blockchain (UTXO/Address Balance).
We have a lot of entities: accounts, balances, assets, orders, feeds, etc.

We need maximum flexibility when querying the state.

The state doesn't have all the appropriated indexes or doesn't have the specific function to query that specific part of the state.

Eg:
We needed to list all the holders of a specific asset.

The index was already there.

... ordered_unique< tag<by_asset_balance>,
            composite_key<
               account_balance_object,
               member<account_balance_object, asset_id_type, &account_balance_object::asset_type>,
               member<account_balance_object, share_type, &account_balance_object::balance>,
               member<account_balance_object, account_id_type, &account_balance_object::owner>
            >,
          ...


So, we just write the function using that index.

Code: [Select]
vector<account_asset_balance> database_api_impl::get_asset_holders( asset_id_type asset_id )const
{
   const auto& bal_idx = _db.get_index_type< account_balance_index >().indices().get< by_asset_balance >();
   const auto range = bal_idx.equal_range( boost::make_tuple( asset_id ) );
   vector<account_asset_balance> result;
   for( const account_balance_object& bal : boost::make_iterator_range( range.first, range.second ) )
   {
     assert( bal.asset_type == asset_id );
     auto account = _db.find(bal.owner);

     account_asset_balance aab;
     aab.name       = account->name;
     aab.account_id = account->id;
     aab.amount     = bal.balance.value;

     result.push_back(aab);
   }

   return result;
}

So if we want to query the state in any imaginable way we have many options each one of them with some trade-offs.

1) Add more boost_multi_index_container for the state with the corresponding functions when needed.
     This will increase memory usage in all nodes since indexes occupy memory space.
     But can be set at compile time or with a custom plugin.

2) Replicate the state in a MySQL/neo4J database.
    Add a function to dump the FULL STATE of the object db.
    And then apply state changes (add/edit/delete objects).
 
    This process need to lock the block processing until we apply the state change in our underlying database.
    Otherwise if we miss a delta-diff our outside-state will be different from the real state.

    Eg: if we just connect to the WS interface, and wait for "object_changed" callback and affect the outside-state accordingly, then our process go down, the blockchain keeps updating objects, we reconnect later and .... we are in problems.

    We have been experimenting with this approach developing a plugin that will call a python function with the changed_objects content.

Code: [Select]
void db_update_plugin::plugin_initialize(const boost::program_options::variables_map& options)
{
  auto script_str = options.at("python-script").as<std::string>();
  if ( !script_str.size() )
    return;

  Py_Initialize();

  my->_main_module = bp::import("__main__");
  my->_global      = my->_main_module.attr("__dict__");

  auto script_path =  bp::str(script_str);

  bp::exec_file(script_path, my->_global, my->_global);

  my->_handler = my->_global["Handler"]();

  database().changed_objects.connect([&]( const std::vector<graphene::db::object_id_type>& ids) {

    bp::list list_ids;
    bp::list list_values;

    for( auto id : ids ) {

      fc::variant vo;
      to_variant(id, vo);
      list_ids.append( vo.get_string() );

      const graphene::db::object *obj = database().find_object(id);
      if ( obj != nullptr ) {
        list_values.append( fc::json::to_string(obj->to_variant()) );
      } else {
        list_values.append( "" );
      }
    }

    bp::call_method<void>(my->_handler.ptr(), "changed_objects", list_ids, list_values);
  });
}

 

    (We have to take care of possible forks in the chain? Or the changed_objects callback also tell us [in case of undo] which objects need to be deleted?) 

3) Create a pluggable db backend.
     Take graphene::db::object_database as the base, create an interface and implement object_database_mysql, object_database_neo4j ... etc.


Offline ElMato

  • Sr. Member
  • ****
  • Posts: 288
    • View Profile
Re: New API for witness_node
« Reply #1 on: December 26, 2016, 07:51:53 pm »
There is a lot more to be discussed but i want to get some feedback from the community.
Also, I would like to join efforts in this because all of us (projects on top of BitShares) need this kind of tools.

Offline xeroc

  • Board Moderator
  • Hero Member
  • *****
  • Posts: 12915
  • ChainSquad GmbH
    • View Profile
    • ChainSquad GmbH
  • BitShares: xeroc
  • GitHub: xeroc
Re: New API for witness_node
« Reply #2 on: December 26, 2016, 07:57:52 pm »
Paging @vikram
I like this and would love to have more advanced api calls (e.g. to query holders and their balances of an asset)
I think this could and should be done as a plugin .. those can be compiled and used by anyone willing to use them and to offer the additional resources (or limit api access with credentials.)
Anyways ... looking forward to your work but i am not enough a c++ dev to give any meaningful advice
Give BitShares a try! Use the http://testnet.bitshares.eu provided by http://bitshares.eu powered by ChainSquad GmbH

Offline abit

  • Committee member
  • Hero Member
  • *
  • Posts: 4473
    • View Profile
    • Abit's Hive Blog
  • BitShares: abit
  • GitHub: abitmore
Re: New API for witness_node
« Reply #3 on: December 27, 2016, 03:46:47 pm »
Just FYI that Steem has moved on to use a SQL database to store irreversible chain state data. More developer can benefit from using that model, as more people know SQL in comparison to C++.

Paging [member=4854]vikram[/member]
I wonder why it's not clickable. Paging [member=4854]vikram[/member]

//Edit:
It's true that it's not too hard (for me) to implement more API's and/or plugins .. but I don't have time to do it, nor discuss the details, sorry.
« Last Edit: December 27, 2016, 03:53:18 pm by abit »
BTS account: abit
BTS committee member: abit
BTS witness: in.abit

Offline xeroc

  • Board Moderator
  • Hero Member
  • *****
  • Posts: 12915
  • ChainSquad GmbH
    • View Profile
    • ChainSquad GmbH
  • BitShares: xeroc
  • GitHub: xeroc
Re: New API for witness_node
« Reply #4 on: December 27, 2016, 05:46:20 pm »
Just FYI that Steem has moved on to use a SQL database to store irreversible chain state data. More developer can benefit from using that model, as more people know SQL in comparison to C++.

Paging [member=4854]vikram[/member]
I wonder why it's not clickable. Paging [member=4854]vikram[/member]

//Edit:
It's true that it's not too hard (for me) to implement more API's and/or plugins .. but I don't have time to do it, nor discuss the details, sorry.
I thought the sql database will be separate from the actual blockchain .. pretty much like a standard library for a separate block explorer
Give BitShares a try! Use the http://testnet.bitshares.eu provided by http://bitshares.eu powered by ChainSquad GmbH

Offline ElMato

  • Sr. Member
  • ****
  • Posts: 288
    • View Profile
Re: New API for witness_node
« Reply #5 on: December 28, 2016, 01:42:00 am »
Just FYI that Steem has moved on to use a SQL database to store irreversible chain state data. More developer can benefit from using that model, as more people know SQL in comparison to C++.

Paging [member=4854]vikram[/member]
I wonder why it's not clickable. Paging [member=4854]vikram[/member]

//Edit:
It's true that it's not too hard (for me) to implement more API's and/or plugins .. but I don't have time to do it, nor discuss the details, sorry.


[member=18687]abit[/member], how steem will implement the SQL database for the irreversible state?
steemd will be pushing data to the SQL database? or an external process will be polling?

Last time i checked steem source code i didn't see any SQL plugin or something like that.

They will model every object as a table with their corresponding indexes?
Or a plain object_id->text(json) table with custom indexes as tables?

 

Offline abit

  • Committee member
  • Hero Member
  • *
  • Posts: 4473
    • View Profile
    • Abit's Hive Blog
  • BitShares: abit
  • GitHub: abitmore
Re: New API for witness_node
« Reply #6 on: December 28, 2016, 05:21:07 pm »
Just FYI that Steem has moved on to use a SQL database to store irreversible chain state data. More developer can benefit from using that model, as more people know SQL in comparison to C++.

Paging [member=4854]vikram[/member]
I wonder why it's not clickable. Paging [member=4854]vikram[/member]

//Edit:
It's true that it's not too hard (for me) to implement more API's and/or plugins .. but I don't have time to do it, nor discuss the details, sorry.


[member=18687]abit[/member], how steem will implement the SQL database for the irreversible state?
steemd will be pushing data to the SQL database? or an external process will be polling?

Last time i checked steem source code i didn't see any SQL plugin or something like that.

They will model every object as a table with their corresponding indexes?
Or a plain object_id->text(json) table with custom indexes as tables?
Pushing. Via a plugin or external script, I'm not sure, it's being developed, the code is not on github so far, but likely we'll see it soon.

By the way, there is already another SQL database developed by a community member on Steem: https://steemit.com/steemit/@arcange/steemsql-com-a-public-sql-server-database-with-all-steemit-blockchain-data .

I thought the sql database will be separate from the actual blockchain .. pretty much like a standard library for a separate block explorer
It's not only for a block explorer.. but yes, you got the idea.
BTS account: abit
BTS committee member: abit
BTS witness: in.abit