Author Topic: [Evaluation] Need for a better python library?  (Read 4522 times)

0 Members and 1 Guest are viewing this topic.

Offline ebit

  • Committee member
  • Hero Member
  • *
  • Posts: 1826
    • View Profile
  • BitShares: ebit
Re: [Evaluation] Need for a better python library?
« Reply #15 on: December 21, 2016, 01:21:05 am »
Vote YES to Growth!
I hope that [member=4465]alt[/member] will create a worker too.
https://github.com/pch957
btsbots is useful for bitshares system to spread.
telegram:ebit521
https://weibo.com/ebiter

Offline bitcrab

  • Committee member
  • Hero Member
  • *
  • Posts: 1895
    • View Profile
  • BitShares: bitcrab
  • GitHub: bitcrab
Re: [Evaluation] Need for a better python library?
« Reply #16 on: December 21, 2016, 06:01:04 am »
In fact the architecture of witness_node + graphene_gui is  good for private node & wallet, but not suitable for public wallet, it's too inefficient.
the client subscribe all chain block data, and process all low level data struct. the api node can't handle too many clients.
I much appreciate your input on this, alt and agree that the API isn't meant to go high-throughput. Still a redundant node from a redundant provider other than OpenLedger is DESPERATELY needed. Otherwise people will think that BitShares went down just because the OL nodes went down.
The subscription model is certainly something that results in plenty of traffic which is why the library that I plan to build will not make heavy use of it. Subscriptions are still useful for writing bots as they let you know about events without the need to poll for new data every block.

Quote
btsbots use a different architecture.
server side pre process the origin low level data, generate a new database suitable for btsbots's use case,  and provide these data through meteor's ddp protocol.
no more api request for witness_node are needed, when client get all the public data include: market orders, operation history, encrypt memo
in fact only broadcast the transaction need a new api request for witness_node.
so I guess this way more better for public wallet, and can handle much more clients.
wow, this seems to have taken quite a while and a lot of passion to build an API wrapper for BitShares. Is that wrapper open source?

I am a user of grapheneexchange python library, trans.bot is running using it, it's easy to use and work well, although there's some minor bugs. if a junior coder like me can use it with little difficulty, then a lot people can also enjoy using it, it definitely make sense, thanks xeroc.

at first I have tried to build the bot based on btsbots, however seems currently btsbots is designed for users to directly run it with some setting, it's not designed as a library for developer to use. and I am not senior enough to select the useful codes to build my own bot, so finally I switch to grapheneexchange.

now my main concern is that, is it possible that we can have the API wrapper based on a more efficient archtechure? maybe now high-throughput is not our main concern, but how about furture?  I think before doing decison at least we need to do some deep discussion on these topics.
Email´╝Ü[email protected]

Offline Chris4210

  • Sr. Member
  • ****
  • Posts: 431
  • Keep Building!
    • View Profile
    • www.payger.com
  • BitShares: chris4210
Re: [Evaluation] Need for a better python library?
« Reply #17 on: December 21, 2016, 11:52:20 am »
I read the proposal and also agree to vote for the worker. I am supporting all projects that help to onboard new developers to BitShares and makes it easier to get started.

I am also curious how the BitUSD payment will work. This could help us with future workers.

Best regards,

Christoph

Vote Chris4210 for Committee Member http://bit.ly/1WKC03B! | www.Payger.com - Payments + Messenger | www.BitShareshub.io - Community based fanpage for the BitShares Blockchain

Offline ElMato

  • Sr. Member
  • ****
  • Posts: 288
    • View Profile
Re: [Evaluation] Need for a better python library?
« Reply #18 on: December 21, 2016, 04:54:28 pm »
In fact the architecture of witness_node + graphene_gui is  good for private node & wallet, but not suitable for public wallet, it's too inefficient.
the client subscribe all chain block data, and process all low level data struct. the api node can't handle too many clients.

btsbots use a different architecture.
server side pre process the origin low level data, generate a new database suitable for btsbots's use case,  and provide these data through meteor's ddp protocol.
no more api request for witness_node are needed, when client get all the public data include: market orders, operation history, encrypt memo
in fact only broadcast the transaction need a new api request for witness_node.
so I guess this way more better for public wallet, and can handle much more clients.


[member=4465]alt[/member], I agree with your view but we need that API inside the witness_node.
Otherwise btsbots or any other application will have to rely on a custom backend.

What we are doing is similar to what you are doing but using GraphQL to make it generic.

http://graphql.org/learn/

Since the underlying database is an object database it makes sense to use a query language like that.

As an example this is how we query for an account balance + history.

Code: [Select]
query getAll($v1 : String!) {
  account(name:$v1) {
    balance {
      quantity
      asset {
        id
      }
    }
    history(start:0, limit:20) {
      id
      __typename
      block {
        timestamp
      }
      ... on NoDetailOp {
        id
      }
      ... on Transfer {
        from {
          name
        }
        to {
          name
        }
        memo {
          from
          to
          nonce
          message
        }
        amount {
          quantity
          asset {
            symbol
            id
          }
        }
        fee {
          quantity
          asset {
            symbol
            id
          }
        }
      }
    }
  }
}

In my opionion it would be amazing to have the witnes_node expose a GraphQL API.
Because In that case you just remove all the "special" functions defined in database_api.hpp/database_api.cpp and just rely in the query language to perform all your data extraction needs.

There is a C++ library that process a graphql query.
https://github.com/graphql/libgraphqlparser

The remaining work would be to process the AST and resolve using the graphene::chain library. 
« Last Edit: December 21, 2016, 05:13:41 pm by ElMato »

Offline nmywn

  • Sr. Member
  • ****
  • Posts: 266
    • View Profile
Re: [Evaluation] Need for a better python library?
« Reply #19 on: December 21, 2016, 05:26:57 pm »
maybe now high-throughput is not our main concern, but how about furture? 
There won't be any future without improving that aspect now. Most traffic go via Openledger, and they constatly fail during crowd invasion ( for example: last BTS peak at 1300).

Offline alt

  • Hero Member
  • *****
  • Posts: 2821
    • View Profile
  • BitShares: baozi
Re: [Evaluation] Need for a better python library?
« Reply #20 on: December 21, 2016, 10:36:19 pm »
I'm very backward in change witness_node. the point is it's must stable and efficient.

so I am not surprised that many data can't get directly from this witness_node api.
for example: get any asset's snapshot at anytime.
for example: dividend.
for example: get the hide deep from different market
of course we can add all these in the witness_node, but maybe it will change more inefficient and unstable. I only need the simplest api:  get a transaction data.

I prefer add a seperate api wrap level, which run in another process, or in another server.
this api level designed  based on what application runs on it.
the application can provide service more efficient.

In fact the architecture of witness_node + graphene_gui is  good for private node & wallet, but not suitable for public wallet, it's too inefficient.
the client subscribe all chain block data, and process all low level data struct. the api node can't handle too many clients.

btsbots use a different architecture.
server side pre process the origin low level data, generate a new database suitable for btsbots's use case,  and provide these data through meteor's ddp protocol.
no more api request for witness_node are needed, when client get all the public data include: market orders, operation history, encrypt memo
in fact only broadcast the transaction need a new api request for witness_node.
so I guess this way more better for public wallet, and can handle much more clients.


[member=4465]alt[/member], I agree with your view but we need that API inside the witness_node.
Otherwise btsbots or any other application will have to rely on a custom backend.

What we are doing is similar to what you are doing but using GraphQL to make it generic.

http://graphql.org/learn/

Since the underlying database is an object database it makes sense to use a query language like that.

As an example this is how we query for an account balance + history.

Code: [Select]
query getAll($v1 : String!) {
  account(name:$v1) {
    balance {
      quantity
      asset {
        id
      }
    }
    history(start:0, limit:20) {
      id
      __typename
      block {
        timestamp
      }
      ... on NoDetailOp {
        id
      }
      ... on Transfer {
        from {
          name
        }
        to {
          name
        }
        memo {
          from
          to
          nonce
          message
        }
        amount {
          quantity
          asset {
            symbol
            id
          }
        }
        fee {
          quantity
          asset {
            symbol
            id
          }
        }
      }
    }
  }
}

In my opionion it would be amazing to have the witnes_node expose a GraphQL API.
Because In that case you just remove all the "special" functions defined in database_api.hpp/database_api.cpp and just rely in the query language to perform all your data extraction needs.

There is a C++ library that process a graphql query.
https://github.com/graphql/libgraphqlparser

The remaining work would be to process the AST and resolve using the graphene::chain library.

Offline xeroc

  • Board Moderator
  • Hero Member
  • *****
  • Posts: 12915
  • ChainSquad GmbH
    • View Profile
    • ChainSquad GmbH
  • BitShares: xeroc
  • GitHub: xeroc
Re: [Evaluation] Need for a better python library?
« Reply #21 on: December 22, 2016, 01:08:07 pm »
[member=4465]alt[/member], the witness node can have so called plugins to deal with those features. Block producing witnesses shouldn't have these plugins enabled at all, so that adding new API calls will not affect the blockchain.
At most it could affect the API providing business
Give BitShares a try! Use the http://testnet.bitshares.eu provided by http://bitshares.eu powered by ChainSquad GmbH

Offline ElMato

  • Sr. Member
  • ****
  • Posts: 288
    • View Profile
Re: [Evaluation] Need for a better python library?
« Reply #22 on: December 26, 2016, 08:14:54 pm »
I wrote a post to see if we can join efforts in upgrading the API of the witness_node or build a parallel solution to query the blockchain state in any possible way.

https://bitsharestalk.org/index.php/topic,23627.0.html

Offline JonnyB

  • Hero Member
  • *****
  • Posts: 636
    • View Profile
    • twitter.com/jonnybitcoin
Re: [Evaluation] Need for a better python library?
« Reply #23 on: February 17, 2017, 02:03:56 am »

Worker
Given that a worker will be voted in for quite some time, I would like to propose a new model of running a worker:
  • I will create a new account, upgrade it to LTM and make it multisig with the committee account and trusted members of the comunity
  • The worker will redeem it's funds on a regular basis and buy up bitUSD from the market (only up to +5% above market)
  • If the market doesn't offer sufficient bitUSD, the worker account will borrow bitUSD at 2.5x collateral
  • For this reasons, the actual pay of the worker is 2.5x the USD value
  • The worker will only pay the agreed amount of money and only in bitUSD to me
  • Every thing that is not paid out after the end of the worker will be settled and returned to the reserve fund

you say " The worker will redeem it's funds on a regular basis and buy up bitUSD from the market "

But if you buy up lots of bitusd and put it in the chainsquad account there will be less bitusd in the marketplace.
I would prefer it if you created all the bitusd to boost the overall supply and help the liquidity issues we have.

Also if you are buying up bitusd with bts, that is the same as dumping bts which will create downward pressure on the bts price.
« Last Edit: February 17, 2017, 02:07:24 am by JonnyBitcoin »
I run the @bitshares twitter handle
twitter.com/bitshares