[Evaluation] Need for a better python library?

[JonnyB]

Worker
Given that a worker will be voted in for quite some time, I would like to propose a new model of running a worker:

• I will create a new account, upgrade it to LTM and make it multisig with the committee account and trusted members of the comunity
• The worker will redeem it's funds on a regular basis and buy up bitUSD from the market (only up to above market)
• If the market doesn't offer sufficient bitUSD, the worker account will borrow bitUSD at 2.5x collateral
• For this reasons, the actual pay of the worker is 2.5x the USD value
• The worker will only pay the agreed amount of money and only in bitUSD to me
• Every thing that is not paid out after the end of the worker will be settled and returned to the reserve fund

you say " The worker will redeem it's funds on a regular basis and buy up bitUSD from the market "

But if you buy up lots of bitusd and put it in the chainsquad account there will be less bitusd in the marketplace.
I would prefer it if you created all the bitusd to boost the overall supply and help the liquidity issues we have.

Also if you are buying up bitusd with bts, that is the same as dumping bts which will create downward pressure on the bts price.

[ElMato]

I wrote a post to see if we can join efforts in upgrading the API of the witness_node or build a parallel solution to query the blockchain state in any possible way.

https://bitsharestalk.org/index.php/topic,23627.0.html

[xeroc]

@alt, the witness node can have so called plugins to deal with those features. Block producing witnesses shouldn't have these plugins enabled at all, so that adding new API calls will not affect the blockchain.
At most it could affect the API providing business

[alt]

I'm very backward in change witness_node. the point is it's must stable and efficient.

so I am not surprised that many data can't get directly from this witness_node api.
for example: get any asset's snapshot at anytime.
for example: dividend.
for example: get the hide deep from different market
of course we can add all these in the witness_node, but maybe it will change more inefficient and unstable. I only need the simplest api:  get a transaction data.

I prefer add a seperate api wrap level, which run in another process, or in another server.
this api level designed  based on what application runs on it.
the application can provide service more efficient.

In fact the architecture of witness_node + graphene_gui is  good for private node & wallet, but not suitable for public wallet, it's too inefficient.
the client subscribe all chain block data, and process all low level data struct. the api node can't handle too many clients.

btsbots use a different architecture.
server side pre process the origin low level data, generate a new database suitable for btsbots's use case,  and provide these data through meteor's ddp protocol.
no more api request for witness_node are needed, when client get all the public data include: market orders, operation history, encrypt memo
in fact only broadcast the transaction need a new api request for witness_node.
so I guess this way more better for public wallet, and can handle much more clients.

@alt, I agree with your view but we need that API inside the witness_node.
Otherwise btsbots or any other application will have to rely on a custom backend.

What we are doing is similar to what you are doing but using GraphQL to make it generic.

http://graphql.org/learn/

Since the underlying database is an object database it makes sense to use a query language like that.

As an example this is how we query for an account balance + history.

Code: [Select]

query getAll($v1 : String!) {
  account(name:$v1) {
    balance {
      quantity
      asset {
        id
      }
    }
    history(start:0, limit:20) {
      id
      __typename
      block {
        timestamp
      }
      ... on NoDetailOp {
        id
      }
      ... on Transfer {
        from {
          name
        }
        to {
          name
        }
        memo {
          from
          to
          nonce
          message
        }
        amount {
          quantity
          asset {
            symbol
            id
          }
        }
        fee {
          quantity
          asset {
            symbol
            id
          }
        }
      }
    }
  }
}

In my opionion it would be amazing to have the witnes_node expose a GraphQL API.
Because In that case you just remove all the "special" functions defined in database_api.hpp/database_api.cpp and just rely in the query language to perform all your data extraction needs.

There is a C++ library that process a graphql query.
https://github.com/graphql/libgraphqlparser

The remaining work would be to process the AST and resolve using the graphene::chain library.

[nmywn]

maybe now high-throughput is not our main concern, but how about furture? 

There won't be any future without improving that aspect now. Most traffic go via Openledger, and they constatly fail during crowd invasion ( for example: last BTS peak at 1300).

[ElMato]

In fact the architecture of witness_node + graphene_gui is  good for private node & wallet, but not suitable for public wallet, it's too inefficient.
the client subscribe all chain block data, and process all low level data struct. the api node can't handle too many clients.

btsbots use a different architecture.
server side pre process the origin low level data, generate a new database suitable for btsbots's use case,  and provide these data through meteor's ddp protocol.
no more api request for witness_node are needed, when client get all the public data include: market orders, operation history, encrypt memo
in fact only broadcast the transaction need a new api request for witness_node.
so I guess this way more better for public wallet, and can handle much more clients.

@alt, I agree with your view but we need that API inside the witness_node.
Otherwise btsbots or any other application will have to rely on a custom backend.

What we are doing is similar to what you are doing but using GraphQL to make it generic.

http://graphql.org/learn/

Since the underlying database is an object database it makes sense to use a query language like that.

As an example this is how we query for an account balance + history.

Code: [Select]

query getAll($v1 : String!) {
  account(name:$v1) {
    balance {
      quantity
      asset {
        id
      }
    }
    history(start:0, limit:20) {
      id
      __typename
      block {
        timestamp
      }
      ... on NoDetailOp {
        id
      }
      ... on Transfer {
        from {
          name
        }
        to {
          name
        }
        memo {
          from
          to
          nonce
          message
        }
        amount {
          quantity
          asset {
            symbol
            id
          }
        }
        fee {
          quantity
          asset {
            symbol
            id
          }
        }
      }
    }
  }
}

In my opionion it would be amazing to have the witnes_node expose a GraphQL API.
Because In that case you just remove all the "special" functions defined in database_api.hpp/database_api.cpp and just rely in the query language to perform all your data extraction needs.

There is a C++ library that process a graphql query.
https://github.com/graphql/libgraphqlparser

The remaining work would be to process the AST and resolve using the graphene::chain library. 

[Chris4210]

I read the proposal and also agree to vote for the worker. I am supporting all projects that help to onboard new developers to BitShares and makes it easier to get started.

I am also curious how the BitUSD payment will work. This could help us with future workers.

Best regards,

Christoph

[bitcrab]

In fact the architecture of witness_node + graphene_gui is  good for private node & wallet, but not suitable for public wallet, it's too inefficient.
the client subscribe all chain block data, and process all low level data struct. the api node can't handle too many clients.

I much appreciate your input on this, alt and agree that the API isn't meant to go high-throughput. Still a redundant node from a redundant provider other than OpenLedger is DESPERATELY needed. Otherwise people will think that BitShares went down just because the OL nodes went down.
The subscription model is certainly something that results in plenty of traffic which is why the library that I plan to build will not make heavy use of it. Subscriptions are still useful for writing bots as they let you know about events without the need to poll for new data every block.

btsbots use a different architecture.
server side pre process the origin low level data, generate a new database suitable for btsbots's use case,  and provide these data through meteor's ddp protocol.
no more api request for witness_node are needed, when client get all the public data include: market orders, operation history, encrypt memo
in fact only broadcast the transaction need a new api request for witness_node.
so I guess this way more better for public wallet, and can handle much more clients.

wow, this seems to have taken quite a while and a lot of passion to build an API wrapper for BitShares. Is that wrapper open source?

I am a user of grapheneexchange python library, trans.bot is running using it, it's easy to use and work well, although there's some minor bugs. if a junior coder like me can use it with little difficulty, then a lot people can also enjoy using it, it definitely make sense, thanks xeroc.

at first I have tried to build the bot based on btsbots, however seems currently btsbots is designed for users to directly run it with some setting, it's not designed as a library for developer to use. and I am not senior enough to select the useful codes to build my own bot, so finally I switch to grapheneexchange.

now my main concern is that, is it possible that we can have the API wrapper based on a more efficient archtechure? maybe now high-throughput is not our main concern, but how about furture?  I think before doing decison at least we need to do some deep discussion on these topics.

[ebit]

Vote YES to Growth!
I hope that @alt will create a worker too.
https://github.com/pch957
btsbots is useful for bitshares system to spread.

[BunkerChainLabs-DataSecurityNode]

So.. I am among the private companies that is getting all kinds of programming work done by Xeroc to work with Bitshares but am not sharing our code because, well, we paid for it. I would prefer if these tools were made available generally to the public though.. it would not have just saved us time and money but I am willing to bet that others would have considered bitshares if the tools were that to make the barrier to entry not so high. This worker is all about that, and this is essential for adoption. I lost count of how may have come an went because of the lack of supporting tools like these. Ethereum largely captures projects because of these type of tools.

If you want to argue otherwise, please look at the past year of adoption and make your case. Good luck with that.

What is also interesting about this worker is that it has been setup with a focus on improving and utilizing our bitassets while also incorporating Committee and trusted member oversight. This can provide a good test if not a standard for future worker proposals that could potentially lead to better ones in the future.

So yes, I am supporting this.. and hope we can have voting support for this to see it through without being short sighted about whatever circumstances Bitshares price goes through and recognizes this as a long term positive impact to the price.

Vote YES to Growth!

[Brekyrself]

The easier it is for others to launch apps built upon BitShares, the better the ecosystem will become.  Big companies will only take notice once there are neat small company app's doing incredible things.  Classic chicken and egg problem.

Create the worker and lets get this moving.

[xeroc]

Thanks you very much for all the feedback! I appreciate this alot! Let me address some points:

So, how is the worker account going to settle its debt (if any)? Will it wait until bitUSD price get back to 5% from feed?

This is a decision I cannot make as the funds are owned by the shareholders. I personally would either sit it out and just keep the open positions indefinitely, or agree on a ratio to stick with and adapt the collateral once in a while. Either way, the increased amount of available bitUSD goes up.

regarding the worker pay advice, in the case that there is no enough bitUSD can be bought from market and finally the worker account borrow bitUSD and pay to worker, how will the worker account be finally settled? waiting someone to force settle?

That would be an option, or just keep the order open. The call position is owned by the shareholders and it doesn't really matter as the shareholders want increased liquidity in bitUSD and this way get it by having their BTS funds locked up as collateral. Of course, the shareholders can vote on closing it eventually and return the BTS into the reserves but it doesn't really change anything except reduce the amount of liquid bitUSD.

whoever is in need of this library should hire developers and code the library according to the specifications of their needs.

That is exaclty what businesses are doing, but BitShares shareholders don't benefit from it.

BitShares has gone this far without this better python library .

That's only partially true. Many businesses approached me and wanted a script for this and that. They all got their script and most of them are non public. Why should they be?
Because I used to release my code for free for the last 3 days doesn't mean that I can do so with code that 3rd parties paid for.

And your proposal is vague as to what this library does and how it will be technically helpful to those who actually need to implement BitShares in their system compare to existing library or code themselves.

Good point. I assumed that people in here know what the piston library (now python-steem) can do and didn't go into detail much.
The point is to have an easy to use library for developers to build ANY application on top of BitShares. Take for instance streemian.com, it uses python-steem in the backend and offers plenty of features for Steem shareholders. Streemian is not the only platform that uses python-steem in the backend. The point here is, because there is an easy to use library, people start buildint stuff and steem has many of those if you take a look at steemtools.com

All in all, judging by your working plan, I think it's more of a hobby work instead of serious development that matches industrial wage standard .

Apologize for not describing my intensions clearly. The point here is that I offer my time and expertise to build something that can grow the ecosystem for cheaper than I ask businesses for. My weekly schedule simply doesn't allow to work on this for more than 10h/week although there is some work that overlaps.
To be clear also, I don't need to work for BitShares, I can go on and keep the work I do private and sell it to every interested party. But this doesn't help BitShares shareholders and does NOT help homebrew developers that want to look into building something just to find out that there is NO easy to use library.

Just FYI, businesses that actually need to use BitShares will more likely to write their own library for obvious security concerns, including but not limited to "it's a hobby work that made from 10 hours a week " , so most of them will still repeat the work themselves.

Sorry for making it appear to be a "hobby", which it certainly is not. As for businessses looking to build there own libraries, well, I founded one of these businesses in Germany and am building a library. The security aspect is rather misplaced because audits can only take place once there is software to actually audit.
Furthermore, please let me know if you see a business that has the knowledge and expertise to actually build stuff on top of BitShares that is not CNX, Blocktrades or BunkerchainLabs.

I think the non-technical stakeholders need to be aware of this perspective - not every public code is useful for many reasons, especially APIs and libraries , which are mostly business and environment specific with special concerns in mind, and " a hobby work that made from 10 hours a week " could not have covered them.

I wonder how you can make this a "hobby" project just because I can't offer more than 10h/week.
Would you be willing to pay 40h/week? For sure, BitShares can't handle that at the current stage and many Shareholders would agree and not even consider this worker. The 10h/week figure is a compromise between the work that I already do, the time that I can offer and the payrate that has at least a slight chance to go through as a worker.

Anyways, I appreciate your input and agree that shareholders should know all the opinions there are.

Oh, btw, did you know that Microsofts' Kinect's success is mostly a result of homebrew developers that appreciate the open source library?

What happens if business do not want to use python but node.js ?  Another proposal to satisfy the imaginary businesses that would somehow bring value to BitShares but can not even pay for 1 developer who works 10 hours a week themselves ?

Sure, those businesses should not even consider working no top of BitShares if they can't afford a single developer for the language they chose to use.
But how much money do you expect them to sink into learning the existing code base and figuring out HOW to actually work with BitShares?
Would you not agree that people can build their stuff more quickly if there was a widely used, audited and open source library to use, like BitcoinJ, BitcoinJS or web3. Non of those have been build and release open source by a company but are sole community products?

How do you plan on building a community of developers if you don't even provide the basic support? How do you plan to bring in businesses that can't even "play with the blockchain" by using a public library or API?

Do not always ask money from BTS holders, but ask what you can do for BTS holders.

You must be new around here. If you want to see what I have provide for BTS holders for free in the time between launch of BitShares 1 and today, just take a look at the 11k+ posts in this forum

In fact the architecture of witness_node + graphene_gui is  good for private node & wallet, but not suitable for public wallet, it's too inefficient.
the client subscribe all chain block data, and process all low level data struct. the api node can't handle too many clients.

I much appreciate your input on this, alt and agree that the API isn't meant to go high-throughput. Still a redundant node from a redundant provider other than OpenLedger is DESPERATELY needed. Otherwise people will think that BitShares went down just because the OL nodes went down.
The subscription model is certainly something that results in plenty of traffic which is why the library that I plan to build will not make heavy use of it. Subscriptions are still useful for writing bots as they let you know about events without the need to poll for new data every block.

btsbots use a different architecture.
server side pre process the origin low level data, generate a new database suitable for btsbots's use case,  and provide these data through meteor's ddp protocol.
no more api request for witness_node are needed, when client get all the public data include: market orders, operation history, encrypt memo
in fact only broadcast the transaction need a new api request for witness_node.
so I guess this way more better for public wallet, and can handle much more clients.

wow, this seems to have taken quite a while and a lot of passion to build an API wrapper for BitShares. Is that wrapper open source?

[ebit]

alt 
I like the idea to pay Workers in bitUSD 

[alt]

In fact the architecture of witness_node + graphene_gui is  good for private node & wallet, but not suitable for public wallet, it's too inefficient.
the client subscribe all chain block data, and process all low level data struct. the api node can't handle too many clients.

btsbots use a different architecture.
server side pre process the origin low level data, generate a new database suitable for btsbots's use case,  and provide these data through meteor's ddp protocol.
no more api request for witness_node are needed, when client get all the public data include: market orders, operation history, encrypt memo
in fact only broadcast the transaction need a new api request for witness_node.
so I guess this way more better for public wallet, and can handle much more clients.

[nmywn]

Additionally, I would like to run a public API endpoint

I'll vote.