Author Topic: more professional price feed?  (Read 31751 times)

0 Members and 1 Guest are viewing this topic.

Xeldal

  • Guest
My feeds are back up today.  They were down for sometime after updating the script I was using caused errors I was unable to reconcile (seemingly) short of re-writing the whole thing, which I have since started doing.   Given that my bots are already polling markets and prices from all over, it shouldn't be too much to translate and expand this to provide another source of feed pricing.   At the moment I'm using Alts script but intend to continue developing my own as a fallback and eventually as a primary source for myself. 

I'd appreciate a returned vote from anyone who may have removed their support during this down time.  Thanks.

Offline Pheonike

As the Bitshares grows up, the amount of resources needed for running a witness will grow. As envisioned early on, witnesses will be mini companies unto themselves. When Bitshares reaches 5 Billion and more, people are going to want a professional team running these nodes and not some guy doing it part-time. So we should start grooming that kind of talent now and not until performance starts to suffer because we have amateur witnesses.
« Last Edit: January 18, 2017, 02:21:42 am by Pheonike »

Offline Thom

camera and phone are different things, but finally they are integrated in iPone, what make this happen? demand to covenience, economy consideration and technology improvement.

so there is not a problem for a witness to do both block generation and price feed, to update skills or to cooperate are both ways to solve the problem. no need to worry that seldom can do both, after the increase of witness pay there will be enough candidates there that can make the tasks done.

I don't care whether you select to use alt's script, to buy data from Bloomberg or to code script yourself. as a proxy and a trader I only care the accuracy of the feeded price, it's not difficult for an active trader to judge whether the feed price is accurate enough, explain and vote accordingly.

workers always bring complex and create new problems so it is not a good choice here.

I tend to agree with you that a worker proposal is not the best option.

I don't think your analogy with the iPhone is a very good one, because a huge, capital intensive corporation, in fact an entire industry of different corporations, took many years to evolve communications technology into what became the iPhone design. We don't have the luxury of waiting for such an evolutionary process.

How many coders and IT engineers have degrees in economics? How many are mathematicians or statisticians? How many are expert or frequent traders? How many of those understand the ins and outs of the server hosting industry and networking?

There is a well recognized benefit to the division of labor. It usually produces better results than jack of all trades workers who don't have the specialty knowledge and therefore miss things or compromise the quality of the job just to get it done.

As I already pointed out, we've had 3 out of four fail at delivery reliable, accurate feeds according the cries from the east.

Splitting the roles into 2 makes a lot of sense. Pay can be focused to where it is needed most when it is needed. Skills appropriate the the task can be brought to bear without compromising quality of either role. There's no guarantee that increasing block pay rate will fix the feed issue. Some witnesses may use the increase to improve server infrastructure and put only a small amount if any towards feeds. Splitting the roles avoids that possibility.

Since there are zero guidelines on what is expected for feed producers I suggest we start there. Why don't those of you who have issues with feeds take some time to think about what your expectations are and communicate them here. Consider feed publishing frequency, sources, a definition of accuracy, whatever you expect from a feed producer. Just saying you want accurate feeds does not help. What you think is accurate others might not and vice-versa.
« Last Edit: January 17, 2017, 05:37:50 pm by Thom »
Injustice anywhere is a threat to justice everywhere - MLK |  Verbaltech2 Witness Reports: https://bitsharestalk.org/index.php/topic,23902.0.html

Offline bitcrab

  • Committee member
  • Hero Member
  • *
  • Posts: 1928
    • View Profile
  • BitShares: bitcrab
  • GitHub: bitcrab
camera and phone are different things, but finally they are integrated in iPone, what make this happen? demand to covenience, economy consideration and technology improvement.

so there is not a problem for a witness to do both block generation and price feed, to update skills or to cooperate are both ways to solve the problem. no need to worry that seldom can do both, after the increase of witness pay there will be enough candidates there that can make the tasks done.

I don't care whether you select to use alt's script, to buy data from Bloomberg or to code script yourself. as a proxy and a trader I only care the accuracy of the feeded price, it's not difficult for an active trader to judge whether the feed price is accurate enough, explain and vote accordingly.

workers always bring complex and create new problems so it is not a good choice here.



Email:bitcrab@qq.com

Offline Thom

First, I'm glad some here are finally realizing there's a helluva lot more to running a witness than producing feeds. Thanks for seeing that @bitcrab.
@DataSecurityNode and @rnglab have the most comprehensive perspectives of anyone else I see posting in this thread. Everyone else has tunnel vision on the feeds issue.

Granted, my own eyes are now open to the feeds problem now. Perhaps I'm guilty of letting others (i.e. wackou) handle the feeds, just like most others use xeroc's feed scripts to handle theirs. My focus has been more on running 4 full nodes at minimal cost. How many other witnesses are running a seed node let alone 3?

Like I've been saying since day 1, the knowledge required to produce feeds is quite different than the knowledge to maintain fault tolerant, up-to-date, decentralized witness and seed nodes. It requires monitoring and automation, lots of IT skills and the ability to gauge when to upgrade, how to spin up new instances and where to get the best hosting and how it should be distributed around the planet. It requires a high degree of security awareness.

Producing feeds require a small amount of those skills. Producing quality and accurate feeds involves finding reliable data sources or developing algorithms as apparently @alt did to monitor trades on exchanges as well as utilize any other spotty and less reliable data from free sources. It requires much more of a trader perspective, and better statistics and math skills to aggregate a final, accurate result.
 
I recognize we need to come up with a better solution to the feeds issue, as well as the overall problem of how to sustain the basic infrastructure of the platform. Feeds are certainly a part of that, especially if we want adoption to increase. But lets not neglect the other important issues in the process of fixing feeds.

Hence decreasing the decentralization of the system since most would use the provided feed.

And if the witness's decided not to use the worker provided feed, what is the point of having a worker provided feed?

What you describe is NOT decentralized if everybody uses the same (paid) feed source. IF a worker proposal is deemed to be the best way to obtain reliable feed data, then multiple sources would be required, or else we have an Achilles heel dependency issue.

We have at least 4 different people (xeroc, abit, alt, wackou) who have created feed scripts, and only 1 claims to have an accurate result, that being alt. Who's to say the trades alt monitors will always be the correct benchmark? Who decides which feed is accurate? Compared with what? Perhaps we need a professional, paid data source as a reference for comparison.

This feed issue is why so many others in the crypto space argue oracles don't work and why the concept of BitAssets won't either. Granted, it's not an easy problem to resolve, but there seems to be a big difference of opinion on the reliability and accuracy of the free sources available. Why else did @alt take the time to develop an algorithm to obtain prices by monitoring trades?

Injustice anywhere is a threat to justice everywhere - MLK |  Verbaltech2 Witness Reports: https://bitsharestalk.org/index.php/topic,23902.0.html

Offline lil_jay890

  • Hero Member
  • *****
  • Posts: 1197
    • View Profile
Plus having BTS pay for a feed that all the witness's end up using kind of goes against the whole decentralization principles.

Who said anything about a single source paid or otherwise for ALL witnesses. It's NOT HELPFUL to jump to such conclusions. Let's stop this blame game and start figuring out the cause of the issue really lies and work together to resolve it.

I was saying that if a worker was put together to purchase a price feed, say from Bloomberg, that all the witness's would tend to use that price feed.  It would be the cheap and easy option for the witness's to do that.  Hence decreasing the decentralization of the system since most would use the provided feed.

And if the witness's decided not to use the worker provided feed, what is the point of having a worker provided feed?

Offline Thom

It actually makes sense to pay separately for producing blocks and for publishing price feeds, because these are two different jobs which require different efforts.

THIS!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Injustice anywhere is a threat to justice everywhere - MLK |  Verbaltech2 Witness Reports: https://bitsharestalk.org/index.php/topic,23902.0.html

Offline Thom

Plus having BTS pay for a feed that all the witness's end up using kind of goes against the whole decentralization principles.

Who said anything about a single source paid or otherwise for ALL witnesses. It's NOT HELPFUL to jump to such conclusions. Let's stop this blame game and start figuring out the cause of the issue really lies and work together to resolve it.
Injustice anywhere is a threat to justice everywhere - MLK |  Verbaltech2 Witness Reports: https://bitsharestalk.org/index.php/topic,23902.0.html

Offline Thom

today a man made margin call happened again for bitCNY, about 50K bitCNY margin called, someone did price manipulation just by selling a little quantity of BTS in yunbi.

proxy bitcrab decided to unvote below witnesses, until they implement professional enough bitCNY price feeding.
spectral delegate-clayop bhuz bue delegate.ihashfury spartako fox elmato rnglab datasecuritynode

don't tell me btc38 always fail to provide price, don't ask me how to generate good price feed, you are witnesses, you should find ways to make that done.

Note - datasecuritynode wasn't being voted for by your proxy before anyway.

I would suggest we create a worker proposal to gain access to more price feed data rather than continuing to rely on bottom of the barrel free data. Vote for it?

I  don't understand why worker proposal are needed here.
if some witnesses have proved that they can provide professional price feed for free(no more than the block generation reward), then why should I vote for the worker propossal?

Why? Because it is now obvious to me the issue is the sources. You are not looking rationally at the issue. If you are making claims that alt can do it, why can't you too?, I would claim alt somehow is getting data from inside the Great Wall and everyone else has to rely on unreliable sources outside.

THIS IS A BIG PROBLEM THAT IS NOT BEING RECOGNIZED BY THE CHINESE

The sequestered info due to the Chinese firewall is a major issue for western witnesses, wake up and listen to us. Our free sources are not reliable. Btc38 has been very bad. My observations over the last year agree with DataSecurityNode's.

We need to get to the bottom of unreliable sources. If the free sources were that good we wouldn't be having this discussion.
Injustice anywhere is a threat to justice everywhere - MLK |  Verbaltech2 Witness Reports: https://bitsharestalk.org/index.php/topic,23902.0.html

Offline bitcrab

  • Committee member
  • Hero Member
  • *
  • Posts: 1928
    • View Profile
  • BitShares: bitcrab
  • GitHub: bitcrab
Sorry for the low response when CNY volume moved from btc38 and yunbi to other exchanges. It is fixed now.

Also I remained publishing acceptable GOLD and SILVER prices when Yahoo started to bring strange values, as I've added an Quandl API key shortly after xeroc added it as a feedsource.

Now I'm fetching GC and SI from Yahoo instead of XAU and XAG as abit suggested. I also applied his patch to derive prices across 3 markets and everything looks good.

thanks, revoted, I am also trying to run a witness in order to understand the witness thing more deeply.
Email:bitcrab@qq.com

Offline bitcrab

  • Committee member
  • Hero Member
  • *
  • Posts: 1928
    • View Profile
  • BitShares: bitcrab
  • GitHub: bitcrab
@alt  runs a bussines which depends on precise and resilient feeds. His code skills along with a real incentive, brings us a good chance to improve the way we fetch and derive external prices.
Besides that I think it would be more productive to fork and improve @xeroc script, as a way to sum the scarce resources we have on this topic. 

Some conditions have to be given for diversity to be enriching, and for fair competition to exist at all. In the meanwhile, I think that cooperation is the way to go. Competition will favor aptitude and commitment only when we become able to incentivize it.


 Some numbers:

  - Witness reward averaged something around $200/month this year.

 - Let's say a regular VPS with 8Gb RAM costs 80 USD per month, and 50 USD for 4Gb.
 - That is a 150 USD reward for witnesses running just one node with only 4Gb  (at its very memory limit).
 - There's a 120 USD incentive to have 8Gb on just one node, with no failover at all.
 - Even running two 4Gb nodes with a proper failover strategy (for ~100 USD), both  nodes could easily crash if RAM becomes not enough.

Automatic fork resolution is working great, but it has not been prooven beyond our current TPS.
With checkpoints and delayed nodes, we have revived the network in less than an hour with no major consequences when forks turned into lost of consensus.
But I wouldn't like to see the network crashing just when the real action starts.

 - It seems to me that having two 8Gb nodes per witness, with failover able to identify and move away from minority forks (when possible with just two nodes), are the minimum technical requirements to handle the upcoming network growth. That means a reward of 40 USD per month at most.
Add seed nodes and the time required to constantly check and fix feeds, having everything up to date and the need to remain informed in a daily basis.

That means good withesses working at a loss, while icentivizing poor performance for a proffit.



This also brings another concern to me: as many good neutral witnesses lack of incentive or resources/time despite they good will, it drives business to run it's own witnesses given their need to have reliable feeds.
I see a flaw here:  business tend to have more resources, voting stake, reward incentives and thus good performance , this raises their motivation and chances to have many witnesses voted in.
As business models may disagree about market parameters that are defined by feed producers, this could lead to unfair competition or other undesirable behavors, specially now that new/unknown players are steadily arriving.

This is one of the reasons why witnesses, committee and proxies were planed as independent powers.



I'm not pushing to rise block rewards here, nor discouraging our most valuable witnesses who also runs business over bitshares.
 I still enjoy being reliable and helpful as a witness, rewards aside, and  I'm glad to see this debate going on in a constructive way, finally.
Just want to sumarize some facts I see.

At this point we really need a good balance between competition and collaborative work to make network and market more stable and to reduce the entry barriers.

this summary convinced me that we need to seriously consider to rise block reward.

while the trading volume keep growing, we need better witness service: stable block generation, reliable fork resolution, better price feed, and maybe even more witness nodes. all these need witnesses to pay more effort and cost.

I believe that many people enjoy contributing to the community as a witness with caring little to the reward, but the growing network should depend not only on good will, but also on reasonable business models. if we demand each witness node to provide high quality service, the block reward should match the effort and cost they have to pay. and then more witness will come and compete.
Email:bitcrab@qq.com

Offline rnglab

  • Full Member
  • ***
  • Posts: 171
    • View Profile
  • BitShares: rnglab
Sorry for the low response when CNY volume moved from btc38 and yunbi to other exchanges. It is fixed now.

Also I remained publishing acceptable GOLD and SILVER prices when Yahoo started to bring strange values, as I've added an Quandl API key shortly after xeroc added it as a feedsource.

Now I'm fetching GC and SI from Yahoo instead of XAU and XAG as abit suggested. I also applied his patch to derive prices across 3 markets and everything looks good.


Offline rnglab

  • Full Member
  • ***
  • Posts: 171
    • View Profile
  • BitShares: rnglab
I agree to let the competition machanisim solve the problem: obviously shareholders appreciate and would like to vote the witnesses that provide more and better service.

and AFAIK, to provide good price feed is also possible via free/open source script, for example, alt has published his script at https://pypi.python.org/pypi/btsprice/0.2.21, it's a good reference.

on the other side, I wonder whether  we need to consider one question: suppose publishing price feed become what witness must do,  shall we increase the block reward? and to how much?

now being a witness can get some reward, but the reward is not very attractive, we need to balance the responsibility and the reward, maybe we need to reestimate what a reward is more reasonable? maybe a more reasonable reward can lead to more active ecosystem?

kindly give your input on this topic.

@Bhuz sorry if I did things carelessly, I' recheck and take actions accordingly.

 @alt  runs a bussines which depends on precise and resilient feeds. His code skills along with a real incentive, brings us a good chance to improve the way we fetch and derive external prices.
Besides that I think it would be more productive to fork and improve @xeroc script, as a way to sum the scarce resources we have on this topic. 

Some conditions have to be given for diversity to be enriching, and for fair competition to exist at all. In the meanwhile, I think that cooperation is the way to go. Competition will favor aptitude and commitment only when we become able to incentivize it.


 Some numbers:

  - Witness reward averaged something around $200/month this year.

 - Let's say a regular VPS with 8Gb RAM costs 80 USD per month, and 50 USD for 4Gb.
 - That is a 150 USD reward for witnesses running just one node with only 4Gb  (at its very memory limit).
 - There's a 120 USD incentive to have 8Gb on just one node, with no failover at all.
 - Even running two 4Gb nodes with a proper failover strategy (for ~100 USD), both  nodes could easily crash if RAM becomes not enough.

Automatic fork resolution is working great, but it has not been prooven beyond our current TPS.
With checkpoints and delayed nodes, we have revived the network in less than an hour with no major consequences when forks turned into lost of consensus.
But I wouldn't like to see the network crashing just when the real action starts.

 - It seems to me that having two 8Gb nodes per witness, with failover able to identify and move away from minority forks (when possible with just two nodes), are the minimum technical requirements to handle the upcoming network growth. That means a reward of 40 USD per month at most.
Add seed nodes and the time required to constantly check and fix feeds, having everything up to date and the need to remain informed in a daily basis.

That means good withesses working at a loss, while icentivizing poor performance for a proffit.



This also brings another concern to me: as many good neutral witnesses lack of incentive or resources/time despite they good will, it drives business to run it's own witnesses given their need to have reliable feeds.
I see a flaw here:  business tend to have more resources, voting stake, reward incentives and thus good performance , this raises their motivation and chances to have many witnesses voted in.
As business models may disagree about market parameters that are defined by feed producers, this could lead to unfair competition or other undesirable behavors, specially now that new/unknown players are steadily arriving.

This is one of the reasons why witnesses, committee and proxies were planed as independent powers.



I'm not pushing to rise block rewards here, nor discouraging our most valuable witnesses who also runs business over bitshares.
 I still enjoy being reliable and helpful as a witness, rewards aside, and  I'm glad to see this debate going on in a constructive way, finally.
Just want to sumarize some facts I see.

At this point we really need a good balance between competition and collaborative work to make network and market more stable and to reduce the entry barriers.


Offline bitcrab

  • Committee member
  • Hero Member
  • *
  • Posts: 1928
    • View Profile
  • BitShares: bitcrab
  • GitHub: bitcrab

There are rules to being able to qualify for election for president. It is not ruleless. In the above however this is being conflated with electioneering. Before they even get into the game, they have rules to follow and know what can and cannot be done. There are laws. Witnesses have nothing but the whim of some people who have no clear guidelines, rules, or to compare to running for President of the USA, laws, that they can then follow to become a candidate.

You want competition to be a factor, then there has to be a playing field that has rules and/or guidelines that can be followed and considered.

what you are talking is just the enter threshold: following the law, and maybe also the candidate should be born in US? ... however, following these is just the precondition to enter the game, it does not guarntee anyone to become the president.

there are some general guidelines for what a president should do - a US president should make US economy grow, should bring happiness to US people, and make US great. but there's no guildlines for how to get this done, no guidelines on how to cope with the illegal immigrants, how to attract overseas investment, whether should to pay more on defense and anti-terrorism...on the contrary, the candidates should give their idea on these issues and try to get support.

similiarly, we also have enter threshold for being a witness - you need to run a witness node. however, running a witness node does not guarntee that you'll be voted as active witness.generally speaking, voters would like to support the witness that provide better service, but I don't think we need to define "better service" on each detail,  if the proxy and witness keep on communicate with each other, the voting precedure will make better service comes out.
Email:bitcrab@qq.com

Offline BunkerChainLabs-DataSecurityNode

whether we can call it competition is not so important, a similar thing is US President election, are there clearly defined what kind of people should be elected president? no, but is this a competition? I think so. the final result comes from competition.
to qualify witness, take the "feed price every 5 min" as an example, at the time that the market price does not change in 30 mins, why should the witness publish price every 5 min?

another example is, several witnesses removed yunbi price when they are told that yunbi price is easy to manipulate, however alt's script uses a different alogrithm: it collect orders from different exchanges, including yunbi, and simulate to put these orders together to generate a result price, so this gives each exchange the weight it should have, right? but before this script comes out, I don't know the price script can be coded like this.

in my view price feed is such an area that it is easy to judge which witness doing well but it's not easy to hard code some rules to determine which nodes is qualified.

as a proxy, in long run I think I will vote witness that
1. generate block with low block missing rate.
2. publish price that reflect market price exactly and globally enough, maybe no need to be very frequent, but when market price changes obviously, need to publish price in time.

maybe I will unvote all the witnesses that does not publish price. however I think we need first reach some conclusion on the reward issue.

I do not like the worker proposal advice, as it depends on trustee, not decentralized enough.

the voting machanisim can work without clearly defined game rule, just as a satisfactory president can comes out from the general election. of course, a lot of communication are needed here between the proxy/stakeholders and the candidates.

There are rules to being able to qualify for election for president. It is not ruleless. In the above however this is being conflated with electioneering. Before they even get into the game, they have rules to follow and know what can and cannot be done. There are laws. Witnesses have nothing but the whim of some people who have no clear guidelines, rules, or to compare to running for President of the USA, laws, that they can then follow to become a candidate.

You want competition to be a factor, then there has to be a playing field that has rules and/or guidelines that can be followed and considered.
+-+-+-+-+-+-+-+-+-+-+
www.Peerplays.com | Decentralized Gaming Built with Graphene - Now with BookiePro and Sweeps!
+-+-+-+-+-+-+-+-+-+-+