Quote from: abit on April 07, 2016, 11:38:18 AM
Yes, some progresses.
Check https://data.sparkfun.com/bitshares_usd_price for a feed_price only stream. Market snapshots stream would be something like that but with much more data, which can be used to calculate scores.
If I'm given a decided formula, I can also stream scores.

This looks great .. !!!

Quote from: tbone on April 07, 2016, 10:42:47 AM

Quote from: abit on April 03, 2016, 10:07:53 AM

Quote from: roadscape on April 02, 2016, 05:49:11 PM

Quote from: abit on March 31, 2016, 08:24:11 AM

Quote from: roadscape on March 31, 2016, 12:27:31 AM
Discussion also here: https://github.com/cryptonomex/graphene/issues/643

At this point I'm thinking @abit's solution would indeed work great as long as he can implement this detail:

Quote from: tonyk on March 22, 2016, 07:29:12 PM
OK, how about a middle ground - taking the snapshot every 10 (20, 30 whatever) minutes BUT also reading the filled orders in that period and using them for the calculation[effectively adding them to the orderbook like they were not filled]?
We can do 2 diff things - either credit them for the whole time period or really check when they were placed and  filled and credit them with the correct real time they were on the book.

####
thisTimeIntervalStart = now() - 10 min
For each filled order in time [now, thisTimeIntervalStart]
      T = OrderFillTime - max(OrderPlacementTime,  thisTimeIntervalStart)
       order_total = size of the Filled Order
####

So each snapshot really needs to be a window of time (say 15mins) that contains every order that existed during that time PLUS its placement time and/or fill time. Is this what you were thinking @abit? Any idea how partially filled orders are handled with this approach?

My idea can be considered that a new snapshot will be taken on every new block, so all unfilled orders will be tracked. For partially filled orders, we'll know when they were created, when they were partially filled, and when they disappeared.

Sounds good.. so this data will be streamed to a log file for processing/scoring by external scripts? It would be ideal to have multiple people running the same script and checking to make sure the numbers are in agreement. Your approach sounds like the most accurate way to get the data.

I'll stream data to somewhere on the Internet.

@roadscape , @abit

Hey guys, how soon do you think this can be completed?  @abit, have you been making progress?  Ronny from @ccedk said he would be interested in utilizing this to reward market makers in his UIAs.  And he will soon be launching some high profile assets suck as Lisk, Digix, and Synerio.  Not to mention, it is critical for us to start bootstrapping key fiat BitAssets.  What do you guys think?

Yes, some progresses.
Check https://data.sparkfun.com/bitshares_usd_price for a feed_price only stream. Market snapshots stream would be something like that but with much more data, which can be used to calculate scores.
If I'm given a decided formula, I can also stream scores.

Quote from: abit on April 03, 2016, 10:07:53 AM

Quote from: roadscape on April 02, 2016, 05:49:11 PM

Quote from: abit on March 31, 2016, 08:24:11 AM

Quote from: roadscape on March 31, 2016, 12:27:31 AM
Discussion also here: https://github.com/cryptonomex/graphene/issues/643

At this point I'm thinking @abit's solution would indeed work great as long as he can implement this detail:

Quote from: tonyk on March 22, 2016, 07:29:12 PM
OK, how about a middle ground - taking the snapshot every 10 (20, 30 whatever) minutes BUT also reading the filled orders in that period and using them for the calculation[effectively adding them to the orderbook like they were not filled]?
We can do 2 diff things - either credit them for the whole time period or really check when they were placed and  filled and credit them with the correct real time they were on the book.

####
thisTimeIntervalStart = now() - 10 min
For each filled order in time [now, thisTimeIntervalStart]
      T = OrderFillTime - max(OrderPlacementTime,  thisTimeIntervalStart)
       order_total = size of the Filled Order
####

So each snapshot really needs to be a window of time (say 15mins) that contains every order that existed during that time PLUS its placement time and/or fill time. Is this what you were thinking @abit? Any idea how partially filled orders are handled with this approach?

My idea can be considered that a new snapshot will be taken on every new block, so all unfilled orders will be tracked. For partially filled orders, we'll know when they were created, when they were partially filled, and when they disappeared.

Sounds good.. so this data will be streamed to a log file for processing/scoring by external scripts? It would be ideal to have multiple people running the same script and checking to make sure the numbers are in agreement. Your approach sounds like the most accurate way to get the data.

I'll stream data to somewhere on the Internet.

@roadscape , @abit

Hey guys, how soon do you think this can be completed?  @abit, have you been making progress?  Ronny from @ccedk said he would be interested in utilizing this to reward market makers in his UIAs.  And he will soon be launching some high profile assets suck as Lisk, Digix, and Synerio.  Not to mention, it is critical for us to start bootstrapping key fiat BitAssets.  What do you guys think?

Quote from: roadscape on April 02, 2016, 05:49:11 PM

Quote from: abit on March 31, 2016, 08:24:11 AM

Quote from: roadscape on March 31, 2016, 12:27:31 AM
Discussion also here: https://github.com/cryptonomex/graphene/issues/643

At this point I'm thinking @abit's solution would indeed work great as long as he can implement this detail:

Quote from: tonyk on March 22, 2016, 07:29:12 PM
OK, how about a middle ground - taking the snapshot every 10 (20, 30 whatever) minutes BUT also reading the filled orders in that period and using them for the calculation[effectively adding them to the orderbook like they were not filled]?
We can do 2 diff things - either credit them for the whole time period or really check when they were placed and  filled and credit them with the correct real time they were on the book.

####
thisTimeIntervalStart = now() - 10 min
For each filled order in time [now, thisTimeIntervalStart]
      T = OrderFillTime - max(OrderPlacementTime,  thisTimeIntervalStart)
       order_total = size of the Filled Order
####

So each snapshot really needs to be a window of time (say 15mins) that contains every order that existed during that time PLUS its placement time and/or fill time. Is this what you were thinking @abit? Any idea how partially filled orders are handled with this approach?

My idea can be considered that a new snapshot will be taken on every new block, so all unfilled orders will be tracked. For partially filled orders, we'll know when they were created, when they were partially filled, and when they disappeared.

Sounds good.. so this data will be streamed to a log file for processing/scoring by external scripts? It would be ideal to have multiple people running the same script and checking to make sure the numbers are in agreement. Your approach sounds like the most accurate way to get the data.

I'll stream data to somewhere on the Internet.

Quote from: abit on March 31, 2016, 08:24:11 AM

Quote from: roadscape on March 31, 2016, 12:27:31 AM
Discussion also here: https://github.com/cryptonomex/graphene/issues/643

At this point I'm thinking @abit's solution would indeed work great as long as he can implement this detail:

Quote from: tonyk on March 22, 2016, 07:29:12 PM
OK, how about a middle ground - taking the snapshot every 10 (20, 30 whatever) minutes BUT also reading the filled orders in that period and using them for the calculation[effectively adding them to the orderbook like they were not filled]?
We can do 2 diff things - either credit them for the whole time period or really check when they were placed and  filled and credit them with the correct real time they were on the book.

####
thisTimeIntervalStart = now() - 10 min
For each filled order in time [now, thisTimeIntervalStart]
      T = OrderFillTime - max(OrderPlacementTime,  thisTimeIntervalStart)
       order_total = size of the Filled Order
####

So each snapshot really needs to be a window of time (say 15mins) that contains every order that existed during that time PLUS its placement time and/or fill time. Is this what you were thinking @abit? Any idea how partially filled orders are handled with this approach?

My idea can be considered that a new snapshot will be taken on every new block, so all unfilled orders will be tracked. For partially filled orders, we'll know when they were created, when they were partially filled, and when they disappeared.

Sounds good.. so this data will be streamed to a log file for processing/scoring by external scripts? It would be ideal to have multiple people running the same script and checking to make sure the numbers are in agreement. Your approach sounds like the most accurate way to get the data.

Quote from: abit on March 31, 2016, 08:24:11 AM

Quote from: roadscape on March 31, 2016, 12:27:31 AM
Discussion also here: https://github.com/cryptonomex/graphene/issues/643

At this point I'm thinking @abit's solution would indeed work great as long as he can implement this detail:

Quote from: tonyk on March 22, 2016, 07:29:12 PM
OK, how about a middle ground - taking the snapshot every 10 (20, 30 whatever) minutes BUT also reading the filled orders in that period and using them for the calculation[effectively adding them to the orderbook like they were not filled]?
We can do 2 diff things - either credit them for the whole time period or really check when they were placed and  filled and credit them with the correct real time they were on the book.

####
thisTimeIntervalStart = now() - 10 min
For each filled order in time [now, thisTimeIntervalStart]
      T = OrderFillTime - max(OrderPlacementTime,  thisTimeIntervalStart)
       order_total = size of the Filled Order
####

So each snapshot really needs to be a window of time (say 15mins) that contains every order that existed during that time PLUS its placement time and/or fill time. Is this what you were thinking @abit? Any idea how partially filled orders are handled with this approach?

My idea can be considered that a new snapshot will be taken on every new block, so all unfilled orders will be tracked. For partially filled orders, we'll know when they were created, when they were partially filled, and when they disappeared.

If it isn't too resource intensive, snapshot on every block sounds great.

Quote from: roadscape on March 31, 2016, 12:27:31 AM
Discussion also here: https://github.com/cryptonomex/graphene/issues/643

At this point I'm thinking @abit's solution would indeed work great as long as he can implement this detail:

Quote from: tonyk on March 22, 2016, 07:29:12 PM
OK, how about a middle ground - taking the snapshot every 10 (20, 30 whatever) minutes BUT also reading the filled orders in that period and using them for the calculation[effectively adding them to the orderbook like they were not filled]?
We can do 2 diff things - either credit them for the whole time period or really check when they were placed and  filled and credit them with the correct real time they were on the book.

####
thisTimeIntervalStart = now() - 10 min
For each filled order in time [now, thisTimeIntervalStart]
      T = OrderFillTime - max(OrderPlacementTime,  thisTimeIntervalStart)
       order_total = size of the Filled Order
####

So each snapshot really needs to be a window of time (say 15mins) that contains every order that existed during that time PLUS its placement time and/or fill time. Is this what you were thinking @abit? Any idea how partially filled orders are handled with this approach?

Since we're trying to incentivize orders placed, not orders filled, I don't think we need to concern ourselves with partial fills except to the extent that a partial fill reduces the number of shares an MM has on the order book and is getting credit for at that moment.  So if an MM reduces the size of their order, it should have the same impact on the scoring as a partial fill. 

Quote from: roadscape on March 31, 2016, 12:27:31 AM
Discussion also here: https://github.com/cryptonomex/graphene/issues/643

At this point I'm thinking @abit's solution would indeed work great as long as he can implement this detail:

Quote from: tonyk on March 22, 2016, 07:29:12 PM
OK, how about a middle ground - taking the snapshot every 10 (20, 30 whatever) minutes BUT also reading the filled orders in that period and using them for the calculation[effectively adding them to the orderbook like they were not filled]?
We can do 2 diff things - either credit them for the whole time period or really check when they were placed and  filled and credit them with the correct real time they were on the book.

####
thisTimeIntervalStart = now() - 10 min
For each filled order in time [now, thisTimeIntervalStart]
      T = OrderFillTime - max(OrderPlacementTime,  thisTimeIntervalStart)
       order_total = size of the Filled Order
####

So each snapshot really needs to be a window of time (say 15mins) that contains every order that existed during that time PLUS its placement time and/or fill time. Is this what you were thinking @abit? Any idea how partially filled orders are handled with this approach?

My idea can be considered that a new snapshot will be taken on every new block, so all unfilled orders will be tracked. For partially filled orders, we'll know when they were created, when they were partially filled, and when they disappeared.

Discussion also here: https://github.com/cryptonomex/graphene/issues/643

At this point I'm thinking @abit's solution would indeed work great as long as he can implement this detail:

Quote from: tonyk on March 22, 2016, 07:29:12 PM
OK, how about a middle ground - taking the snapshot every 10 (20, 30 whatever) minutes BUT also reading the filled orders in that period and using them for the calculation[effectively adding them to the orderbook like they were not filled]?
We can do 2 diff things - either credit them for the whole time period or really check when they were placed and  filled and credit them with the correct real time they were on the book.

####
thisTimeIntervalStart = now() - 10 min
For each filled order in time [now, thisTimeIntervalStart]
      T = OrderFillTime - max(OrderPlacementTime,  thisTimeIntervalStart)
       order_total = size of the Filled Order
####

So each snapshot really needs to be a window of time (say 15mins) that contains every order that existed during that time PLUS its placement time and/or fill time. Is this what you were thinking @abit? Any idea how partially filled orders are handled with this approach?

Quote from: roadscape on March 25, 2016, 11:15:04 PM

@abit, that would be great if this could be a plugin for the node. It will be much more efficient and accurate than doing this thru API.

Quote from: tbone on March 24, 2016, 02:54:59 AM
@roadscape: Many orders will be on the books for much less than 10 minutes.  So I think sampling every 10 minutes will yield somewhat arbitrary results.  On the other hand, I imagine constant monitoring would be too resource intensive?  If so, how about sampling every 1 minute?  I don't think it would yield perfect results, but probably more than good enough, and I'm guessing without being too unreasonably resource intensive.  Thoughts?  @abits, can you comment on this as well?

@tbone It could probably be done every minute. If we have many quickly placed/filled orders at high volumes then it would not make sense to use the sampling approach tho. But I figured most the orders getting rewarded would be bigger walls that don't move quickly. But there are workarounds, I just wanted to explore the limits of that approach and get feedback from traders.

Quote from: tonyk on March 24, 2016, 03:37:52 AM

Your middle ground is a good idea, the main issue--I don't think it's possible to get all this info solely via RPC calls; it would require extra processing. This is doable but at that point I'd weigh it against the continuous approach.
However @abit's solution (processing data straight in the node) is probably the most proper way to accomplish this.

@roadscape too be more efficient, at first I'll write a plugin which provides basic snapshot data only via RPC (E.G. what orders exist at a given time and what the feed price is, or data sets for a given time range), then you and/or others can do more processing (easier than querying via current APIs).
Implementing all processing logic in the plugin sounds not a good idea for me.

Quote from: abit on March 23, 2016, 07:45:18 PM
@roadscape I prefer to calculate with full data rather than with sample data. I'm going to write a snapshot plugin.

Data structure of one market pair (for example BTS/USD):
* a calculation period = snapshots
* a snapshot = time stamp, feed price, ask orders, bid orders
* a order = direction(ask/bid), price, volume, owner, create_time

Calculations:
[Note: the algorithm below is based on feed price, but not on "best prices"]
* at the beginning of a calculation period, take a snapshot, set everyone's score = 0
* Every time (after a new block is produced) when a new order is created or cancelled, or feed price is updated, or reached the end of a calculation period, take a new snapshot.
-> let LT= last snapshot time
     let T = current snapshot time
     let LFP = feed price of last snapshot
-> for each order,
     let LV=volume of last snapshot,
     calculate LV' = function1 (direction, price, LFP, LV), to filter out unqualified orders
     calculate new score gained by this order after last snapshot
          NOS = function2 (direction,LV', LT, T, create_time, price, LFP)
          note: create_time may be needed here to judge whether this order is qualified.
-> for each account(owner) on each side,
     let LAS = score on last snapshot
     calculate TLV = sum(LV'), total volume of qualified orders at last snapshot
     calculate ELV = function3(TLV), to filter out unqualified account if total volume is too low
     calculate TNS = sum(NOS), total new score gained
     calculate ENS = function4(TNS,ELV), to filter out too low total score/volume, and perhaps set a cap on new score
     calculate AS = LAS + ENS, final score at this snapshot
* at the end of a calculation period, for each side, we have a set of (account, score), so we can calculate the final rewards.

//Update: I'm going to write a plugin to provide snapshot data, so you guys can use it to do whatever calculation as you like.

@abit, that would be great if this could be a plugin for the node. It will be much more efficient and accurate than doing this thru API.

Quote from: tbone on March 24, 2016, 02:54:59 AM
@roadscape: Many orders will be on the books for much less than 10 minutes.  So I think sampling every 10 minutes will yield somewhat arbitrary results.  On the other hand, I imagine constant monitoring would be too resource intensive?  If so, how about sampling every 1 minute?  I don't think it would yield perfect results, but probably more than good enough, and I'm guessing without being too unreasonably resource intensive.  Thoughts?  @abits, can you comment on this as well?

@tbone It could probably be done every minute. If we have many quickly placed/filled orders at high volumes then it would not make sense to use the sampling approach tho. But I figured most the orders getting rewarded would be bigger walls that don't move quickly. But there are workarounds, I just wanted to explore the limits of that approach and get feedback from traders.

Quote from: tonyk on March 24, 2016, 03:37:52 AM

Quote from: roadscape on March 23, 2016, 01:41:19 PM

Quote from: tonyk on March 22, 2016, 07:29:12 PM

Quote from: roadscape on March 22, 2016, 02:24:45 PM
tonyk Your proposed changes make sense, but continuous monitoring is much more complex than sampling*. What does it capture that sampling can't? And what if samples were e.g. 15 mins apart?

@roadscape
OK, how about a middle ground - taking the snapshot every 10 (20, 30 whatever) minutes BUT also reading the filled orders in that period and using them for the calculation[effectively adding them to the orderbook like they were not filled]?
We can do 2 diff things - either credit them for the whole time period or really check when they were placed and  filled and credit them with the correct real time they were on the book.

####
thisTimeIntervalStart = now() - 10 min
For each filled order in time [now, thisTimeIntervalStart]
      T = OrderFillTime - max(OrderPlacementTime,  thisTimeIntervalStart)
       order_total = size of the Filled Order
####

My question is, what scenario are you trying to avoid (or create) by doing this?

If your order is on the books for 120 mins, and is *completely* filled at minute 125, you would not get credit for those last 5 minutes (assuming 10-minute snapshot interval). To me this doesn't seem like a problem.

If you expect orders to be on the books for less than 10 minutes at a time, I could see why we would need to be tracking this more detailed order activity.

My original line of thinking was a simple "sharedrop" of points onto order book participants at a regular interval.

My assumptions for MM subsidies:
1) The actual market activity doesn't matter nearly as much as creating a nicely-shaped order book.
2) 'Sampling' the order book every ~10 minutes is at least 95% as accurate as analyzing continuous data.

Obviously (I thought), I am trying to avoid orders being on the order book for 30% -99% of the time between sample taking, and such orders getting no credit at all. To say nothing about such orders being filled first, means they had the best prices of all orders!!!
The less the time between sample taking the less an issue this becomes, but your original proposal was every 1 h. (way too long in my view). The smaller the time between each sample taking the less an issue it is. (so if you cut it to 3-5-7 min, this is a one way to do it)
Yet, again the proposed solution will yield ' near perfect'* results even if you sample far less frequently... and the computations involved seem to take not too much recourses.

* It is not perfect, cause if you sample once every hour, you should also credit the "placed and subsequently cancelled" orders, meeting all other criteria, in that time frame.

Your middle ground is a good idea, the main issue--I don't think it's possible to get all this info solely via RPC calls; it would require extra processing. This is doable but at that point I'd weigh it against the continuous approach.
However @abit's solution (processing data straight in the node) is probably the most proper way to accomplish this.

Do we need to subsidize force settlement orders and/or call orders (collateral holders)?

Quote from: roadscape on March 23, 2016, 01:41:19 PM

Quote from: tonyk on March 22, 2016, 07:29:12 PM

Quote from: roadscape on March 22, 2016, 02:24:45 PM
tonyk Your proposed changes make sense, but continuous monitoring is much more complex than sampling*. What does it capture that sampling can't? And what if samples were e.g. 15 mins apart?

@roadscape
OK, how about a middle ground - taking the snapshot every 10 (20, 30 whatever) minutes BUT also reading the filled orders in that period and using them for the calculation[effectively adding them to the orderbook like they were not filled]?
We can do 2 diff things - either credit them for the whole time period or really check when they were placed and  filled and credit them with the correct real time they were on the book.

####
thisTimeIntervalStart = now() - 10 min
For each filled order in time [now, thisTimeIntervalStart]
      T = OrderFillTime - max(OrderPlacementTime,  thisTimeIntervalStart)
       order_total = size of the Filled Order
####

My question is, what scenario are you trying to avoid (or create) by doing this?

If your order is on the books for 120 mins, and is *completely* filled at minute 125, you would not get credit for those last 5 minutes (assuming 10-minute snapshot interval). To me this doesn't seem like a problem.

If you expect orders to be on the books for less than 10 minutes at a time, I could see why we would need to be tracking this more detailed order activity.

My original line of thinking was a simple "sharedrop" of points onto order book participants at a regular interval.

My assumptions for MM subsidies:
1) The actual market activity doesn't matter nearly as much as creating a nicely-shaped order book.
2) 'Sampling' the order book every ~10 minutes is at least 95% as accurate as analyzing continuous data.

Obviously (I thought), I am trying to avoid orders being on the order book for 30% -99% of the time between sample taking, and such orders getting no credit at all. To say nothing about such orders being filled first, means they had the best prices of all orders!!!
The less the time between sample taking the less an issue this becomes, but your original proposal was every 1 h. (way too long in my view). The smaller the time between each sample taking the less an issue it is. (so if you cut it to 3-5-7 min, this is a one way to do it)
Yet, again the proposed solution will yield ' near perfect'* results even if you sample far less frequently... and the computations involved seem to take not too much recourses.

* It is not perfect, cause if you sample once every hour, you should also credit the "placed and subsequently cancelled" orders, meeting all other criteria, in that time frame.

Quote from: cylonmaker2053 on March 24, 2016, 01:01:38 AM

Quote from: roadscape on March 23, 2016, 01:41:19 PM
@tbone @cylonmaker2053 currently the scoring bonus is linear: 100% bonus @ the midpoint, and 0% bonus at 5% off. This could be scaled to a wider range, and we could also use a curve instead of a line (creating a "long tail") for the bonus.

yes, wider margin (like 20%+) would be the most important change. linear is fine as long as the margin is wider, but curved with weights trailing off towards the tails would be best.

I agree with these points.

Quote from: roadscape on March 23, 2016, 01:41:19 PM
@abit if you think any part of this can be done from within graphene, that's great. I've been looking at it from an API perspective.

@tbone @cylonmaker2053 currently the scoring bonus is linear: 100% bonus @ the midpoint, and 0% bonus at 5% off. This could be scaled to a wider range, and we could also use a curve instead of a line (creating a "long tail") for the bonus.

Quote from: tonyk on March 22, 2016, 07:29:12 PM

Quote from: roadscape on March 22, 2016, 02:24:45 PM
tonyk Your proposed changes make sense, but continuous monitoring is much more complex than sampling*. What does it capture that sampling can't? And what if samples were e.g. 15 mins apart?

@roadscape
OK, how about a middle ground - taking the snapshot every 10 (20, 30 whatever) minutes BUT also reading the filled orders in that period and using them for the calculation[effectively adding them to the orderbook like they were not filled]?
We can do 2 diff things - either credit them for the whole time period or really check when they were placed and  filled and credit them with the correct real time they were on the book.

####
thisTimeIntervalStart = now() - 10 min
For each filled order in time [now, thisTimeIntervalStart]
      T = OrderFillTime - max(OrderPlacementTime,  thisTimeIntervalStart)
       order_total = size of the Filled Order
####

My question is, what scenario are you trying to avoid (or create) by doing this?

If your order is on the books for 120 mins, and is *completely* filled at minute 125, you would not get credit for those last 5 minutes (assuming 10-minute snapshot interval). To me this doesn't seem like a problem.

If you expect orders to be on the books for less than 10 minutes at a time, I could see why we would need to be tracking this more detailed order activity.

My original line of thinking was a simple "sharedrop" of points onto order book participants at a regular interval.

My assumptions for MM subsidies:
1) The actual market activity doesn't matter nearly as much as creating a nicely-shaped order book.
2) 'Sampling' the order book every ~10 minutes is at least 95% as accurate as analyzing continuous data.

@roadscape: Many orders will be on the books for much less than 10 minutes.  So I think sampling every 10 minutes will yield somewhat arbitrary results.  On the other hand, I imagine constant monitoring would be too resource intensive?  If so, how about sampling every 1 minute?  I don't think it would yield perfect results, but probably more than good enough, and I'm guessing without being too unreasonably resource intensive.  Thoughts?  @abits, can you comment on this as well?