Percentage based transfer fee [BSIP10] implemented

[tbone]

One scenario need to be taken into consideration:
Since CERs of smart coins change frequently, there may be race conditions when a transfer operation and a price feed operation appear at a same block.
Usually a user need some seconds to confirm her transaction, or say, after the "Please confirm the transaction" page is shown, before the user clicks 'OK', if CER changed, the transaction would probably fail due to insufficient fee provided. It will cause negative user experience.

Imo this should be included in BSIP.

I have an idea to solve this issue (sure it will make the whole thing more complicated):
* For smart coins, show a message "you need to confirm in 60 seconds" on the "please confirm" page, if the user failed to confirm in 60 seconds, disable the "OK" button.
* Maintain the "worst" CERs of every smart coins in last 3 minutes somewhere in witness_node. The fee is only required to be not lower than the lowest one.

Thoughts?
@jakub @svk

@abit @svk

Maybe there is a third option:
In case CER changes before the user has a chance to confirm the transfer, just charge the user more (or less) than the amount displayed on the confirmation screen. This way it will only fail in one situation which is quite rare: if the user does not have sufficient funds in their account to cover the unexpected increase in the transfer fee.

This is obviously not perfect (as occasionally there will be a discrepancy between the transfer fee displayed on the confirmation screen and the actual fee charged) but I think it's the best solution available from the UX perspective.
I think people are usually not bothered much about the exact value of the transfer fee as long as it is within some acceptable limits.

Sorry, we're unable to do it. Fee is a field in the transaction/operation which is signed by the user. We can't modify it, nor deduct even one Satoshi more than the amount in the transaction/operation.

Another option would be deduct some fee from the fee pool, if the issuer approves. But it's hard to judge why the fee provided by the user is insufficient -- due to CER change or something else (maybe attack).

Aren't we talking about a scenario where the CER changes after the user initially submits the transaction, but BEFORE they confirm it?  If so, then can't we present them with a new confirmation which includes the updated fee and a message explaining why the fee is slightly higher than expected?

[jakub]

[KNOWN ISSUES/LIMITATIONS]
* Unable to apply percentage base fee mode to BTS

Please be aware of this^ limitation.

I wish we had some choice regarding this, nevertheless I think percentage-based fee mode for BTS is not crucial - there could be a flat transfer fee for BTS which is kept quite low.
EDIT: as abit rightly pointed it out, there is only one global flat transfer fee for all assets so we cannot make the flat fee low for BTS only.

The crucial things are SmartCoins (both public and private) and UIAs - these are fully covered.

As for FBAs - probably yes but we cannot say it for sure until FBAs are implemented.

[jakub]

Sorry, we're unable to do it. Fee is a field in the transaction/operation which is signed by the user. We can't modify it, nor deduct even one Satoshi more than the amount in the transaction/operation.

Another option would be deduct some fee from the fee pool, if the issuer approves. But it's hard to judge why the fee provided by the user is insufficient -- due to CER change or something else (maybe attack).

OK, I guess we'll have to live with that as it is.
If I understand it correctly, it will cause a negative UX in a very rare situation: only when there is a CER update between the moment you hit "Transfer" and the moment you hit "Confirm".

So my advice is to ignore the problem at this stage and if it becomes annoying in the future, we'll have to revise it, e.g. by introducing a 60 sec delay before the "Confirm" button gets disabled.

[abit]

One scenario need to be taken into consideration:
Since CERs of smart coins change frequently, there may be race conditions when a transfer operation and a price feed operation appear at a same block.
Usually a user need some seconds to confirm her transaction, or say, after the "Please confirm the transaction" page is shown, before the user clicks 'OK', if CER changed, the transaction would probably fail due to insufficient fee provided. It will cause negative user experience.

Imo this should be included in BSIP.

I have an idea to solve this issue (sure it will make the whole thing more complicated):
* For smart coins, show a message "you need to confirm in 60 seconds" on the "please confirm" page, if the user failed to confirm in 60 seconds, disable the "OK" button.
* Maintain the "worst" CERs of every smart coins in last 3 minutes somewhere in witness_node. The fee is only required to be not lower than the lowest one.

Thoughts?
@jakub @svk

@abit @svk

Maybe there is a third option:
In case CER changes before the user has a chance to confirm the transfer, just charge the user more (or less) than the amount displayed on the confirmation screen. This way it will only fail in one situation which is quite rare: if the user does not have sufficient funds in their account to cover the unexpected increase in the transfer fee.

This is obviously not perfect (as occasionally there will be a discrepancy between the transfer fee displayed on the confirmation screen and the actual fee charged) but I think it's the best solution available from the UX perspective.
I think people are usually not bothered much about the exact value of the transfer fee as long as it is within some acceptable limits.

Sorry, we're unable to do it. Fee is a field in the transaction/operation which is signed by the user. We can't modify it, nor deduct even one Satoshi more than the amount in the transaction/operation.

Another option would be deduct some fee from the fee pool, if the issuer approves. But it's hard to judge why the fee provided by the user is insufficient -- due to CER change or something else (maybe attack).

[jakub]

One scenario need to be taken into consideration:
Since CERs of smart coins change frequently, there may be race conditions when a transfer operation and a price feed operation appear at a same block.
Usually a user need some seconds to confirm her transaction, or say, after the "Please confirm the transaction" page is shown, before the user clicks 'OK', if CER changed, the transaction would probably fail due to insufficient fee provided. It will cause negative user experience.

Imo this should be included in BSIP.

I have an idea to solve this issue (sure it will make the whole thing more complicated):
* For smart coins, show a message "you need to confirm in 60 seconds" on the "please confirm" page, if the user failed to confirm in 60 seconds, disable the "OK" button.
* Maintain the "worst" CERs of every smart coins in last 3 minutes somewhere in witness_node. The fee is only required to be not lower than the lowest one.

Thoughts?
@jakub @svk

@abit @svk

Maybe there is a third option:
In case CER changes before the user has a chance to confirm the transfer, just charge the user more (or less) than the amount displayed on the confirmation screen. This way it will only fail in one situation which is quite rare: if the user does not have sufficient funds in their account to cover the unexpected increase in the transfer fee.

This is obviously not perfect (as occasionally there will be a discrepancy between the transfer fee displayed on the confirmation screen and the actual fee charged) but I think it's the best solution available from the UX perspective.
I think people are usually not bothered much about the exact value of the transfer fee as long as it is within some acceptable limits.

[abit]

The scenario you describe is a potential problem even without percentage-based fees, correct?

Correct, when one doesn't pay fee in BTS but in other assets.

[roadscape]

One scenario need to be taken into consideration:
Since CERs of smart coins change frequently, there may be race conditions when a transfer operation and a price feed operation appear at a same block.
Usually a user need some seconds to confirm her transaction, or say, after the "Please confirm the transaction" page is shown, before the user clicks 'OK', if CER changed, the transaction would probably fail due to insufficient fee provided. It will cause negative user experience.

Imo this should be included in BSIP.

I have an idea to solve this issue (sure it will make the whole thing more complicated):
* For smart coins, show a message "you need to confirm in 60 seconds" on the "please confirm" page, if the user failed to confirm in 60 seconds, disable the "OK" button.
* Maintain the "worst" CERs of every smart coins in last 3 minutes somewhere in witness_node. The fee is only required to be not lower than the lowest one.


Thoughts?
@jakub @svk

I think the simplest solution may be to overpay fees by 1% or so. This would give a buffer for CER movement.

The scenario you describe is a potential problem even without percentage-based fees, correct?

[abit]

no, I am afraid that's not a solution
this kinds of complexity/uncertainty will make users felt more worse

Sorry I didn't get what you said.

[alt]

no, I am afraid that's not a solution
this kinds of complexity/uncertainty will make users felt more worse

[abit]

let me ask a question for the percentage fees.

suppose we have known the current rate for CNY is 0.02CNY/BTS, the min fee is 10 BTS, the high fee is 20 BTS, percentage is 0.1%

If I want to transfer 200 CNY, what fees information should package in the transaction?
is it 10 BTS or 0.2 CNY?

if this is an offline operation, such as signed from a cold wallet, or pay with a paper wallet.
when generate the operation,  the rate is 0.02 CNY/BTS
after signed, you want to broadcast, the price  have changed
1) if you pay fees with 10BTS, and price change to 0.019CNY/BTS, it's not enough, because now you need pay 0.2/0.019 BTS
2) if you pay fees with 0.2CNY, and price change to 0.021 CNY/BTS, it's still not enough, because now you need pay at least 10BTS*0.021CNY/BTS

Thanks for pointing out this issue. It's what I just posted above. And a potential solution described.
Do you have better idea? If yes, please let us know, thanks!

Usually, if you are dealing with cold/offline wallets, you shouldn't be surprised if need to pay more, it's the trade off for better security. So what we need are some guidelines in "best practice of using cold/offline/paper wallets".

[alt]

let me ask a question for the percentage fees.

suppose we have known the current rate for CNY is 0.02CNY/BTS, the min fee is 10 BTS, the high fee is 20 BTS, percentage is 0.1%

If I want to transfer 200 CNY, what fees information should package in the transaction?
is it 10 BTS or 0.2 CNY?

if this is an offline operation, such as signed from a cold wallet, or pay with a paper wallet.
when generate the operation,  the rate is 0.02 CNY/BTS
after signed, you want to broadcast, the price  have changed
1) if you pay fees with 10BTS, and price change to 0.019CNY/BTS, it's not enough, because now you need pay 0.2/0.019 BTS
2) if you pay fees with 0.2CNY, and price change to 0.021 CNY/BTS, it's still not enough, because now you need pay at least 10BTS*0.021CNY/BTS

[abit]

To @xeroc, @cube and others who are testing or going to test my code:

I'm about to add an extension to the fee structure for easier future extension. After this change, something in the test chain will become incompatible. If replay doesn't work, need to reset the chain to a state before the first propose_fee_change after applied my previous change.

I will post here when the work is done.

I won't add hard fork logic for this change since it's too much work and not very useful imo.

Sorry for the inconvenience, and thanks for support!

Done. @xeroc @cube

[abit]

One scenario need to be taken into consideration:
Since CERs of smart coins change frequently, there may be race conditions when a transfer operation and a price feed operation appear at a same block.
Usually a user need some seconds to confirm her transaction, or say, after the "Please confirm the transaction" page is shown, before the user clicks 'OK', if CER changed, the transaction would probably fail due to insufficient fee provided. It will cause negative user experience.

Imo this should be included in BSIP.

I have an idea to solve this issue (sure it will make the whole thing more complicated):
* For smart coins, show a message "you need to confirm in 60 seconds" on the "please confirm" page, if the user failed to confirm in 60 seconds, disable the "OK" button.
* Maintain the "worst" CERs of every smart coins in last 3 minutes somewhere in witness_node. The fee is only required to be not lower than the lowest one.


Thoughts?
@jakub @svk

[abit]

I don't foresee this needing a lot of GUI work, a day at most (a day being 8 hours). As long as there's a flag on the asset that says what type of fee to use, and only transfers are affected, it should be trivial to add a switch to whichever transfer operation type that asset uses.

Fees need a little bit of work but not a whole lot, assuming the percent fee structure is easily available either in the asset object or in global parameters. We estimate fees using the global parameters and the operation type, so that would need to be extended to handle percent fees but that's not hard. And final operation fees are fetched from the witness node so that should not require any work on the GUI.

@svk Imo here is a list of work to be done in GUI:
* add an selection box on the "create asset" page and "update asset" page, so the issuer can set fee mode easily
* show transfer fee mode on asset details page, and maybe somewhere related
* use new operation to do all transfers (although the original operation can still be used on assets with flat mode)
* extend fee estimation to handle percent fees.
* the fee schedule page

Did I miss something?

I think Jakub will provide a document about data structure changes and API changes. Or you can dig into my code (after CNX reviewed)

Thanks for support!

OK.

How is the fee mode switched? Does it use the existing flags and permissions structure?

A new field in "extensions". Please check the examples in OP for reference (haven't been reviewed so far, so don't rely on it right now).
No permission involved so far.
Please let me know if I missed something. Thanks.

[jakub]

I think Jakub will provide a document about data structure changes and API changes. Or you can dig into my code (after CNX reviewed)

I'll surely do that once the worker proposal goes through.